undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Bar
  4. Error 406 with crawler test

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Error 406 with crawler test

Moz Bar
3
10
3.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • ArchieDonnithorne
    ArchieDonnithorne last edited by Oct 19, 2015, 3:51 PM

    hi to all. I have a big problem with the crawler of moz on this website: www.edilflagiello.it.

    On july with the old version i have no problem and the crawler give me a csv report with all the url but after we changed the new magento theme and restyled the old version, each time i use the crawler, i receive a csv file with this error:

    "error 406"

    Can you help me to understan wich is the problem? I already have disabled .htacces and robots.txt but nothing.

    Website is working well, i have used also screaming frog as well.

    1 Reply Last reply Reply Quote 1
    • ArchieDonnithorne
      ArchieDonnithorne @DirkC last edited by Oct 24, 2015, 1:50 PM Oct 24, 2015, 1:50 PM

      thank you very much Dirk. this sunday i try to fix all the error and next i will try again. Thanks for your assistance.

      1 Reply Last reply Reply Quote 0
      • DirkC
        DirkC @ArchieDonnithorne last edited by Oct 23, 2015, 11:22 AM Oct 23, 2015, 11:22 AM

        I noticed that you have a Vary: User-Agent in the header - so I tried visiting your site with js disabled & switched the user agent to Rogerbot. Result: the site did not load (turned endlessly) and checking the console showed quite a number of elements that generated 404's. In the end - there was a timeout.

        Try screaming frog - set user agent to Custom and change the values to

        Name:Rogerbot

        Agent: Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)

        It will be unable to crawl your site. Check your server configuration - there are issues in how you deal with the Mozbot useragent.

        Check the attached images.

        Dirk

        2emkZWc qWVIqVo jb5FpQK

        1 Reply Last reply Reply Quote 0
        • ArchieDonnithorne
          ArchieDonnithorne last edited by Oct 23, 2015, 10:25 AM Oct 23, 2015, 10:25 AM

          nothing. After i fix all the 40x error the crawler is always empty. Any other ideas?

          DirkC 1 Reply Last reply Oct 23, 2015, 11:22 AM Reply Quote 0
          • ArchieDonnithorne
            ArchieDonnithorne last edited by Oct 22, 2015, 3:04 AM Oct 22, 2015, 3:04 AM

            thanks, i'm wait another day

            1 Reply Last reply Reply Quote 0
            • axel.loesken
              axel.loesken last edited by Oct 21, 2015, 1:01 PM Oct 21, 2015, 12:56 PM

              I know the Crawl Test reports are cached for about 48 hours so there is a chance that the CSV will look identical to the previous one for that reason.

              With that in mind, I'd recommend waiting another day or two before requesting a new Crawl Test or just waiting until your next weekly campaign update, if that is sooner 🙂

              1 Reply Last reply Reply Quote 0
              • ArchieDonnithorne
                ArchieDonnithorne last edited by Oct 21, 2015, 12:37 PM Oct 21, 2015, 12:37 PM

                i have fixed all error but csv is always empty and says:

                http://www.edilflagiello.it,2015-10-21T13:52:42Z,406 : Received 406 (Not Acceptable) error response for page.,Error attempting to request page

                here the printscreen: http://www.webpagetest.org/result/151020_QW_JMP/1/details/

                Any ideas? Thanks for your help.

                1 Reply Last reply Reply Quote 0
                • ArchieDonnithorne
                  ArchieDonnithorne @DirkC last edited by Oct 20, 2015, 6:23 AM Oct 20, 2015, 6:23 AM

                  thanks a lot guy! I'm going to check this errors before next crawling.

                  1 Reply Last reply Reply Quote 0
                  • axel.loesken
                    axel.loesken last edited by Oct 19, 2015, 6:37 PM Oct 19, 2015, 6:37 PM

                    Great answer Dirk! Thanks for helping out!

                    Something else I noticed is that the site is coming back with quite a few errors when I ran it through a 3rd party tool, W3C Markup Validation Service and it also was checking the page as XHTML 1.0 Strict which looks to be common in other cases of 406 I've seen.

                    1 Reply Last reply Reply Quote 0
                    • DirkC
                      DirkC last edited by Oct 19, 2015, 6:28 PM Oct 19, 2015, 6:17 PM

                      If you check your page with external tools you'll see that the general status of the page is 200- however there are different elements which generate a 4xx error (your logo generates a 408 error - same for the shopping cart) - for more details you could check this http://www.webpagetest.org/result/151019_29_14E6/1/details/.

                      Remember that Moz bot is quite sensitive for errors -while browsers, Googlebot & Screaming Frog will accept errors on page, Moz bot stops in case of doubt.

                      You might want to check the 4xx errors & correct them - normally Moz bot should be able to crawl your site once these errors are corrected. More info on 406 errors can be found here. If you have access to your log files you could check in detail which elements are causing the problems when Mozbot is visiting your site.

                      Dirk

                      ArchieDonnithorne 2 Replies Last reply Oct 24, 2015, 1:50 PM Reply Quote 1
                      • 1 / 1
                      1 out of 10
                      • First post
                        1/10
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • Alberto_Diaz

                        Unsolved New Moz Bar not working in my domain

                        moz bar

                        Hello, I have installed the new version of Moz Bar in Google Chrome.
                        My problem is that it analyzes all domains except mine: https://elchesemueve.com
                        I also can't find my country (Spain) to configure the site.

                        Moz Bar | Nov 16, 2024, 8:06 AM | Alberto D.
                        0
                      • ravigaurav

                        Unsolved How Should I Increase DA of my Website?

                        increase da

                        Google itself has been troubled by ranking factors. On top of that, the https://moz.com/ has worn a new garland of domain authority. It is even more regrettable when a website DA suddenly decreased. As recently on 22nd September, my website's DA was 16. As usual, on September 23, I posted a new informational blog related to QuickBooks antivirus exclusions on my website. The very next day my website DA was decreased. Can anyone who knows SEO help me understand why this always happens specially to me every time? I read the whole forum and all I found out is how my site's DA is decreased.
                        Can anyone let us know whether we can increase the Domain Authority of our website only by creating backlinks or there can be some other activities also?

                        Moz Bar | Oct 26, 2024, 4:48 PM | ravigaurav
                        2
                      • rajas2019

                        How Much Time It Will Take To Lower the Spam Score?

                        I'm facing an issue with my website. Due to little to no knowledge about link building and backlink, i created backlinks without checking the quality and spam score of the sites. Now there are many sites linking to my website but the overall spam score is very high of my website and my domain is reviewheart.com that i'm talking about. I have created a list and disavowed all he poor linking domains but still no improvement shown. Can anyone have the idea how much time moz will take to show the updated spam score as i have disavowed all the low quality spammed linking site?

                        Moz Bar | Jul 4, 2024, 9:36 AM | rajas2019
                        2
                      • jbcorcoran

                        How do I see amount of traffic to a particular page on a website?

                        I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?

                        Moz Bar | May 8, 2024, 7:32 PM | jbcorcoran
                        6
                      • seoanalytics

                        Spam score of 28 %

                        Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,

                        Moz Bar | Apr 8, 2024, 4:50 PM | seoanalytics
                        1
                      • KanikaG

                        Keyword Ranking Report is Different than Real Result on Google

                        Hi Guys,
                        I am getting MOZ reports on a few keywords and that is good But when My client search for the same keywords he doesn't get the same rank, not even on that page. Let me explain in more details:
                        I have a keyword that shows on rank #5 in Google Mobile for a Local location for the current week. The client is from the same location as well. But when client search for that keyword from his mobile he doesn't get the keyword listed on that page, not even on other pages. I checked with other online keyword rank checker tool and all are showing the result similar to the Moz ranking report. It's really frustrating for my client. Google analytics shows that mobile users and organic reach are increasing so I am sure MOZ and other tools are showing us correct result. What could be the possible reason that he is not getting the same result as showing by MOZ? I am not from his location so how can I check the keyword result for the selected city using MOZ that is accurate? Any help would be highly appreciated. Please help. Thanks

                        Moz Bar | Apr 29, 2024, 12:13 PM | KanikaG
                        1
                      • alisacromer

                        Need to solve "Oops our crawlers were unable to access" url for new campaign

                        I'm putting the url designfirstkitchenandbath.com and getting the "oops! our crawlers were unable to access the site. Since this site is a potential client, which shows up online, I can't get access to fix the code, plus while I can write a little html I don't feel comfortable working with hard, live code on someonelse's site. Anyone have a simple solution?

                        Moz Bar | Nov 13, 2016, 7:49 PM | alisacromer
                        0
                      • alexbudko

                        SEOMOZ Toolbar for FIREFOX [Moz Extension Firefox]

                        SEOMOZ toolbar for firefox - where to find it - i could find only for chrome

                        Moz Bar | Mar 27, 2024, 8:03 AM | alexbudko
                        10

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.