undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Bar
  4. Error 406 with crawler test

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Error 406 with crawler test

Moz Bar
3
10
3.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • ArchieDonnithorne
    ArchieDonnithorne last edited by Oct 19, 2015, 3:51 PM

    hi to all. I have a big problem with the crawler of moz on this website: www.edilflagiello.it.

    On july with the old version i have no problem and the crawler give me a csv report with all the url but after we changed the new magento theme and restyled the old version, each time i use the crawler, i receive a csv file with this error:

    "error 406"

    Can you help me to understan wich is the problem? I already have disabled .htacces and robots.txt but nothing.

    Website is working well, i have used also screaming frog as well.

    1 Reply Last reply Reply Quote 1
    • ArchieDonnithorne
      ArchieDonnithorne @DirkC last edited by Oct 24, 2015, 1:50 PM Oct 24, 2015, 1:50 PM

      thank you very much Dirk. this sunday i try to fix all the error and next i will try again. Thanks for your assistance.

      1 Reply Last reply Reply Quote 0
      • DirkC
        DirkC @ArchieDonnithorne last edited by Oct 23, 2015, 11:22 AM Oct 23, 2015, 11:22 AM

        I noticed that you have a Vary: User-Agent in the header - so I tried visiting your site with js disabled & switched the user agent to Rogerbot. Result: the site did not load (turned endlessly) and checking the console showed quite a number of elements that generated 404's. In the end - there was a timeout.

        Try screaming frog - set user agent to Custom and change the values to

        Name:Rogerbot

        Agent: Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)

        It will be unable to crawl your site. Check your server configuration - there are issues in how you deal with the Mozbot useragent.

        Check the attached images.

        Dirk

        2emkZWc qWVIqVo jb5FpQK

        1 Reply Last reply Reply Quote 0
        • ArchieDonnithorne
          ArchieDonnithorne last edited by Oct 23, 2015, 10:25 AM Oct 23, 2015, 10:25 AM

          nothing. After i fix all the 40x error the crawler is always empty. Any other ideas?

          DirkC 1 Reply Last reply Oct 23, 2015, 11:22 AM Reply Quote 0
          • ArchieDonnithorne
            ArchieDonnithorne last edited by Oct 22, 2015, 3:04 AM Oct 22, 2015, 3:04 AM

            thanks, i'm wait another day

            1 Reply Last reply Reply Quote 0
            • axel.loesken
              axel.loesken last edited by Oct 21, 2015, 1:01 PM Oct 21, 2015, 12:56 PM

              I know the Crawl Test reports are cached for about 48 hours so there is a chance that the CSV will look identical to the previous one for that reason.

              With that in mind, I'd recommend waiting another day or two before requesting a new Crawl Test or just waiting until your next weekly campaign update, if that is sooner 🙂

              1 Reply Last reply Reply Quote 0
              • ArchieDonnithorne
                ArchieDonnithorne last edited by Oct 21, 2015, 12:37 PM Oct 21, 2015, 12:37 PM

                i have fixed all error but csv is always empty and says:

                http://www.edilflagiello.it,2015-10-21T13:52:42Z,406 : Received 406 (Not Acceptable) error response for page.,Error attempting to request page

                here the printscreen: http://www.webpagetest.org/result/151020_QW_JMP/1/details/

                Any ideas? Thanks for your help.

                1 Reply Last reply Reply Quote 0
                • ArchieDonnithorne
                  ArchieDonnithorne @DirkC last edited by Oct 20, 2015, 6:23 AM Oct 20, 2015, 6:23 AM

                  thanks a lot guy! I'm going to check this errors before next crawling.

                  1 Reply Last reply Reply Quote 0
                  • axel.loesken
                    axel.loesken last edited by Oct 19, 2015, 6:37 PM Oct 19, 2015, 6:37 PM

                    Great answer Dirk! Thanks for helping out!

                    Something else I noticed is that the site is coming back with quite a few errors when I ran it through a 3rd party tool, W3C Markup Validation Service and it also was checking the page as XHTML 1.0 Strict which looks to be common in other cases of 406 I've seen.

                    1 Reply Last reply Reply Quote 0
                    • DirkC
                      DirkC last edited by Oct 19, 2015, 6:28 PM Oct 19, 2015, 6:17 PM

                      If you check your page with external tools you'll see that the general status of the page is 200- however there are different elements which generate a 4xx error (your logo generates a 408 error - same for the shopping cart) - for more details you could check this http://www.webpagetest.org/result/151019_29_14E6/1/details/.

                      Remember that Moz bot is quite sensitive for errors -while browsers, Googlebot & Screaming Frog will accept errors on page, Moz bot stops in case of doubt.

                      You might want to check the 4xx errors & correct them - normally Moz bot should be able to crawl your site once these errors are corrected. More info on 406 errors can be found here. If you have access to your log files you could check in detail which elements are causing the problems when Mozbot is visiting your site.

                      Dirk

                      ArchieDonnithorne 2 Replies Last reply Oct 24, 2015, 1:50 PM Reply Quote 1
                      • 1 / 1
                      1 out of 10
                      • First post
                        1/10
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • BarshaEmiactech

                        Unsolved Mozbar extension now showing spam score

                        mozbar 2024

                        I have using extension and login my account and set free trial of one month but in this case i can't see spam score in Moz bar so how I can solve this

                        Moz Bar | Dec 1, 2024, 6:26 PM | BarshaEmiactech
                        0
                      • Alberto_Diaz

                        Unsolved New Moz Bar not working in my domain

                        moz bar

                        Hello, I have installed the new version of Moz Bar in Google Chrome.
                        My problem is that it analyzes all domains except mine: https://elchesemueve.com
                        I also can't find my country (Spain) to configure the site.

                        Moz Bar | Nov 16, 2024, 8:06 AM | Alberto D.
                        0
                      • Josh_Eligon

                        Unsolved MozBar 2024 How it Works Now

                        mozbar 2024

                        I am a premium user and i am still not able to use mozbar
                        ec22b5ba-4430-4fc7-9825-7fce7b1ca300-image.png I don't know how its works.

                        Moz Bar | Nov 21, 2024, 3:53 AM | Josh_Eligon
                        0
                      • ravigaurav

                        Unsolved How Should I Increase DA of my Website?

                        increase da

                        Google itself has been troubled by ranking factors. On top of that, the https://moz.com/ has worn a new garland of domain authority. It is even more regrettable when a website DA suddenly decreased. As recently on 22nd September, my website's DA was 16. As usual, on September 23, I posted a new informational blog related to QuickBooks antivirus exclusions on my website. The very next day my website DA was decreased. Can anyone who knows SEO help me understand why this always happens specially to me every time? I read the whole forum and all I found out is how my site's DA is decreased.
                        Can anyone let us know whether we can increase the Domain Authority of our website only by creating backlinks or there can be some other activities also?

                        Moz Bar | Oct 26, 2024, 4:48 PM | ravigaurav
                        2
                      • seoanalytics

                        Spam score of 28 %

                        Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,

                        Moz Bar | Apr 8, 2024, 4:50 PM | seoanalytics
                        1
                      • alisacromer

                        Need to solve "Oops our crawlers were unable to access" url for new campaign

                        I'm putting the url designfirstkitchenandbath.com and getting the "oops! our crawlers were unable to access the site. Since this site is a potential client, which shows up online, I can't get access to fix the code, plus while I can write a little html I don't feel comfortable working with hard, live code on someonelse's site. Anyone have a simple solution?

                        Moz Bar | Nov 13, 2016, 7:49 PM | alisacromer
                        0
                      • mediaxplain.nl

                        Moz crawler only crawls one page?!

                        Hello there, I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled. We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working. Hopefully someone can point us at the right direction. Thanks in advance, Jeremy

                        Moz Bar | Sep 4, 2015, 2:18 PM | mediaxplain.nl
                        0
                      • spinoki

                        Getting 'Sorry, but that URL is inaccessible' error msg when trying to run On-Page Grader

                        I just signed up for MOZ Pro for the first time today.  Tried to run the 'on-page grader' tool on some of my pages but I'm getting a 'Sorry, but that URL is inaccessible' error msg.  I have verified against the robot.txt file that the pages are NOT blocking any crawlers. Can anybody help?

                        Moz Bar | Jul 31, 2014, 6:21 PM | spinoki
                        0

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.