Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
In MozBar Page Analysis under General Attributes Country is empty or wrong
Some of our clients sites have a top level domain for Mexico, but in the Mozbar they doesn't show any country, is this related to a known bug? https://www.edacom.mx/ https://www.dropbox.com/s/a8jgr7ztk1im15k/Captura de pantalla 2019-08-15 12.41.10.png?dl=0 https://www.cliento.mx/ https://www.dropbox.com/s/a8jgr7ztk1im15k/Captura de pantalla 2019-08-15 12.41.10.png?dl=0 In the search console: "International Targeting report" the target is Ok to Mexico.
Moz Bar | | Andrea_Ugalde1 -
Moz Bar Not Showing DA?
Hello all, This could be something to do with our site or the Moz bar on Chrome, I just need to know which it is so if it is our website we can look into it further. On certain sections of our website the Moz bar doesn't display any Domain Authority, not even zero, the bar just isn't present. These types of pages are php which pull in data through a feed daily. Speaking to an SEO expert they said it could be where the page is being updated so frequently, or it could be something more sinister and technically not quite right. Does anyone have any ideas? Is the Moz bar just not working for these types os pages or is it more likely something to do with my site? Ironically it's these pages which I'm having trouble with that are not showing in SERPs! Thanks! 3Foorka
Moz Bar | | HB176 -
Has using Moz got me banned from Google search?
Dear fellow Mozzers, For the last couple of weeks when ever I do a google search I have to enter a captcha code. Now I hardly use google anymore which is annoying ! Then I went onto support.google.com and saw this: https://support.google.com/websearch/answer/86640?hl=en&ref_topic=3378866 "Automated traffic includes: Sending searches from a robot, computer program, automated service, or search scraper Using software that sends searches to Google to see how a website or webpage ranks on Google" Is Moz alone doing this? Thanks in advance for your response. Ash
Moz Bar | | -Ash-0 -
Getting 'Sorry, but that URL is inaccessible' error msg when trying to run On-Page Grader
I just signed up for MOZ Pro for the first time today. Tried to run the 'on-page grader' tool on some of my pages but I'm getting a 'Sorry, but that URL is inaccessible' error msg. I have verified against the robot.txt file that the pages are NOT blocking any crawlers. Can anybody help?
Moz Bar | | spinoki0 -
Checking Do-follow using Moz Bar
Does anyone know how to check if a link is a do-follow or no-follow using the Moz bar? I believe there's a function that highlights and color codes the link so as to tell? Can't seem to turn this function on in my Moz Bar settings. Am I missing something completely obvious?
Moz Bar | | Gavo0 -
How Does Moz Shoot Whiteboard Fridays?
Okay, sort of a meta question here... My company, a global test preparation company, is looking into new ways to deliver online course content (both pre-recorded and live streaming). Around the office we have looked at dozens of examples of companies that do video presenting or teaching, and then I realized that I watch a great example of high-quality online video every week: Whiteboard Fridays! They look and sound fantastic! Moz team, what are you willing to share re: the setup you use? I'm looking for specific direction about lighting, sound, and equipment. Anything you share would be VERY much appreciated. Thanks!
Moz Bar | | ScottShrum
Scott 3-methods-fueled-by-data-and-tools-to-earn-more-and-better-links-whiteboard-friday0 -
How do you stop Moz crawling a page?
Hello, I have a contact form which generates thousands of duplicate crawl errors. I'm going to use to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error? Thanks!
Moz Bar | | Seaward-Group0
