Possible Crawling Problem with Screaming Frog and Moz Crawlers
-
So I'm not sure if what I'm seeing is a problem or not.
As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors.
When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content.
There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it.
I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them.
Thanks
-
Thank you for the response.
I've ran two MOZ crawl reports today, one with mysite.com and one www.mysite.com. Both returned 1 result for mysite.com and www.mysite.com respectively, with a 200 status code, but no meta data. I know that I've successfully crawled www.mysite.com about a month ago with no problems. I have made small changes here and there but nothing is jumping out at me as wrong.
Screaming Frog is currently crawling my site successfully about 1/10 tries. The successful tries it sees 163 Total URL Encountered (its a small site) and the other 9/10 times it shows exactly 1 URL (the one i entered) and no meta data. There doesn't seem to be any pattern when it successfully crawls and when it doesn't make it past the first page.
Google WMT is currently showing No Data Available for both internal links and links to your site which is a little concerning. Everything else in WMT looks ok.
-
Two possible simple key-in items to consider: make sure the URL is inputted w/ the full url (not just mysite.com) and/or ensure to click any options for including root or sub-domains so its not just looking at a single page.
-
If you PM me the domain I can take a look myself.
Does the robots.txt have anything funny in there?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rel="canonical" tag is implemented in my product pages, but still getting canoncal error for products in Moz. What is the problem? me or MOZ?
I have included the rel="canonical" tag in all my product pages, but still getting canonical error in MOZ reports for more than 6 month ! I would like to know if my code is wrong or the MOZ report system is not working properly. Here is an example of my canonical code in line 84 rel="canonical" href="http://www.doornmore.com/slab-single-door-80-fiberglass-courtlandt-1-panel-arch-lite-glass.html" /> Thanks Shayann
Moz Pro | | Shayann0 -
Crawl Diagnostics - Crawling way more pages than my site has?
Hello all, I'm fairly new here, more of a paid search guy dabbling in SEO on the side. I have a client that I have in SEOMoz and the Crawl Diagnostics report is showing 10,000+ pages crawled and I think the site has at most 800 pages (e-commerce site using freewebstore.org as the platform). Any reasons this would be happening?
Moz Pro | | LodestoneGen0 -
Problems with google chrome
do anybody have a problem navigating Seomoz webpage and scrolling down. for example everytime that I came to ask a question and post, I write down later select my topic. and when I scroling down the web page just push up. and don't let me go to the button to "aks a question" from one moment I though was my computer but at the office and at home using chrome, I have the same issue. Anybody have the same issue?
Moz Pro | | lnietob0 -
Confusion about how SEOMOZ crawler works...
So according to my SEOMOZ dashboard, I'm ranking between #3-4 for one of my keywords. My keyword is 'Boston Wedding Photographer'. My site is http://www.symbolphoto.com I show up in google places, true. But i was wanting to rank organically. Am i right in the assumption that Google Places and Google Organic are not the same thing? SEOMOZ claims 3,4th but not organically(Assuming they aren't the same thing) I get pretty good traffic right now being in Places, but i can't help but feel that organically ranking would bring more traffic. Any suggestions or advice is greatly appreciated. TIA! -Brendan
Moz Pro | | symbolphoto0 -
Adjusting SEOmoz Crawling Speed
How do you adjust the SEOmoz crawling speed? SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
Moz Pro | | cappuccino891 -
Weird campaign problem
I have a campaign that is showing a 302 redirect for my site with the page title of: http://www.example.com/www.example.com and no URL under it. what could be causing this? Thanks!
Moz Pro | | questfore0 -
Is The Crawl Diagnostic tool working correctly?
The Crawl Diagnostic tool shows issues and displays a graph but they don't display the page specific results/suggestion like it used to. I get the "Congratulations, there are no pages affected by this issue!" message.
Moz Pro | | -PAUL-0