Crawler Accessibilit
-
In Insights section of MOZ campaign, I'm seeing this: https://imgur.com/Gu2K9dz
Here are the contents of robots.txt:
User-agent: *
Disallow: /wp-admin/Sitemap: http://website.com.com/sitemap_index.xml
Can you please let me know what is wrong here?
-
Hi there!
Thanks so much for the great question! The data that populates in the homepage insights module in your Campaign is pulled when your Campaign is initially created and we do not update that info with each crawl like your other data. If something has changed since the Campaign was initially created, that module will not reflect the change. Sorry for that confusion!
If you'd like me to look into this further for your specific Campaign, I would be happy to do so! I would just need you to send an email on over to help@moz.com with the Campaign name so we can investigate.
I hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Website Traffic
Greetings All. I'm working on a new business website for a client, and I've been accessing the site numerous times daily (troubleshooting, confirming changes, etc.). I've been using Google search to access the site, and I use a VPN so that my IP would be random. So I would presume that the site traffic should be increasing. But on the last Moz Pro crawl, the site traffic was still listed as 0.
Getting Started | | depawl52
Is there a minimum amount of traffic required before Moz recognizes it or is something else going on?
Thank you.0 -
Unsolved Domain Overview - no NZ?
Why on Domain Overview are there only 4 countries to choose from? I am in NZ and need to check websites that operate businesses in NZ not Canada, US or Australia?
Getting Started | | KapitiGirl0 -
Using the free domain analysis tool - what would cause "Bummer no data found"
When I enter my domain in the free analysis tool, I get a "bummer, no data found". I am certain whatever is causing that to happen is causing other SEO problems https://academicanv.org
Getting Started | | verdet32323 -
Our crawler was not able to access the robots.txt file on your site
I've submitted my website to be crawled by Moz and done everything I can according to the troubleshooting guides. Please help! https://digitalbutter.co.za/robots.txt
Getting Started | | DigitalButter0 -
Has anyone purchased the MOZ SEO courses? Are they good?
I'm looking to learn more about SEO. Has anyone purchased the courses available through MOZ? If so, are they useful?
Getting Started | | Stevepair0 -
Site with 2 domains - 1 domain SEO opimised & 1 is not. How best to handle crawlers?
Situation: I have a dual domain site:
Getting Started | | DGAU
Domain 1 - www.domain.com is SEO optimised with product pages and should of course be indexed.
Domain 2 - secure.domain.com is not SEO optimised and simply has checkout and payment gateway pages. I've discovered that Moz automatically crawls Domain 2 - the secure.domain.com site and consequently picks up hundreds of errors.
I have put an end to this by adding a robots.txt to stop rogerbot and dotbot (mozs crawlers) from crawling domain 2. This fixes my errors in Moz reports however after doing more research into 'Crawler Control' I figure this might be the best option. My Question: Instead of using robots.txt to stop moz from crawing all of Domain 2 should I use on each page of domain 2? I believe this would then allow moz and google to crawl Domain 2 but also tell them both not to index it.
My understanding is that this would be best, and might even help my overall SEO by telling google not to give any SEO value to the Domain 2 pages?0 -
Can't track my site, keep getting "Ooops. Our crawlers are unable to access that URL"
Hello, So i keep getting this message and I went to hurl.it and I get 200 response. But it appears its not my actual homepage bc it says the body is empty and in the title it says "COMING SOON" which is not what my actual homepage says. Does anyone know what this means?? Thank you in advance! Rena
Getting Started | | Palila-Studio0 -
Does the Moz crawler have a static ip address?
We block AWS from crawling our site. Does the moz crawler use a static ip address that we could allow? I'm currently not able to add a campaign because Moz can't connect to our site.
Getting Started | | uShip0