When will Dotbot see changes to my robots.txt file?
-
Dotbot doesn't want to know anything about my site, my Page Authotize is still 1 and my DA is the same. How long does it take to start getting results? In all the websites we have been working on, we have no problems, but in ours it has come up.
-
Thank you very much for your response, the truth is that we are appearing in different advertising media, and google has the links indexed. In fact we have now added the url to our moz profile and added it to the Link Tracking Lists tool.
It will be a matter of time
-
Hi - thanks for contacting us,
When you see a DA of 1, this will usually mean that we haven't indexed that site yet or we haven't found any inbound links for that particular site. That can happen for a number of reasons, such as we haven't found any external links to the site or the site is blocking our crawler or the site is fairly new.
When we compile our index by first crawling the more valuable pages we find and moving on from there. Specific pages may not be crawled if they have lower quality links on them or appear to be part of an unintentional link farm (for example, things like programmed calendars). If you’re wondering why we’re not identifying links to your site, I would suggest checking to make sure we’ve indexed the sites and pages you know are linking to your site.
Also, it’s important to keep in mind that with Link Explorer, new data is added all the time, so please keep an eye out for new data to be available for your site!
I hope this helps - please let me know if you have any other questions!
-
Hi there,
You just need to wait for a little bit longer, so Moz can crawl your site. Meanwhile, you check all your links in Google Search Console.
Ross
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
Site: www.kpmg.us Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter". Updated robots.txt to allow rogerbot full access: User-agent: rogerbot
Link Explorer | | KPMG-Search-Social
Disallow: Any ideas how to get roger to crawl my site????1 -
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Link Explorer | | gunawanjord0 -
How is Moz DA affected by spam links? Disavow file?
So it does not appear that moz let's you upload your disavow file. So when moz calculates your DA how do spammy links factor in? After digging through our GA it appears our site was hit with the 2016 penguin update and never recovered. Our weekly visitors were 2k, then dropped to 500 and have stayed close to that level for a while. We've used the disavow tool, without success over the past 3 years. During that time we have done link out reach and built around 10 legit good quality DA links since. But we have not recovered. At this point i'm thinking I should just remove the disavow file. Moz says our spam score for our domain is 5%.
Link Explorer | | jessicapremier0 -
The New Link Explorer (which will replace Open Site Explorer) is Now in Beta
Howdy Moz friends, Today, the Moz team is making a new tool -- Link Explorer -- available in private beta for Moz Pro subscribers (including those taking a free trial). The tool is still a little ways out from public launch, but we wanted to get your feedback to help make it the best product it can be. What's Link Explorer? In essence, it's a replacement for Open Site Explorer (Moz's tool for link discovery, competitive analysis, and link building) that addresses many of its most pernicious challenges, such as: Daily updates - no longer will you need to wait a month or two to see a new DA score or the links you built last week. Link Explorer updates every 24 hours with all the new links we've discovered that day, and gives a new DA score each night. A MUCH bigger index - OSE has always been known for having quality links, but quantity has been an issue. No longer. The new tool's link index is more than 20 times larger than Open Site Explorer's, covering trillions of links across hundreds of billions of pages, while maintaining a focus on high quality domains and pages. Additional functionality - new graphs (like link growth over time), new charts (like gained+lost links), new filters and sorts, and some new kinds of data coming soon. Improved metrics - Domain Authority and Page Authority have both been upgraded to have better correlations with Google's rankings (and they now update every 24 hours) Insanely fast - page load times on the new tool are almost as fast as Google's results 🙂 Less time waiting means more time to dig into results Link Tracking Lists - check a box next to any links of interest and you can build lists in the tool to track them over time, see whether/when they link, prioritize your outreach efforts, and (in the future) get aggregated data and alerts about those links There's much more to come, but we'd love for you to check out Link Explorer, find bugs, report things you love (and don't), and help us make it the best possible product for you and your teams. You can leave feedback here in this Q+A thread, email help@moz.com, or send feedback through the feedback form in Link Explorer.
Link Explorer | | randfish18 -
612 : Page banned by error response for robots.txt
Hi all,
Link Explorer | | ME5OTU
I ran a crawl on my site https://www.drbillsukala.com.au and received the following error "612 : Page banned by error response for robots.txt." Before anyone mentions it, yes, I have been through all the other threads but they did not help me resolve this issue. I am able to view my robots.txt file in a browser https://www.drbillsukala.com.au/robots.txt.
The permissions are set to 644 on the robots.txt file so it should be accessible
My Google Search Console does not show any issues with my robots.txt file
I am running my site through StackPath CDN but I'm not inclined to think that's the culprit One thing I did find odd is that even though I put in my website with https protocol (I double checked), on the Moz spreadsheet it listed my site with http protocol. I'd welcome any feedback you might have. Thanks in advance for your help.
Kind regards0 -
Crawl a node js page - Why can I only see my frontpage?
Hi When i am trying to crawl my website ( https://www.doorot.com/ ) it can only find my frontpage. It's a node js page. Any one had the same problem or know how to crawl my site in order to see all my pages? Kasper
Link Explorer | | KasperClio1 -
Moz crawler showing pages blocked by robots.txt
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc. However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
Link Explorer | | ahyde0 -
Psst Did you see Open Site Explorer's New Link Building Opportunities Section??
Check out Rand's blog post about it for all the details: http://moz.com/blog/open-site-explorers-new-link-building-opportunities-section. We'd love for you to check it out and leave us your feedback! Thanks,
Link Explorer | | jennita
Jen3