Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Zendesk robots.txt
-
Hello!
We have a Zendesk support site at support.zspace.com - our Moz crawl report is saying that there is 85 temporary redirect issues, mostly coming from our support site.
My question is, does the Moz crawler use robots.txt?
We have currently Disallow: /search in our robots.txt, but not /search/*
Most of the temporary redirect URL's in the crawl error report look like this...
https://support.zspace.com/hc/en-us/search/click?data=BAh7CjoHaWRp
So would we need to add "Disallow: /hc/en-us/search/*" to get Moz crawler to ignore these?
-
Thanks Yossi, this is kind of what I expected I think. I guess the question should have been "has anyone had Moz crawl issues with their Zendesk support site"?
The main issue with our support site is that Zendesk does not allow access to the robots.txt file so there is no way to add regular expressions like the wildcard search/* to it.
I will re-post the question as above.
-
Hi
Mozs crawler aka the RogerBot does obeys the robots.txt and Moz states that under the help section here.
Looks like the problem you are having is connected to your robots.txt statements/directives.
In order to block a directory **and its content,**you need to use the category name following with a forward slash (even if you have a redirect from xxx.com/search to xxx.com/search/)
(Check out Google robots.txt guidelines)if you want to block the directory "search" and its content, use Disallow : /search/ (no need for asterisk at the end)
if you use "Disallow: /hc/en-us/search/" (again, no need for asterisk) you will block all the content under xxx.com/hc/en-us/search/
So, for example if you have content that you want to block under xxx.com/hc/fr-FR/search/ it will not be blocked because your statement/directive limits it to a very specific "search" directory which is located under "en-US".What is it that you want to block exactly?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved How Should I Increase DA of my Website?
Google itself has been troubled by ranking factors. On top of that, the https://moz.com/ has worn a new garland of domain authority. It is even more regrettable when a website DA suddenly decreased. As recently on 22nd September, my website's DA was 16. As usual, on September 23, I posted a new informational blog related to QuickBooks antivirus exclusions on my website. The very next day my website DA was decreased. Can anyone who knows SEO help me understand why this always happens specially to me every time? I read the whole forum and all I found out is how my site's DA is decreased.
Moz Bar | Oct 26, 2024, 4:48 PM | ravigaurav
Can anyone let us know whether we can increase the Domain Authority of our website only by creating backlinks or there can be some other activities also?2 -
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | May 8, 2024, 7:32 PM | jbcorcoran6 -
Spam score of 28 %
Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,
Moz Bar | Apr 8, 2024, 4:50 PM | seoanalytics1 -
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
Moz Bar | Oct 3, 2018, 1:21 PM | AllAboutShapewear2 -
Does "Disallow: /xmlrpc.php" in robots.txt affect moz tools ability to fetch DA?
Just checked a website for Domain Authority using Moz' tool, however it returned 1 for DA, which should be unlikely. I have been trying to find the problem and found "Disallow: /xmlrpc.php" in robots.txt. Could this affect Moz' tools ability to get the required data?
Moz Bar | Jul 5, 2017, 2:37 AM | Foli0