Google robots.txt test - not picking up syntax errors?
-
I just ran a robots.txt file through "Google robots.txt Tester" as there was some unusual syntax in the file that didn't make any sense to me...
e.g. /url/?*
/url/?
/url/*and so on. I would use ? and not ? for example and what is ? for! - etc.
Yet "Google robots.txt Tester" did not highlight the issues...
I then fed the sitemap through http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php and that tool actually picked up my concerns.
Can anybody explain why Google didn't - or perhaps it isn't supposed to pick up such errors?
Thanks, Luke
-
Many thanks Beau - much appreciated.
-
Hey Luke,
It appears that in each of the three examples, there was a plausible case for each example. Let's cover each:
- For /url/?* , it can be expressed that a URL can offer a trailing slash and then a query string, see examples here.
- with /url/? , this covers examples of the above and in addition, would plausibly block product pages that generate query strings, similar to this example from H&M. In essence, only allowing the product page to be seen.
- /url/* , well, that's just anything and everything after the trailing slash.
I guess the question you should ask yourself is "Is this the best approach for the issue?"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I'm noticing that URL that were once indexed by Google are suddenly getting dropped without any error messages in Webmasters Tools, has anyone seen issues like this before?
I'm noticing that URLs that were once indexed by Google are suddenly getting dropped without any error messages in Webmasters Tools, has anyone seen issues like this before? Here's an example:
Intermediate & Advanced SEO | | nystromandy
http://www.thefader.com/2017/01/11/the-carter-documentary-lil-wayne-black-lives-matter0 -
Google Webmaster tools -Fixing over 20,000+ crawl errors
Hi, I'm trying to gather all the 404 crawl errors on my website after a recent hacking that I've been trying to rectify and clean up. Webmaster tools states that I have over 20 000+ crawl errors. I can only download a sample of 1000 errors. Is there any way to get the full list instead of correcting 1000 errors, marking them as fixed and waiting for the next batch of 1000 errors to be listed in Webmaster tools? The current method is quite timely and I want to take care of all errors in one shot instead of over a course of a month.
Intermediate & Advanced SEO | | FPK0 -
Google Indexing our site
We have 700 city pages on our site. We submitted to google via a https://www.samhillbands.com/sitemaps/locations.xml but they only indexed 15 so far. Yes the content is similar on all of the pages...thought on getting them to index the remaining pages?
Intermediate & Advanced SEO | | brianvest0 -
Meta robots or robot.txt file?
Hi Mozzers! For parametric URL's would you recommend meta robot or robot.txt file?
Intermediate & Advanced SEO | | eLab_London
For example: http://www.exmaple.com//category/product/cat no./quickView I want to stop indexing /quickView URLs. And what's the real difference between the two? Thanks again! Kay0 -
Why, oh why does Google hate us?
My URL is:
Intermediate & Advanced SEO | | candylotus
www.drupalgeeks.org I have tried my very best to cover all the usual SEO items.... But nothing... We are a legitimate company offering a legitimate service. Ideally we would come up in the results for: "Drupal Developers"
"Drupal Development" and
"Drupal Designers" Yet, we cannot break the top 50 for any of these.... We have:
Optimized Meta Tags
Written Quality Content
Maintained a Social Presence in Twitter, LinkedIn, Facebook, Pinterest, Youtube, and Google Plus
Blogged with some consistency
Guest Blogged
Canonicalization
And dozens of other things And we are all around nice people and a good company....... Why oh why? Can anyone take a look and see if there is something blatantly obvious I am missing? Are we using "drupal" too much?? Thank you in advanced for any assistance. Candice1 -
Google badge extracted to SERPs
It's a while ago that (i thought) I read the following information on the Google badge. Here https://developers.google.com/+/plugins/badge/ we have the implementation guide, however I was under the impression the Google badge could be thereafter extracted into SERPs so the user could follow etc... direct from SERPs. I can't find anything confirming this. I think that it might clash with authorship data which does a similar job, but where a site page is not relevant to authorship at all, I would have thought linking back to the G+ page from SERPs was a sensible option. Can anyone confirm the Google badge can appear in SERPs?
Intermediate & Advanced SEO | | richcowley0 -
Hitting the top of Google Places!
Hi there, I am quite confused about how to get a good rankings on Google Places? It will be much appreciated to get an answers from PROs on following questions: On which websites I need to submit my details in order to rank well? Are there any rules which my website need to follow in order to rank well in GP? **Which techniques are you implementing in order to find proper websites influencing on Google Places? ** If there any 3rd party websites which helps to do so? Could Google Adwords have an impact on my presence in Google Places? Any major mistakes I should to avoid?**** Your detailed answers will help a lot, I am really upset about my knowledge about GP. Cheers, Russel
Intermediate & Advanced SEO | | smokin_ace0 -
Google.ca vs Google.com Ranking
I have a site I would like to rank high for particular keywords in the Google.ca searches and don't particularly care about the Google.com searches (it's a Canadian service). I have logged into Google Webmaster Tools and targeted Canada. Currently my site is ranking on the third page for my desired keywords on Google.com, but is on the 20th page for Google.ca. Previously this change happened quite quickly -- within 4 weeks -- but it doesn't seem to be taking here (12 weeks out and counting). My optimization seems to be fine since I'm ranking well on Google.com: not sure why it's not translating to Google.ca. Any help or thoughts would be appreciated.
Intermediate & Advanced SEO | | seorm0