How many times robots.txt gets visited by crawlers, especially Google?
-
Hi,
Do you know if there's any way to track how often robots.txt file has been crawled?
I know we can check when is the latest downloaded from webmaster tool, but I actually want to know if they download every time crawlers visit any page on the site (e.g. hundreds of thousands of times every day), or less.
thanks...
-
Your web server logs keep track of every file that is serves and to whom it provides the data. The raw log files are difficult to read unless you are used to viewing that type of code. The process varies based on your server type. Ask your web host or developer for guidance.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Adding your sitemap to robots.txt
Hi everyone, Best practice question: When adding your sitemap to your robots.txt file, do you add the whole sitemap at once or do you add different subcategories (products, posts, categories,..) separately? I'm very curious to hear your thoughts!
Technical SEO | | WeAreDigital_BE0 -
Google is shortening many of my title tags although they are already quite concise
Hi, Title tags of our website are being truncated by Google even though they can be very short (sometimes < 40 characters) and with very few capital letters. We would like to understand why. Example: Principal component analysis (pca) in abcde - OurBrand shows up as: Principal component analysis (pca) in abcd... - OurBrand where abcde is the name of a very common software (5 characters), and OurBrand is a 6 characters long string (could be used in either lower case or upper case). Even when removing the brackets around pca, truncation still occurs... Any clue why?
Technical SEO | | trigaudias1 -
What can i do to get google to visit my site more often
Hi, i am having serious problems since i upgraded my website from joomla 1.5 to 3.0 We have dropped down the rankings from page one for the word lifestyle magazine, and we have dropped down in rankings for other very important words including gastric band hypnotherapy and i am starting to regret having the site upgraded. i am finding the google is taking its time visiting my site, i know this for two reasons, one i have checked the cache and it is showing the 2nd july and i have checked articles that we have written and they are still not showing. example if i put this article name in word for word it does not come up, Carnival Divert Ships In The Caribbean Due To bad Weather this was an article that was done yesterday. in the old days before the upgrade that would have been in google now. these problems are costing us the loss of a great deal of traffic, we are losing around 70% of our traffic since the upgrade and would be grateful if people could give me advice on how to turn things around. we add articles all the time. each day we add a number of articles, i was considering changing the front page in the middle and having a few paragraphs of the latest story to get google to visit more often. i know this would look messy but i am running out of ideas. any help would be great
Technical SEO | | ClaireH-1848860 -
Google description problem
Hi all, My website is www.ipbskinning.com I'm having a problem with how my site is appearing in google. I have this in the head of my website: <meta name='<a class="attribute-value">description</a>' content='<a class="attribute-value">Free and Custom IPB Skins for Invision Power Board.</a>'/> Yet when I google 'ipbskinning' it says: Solid Skins. 1We test all our skins in all browsers to insure that they are compatible. This ensures that your users have the best user experience. which is random text from the content of my site. Any idea why this is happening? Thanks a lot all
Technical SEO | | pezza34340 -
How can we get google to know we are a lifestyle magazine
hi, i have a website which is a lifestyle magazine www.in2town.co.uk before we had to build out site from scratch due to a major error from our hosting company we ranked number one for the word lifestyle magazine but since then we have always ranked number five. This week that has dropped to number seven and i would like to know, what do we need to do to let google know we are a lifestyle magazine. I am not sure what we need to do. i know we should have important keywords on the home page but i am not sure where to put these without making the site look silly. can anyone let me know if we should have an introduction and if so where shall we put this. we would really like to increase our page rank for lifestyle magazine but are struggling to know where to put text that will not make the site look silly. Also, can anyone let me know what other keywords we should be aiming for please; We are currently making major changes to our site to make it better.
Technical SEO | | ClaireH-1848860 -
Getting Listed on Google Satellite Maps
How does a website get listed in the satellite version of Google Maps? I have clients that are ranking number 1 in the pack 7 with Google Places and local SEO but they don't show up on the big map. Click on this link to see what I mean http://connect7.biz/google-maps.jpg I have circled the listings in red. How are these businesses getting listed in the satellite version of Google Maps? Thanks
Technical SEO | | fun52dig
Gary0 -
Getting subdomains unindexed
If i turn an application off displaying a 503 error will that get my site unindexed from search engines?
Technical SEO | | kchandler0 -
Robot.txt pattern matching
Hola fellow SEO peoples! Site: http://www.sierratradingpost.com robot: http://www.sierratradingpost.com/robots.txt Please see the following line: Disallow: /keycodebypid~* We are trying to block URLs like this: http://www.sierratradingpost.com/keycodebypid~8855/for-the-home~d~3/kitchen~d~24/ but we still find them in the Google index. 1. we are not sure if we need to specify the robot to use pattern matching. 2. we are not sure if the format is correct. Should we use Disallow: /keycodebypid*/ or /*keycodebypid/ or even /*keycodebypid~/? What is even more confusing is that the meta robot command line says "noindex" - yet they still show up. <meta name="robots" content="noindex, follow, noarchive" /> Thank you!
Technical SEO | | STPseo0