Robots.txt Download vs Cache
-
We made an update to the Robots.txt file this morning after the initial download of the robots.txt file. I then submitted the page through Fetch as Google bot to get the changes in asap.
The cache time stamp on the page now shows Sep 27, 2013 15:35:28 GMT. I believe that would put the cache time stamp at about 6 hours ago. However the Blocked URLs tab in Google WMT shows the robots.txt last downloaded at 14 hours ago - and therefore it's showing the old file.
This leads me to believe for the Robots.txt the cache date and the download time are independent. Is there anyway to get Google to recognize the new file other than waiting this out??
-
No to my knowledge. You will have to wait. Anyway, Google could have already download the new robots but while the reports are showing the older file. Those reports always take a while until refreshing completely.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
www vs non-www
We have 2 variations of domains. www and non-www
Technical SEO | | aaaannieee
Both can be seen by users and have been linked to on press releases, but only the www one has data on Google Search Console.
In the case, what is the best practice for us?0 -
Spanish United States Vs Puerto Rico Hreflang
Hey Moz, So we are trying to figure out weather it is the same if we have Hreflang for "US-ES" vs "US-PR", IF we do "US-PR" for Puerto Rico for its own links we then have to create 3 parts to our site, PR Spanish PR English US Spanish We looked at Apple as an example and they had a "Latin America" for their Hreflang and labeled everything has either "es-419" is that the same concept as having just "us-es" for Puerto Rico? ( see attached screenshot ) We are trying to figure out what would be more effective and weather or not "US-ES" search results will appear for Puerto Rico also. PZVwg16
Technical SEO | | uBreakiFix0 -
#1 rankings on both HTTP and HTTPS vs duplicate content
We're planning a full migrate to HTTPS for our website which is accessible today by both **www.**website.com, **http://**www.website.com as well as **https://**www.website.com. After the migrate the website will only be accessible by https requests and every other request (Ex. www or http) will be redirected to the same page but in HTTPS by 301 redirects. We've taken a lot of precautions like fixing all the internal links to HTTPS instead of HTTP etc. My questions is: What happened to your rankings for HTTP after making a full migrate to HTTPS?
Technical SEO | | OliviaStokholm0 -
Cache Not Working on Our Site
We redesigned our site (www.motivators.com) back in April. Ever since then, we can't view the cache. It loads as a blank, white page but the cache text is at the top saying: "This is Google's cache of http://www.motivators.com/. It is a snapshot of the page as it appeared on Jul 22, 2013 15:50:40 GMT. The current page could have changed in the meantime. Learn more. Tip: To quickly find your search term on this page, press Ctrl+F or ⌘-F (Mac) and use the find bar." Has anyone else ever seen this happen? Any ideas as to why it's happening? Could it be hurting us? Advice, tips, suggestions would be very much appreciated!
Technical SEO | | Motivators0 -
What is the difference between 301 redirect to 404 vs just 404.
A bunch of pages on my site are set to 301 redirect to our 404 page. Intuitively, I feel like they should all just 404 from the page's url and not redirect to the 404 page. How do I explain to my developer that they should not redirects but should just 404? Is there much of a difference between the redirect first vs 404 first? Thanks!
Technical SEO | | gaytravel0 -
Thesis Vs Yoast Wordpress SEO Plugin
Hi All. I've noticed some issues between the thesis themes and Yoast's plugin, namely, thesis doesn't recognise Yoast's plugin in for title tag and meta description. Is there a way to overide this?
Technical SEO | | PerchDigital0 -
Panda Update Question - Syndicated Content Vs Copied Content
Hi all, I have a question on copied content and syndicated content - Obviously copying content directly form another website is a big no no, but wanted to know how Google views syndicated content and if it views this differently? If you have syndicated content on your website, can you penalised from the lastest Panda update and is there a viable solutiion to address this? Mnay thanks Simon
Technical SEO | | simonsw0