What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
-
Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:
My current robot.txt settings are as follows;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: */xmlrpc.php Disallow: */wp-*.php Disallow: */trackback/ Disallow: *?wptheme= Disallow: *?comments= Disallow: *?replytocom Disallow: */comment-page- Disallow: *?s= Disallow: */wp-content/ Allow: */wp-content/uploads/ ```What to I need to allow/disallow to allow Google spiders to properly read my website?
-
Remove the following line:
Disallow: */wp-content/ And add: Allow: /wp-includes/js/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens if we remove all the links to internal pages from our homepage?
Hi Moz community, We wanna give a try by removing all the links from homepage to internal pages and keep just a free trial button. Will this impact our SEO anyway? We have nearly 15 important internal pages at 2nd and 3rd hierarchy level. They may drop in rankings but we want to risk for few days to understand how it works. Your opinion please! Thanks
Algorithm Updates | | vtmoz0 -
Google ranking impact: Returning visitor vs New visitor
Hi all, If a website's traffic increase in "New visitors"; will this impact rankings? Do the website overall traffic affect rankings? How much this is related with ranking improvement for main keywords? Just because thousands of visits increased for website, will it count as a strong ranking improvement signal? Thanks
Algorithm Updates | | vtmoz0 -
Using Google to find a discontinued product.
Hi Guys. I mostly use this forum for business questions, but now it's a personal one! I'm trying to find a supplier that might still have discontinued product. It's the Behritone C5A speaker monitor. All my searches bring up a plethora of pages that appear to sell the product... but they have no stock. (Wouldn't removing these pages make for a better internet?) No 2nd hand ones on eBay 😞 Do you have any suggestion about how I can get more relevant results... i.e find supplier that might still have stock? Any tips or trick I may be able to use to help me with this? Many thanks in advance to an awesome community 🙂 Isaac.
Algorithm Updates | | isaac6631 -
A company claiming to have a proprietary software that replicates Google algorithm?
Hi all, Unfortunately, getting into a bit of a p*ssing match 😞 with a company trying to compete for the business of one of our clients and just wanted to some feedback from the community here. The company competing for the client's business claims to have spent $1 million to replicate Google's algorithm so they create a replica site (not sure I understand this) of the client site, then test and optimize on-page SEO changes in their software to determine whether the on-page changes are ideal. Sounds fishy to me. Thoughts?
Algorithm Updates | | RickyShockley0 -
Best and easiest Google Depersonalization method
Hello, Moz hasn't written anything about depersonalization for years. This article has methods, but I don't know if they are valid anymore. What's an easy, effective way to depersonalize Google search these days? I would just log out of Google, but that shows different ranking results than Moz's rank tracker for one of our main keywords, so I don't know if that method is correct. Thanks
Algorithm Updates | | BobGW0 -
A Google Update Happened?
I'm curious to know what us MOZ folks have to say about an update on Google. Article here: http://searchengineland.com/big-google-search-update-happening-chatter-thinks-258142 Any ideas?
Algorithm Updates | | Chenzo0 -
Recent Google algorithm update?
Two of our clients have experienced a huge dip in organic rankings during the past week or so and we haven't done anything that would cause this. Have there been any major Google changes reported lately? I'm not seeing anything reported here: https://moz.com/google-algorithm-change. Thanks for your input. Eric
Algorithm Updates | | EricFish0 -
Google webmaster tool content keywords Top URLs
GWT->Optimization->Content Keywords section... If we click on the keyword it will further shows the variants and Top URLs containing those variants. My problem is none of the important pages like product details pages, homepage or category pages are present in that Top URLs list. All the news, guides section url's are listed in the Top URLs section for most important keyword that is also present in my domain name. How to make google realize the important pages for the important keyword?
Algorithm Updates | | BipSum0