Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have a Category and Tag In My Blogs
I have use category and Tags in my blogs. Now i have an problem with blog URL and Tags URL. My blog URLs is also show in Tags page and both the content is same. For Example: My Blog URL is: https://www.example.com/advice-how-to-do-batting And Tag Page URL is : https://www.example.com/advice-batting in that - https://www.example.com/advice-how-to-do-batting The URLs contain same content. No should i write two different meta title and description for above two URLs pages. As there might more blog added under Tags pages with different topics and title. Request on Thought Please.
Technical SEO | | ProcessSEO0 -
How to use robots.txt to block areas on page?
Hi, Across the categories/product pages on out site there are archives/shipping info section and the texts are always the same. Would this be treated as duplicated content and harmful for seo? How can I alter robots.txt to tell google not to crawl those particular text Thanks for any advice!
Technical SEO | | LauraHT0 -
How to add specific Tumblr blogs into a disavow file?
Hi guys, I am about to send a reconsideration letter and still finalizing my disavow file. The format of the disavow is Domain:badlink.com (stripping out to the root domain) but what about those toxic links that are located in tumblr such as: badlink.tumblr.com? The issue is that there are good tumblr links we got so I don't want to add just tumblr.com so do you guys think I will have issues submitting badlink.tumblr.com and not tumblr.com? Thank you!
Technical SEO | | Ideas-Money-Art0 -
Duplicate Page Title for multilingual wordpress site
Hello all, I have received my first crawl reports and I find a lot of errors of duplicate page title. In the wordpress site I use the qtranslate plugin in order to have the site in 2 languages. I also use the Yoast SEO plugin in order to put titles, description and keywords to each web page. By looking deeply in the duplicate page title errors I think I found that the problem is that every web page takes the same SEO Title for each language. But I am not 100% sure. I tried to use some shortcodes of the qtranslate plugin like the following ABOUT [:en]About in order to indicate and give different titles per language for one web page but that doesn't seem to work. Does anybody here has experienced the same problem as me? Do you have any suggestions about how to ressolve the problem of the duplicate page title? I can give you the URL of the website if you need it to have a look. Thank you in advanced for your help. I really appreciate that. Regards, Lenia
Technical SEO | | tevag0 -
Unnecessary pages getting indexed in Google for my blog
I have a blog dapazze.com and I am suffering from a problem for a long time. I found out that Google have indexed hundreds of replytocom links and images attachment pages for my blog. I had to remove these pages manually using the URL removal tool. I had used "Disallow: ?replytocom" in my robots.txt, but Google disobeyed it. After that, I removed the parameter from my blog completely using the SEO by Yoast plugin. But now I see that Google has again started indexing these links even after they are not present in my blog (I use #comment). Google have also indexed many of my admin and plugin pages, whereas they are disallowed in my robots.txt file. Have a look at my robots.txt file here: http://dapazze.com/robots.txt Please help me out to solve this problem permanently?
Technical SEO | | rahulchowdhury0 -
If I reply to a PR7 blog does it create a backlink?
Hi, I'm going through my blog spam shite and I've come across some PR7 questions, if I reply which I think they want me to, does it create a backlink trust? I have a few PR4's from SEO companies outside the UK also, will this help my trust? Thanks Mark http://www.thehairroom.co.uk/blog
Technical SEO | | smoki6660 -
Weebly Blog
Hello, Previously I asked if I should remove my blog from Weebly if I cannot change the page titles as they are being reported as duplicates. The answer was,yes and I headed to Weebly to remove it. Upon looking closer, every page and post does in fact show a different address. Now Im confused, I dont see duplicates but the campaign lists all the blog pages and states they are duplicates. I'm not sure what to do. Am I misunderstanding duplicate page? Thank you.
Technical SEO | | Gardengirl0 -
Guest Blog Posts and PR
Let's say I do a guest blog post on a PR5 site. Let's also say the post is deep within the site, and the page rank that flows to it is only maybe PR1 or 2. Let's also say it doesn't get many links to it so it forever stays that way. Will Google realize even though the PR of that page is low, it's on a PR5 site, thus giving it more link power than it the overall site were a PR2 site? Did that make any sense?
Technical SEO | | UnderRugSwept0