Wordpress Blog Blocked by Metarobots
-
Upon receiving my first crawl report from new pro SEOMoz acc (yaay!) I've found that the wordpress blog plugged into my site hasn't been getting crawled due to being blocked by metarobots.
I'm not a developer and have very little tech expertise, but a search dug up that the issue stemmed from the wordpress site settings > privacy > Ask search engines not to index this site option being selected.
On checking the blog "Allow search engines to index this site" was selected so I'm unsure what else to check. My level of expertise means I'm not confident going into the back end of the site and I don't have a tech guy on site to speak to.
Has anyone else had this problem? Is it common and will I need to consult a developer to get this fixed?
Many thanks in advance for your help!
-
I didn't think there were any issues with the blog being crawled. I'm not seeing any errors in webmaster tools, and I'm def not doing anything tricky on the server side.
I don't even go near that stuff for fear of breaking summat.
Really appreciate your help Barry.
All the best,7
Pete
-
There shouldn't be a robots.txt file on the /blog section anyway, should always be in the root. It was just something to have a look at.
I'm having a look just now and also don't see any problems.
You've nothing in the robots.txt file and nothing in meta-robots for the header.
There's 42 pages in the site: command and a similar number in your sitemap.xml so I presume that's right. 6 pages in site:/blog which again looks right.
I've tried using SEOmoz's tools on your site though and it just tells me that your site doesn't resolve. edit Managed to get it to resolve on the 3rd try for a crawl, but using the on page report card checker it's still giving me problems.
You're definitely returning a 200 message with a site when I check using any other tool though, so I'd get in touch with SEOmoz directly and see what's wrong with their tool - help@seomoz.org
Just to confirm you're not doing anything tricky server side to prevent scraping are you?
-
Hi Barry,
Thanks for the reply, I'm checking out your recommendations now..
I checked http://debtmadesimple.co.uk/robots.txt and there is no Disallow for the blog.
I tried http://debtmadesimple.co/uk/wp-install/robots.txt I can't access the file you speak of.
I will try and download the plugin you mentioned, it would be good to get access to the robot file nonetheless.
Thanks again!
Pete
-
Hi Zach,
First I'd like to thank you for the speedy reply, I really appreciate your help.
The URL of the blog is http://www.debtmadesimple.co.uk/blog/.
Thanks again!
Pete
-
If you're not taking Zach up on his offer, have a look at http://yoursite.com/robots.txt and see if it has
User-agent: *
Disallow: (your blog url in here)If it does you'll need to edit your robots.txt file to not have anything you don't want disallowed in the disallow section. You can do this via ftp.
If it's in WP itself there may be another robots.txt file at http://yoursite.com/wp-install/robots.txt which, in theory, could also be preventing crawling if it has anything disallowed in there.
Again, editable via ftp or maybe this plugin - http://wordpress.org/extend/plugins/wp-robots-txt/
As it already says that it should be public probably not WP, but worth a look anyway.
-
I'm a WP developer and an SEO, i'd be more than willing to do some troubleshooting here on the forums for you. If the settings>privacy is checked to allow search engines to crawl, then I doubt it's a WordPress issue in itself, though a plugin could do this.
What is the URL of your site? You may have a robots.txt that is blocking search engine crawlers, i've also seen a thing where all URLs on the site are noinexed and nofollowed.
Let me know and i'll take a quick look for you.
Zach
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Extremely slow TTFB (WordPress)
Hi guys. I’m not sure if this is a relevant place to post such question, but I’m really desperate. The TTFB on my website is ridiculously slow no matter what I do. I’m using Apache and the latest version of WordPress on a VPS (NameCheap). 2 CPU Cores 2 GB RAM 40 GB SSD RAID 10 1000 GB/mo Bandwidth GZIP compression is enabled, images are optimized, CSS, JS, HTML optimized, caching enabled (WordPress plugin), CDN is enabled (Cloudflare), PHP updated to the latest version. As I said, no matter what I do, TTFB remains very slow (most of the time the TTFB of my website is between 1.5 - 3 seconds). Google Page-speed Insights score is 86 for mobile and 95 for desktop (with 1.5s TTFB). What do I do about it? Will it affect the rankings of my website? Thank you so much.
Technical SEO | | AslanBarselinov0 -
Swapping Homepages in WordPress
Hi, Th situation is: I've cloned my homepage & added new content to the cloned page I've then updated the WordPress settings to make the cloned page the new homepage Will I lose PA as the new cloned page is in effect a new article?
Technical SEO | | jasongmcmahon0 -
I have a Category and Tag In My Blogs
I have use category and Tags in my blogs. Now i have an problem with blog URL and Tags URL. My blog URLs is also show in Tags page and both the content is same. For Example: My Blog URL is: https://www.example.com/advice-how-to-do-batting And Tag Page URL is : https://www.example.com/advice-batting in that - https://www.example.com/advice-how-to-do-batting The URLs contain same content. No should i write two different meta title and description for above two URLs pages. As there might more blog added under Tags pages with different topics and title. Request on Thought Please.
Technical SEO | | ProcessSEO0 -
Google ranks my sitemap.xml instead of blog post
Hello, For some reason Google shows sitemap results when i search for my blog url website.com/blog/postwhy is Google ranking my sitemap but not a post, especially when i search for full URL? Thanks
Technical SEO | | KentR0 -
Why I am a seeing an error for duplicate content for any categories and tags on my Wordpress blog?
When I look under "Crawl Diagnostics" I see I have 12 errors for duplicate content and there are all from tags and categories. I am assuming that search engines are reading the content in the tags and categories as duplicate. Should I set my categories to "no-index?"
Technical SEO | | brytewire0 -
Business/Personal Blog Duplicate Content
Quick Question. I am in the process of launching a new website for my IT business which will include a blog. I also want to start up my personal blog again. I want to publish some blog posts to both my business and personal blogs but I don't want to have any duplicate content issues. I am not concerned with building the SERPs of my personal blog but I am very focused on the business blog/site. I am looking for some ideas of how I can publish content to both sites without getting hurt by duplicate content. Again, I am not concerned with building up the placement of my personal site but I do want to have a strong personal site that helps build my name. Any help on this would be great. Thanks!
Technical SEO | | ZiaTG0 -
How to write 301 redirects in WordPress
I've successfully migrated new site to new domain (www.cmsearchmarketing.com) But I cannot get 301 redirects for pages and blog posts to redirect from the old domain (www.creativemindsearchmarketing.com). And it's my understanding I need to do a 301 for each page to maintain SEO. Here's what I've tried: RewriteCond %{QUERY_STRING} ^p=975$RewriteRule ^index.php$ http://www.cmsearchmarketing.com/top-5-questions-to-ask-an-seo-firm-before-signing-up/? [R=301,L] BEGIN WordPress<ifmodule mod_rewrite.c="">RewriteEngine OnRewriteBase /RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-dRewriteRule . /index.php [L]</ifmodule># END WordPress #AND ALSO# Use PHP5 Single php.ini as defaultAddHandler application/x-httpd-php5s .php BEGIN WordPress<ifmodule mod_rewrite.c="">RewriteEngine OnRewriteBase /RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-dRewriteRule . /index.php [L]</ifmodule># END WordPress redirect 301 /top-5-questions-to-ask-an-seo-firm-before-signing-up http://www.cmsearchmarketing.com/top-5-questions-to-ask-an-seo-firm-before-signing-up/ Any suggestions would be appreciated. _Cindy P.S. Maybe some other issues are in the way: --Old site is WP-Remix theme no longer supported, and latest WP version is 2.9.1 -- Old domain (www.creativemindsearchmarketing.com) is the primary account on BlueHost …and the new domain (www.cmsearchmarketing.com) is an addon, so the new domain's directory is within root of old domain. -- in root domain of old site there are other "handler files" that also have base file rewrites, if this is an issue: name of this file in root directory is:
Technical SEO | | CeCeBar
.htaccess.addHandlerBak -FrontPage- <limit get="" post="">order deny,allowdeny from allallow from all</limit><limit put="" delete="">order deny,allowdeny from all</limit>AuthUserFile /home/creatjo7/public_html/_vti_pvt/service.pwdAuthGroupFile /home/creatjo7/public_html/_vti_pvt/service.grp# BEGIN WordPress<ifmodule mod_rewrite.c="">RewriteEngine OnRewriteBase /RewriteCond %{REQUEST_FILENAME} !-fRewriteCond %{REQUEST_FILENAME} !-dRewriteRule . /index.php [L]</ifmodule> END WordPressAuthName creativemindsearchmarketing.comIndexIgnore .htaccess /.?? *~ *# /HEADER /README /_vti0 -
Ping Services for Wordpress
I have been trying to research adding ping services to the default list in wordpress, which only includes http://rpc.pingomatic.com/ A lot of the articles that I have come across seem to be quite old, can anyone tell me if this is still relevant and if so, which services do you find to be worthwhile using? Thanks
Technical SEO | | pugh0