Help, a certain directory is not being indexed
-
Before I start, dont expect this to be too easy. This really has me puzzled and am surprised I am still yet to find a solution for it. Get ready.
We have a wordpress website, launched over 6 months ago and have never had an issue getting content such as pages and post pages and categories indexed. However, I some what recently (about 2 months ago) installed a directory plugin (Business Directory Plugin) which lists businesses via unique urls that are accesible from a sub folder. Its these business listings that I absolutely cannot get indexed.
The index page to the directory which links to the business pages is indexed, however for some reason google is not indexing all the listing pages which are linked to from this page. Its not an issue of the content being uncrawlable or at least dont think so as when I run crawlers on my site such as xml sitemap crawlers it finds all the pages including the directory pages so I am sure its not an issue of the search engines not finding the content.
I have created xml sitemaps and uploaded to webmaster tools, tools recongises that there are many pages in the xml sitemap but google continues to only index a small percentage (everything but my business listings).
The directory has been there for about 8 weeks now so I know there is a issue as it should of been indexed by now.
See our main website at www.smashrepairbid.com.au and the business directory index page at www.smashrepairbid.com.au/our-shops/
To throw in a curve ball, in looking into this issue and setting up tools we noticed a lot of 404 error pages (nearly 4,000). We were very confused where these were coming from as they were only being generated from search engines - humans could not access the 404s and so we are guessing se's were firing some javascript code to generate them or something else weird. We could see the 404s in the logs so we know they were legit but again feel it was only search engines, this was validated when we added some rules to robots.txt and we saw the errors in the logs stop. We put the rules in robots txt file to try and stop google from indexing the 404 pages as we could not find anyway to fix the site / code (no idea what is causing them). If you do a site search in google you will see all the pages that are omitted in the results.
Since adding the rules to robots, our impressions shown through tools have jumped right up (increased by 5 times) so thought this was a good indication of improvement but still not getting the results we want.
Does anyone have any clue whats going on or why google and other se's are not indexing this content? Any help would be greatly appreciated and if you need any other information to assist just ask me.
Really appreciate anyone who can spare their time to help me, I sure do need it.
Thanks.
-
OK issue resolved!
Lynn thank you - was the relative url in the canonical tag that played havoc Changing it to absolute is now causing the pages to be indexed.
Lesson learnt.
-
Hey Kane,
The /shops url was a old url that had a directory in it. We blocked it in the robots as it was generating tons of 404 errors. In webmaster tools we can see thousands of 404 errors within that directory so we deleted it all and tried to block se's from throwing the errors (like i described in initial post).
A number of those listing do have very little information however there are a bunch that do have great content which is why I am not sure if that is the case. I will keep an eye on this though and also check about the logs and let you know what that says.
-
Thanks Lynn.
I have taken on your recommendation and changed the canonical tag to be absolute. Thanks for your help we will see how it goes.
-
As Lynn said, relative canonical tags could absolutely cause issues. That said, I'm seeing absolute URLs in the canonical tag now, so you may have fixed that in the past few days.
Also, I do see the Our Shops pages indexed when I search for site:smashrepairbid.com.au, but I don't see any other pages in the /our-shops/ directory aside from www.smashrepairbid.com.au/our-shops/?action=search
Your robots.txt is currently blocking /shops/. I don't think that would cause an issue but would be nice to remove that if it's not needed...
There's almost zero content on the pages I glanced at, eg. http://www.smashrepairbid.com.au/our-shops/1263/bakker-towing/ and http://www.smashrepairbid.com.au/our-shops/1616/coastal-towing-service/. When you look at it from Google's perspective, there's very little value being added by these pages. No unique photos, no phone number, no website, etc. There's a million local business scrapers that have more content than this, so why should they bother indexing these pages?
Try pulling up your logs and seeing if these URLs have been requested by Google's spiders. Here's a good guide from Ian Lurie on how to do that in Excel: http://www.portent.com/blog/analytics/how-to-read-a-web-site-log-file.htm
If the spiders are crawling those shop URLs but aren't indexing them, I think the first thing to do is add way more content to the pages.
-
Hi Trent,
Having a quick look I saw that you have relative urls in your canonical tag and this could be problematic. I think it would be worth making those urls absolute to avoid any confusion on Google's part in determining what page or page version should be indexed.
Cannot say for sure if this is the problem, but worth looking into.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Single Page on my client's website is not crawling and indexing new changes. What could be the possible reason?
I made several changes on client's website on different pages, changed titles, add content on few pages, moved blog from subdomain to sub directory. Everything is crawled but there is one page on the website (not part of the blog) that isn't getting crawled in Google and picking up changes. The last crawl of the website is 2 days back whereas that page was last crawled on 30th sep. I just wanted to know the possible reasons and has anyone encountered this before?
On-Page Optimization | | MoosaHemani0 -
Is it convinient to use No-Index, Follow to my Paginated Pages?
I have a website http://www.naukrigulf.com and it has a lot of Paginated pages on its SERP and most of paginated pages are getting indexed in Google SERP. Is it beneficial to use No-Index, Follow to keep the link equity to main (first page), although we have already used rel=next and rel=prev. If Answer is "yes" is their any harm by using no-index, follow with rel=next, rel=prev.
On-Page Optimization | | vivekrathore0 -
Catergories not appaearing in google help!
Hi I call upon the seomoz experts again this time regarding getting our categories to show in google when you type in our company name Tidy Books. Ive added 3 pictures 1 of our UK store www.tidy-books.co.uk 2 of US store - www.tidy-books.com 3 of a competitor who has over taken us and had the Google mark up we require. Would some love help some help in the last few months we've lost a few places and traffic is going down Thanks everyone ItiqzWV 6g8e1Zy I2qEJCc
On-Page Optimization | | tidybooks0 -
Do sites built with WordPress work in China get indexed by Baidu?
I think that WordPress.com is banned, but are websites that are created by WordPress get banned?
On-Page Optimization | | CoGri0 -
Why Aren’t All My XML Sitemap Images Indexed in Webmaster Tools?
Hi, Here is our main sitemap http://www.vistastores.com/newsitemap/main_sitemap.xml We have submitted all category wise sitemap having Image Tags : For eg - Ac Category http://www.vistastores.com/newsitemap/window_ac_sitemap.xml contains iamge tag - image:imageimage:locimage:captionimage:title</image:title></image:caption></image:loc></image:image> All our 142 category pages includes these format. Still the sitemap report on 4-Apr-2013 says: Sitemaps content Web pages:
On-Page Optimization | | CommercePundit
Submitted 14,569
Indexed 11,219 Images:
Submitted 21,442
Indexed 11,762 You can see major difference in submitted v/s indexed. I have looked into Jay Simpson question - http://www.seomoz.org/q/any-idea-why-our-sitemap-images-aren-t-indexed to find this answer but didn't get Perfect & clear answer. I need urgent answer to fix this issue..... K0NDuw5s.jpg0 -
Help on Avoid Keyword Self-Cannibalization
i ran the on page test and i got the avoid keyword self-canabilzation alert one example is for http://www.deporvillage.com/shimano maybe is because a'm not a native speaking english person but i do not understand what it exactly means the targeted keyword for this page is shimano and the alert i got for the links like the one below that i have in my page http://www.deporvillage.com/ciclismo/zapatillas-shimano which has a anchor text like zapatillas ciclismo shimano
On-Page Optimization | | deporvillage0 -
I changed my site from HTML to PHP and I need to get some help.
Ok...so the other day I went from HTML to PHP in every part of my website. I want to know the best option for me for redirecting my pages from HTML to php. I had my site scanned with SEOMoz and I was given many 404 errors which is not at all good. I do not have any pages of my site linking to any of these html pages. All of the site links have been updated. I have checked 3 times. I have never created a robots.txt file so I would love to get a little help with this part. I was thinking it would be best to tell Google not to worry about these pages in the file. I kept the pages up and I plan to remove all code with them so that no content shows up if someone visits but the issue with that is my site is already indexed as HTML. I want to have the HTML pages redirect to the PHP without worrying that my visitors will land on my site via Google onto an HTML page. I hope I am making sense. What is the best advice you can give me. I need all pages to redirect to PHP. I used an htaccess redirect from all HTML to PHP but when I get so many of them added I get an error on my site saying too many redirects. Seriously need help.
On-Page Optimization | | TrendyHost0 -
Google is indexing spam pages from my site. What is the most effective way to get ride of the search results? Pages are deleted now but should I do something more?
A long time ago I created a forum (Invision Power Board) and it got full of spam. Massive amounts! /forum/ I've now deleted the forum but the spam pages are still indexed on Google. Can I do something else to hurry up the process to get ride of them?
On-Page Optimization | | ocarlsson0