URLs appear in Google Webmaster Tools that I can't find on my own site?!?

[clickyleap]

Hi,

I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.

For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:

http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.

I think I need to:

1. Find how Googlebot is finding these urls?

2. Find out how to keep out of index (e.g. robots.txt, canonical etc....

Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?

Thanks,

Darrell

[johncallaghan]

No problem, glad it resolved the problem.

There are a number of possibilities, probably through one of the following;

• XML sitemap
• Faceted navigation
• Magento pinged Google when the page was created

[clickyleap]

Cheers John, sorted the issue! Appreciate your expertise.

[clickyleap]

Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.

How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?

[johncallaghan]

What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.

Setting the visibility to "Not Visible Individually" should resolve this issue.

[PaddyDisplays]

I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?

check the url in the open site explore too, it might tell you if any urls are linking to it