URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How does Google's AJAX Announcement Impact the likes of AngularJS?
Google's announcement last month about depreciating their AJAX crawl directive and Distilled's recent article have got me thinking a lot about how this change impacts frameworks like AngularJS. For those of you that use or are considering using frameworks like AngularJS, does this change impact you? Has it changed your mind about services like Prerender etc? All discussions relating to AJAX crawling welcome. Some resources to get started: https://prerender.io/js-seo/angularjs-seo-get-your-site-indexed-and-to-the-top-of-the-search-results/ https://www.distilled.net/resources/prerender-and-you-a-case-study-in-ajax-crawlability/
Web Design | | ecommercebc1 -
Interlinking using Dynamic URLs Versus Static URLs
Hi Guys, Could you kindly help us in choosing best approach out of mentioned below 2 cases. Case. 1 -We are using: We interlink our static pages(www.abc.com/jobs-in-chennai) through footer, navigation & by showing related searches. Self referential Canonical tags have been implemented. Case. 2 -We plan to use: We interlink our Dynamic pages(www.abc.com/jobs-in-chennai?source=footer) through footer, navigation & by showing related searches. Canonical tags have been implemented on dynamic urls pointing to corresponding static urls Query 1. Which one is better & expected to improve rankings. Query 2. Will shifting to Case 2 negatively affect our existing rankings or traffic. Regards
Web Design | | vivekrathore0 -
I need help with international SEO for two sites?
I'll try to keep this clear... I am working with an company based in Germany, they own company.com/de and company.com/en, and that's how they are currently structuring their domains. They also own companyusa.com that they really want to show up in USA only. They want to keep company.com/en for England/english speaking Europe and company.com/de for their German audience in Germany. They are wanting us to optimize/SEO for companyusa.com, and they want that URL to show up as the top google search in the USA for their "company" keyword. What is showing up now is www.company.com/en 1st in Google because it's been around longer and it has more domain authority. What is the best practice for us optimize companyusa.com so that it is the top dog in the USA while not messing up the other domains? Should we merge? Subfolders all around? Thanks for all the input.
Web Design | | Rocket.Fuel0 -
Can anyone recommend a firm for cross-browser testing?
I'd like to outsource the cross-browser testing and get complete functionality testing across browsers too. I work for an online travel tour operator, so we take a lot of reservations online. I don't totally trust my internal resources using tools like Selenium because we're adding a lot of new functionality to our new site - saved itineraries, wish list, some upselling and cross selling functionality, selling insurance, etc. So, while of course I want to be sure everything renders properly across browsers, I also feel like the investment in having someone check booking process in all browsers/technologies is critical especially since many of our customers are accessing our sites using older browsers. Thoughts??
Web Design | | CityWonders0 -
What's so great about Thesis framework?
I keep hearing about how great Thesis is for SEO. But when I look at the code, it doesn't look like anything special to me -- they followed the basics (proper title, header usage, etc.), pages load quickly, and they packaged things like title and meta control with the theme itself, but none of those things seem particularly special to me. Plenty of SEO plugins give you the same control over title & meta (and the best ones go beyond what Thesis offers) and it's easy to make sure the code is clean. What am I missing?
Web Design | | EricOliver1 -
Considering site navigation options
I am working on a site redesign and re evaluating concepts I haven't thought about for a few years. I generally see site navigation that is either "top-down" or "left bar". Top down navigation normally uses the left nav. for search refinements. The benefit of top nav. is that it clears up the center of the page for non navigation content. The drawback is that you can't fit as many categories in a top nav. Left side nav. can hold a long list of categories, but subcategories are often in the center of the page. In the past, I have preferred to use left nav. with a multi level scroll over search refinement. I believe this allowed users to get to their destination page with fewer clicks. (I have always believed that every required additional click causes lost customers). I also believe that this has caused me to get more juice flowing to deeper pages on sites and better long-tail conversion. This means I have had pages with a LOT of links. With this method, I have tightly controlled my categories. What on other sites are often dynamic search refinements, are on my sites additional categories. I am considering making a site with a top down navigation system. I like the additional screen space in the center I get to work with. Is my assumption about pages created by search refinement wrong? Is it ok for SEO to have a left nav that has a bunch of search refinements that are dynamically created?
Web Design | | EugeneF0 -
Should I Remove URL extentions for SEO?
We are having a developer design our website with Magento. I noticed the main pages such as About Us have no file extention in the URL. But the product pages have a .html file extention. I was once told to remove the file extentions. Are there benefits to removing the .html file extension and if so, is there a way we can do this using Magento?
Web Design | | hfranz0 -
Mobile sites! Any advice or suggestions for building one?
Hello Mozland, I'm looking to have a mobile site built. We have a fair amount of traffic coming from smart phones, tablets etc and want to capture this traffic better. Do you have any advice on how to have it built properly, if/how it can be optimised, what good features to include, or anything else you think might be of help? Many thanks Martin
Web Design | | Martin_S0