URLs appear in Google Webmaster Tools that I can't find on my own site?!?
-
Hi,
I have a Magento e-commerce site (clothing) and when I had a look through some of the sections in Google Webmaster Tools I found URLs that I can't find on my site.
For example, a product url maybe http://www.example.co.uk/product-url/ which is fine. In that product there maybe three sizes of the product (Small, Medium, Large) and for some reason Googlebot is sometimes finding a url like:
http://www.example.co.uk/product-url/1202/ has been found and when clicked on is a live url (Status code: 200) with is one of the sizes (medium). However I have ran a site crawl in Screaming Frog and other crawl tests and can't seem to find where Googlebot is finding these URLs.
I think I need to:
1. Find how Googlebot is finding these urls?
2. Find out how to keep out of index (e.g. robots.txt, canonical etc....
Any help would be much appreciated and I'm happy to share the URL with members if they think they can have a look and help with this problem. I can share specific URLs which might make the issue seem clearer, let me know?
Thanks,
Darrell
-
No problem, glad it resolved the problem.
There are a number of possibilities, probably through one of the following;
- XML sitemap
- Faceted navigation
- Magento pinged Google when the page was created
-
Cheers John, sorted the issue! Appreciate your expertise.
-
Thanks John, your reply was really helpful and I've now done that for the 4000 simple product and now those URLs are returning 404 pages, which is great. Well, just going to see if I can find a mass import 301 redirect extension for Magento to 301 redirect these urls to the homepage so I can redirect them rather than leave as 404 pages.
How do you think Googlebot found those pages as there is no links to them? Maybe through a link when the simple products were loaded to the cart?
-
What is the visibility set to on the simple products for different sizes? If it's set to "Catalog" it will still be crawlable but not appear in your website's internal search results.
Setting the visibility to "Not Visible Individually" should resolve this issue.
-
I had a similar issue (not Magento), turns out it was in the sitemap that was submitted to WMTs, did you check there?
check the url in the open site explore too, it might tell you if any urls are linking to it
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I find why I have a 302 redirect?
I have a 10 year old website and I have just recently changed to a https: version and my host ionos setup a 301 redirect. I have noticed on my Moz reports that as well as a 301 redirect there is also a 302. How do I find why I have a 302 redirect? I have looked in my htaccess file and it says:- RewriteCond %{SERVER_PORT} 80
Web Design | | Ast9858
RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R,L] I'm assuming this is the 301 redirect, where will I find the 302 redirect. Please note that I am using Wordpress that was not setup through ionos. Does anyone know where a 302 redirect could also be installed or how I find why its there or where its coming from? Thanks Andrew0 -
Site Speed for Magento Site
We have our site all ready for test users but the site speed is painfully slow. Our developers are a bit stymied about what to do. Is anyone a site speed expert or know of anyone good that can evaluate our site and tell us what's going wrong? Thanks!
Web Design | | EcomLkwd1 -
Responsive design and Google analytics mobile tracking codes?
Hi all, We are currently rebuilding a site using responsive design, however i have just had a thought. On another site where we have a mobile site under a sub-directory we utilise mobile tracking codes as we found that this was far more accurate for recording visits. On a responsive design site evidently all pages, desktop and mobile, will be under the same URL, yet the content will adjust to the screen size of the device. Should we also change the tracking code to be mobile code on the lower resolutions or would the same code be sufficient?
Web Design | | Sarbs0 -
How do I optimize a site designed to be one scrolling page of content?
Our website uses section ID's as its navigation so all the content is on one page. When you click About Us, the page scrolls down to About Us. Products, the page scrolls to Products section, and etc. I am getting crawl errors for meta descriptions but will this go away once the main domain has this info? We just added the meta keywords and description to the header and since the navigation sections use the same page, I assume it will correct the errors. Any other advice on optimizing for site designs like ours would be great. www.theicecubekit.com is the site. Thanks,
Web Design | | bangbang
Chris0 -
Site down for more than a month - lost rankings
Hello, We have run into a situtation where we had multiple pages setup for different keywords but didn't realize that we had a name server issue that has caused the pages to be down for the last month or so (2-3 weeks on the low side.) The rank finder was still working fine, but the offline page was never reported. We realized the situation recently and have since gotten the sites back online under the new nameservers. Most of these sites were ranking 1 and 2 spots in their keywords, and now are no where to be found in the Google Index. Should I do anything differently, or just put the sites back online and wait it out? I have seen in different places that it may only take 2 weeks to come back, but it's possible that Google has marked the sites as 'not quality' because of their downtime and it will be even harder to get them to rank again. Can anyone shed any light on this situation? Any information is appreciated. Thanks in advance.
Web Design | | EQ-Richie0 -
Can anyone recommend a firm for cross-browser testing?
I'd like to outsource the cross-browser testing and get complete functionality testing across browsers too. I work for an online travel tour operator, so we take a lot of reservations online. I don't totally trust my internal resources using tools like Selenium because we're adding a lot of new functionality to our new site - saved itineraries, wish list, some upselling and cross selling functionality, selling insurance, etc. So, while of course I want to be sure everything renders properly across browsers, I also feel like the investment in having someone check booking process in all browsers/technologies is critical especially since many of our customers are accessing our sites using older browsers. Thoughts??
Web Design | | CityWonders0 -
How can the Web site designer and the SEO strategist work together peacefully?
The organization I work for has decided to re-design or re-develop the existing company Web site. My part in this project is to come up with new features to add to the site, as well as making the site SEO-friendly (copywriting, link-building, keyword research, etc.). I don’t know a thing about Web site design, coding, format, etc., and I guess I will have to work with a designer on this project. How would I go about finding a Web site designer? Should they have some SEO knowledge? How much designer, coding and site structure knowledge should I have? And how do we not infringe on one another as we work together? (Sorry so many questions.)
Web Design | | Obie0 -
Can i do this? Will Google penalize me?
I have a page for a Criminal Defense Attorney and i set up a list of the type of criminal charges he is certified to deal with. I wanted to use title tags and put the Keyword "Miami Criminal Defense Attorney" & "Miami Traffic Defense Lawyer"... My question is will Google penalize me for plugging the same Key words over and over on the title tag for each ?? CHECK THE IMAGE to see what I'm talking about... thanks guys. x97dl
Web Design | | marig0