How best to fix 301 redirect problems
-
Hi all
Wondering if anyone could help out with this one. Roger Bot crawler has just performed it's weekly error crawl on my site and I appear to have 18,613 temp redirect problems!! Rather, the same 1 problem 18,613 times.
My site is a magento store and the errors it is giving me is due to the wishlist feature on the site. For example, it is trying to crawl links such as index.php/wishlist/index/add/product/29416/form_key/DBDSNAJOfP2YGgfW (which would normally add the item to one's wishlist). However, because Roger isn't logged into the website it means that all these requests are being sent to the login url with the page title of Please Enable Cookies.
Would the best way to fix this be to enable wishlists for guests? I would rather not do that but cannot think of another way of fixing it.
Any other Magento people come across this issue?
Thanks, Carl
-
thanks. I completely forgot about the robots.txt method of fixing it. Will give it a go. I will add comparison in there too just to be safe.
-
Hey Carl
I have this problem a lot on my Magento stores since theres a ton of ways to access the same product and it can be an SEO nightmare. If you have FTP access I would recommend adding this to the robots.txt file:
Disallow: /wishlist/
This will stop the bots from crawling everything passed /wishlist/. Here's a good resource if you need further assistance: http://www.magentocommerce.com/knowledge-base/entry/setting-up-a-robotstxt-file
Hopefully that helps!
Good luck
-
Thanks will try and work out how to do that. Magento is a pain to make minor changes to.
Hopefully the crawl stats will look a little healthier next week !!
-
Use Rel="no index, no follow" on the link and roger will ignore the link. Job done
You wouldn't get any advantage of seeing the added items in someones wishlist as it would be duplicate content I'm presuming.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Forced Redirects/HTTP<>HTTPS 301 Question
Hi All, Sorry for what's about to be a long-ish question, but tl;dr: Has anyone else had experience with a 301 redirect at the server level between HTTP and HTTPS versions of a site in order to maintain accurate social media share counts? This is new to me and I'm wondering how common it is. I'm having issues with this forced redirect between HTTP/HTTPS as outlined below and am struggling to find any information that will help me to troubleshoot this or better understand the situation. If anyone has any recommendations for things to try or sources to read up on, I'd appreciate it. I'm especially concerned about any issues that this may be causing at the SEO level and the known-unknowns. A magazine I work for recently relaunched after switching platforms from Atavist to Newspack (which is run via WordPress). Since then, we've been having some issues with 301s, but they relate to new stories that are native to our new platform/CMS and have had zero URL changes. We've always used HTTPS. Basically, the preview for any post we make linking to the new site, including these new (non-migrated pages) on Facebook previews as a 301 in the title and with no image. This also overrides the social media metadata we set through Yoast Premium. I ran some of the links through the Facebook debugger and it appears that Facebook is reading these links to our site (using https) as redirects to http that then redirect to https. I was told by our tech support person on Newspack's team that this is intentional, so that Facebook will maintain accurate share counts versus separate share counts for http/https, however this forced redirect seems to be failing if we can't post our links with any metadata. (The only way to reliably fix is by adding a query parameter to each URL which, obviously, still gives us inaccurate share counts.) This is the first time I've encountered this intentional redirect thing and I've asked a few times for more information about how it's set up just for my own edification, but all I can get is that it’s something managed at the server level and is designed to prevent separate share counts for HTTP and HTTPS. Has anyone encountered this method before, and can anyone either explain it to me or point me in the direction of a resource where I can learn more about how it's configured as well as the pros and cons? I'm especially concerned about our SEO with this and how this may impact the way search engines read our site. So far, nothing's come up on scans, but I'd like to stay one step ahead of this. Thanks in advance!
Technical SEO | | ogiovetti0 -
Problems with canonical urls / redirect (magento webshop)
Hi all, We're running a Magento webshop and we discover some strangs things regarding canonical urls and redirects after using the Amasty improved navigation extension. To clarify, please check these four urls. They contain the same content (the same product page). https://www.afwerkingshop.be/gyproc-gipskartonplaat-ak-2600x1200x9-5mm.html https://www.afwerkingshop.be/wanden/gyproc-gipskartonplaat-ak-2600x1200x9-5mm.html https://www.afwerkingshop.be/wanden/gipsplaten/gyproc-gipskartonplaat-ak-2600x1200x9-5mm.html https://www.afwerkingshop.be/wanden/gipsplaten/standaard/gyproc-gipskartonplaat-ak-2600x1200x9-5mm.html All these four pages have different canoncials (the page url). Obviously, that's not good. However, in Google (site:...) url (1) is the only one that's indexed. Thereby, if I visit the productpage by first going to a category page (fe. www.afwerkingshop.be/wanden.html), I'm redirected to url (1), but the canonical url is www.afwerkingshop.be/last_visited_category_name/product. So, the canonical seems dynamic depending on the last visited category. And still, only url (1) is indexed. Additionally, all aforementioned pages contain . Is anyone familiar with this issue? And more important, will it cause problems in future? Thanks in advance. Kind regards, Chendon
Technical SEO | | RBijsterveld0 -
What is best practice for fixing urls that have duplicate content, non-static and other issues?
Hi, I know there are several good answers regarding duplicate content issues on this website already, however I have a question that involves the best way to avoid negative SEO impacts if I change the urls for an ecommerce site. Basically a new client has the following website http://www.gardenbeauty.co.uk and I notice that it suffers from duplicate content due to the http://www version and the non www version of the pages - this seems quite easy to fix using the guidance on this website. However I notice that the product page urls are far from ideal in that they have several issues including:- (a) they are mostly too long (b) don't include the keyword terms (in terms of best practice) (c) they don't use Static URLS An example of one these product urls would be http://www.gardenbeauty.co.uk/plant-details.php?name=Autumn Glory&p_genus=Hebe&code=heagl&category=hebe I'd like to address these issues, but the pages rank highly for the products themselves, therefore my question is what would you recommend I do to fix the urls without risking the high positions that many of these product pages have? thanks, Ben
Technical SEO | | bendyman0 -
301 redirect or maual edit of new urls
Hello forum! I will get right to the point,I have a 4 year old PR4 site with lots of links (vacation rentals marketplace, like Homeaway), In about a month from now new CMS will be ready and I will be doing redesign of the site. The problem that I have is (as many of you can guess) losing all the old links that rank high = losing traffic / revenue. Two posiblle solutions here: 1. 301 redirect for each page that ranks high - point it to new url 2. Manually editing new urls created by new CMS and making them to be the same as old ones. This means that some number of urls (the ones that rank high and generate traffic) would be exactly the same while other ones would be generated by CMS thus dufferent in many ways (unicode,different keywords etc.) What would You do here? I am more for 301 redirect but I read all kinds of horror stories in drop of SERP. Thank You for help and advices in advance.
Technical SEO | | Gregos0 -
301 redirects and seo..
I bought a domain and it has nice traffic. It only has about 5 main pages in php When i got the site i switched to html because php was overkill. I did the 301 and google deleted the php files and replaced with html version when i check site:domain.com It has been about 7 days. I DID NOT use 301 for each of the 5 pages to go php to html instead is used this code RewriteEngine On
Technical SEO | | samerk
RewriteCond %{HTTP_HOST} ^mydomain.com
RewriteRule (.) http://www.mydomain.com/$1 [R=301,L]
RedirectMatch 301 (.).php$ http://www.mydomain.com$1.html So basically if you load php it will load the html version. dog.php > dog.html Is this OKAY? or should it be done differently.... worried! Thanks !0 -
302 to 301 redirect confirmation
Hi guys, Fairly sure of the answer from what I've read so far, but I just wanted to doublecheck I have it right. Page A gets a significant amount of referring, followed traffic, and also ranks in Google. Page A uses a 302 redirect to Page B (on a completely different domain), which means that 0% of Page A's link juice is being passed on to Page B. If I were to change the 302 redirect to a 301 redirect, then the link juice passed on to Page A from the followed, referring traffic will be (mostly) passed on to Page B. Is that correct? Cheers, Jez
Technical SEO | | jez0000 -
Canonical tag, CNAME and 301 redirect
I have a website with a couple of domains pointing to one IP address. Let's say I have two domains www.example.com and www.example.ca I also see during my SEO analysis that the example.com and the www.example.com (same for the example.ca and the www.example.ca) are triggering server responses. How do I deal with this issue for best SEO. Canonical links? CNAME, or 301 redirects? thanks
Technical SEO | | casper4340 -
Do 301 redirects pass page rank quickly
Hi I have been asked to carry out a site audit for a potential client. The site has that many issues I don't where to start in explaining them however, there is one question we are debating and would like to get a second opinion on it. The site I am auditing used to have a homepage rank 7. The site has currently had a redesign (new template with new URLs) and now the root domain 301 redirects to a sub folder two levels deep (not ideal I know!). This happened about a month ago and we are still getting N/A for toolbar page rank. The question is, does Google page rank transfer quicker than normal due to the redirects? or do we still have to wait on the next Google Page Rank update? Thanks in advance Gavelect
Technical SEO | | Equatorites0