How best to fix 301 redirect problems
-
Hi all
Wondering if anyone could help out with this one. Roger Bot crawler has just performed it's weekly error crawl on my site and I appear to have 18,613 temp redirect problems!! Rather, the same 1 problem 18,613 times.
My site is a magento store and the errors it is giving me is due to the wishlist feature on the site. For example, it is trying to crawl links such as index.php/wishlist/index/add/product/29416/form_key/DBDSNAJOfP2YGgfW (which would normally add the item to one's wishlist). However, because Roger isn't logged into the website it means that all these requests are being sent to the login url with the page title of Please Enable Cookies.
Would the best way to fix this be to enable wishlists for guests? I would rather not do that but cannot think of another way of fixing it.
Any other Magento people come across this issue?
Thanks, Carl
-
thanks. I completely forgot about the robots.txt method of fixing it. Will give it a go. I will add comparison in there too just to be safe.
-
Hey Carl
I have this problem a lot on my Magento stores since theres a ton of ways to access the same product and it can be an SEO nightmare. If you have FTP access I would recommend adding this to the robots.txt file:
Disallow: /wishlist/
This will stop the bots from crawling everything passed /wishlist/. Here's a good resource if you need further assistance: http://www.magentocommerce.com/knowledge-base/entry/setting-up-a-robotstxt-file
Hopefully that helps!
Good luck
-
Thanks will try and work out how to do that. Magento is a pain to make minor changes to.
Hopefully the crawl stats will look a little healthier next week !!
-
Use Rel="no index, no follow" on the link and roger will ignore the link. Job done
You wouldn't get any advantage of seeing the added items in someones wishlist as it would be duplicate content I'm presuming.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is best practice for fixing urls that have duplicate content, non-static and other issues?
Hi, I know there are several good answers regarding duplicate content issues on this website already, however I have a question that involves the best way to avoid negative SEO impacts if I change the urls for an ecommerce site. Basically a new client has the following website http://www.gardenbeauty.co.uk and I notice that it suffers from duplicate content due to the http://www version and the non www version of the pages - this seems quite easy to fix using the guidance on this website. However I notice that the product page urls are far from ideal in that they have several issues including:- (a) they are mostly too long (b) don't include the keyword terms (in terms of best practice) (c) they don't use Static URLS An example of one these product urls would be http://www.gardenbeauty.co.uk/plant-details.php?name=Autumn Glory&p_genus=Hebe&code=heagl&category=hebe I'd like to address these issues, but the pages rank highly for the products themselves, therefore my question is what would you recommend I do to fix the urls without risking the high positions that many of these product pages have? thanks, Ben
Technical SEO | | bendyman0 -
Is 301 redirect replacing or adding PA on a page ?
Hello Moz community, I am re-structuring information architecture on a website. The homepage targets keyword A
Technical SEO | | Sindicic_Alexis
There is a page /page-on-B/ that targets keyword B
There is also an older page /page-on-kinda-B/ that targets a keyword which is really close to B ( i think they should have been worked on the same page...) I want to re-optimise the homepage on keyword B because i know it is worth the effort. If i re-write the homepage to talk about B and do a 301 redirect of : /page-on-B/
/page-on-kinda-B/ to the homepage is it going to add page authority to the homepage ? is it the right way to proceed ?
Thanks for your support !0 -
How to create a delayed 301 redirect that still passes juice?
My company is merging one of our sites into another site. At first I was just going to create a 301 redirect from domainA.com to domainB.com but we decided that would be too confusing for customers expecting to see domainA.com so we want to create a page that says something like "We've moved. please visit domainB.com or be redirected after 10 seconds". My question is, how do I create a redirect that has a delay and will this still pass the same amount of juice that a regular 301 redirect would? I've heard that meta refreshes are considered spammy by Google.
Technical SEO | | bewoldt0 -
301 Redirect Question for trailing slash & index.php
1. Can you show me how to do a redirect for forhttp://www.exercisebiology.com to http://www.exercisebiology.com/ I want to add the trailing slash. 2. Also how to redirect from http://www.exercisebiology.com/in... to http://www.exercisebiology.com/. When I tried to redirect the index.php last time, I couldn't login to the forums. Everytime when someone tries to login to my forums, it takes them the to the home page. I did it through the hostgator CP. I am thinking someone could help me here. Thank you so much!
Technical SEO | | anoopbal0 -
301 redirects and Dynamic URLs
I just ran my first diagnostic and one of my primary immediate problems are duplicate titles and duplicate content. My guess it that because the root URL http://sitename.com (which has not yet been redirected to www...) has generated an entire tree of content which is identical to the tree rooted at http://www.sitename.com. QUESTION: Do I need to do a redirect simply for the root url (sitename.com -> www.sitename.com) or do I now need to develop specific 301 redirects for each of the sub-nodes/pages? ie sitename.com/?q=about-us -> www.sitename.com/?q=about-us sitename.com/?q=our-team -> www.sitename.com/?q=our-team etc.
Technical SEO | | Barrycliff680 -
Redirect or not to redirect
We are rebuilding a website and try to get rid of errors. The content remains exactly the same but we correct the code and make it load faster. The site has quite many backlinks and I can't decide whether to remove .html endings from the urls and 301 redirect to the new ones or leave them with the older ending. If I remove the endings how much of the link juice will be passed? Anyone any idea?
Technical SEO | | sesertin0 -
Hundreds Of 404s Best Dealt Wtih 301-Palooza?
Was looking at a site's webmaster tools where there are a few hundred 404s for a site. God only knows where all those urls came from or what they once were or what. Two questions; How bad is all this for the site's overall ranking? Should I just make sure they go nowhere and 301 them all to the homepage or some other possibly relevant page? Anything bad that could happen with 301ing all this away? Also, the site has no sitemap submitted to Google. Any tips on that? Thanks!
Technical SEO | | 945010 -
Minisites - 301 Redirect or Links to Main site
Not sure whether this is considered black hat or not but I know it is done and I would like to know which is the most effectrive method. If you were to acquire multiple sites in the same niche to your main site (either by buying existing sites or perhaps registering expired domains) which already had strong aged backlinks, is it better to either: a) 301 the new domain to the main site (or a subpage perhaps) b) create 'minisites' on the new domains (trying to mirror the URL structure of the previous incarnation if possible to scoop up and remaining inbound backlink juice, on seperate IPs to the main site as well) and then place several links to the main site & subpages. Would the decay of link juice through 301's mean you lose benefit that way or is it the same as a normal link? Would the 301 method mean any IBL's into URL's other than the homepage be lost? The homepage of the minisite will likely have 4 or 5 internal links so will this dilure the effect of the links to the main site? Thanks in advance!
Technical SEO | | OzDave0