Magento Core_URL_Rewrite Problems
-
Hi Everyone,
We are currently caught between a rock and a hard place with Magento and are wondering if anyone else had similar problems and could share their advice.
Our Core_URL_Rewrite now containt 1.3 million records for an account that has 12000 products on 4 different store views. This has ballooned past the point that we are no longer able to reindex our URL Management.
The option that is being suggested to us is to truncate the table and start over, though this will essentially kill our SEO for those pages.(Which as there are duplicates, I can only imagine how much they are going to be penalized by it)
Would anyone have any advice other than truncating and starting over?
Any advice would be greatly appreciated.
Thanks!
-
Hi,
I found the exact problem you are facing with a solution on this link
http://magento.stackexchange.com/questions/17553/magento-core-url-rewrite-table-excessively-large
There are patch codes available on this link, however do read this reply on this page
Bugs in earlier (and possibly current) versions of Magento is one. Another is there's logic in this table that tries to track changes to the URL key value so that 301/302 rewrites are setup for old products. Because of this, and complicating things, truncating the table and regenerating may make existing URL rewrites go away, and this will have an unknown effect on your search engine listing (not necessity bad, just hard to predict).
My general advice to clients who ask is
-
Leave the giant growing table as is if you don't have a good handle on your URL/SEO situation
-
Until the table size starts being a problem (generating site maps, for example). When that happens, get a handle on your URL/SEO situation.
-
Once you have a handle on your URL/SEO situation, backup the table, then truncate the table and regenerate. Address any URL/SEO problems caused by the truncating.
-
Automate step 3
Trying to fix this on the Magento code level is admirable, but you'll be swimming upstream. Sometimes it's better to accept that "That's just Magento being Magento", and to solve the problem with and external process.
I hope this helps, if you have further questions, then post a response, I will be happy to answer.
Regards,
Vijay
-
-
I'm not sure the answers previously presented are related to the issues you're having. Having worked with Magento for a long time, this can be an issue that occurs over and over again.
To answer your initial question, truncating your core_url_rewrite table will remove all of these URLs, but it'll only delay the problem until it reoccurs again in the future (unless you've had a problem in the past which has been rectified). You're also correct in that any rewrites in the system previously there will disappear, so you'll probably end up with a lot of crawl issues appearing in Search Console.
Your best move would be to find out why you have so many URLs in there in the first place. Do you have a huge product catalog with multiple stores? Or is this something to do with an issue in your Magento version or some setup issues. The most common time this usually occurs is if two products get added to your site with the same URL Key. Every time the reindex process runs, your core_url_rewrite table will grow. You could check this by looking at the number of rows in the table, reindexing the site and if it grows further, then it's likely to be the problem. The quickest way to fix this is to ensure all URL key are unique.
There's also an article here about duplicate keys - https://firebearstudio.com/blog/magento-url-reindex-core_url_rewrite-duplicates-patch.html - this should hopefully clear the issue.
I hope this helps! If it doesn't solve the problem, then sending over a little more information around the number of stores, catalog site and the split between system generated URL rewrites and custom URL rewrites would be great so we can try to help further!
Thanks,
Lewis -
This is an issue to to set-up. When you set up multiple ecommerce websites on Magento as 'Stores', then all SKUs will load on other domains. if they were set-up as 'Websites' then this would alleviate the issue. However, with Stores you are able to share shopping carts (i.e. Add a product from website A and checkout on website B).
What I did was turn off the XML cron jobs and set-up cross-domain canonicals. Also make sure your session IDs (/?SID=) are working properly. Not sure if this solves the technical issues, but should help clear up dupe content.
-
Is it creating a new url for each option (size, color, etc) as well as what page it shows up on or other various sort orders (by price, by size, etc.) and session id's that you could exclude? Are you sure they are truly duplicates?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing the Trailing Slash in Magento
Hi guys, We have noticed trailing slash vs non-trailing slash duplication on one of our sites. Example:
Intermediate & Advanced SEO | | brandonegroup
Duplicate: https://www.example.com.au/living/
Preferred: https://www.example.com.au/living So, SEO-wise, we suggested placing a canonical tag on all trailing slash pointing to non-trailing slash. However, devs have advised against removing the trailing slash from some URLs with a blanket rule, as this may break functionality in Magento that depends on the trailing slash. The full site would need to be tested after implementing a blanket rewrite rule. Is any other way to address this trailing slash duplication issue without breaking anything in Magento? Keen to hear from you guys. Cheers,0 -
Why isn't the canonical tag on my client's Magento site working?
The reason for this mights be obvious to the right observer, but somehow I'm not able to spot the reason why. The situation:
Intermediate & Advanced SEO | | Inevo
I'm doing an SEO-audit for a client. When I'm checking if the rel=canonical tag is in place correctly, it seems like it: view-source:http://quickplay.no/fotball-mal.html?limit=15) (line nr 15) Anyone seing something wrong with this canonical? When I perform a site:http://quickplay.no/ search, I find that there's many url's indexed that ought to have been picked up by the canonical-tag: (see picture) ..this for example view-source:http://quickplay.no/fotball-mal.html?limit=15 I really can't see why this page is getting indexed, when the canonical-tag is in place. Anybody who can? Sincerely 🙂 GMdWg0K0 -
72KB CSS code directly in the page header (not in external CSS file). Done for faster "above the fold" loading. Any problem with this?
To optimize for googles page speed, our developer has moved the 72KB CSS code directly in the page header (not in external CCS file). This way the above the fold loading time was reduced. But may this affect indexing of the page or have any other negative side effects on rankings? I made a quick test and google cache seems to have our full pages cached, but may it affect somehow negatively our rankings or that google indexes fewer of our pages (here we have some problems with google ignoring about 30% of our pages in our sitemap".)
Intermediate & Advanced SEO | | lcourse0 -
Next Gen Gallery Crawler Problem
I use the Next gen gallery plugin on my wordpress sites. The moz crawler reports a ton of high importance issues with this plugin because it creates duplicate pages It will have domain.com/page, domain.com/page/gallery, domain.com/page/gallery/1/, domain.com/page/gallery/2/ This is a pretty popular plugin so I am hoping there is some way of fixing this relatively easy. I can imagine i need to set up a rel canonical but there does not seem to be an easy way to do so. Thoughts?
Intermediate & Advanced SEO | | Atomicx0 -
Problem with Google reading https homepage?
Hi Moz Community, In July, we changed our homepage to https via a 301 redirect from http (the only page on our site with https). Our homepage receives an A grade in the ‘On Page Grader’ by Moz for our desired keyword. We have increased our backlink efforts directly to our homepage since we switched to the SSL homepage. However, we still have not increased in search ranking for our specific keyword. Is there something we could have missed when doing the 301 redirect (submitting a new sitemap, changing rotbots.txt files, or anything else??) that has resulted in Google not correctly accessing the https version? (the https page has been indexed by Google). Any help would be greatly appreciated.
Intermediate & Advanced SEO | | G.Anderson0 -
Whats the best way to remove search indexed pages on magento?
A new client ( aqmp.com.br/ )call me yestarday and she told me since they moved on magento they droped down more than US$ 20.000 in sales revenue ( monthly)... I´ve just checked the webmaster tool and I´ve just discovered the number of crawled pages went from 3.260 to 75.000 since magento started... magento is creating lots of pages with queries like search and filters. Example: http://aqmp.com.br/acessorios/lencos.html http://aqmp.com.br/acessorios/lencos.html?mode=grid http://aqmp.com.br/acessorios/lencos.html?dir=desc&order=name Add a instruction on robots.txt is the best way to remove unnecessary pages of the search engine?
Intermediate & Advanced SEO | | SeoMartin10 -
Google Sitemap only indexing 50% Is that a problem?
We have about 18,000 pages submitted on our Google Sitemap and only about 9000 of them are indexed. Is this a problem? We have a script that creates a sitemap on a daily basis and it is submitted on a daily basis. Am I better off only doing it once a week? Is this why I never get to the full 18,000 indexed?
Intermediate & Advanced SEO | | EcommerceSite0 -
Bridge page problem
Hello I run this website, http://www.bestillkredittkort.no (Norwegian website) I'm working all i can to make it rank, so i wanted to test adwords to see how good the page converted.
Intermediate & Advanced SEO | | katal
.
Problem is my page got labeled bridge page. I have read the Google guideline for fixing bridge page and tried to fix what they suggest to make them accept the page, but its not working. I might think there's a underlying problem here, but im not sure how to fix it. I've even seen People from the same niche running adwords campaign with less content on the target page. Last time i tried to recheck if my site would get approved. I ran over the quality score tab. And when it was in pending it showed 10/10 in every aspect. was that just a sample ? I'm realy confused on this one. And im afraid it might be a problem witht the page that can destroy my seo efforts. Anyone have any suggestion or feedback on this one?0