Magneto site with many pages
-
just finsihed scan to a magento site.
off course I am getting thousand of pages that are dynamic.
search pages and other.
checking with site command on Google I see 154,000 results
which pages it is recommended to block?
some people are talking about blocking the search pages and some actually talking about allowing them?
any answer on this?
Thanks
-
There was no effect on the rankings, no significant ups or downs. But now when I do the site:www.domain.com command in Google, I see just the pages that I want Google to index.
It can only help in the long run I guess.
-
Hi
sorry for responding late.
what was the affect on your results when u did block all those pages?
Thanks
-
yes my thought is blocking through robot.txt
-
I've had similar problems with a few Magento sites. This is a standard list I use in my robots.txt files (below.) I hope it helps.
You don't have to include all the Magento folders like 'app' 'lib' 'var' and 'admin' t etc hey are just there to be thorough.
I think you'll get the idea. I've brought the number of indexed pages down from half a million to just a few thousand using these.
Disallow: /*? Disallow: /*.js$ Disallow: /*.css$ Disallow: /404/ Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /catalog/product_compare/ Disallow: /catalogsearch/ Disallow: /catalogsearch/advanced/ Disallow: /catalogsearch/term/ Disallow: /catalogsearch/term/popular/ Disallow: /cgi-bin/ Disallow: /checkout/ Disallow: /checkout/cart/ Disallow: /contacts/ Disallow: /contacts/index/ Disallow: /contacts/index/post/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ Disallow: /downloader/ Disallow: /install/ Disallow: /js/ Disallow: /lib/ Disallow: /magento/ Disallow: /newsletter/ Disallow: /pkginfo/ Disallow: /private/ Disallow: /poll/ Disallow: /report/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /skin/ Disallow: /tag/ Disallow: /var/ Disallow: /wishlist/
-
Hi there! When you say block, do you mean through your robots.txt?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Related Keywords: How many separate pages?
We have an attorney website. There is a practice area that our research shows many different 2-4 word length keyword queries for. The keywords are all very different, but they end up in the same kind of legal action. We're wondering whether we should write many different pages, perhaps 10, to cover all the basic different keyword categories, or whether we should just write a few pages. In the latter situation, many of the target key words would be mentioned in the text, but wouldn't get placement in a url or title tags. One basic problem is that since the keyword queries are made up of different words, but result in the same kind of legal action and applicable law, the content of the pages might be similar with the only difference being a paragraph that speaks to that specific key word. The rest of the content would be quite similar among the pages, i.e. "here is the law that applies, contact us." Also, some of the keywords, like the name of the law, would have to be repeated on all the pages.
Intermediate & Advanced SEO | | RFfed90 -
SEO implications of off-site secure Donation page
Hi Mozzers, I have a non-profit client that defends wildlife and public lands in the western US. The huge website is currently not responsive so we are working on that. In the meantime, we will be making the Action pages (such as Donations, Sign Petition, Get Newsletter) pages responsive. This will be housed under a new domain. My question is, what are SEO best practices for doing this? Does it negatively impact SEO to have a visitor "booted" from a site to a second secure site? Does Google know that the Donation site is in fact fulfillment and is linked to the original site? Also, what about domain implications? Is it best to have the name of the non-profit in the domain or sub, like this: https://saveanimals.secure.com/donate? Thanks everyone!
Intermediate & Advanced SEO | | CalamityJane770 -
Too many on page links
Hi I know previously it was recommended to stick to under 100 links on the page, but I've run a crawl and mine are over this now with 130+ How important is this now? I've read a few articles to say it's not as crucial as before. Thanks!
Intermediate & Advanced SEO | | BeckyKey1 -
Want to merge high ranking niche websites into a new mega site, but don't want to lose authority from old top level pages
I have a few older websites that SERP well, and I am considering merging some or all of them into a new related website that I will be launching regardless. My old websites display real estate listings and not much else. Each website is devoted to showing homes for sale in a specific neighborhood. The domains are all in the form of Neighborhood1CityHomes.com, Neighborhood2CityHomes.com, etc. These sites SERP well for searches like "Neighborhood1 City homes for sale" and also "Neighborhood1 City real estate" where some or all of the query is in the domain name. Google simply points to the top of the domain although each site has a few interior pages that are rarely used. There is next to zero backlinking to the old domains, but each links to the other with anchor text like "Neighborhood1 Cityname real estate". That's pretty much the extent of the link profile. The new website will be a more comprehensive search portal where many neighborhoods and cities can be searched. The domain name is a nonsense word .com not related to actual key words. The structure will be like newdomain.com/cityname/neighborhood-name/ where the neighborhood real estate listings are that would replace the old websites, and I'd 301 the old sites to the appropriate internal directories of the new site. The content on the old websites is all on the home page of each, at least the content for searches that matter to me and rank well, and I read an article suggesting that Google assigns additional authority for top level pages (can I link to that here?). I'd be 301-ing each old domain from a top level to a 3rd level interior page like www. newdomain/cityname/neighborhood1/. The new site is better than the old sites by a wide margin, especially on mobile, but I don't want to lose all my top positions for some tough phrases. I'm not running analytics on the old sites in question, but each of the old sites has extensive past history with AdWords (which I don't run any more). So in theory Google knows these old sites are good quality.
Intermediate & Advanced SEO | | Gogogomez0 -
Backlinking 3 sites from same domain and backlinking main site too
Hello, we have 4 sites, in which 1 is a main site and rest 3 are niche sites All these 3 sites have dofollow links to main site from home page We got a high quality backlink - through which all 3 niche sites have got it from that domain Is it worth to add backlink from that domain to main site too, despite the fact the 3 sites already have recvd it and they all link to main site many thanks
Intermediate & Advanced SEO | | Modi0 -
How many inner links on one page?
I have seen Matt Cutts video about links per page and know that too many links "may" harm the flow of link juice. But what should e-commerce sites do? We have category pages with more than a few thousands products in each of them. So linking to each of them dilutes the PR flow? We could use pagination, but doesn't it give a disadvantage in user experience when he needs to go 10 links deep to reach a product? And Google robots won't update the information frequently because it will be on the lowest part of our site? Now our goal is to make all our products appear like Facebook scroll down page. We know that Google doesn't use Ajax to see more links so robots and all the users that don't have JavaScript could see the paginated results. Is it a good way to put all products and links like this?
Intermediate & Advanced SEO | | komeksimas1 -
How many links home on a page?
We are planning on a mega menu which will have around 300 links and a mega slider which will have around 175 links if our developer has their way. In all I could be looking at over 500 links from the home page. The Mega Menu will flatten the site link structure out but I am worried this slider on the home page which is our 4th most visited page behind our 3 core category pages. What are your thoughts?
Intermediate & Advanced SEO | | robertrRSwalters0 -
To many on page links with ABC search
My client site http://www.tshirtsubway.com has a ABC quick find selector on the homepage of the site and throughout the site and as a result is is showing an error of to many links on the SEO moz error crawls reports. I wanted some advice on improving this and perhaps looking for an alternative also looking at the current setup and asking is this wrong.
Intermediate & Advanced SEO | | onlinemediadirect0