Question about robots.txt
-
Solved!
-
Just a friendly reminder. Please don't delete your question after it's been answered. It's very likely that someone in the future could have the same question and they would have been able to find the answer if you hadn't deleted the question.
-
Consider deleting all of this:
Disallow: /&limit
Disallow: /?limit
Disallow: /&sort
Disallow: /?sort
Disallow: /?route=checkout/
Disallow: /?route=account/
Disallow: /?route=product/search
Disallow: /?route=affiliate/
Disallow: /?marca
Disallow: /&manufacturer
Disallow: /?manufacturer
Disallow: /?filter
Disallow: /&filter
Disallow: /?order
Disallow: /&order
Disallow: /?price
Disallow: /&price
Disallow: /?filter_tag
Disallow: /&filter_tag
Disallow: /?mode
Disallow: /&mode
Disallow: /?cat
Disallow: /&cat
Disallow: /?product_id
Disallow: /&product_id
Disallow: /?route=affiliate/
Disallow: /*?keywordThose rules are telling Google not to crawl domain.com/EVERYTHING(then the URL parameter). This could be where the issue stems from. If you're worried about URLs with these things ranking, consider implementing canonical tags instead to point to the proper pages
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help recover lost traffic (70%) from robots.txt error.
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change. Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated. On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL? Thanks,
On-Page Optimization | | akin671 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
Question Re Cornerestone Page And Anchor Text For Internal/External Links
Suppose I create a cornerstone page with the targeted keyword "Dog Collars". I write a dozen articles on various dog collars and point a link from each article to my cornerstone page. Should the anchor text for the links from each of those articles to the cornerstone page be "Dog Collars" or should they vary, but still be relevant to "Dog Collars" for best SEO? Should half of them be "Dog Collars" and the other half various? Also, if I have 12 articles and all of the anchor text is already "Dog Collars", should I go back and change them so that they all don't say the same thing? If hope my question makes sense ... thanks in advance. I will give thumbs up for helpful responses and suggestions 😉
On-Page Optimization | | Humanovation0 -
Crosslinking Question - Footer or Header?
We own 9 fashion blogs that focus on different niches. We have Your Next Shoes focusing on shoes, Your Next Dress focusing on dresses, Your Next Handbag focusing on handbags, and so on. Previously we had links in the blog header to the other sister sites. Your Next Shoes had text links to its 8 sister sites, and so on. However, we were told that this could cause a cross-linking penalty, so we removed these links. My understanding now is that the advice we were given is not necessarily correct. We think it is useful for our visitors to see that we also operate other blogs, so we are thinking of adding back the links - this time to our footer. Basically I am looking for advise on the best way to go about this.
On-Page Optimization | | Jantaro0 -
Internal Link Title and Anchor Text Question
So, I am kind of a newbie with all of this stuff, so please bear with me! If you have a chance to look into the domain it's www.noip.com I have checked the internal followed and 301 links. It seems like it is pulling in all of our navigation a funky way. all of those link have no data for the title and the anchor text is Home, Contact Us, Forgot Password?, ETC. Is this correct? I have tried to look into the way other websites are doing it and it seems wrong. I tried to talk to my developer, but he is not so sure. Do we simply need to make more copy on our site that has internal links within the copy with anchor text that is for keywords that I would like to be ranking for? Like Managed DNS? and then that page goes to the managed DNS page? Or what? Sorry if i am not being totally clear. Any help and advice is 110% welcomed!!! Natalie
On-Page Optimization | | vwnatalie0 -
New website launch - Dupilicate content question
Hi, We are going to be launching a US version of our UK E-commerce web store. The US version will be a standalone site, separate URL/hosting etc. My question is that the US website will carry the same products as the UK site - will the product copy have to be written differently for the US website in order to avoid any SEO duplicate copy? Thanks
On-Page Optimization | | WalesDragon0 -
Can duplicate content issues be solved with a noindex robot metatag?
Hi all I have a number of duplicate content issues arising from a recent crawl diagnostics report. Would using a robots meta tag (like below) on the pages I don't necessarily mind not being indexed be an effective way to solve the problem? Thanks for any / all replies
On-Page Optimization | | joeprice0 -
Photogallery and Robots.txt
Hey everyone SEOMOZ is telling us that there are to many onpage links on the following page: http://www.surfcampinportugal.com/photos-of-the-camp/ Should we stop it from being indexed via Robots.txt? best regards and thanks in advance... Simon
On-Page Optimization | | Rapturecamps0