How to handle blank, auto generated system pages/urls
-
Hi Guys
Our backend system has been creating listing pages based on out of date and irrelevant data meaning we have hundreds of thousands of pages that are blank but currently indexable and active. They're almost impossible to access from the front end and have 0 traffic pointing at them but you can access these pages if you have the URL and i'm pretty sure due to the site architecture, google is crawling them regardless. For the most part, I think its likely best to 301 these pages to the most closely related page on the site but I'm concerned we're wasting crawl budget here. We don't want these pages to be crawled or found. Would a sound solution be to make them inactive, no-index and create a custom 404 in the event anyone (or the crawler) managed to get to them? Would this enormous increase in 404 pages cause us issues?
Many thanks
-
Thanks for such a speedy reply! Its such a daunting task as there's literally thousands and thousands of pages so we want to be sure we're doing the right thing. I appreciate your help. Now i'll investigate blocking within the robots.txt and using google search console to remove the URLs
-
First, do not 404 them, use a 410 error code instead as that denotes intended permanent deletion. In addition, I would also block the files/folder in robots.txt. Finally, I would use Google Search Console to remove these URLs. Good luck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best comments system / plugin for websites
Hi, What is the best comments system / plugin for websites that not harm seo Thanks Roy
Intermediate & Advanced SEO | | kadut0 -
How many links to the same page can there be for each page?
I need to know if I can add more than 2 equal links on the same page, for example 1 link in the header, another in the body and one in the footer
Intermediate & Advanced SEO | | Jorgesep0 -
No designated 404 page, but any made-up URL path displays homepage Good / Bad?
I have a custom website where if you type in companyxyz.com/_any-made-up-url _it displays the homepage. So then you will see the homepage and in the URL bar the made up URL path remains visible "companyxyz.com/any-made-up-url" Is this good or bad or not an issue?
Intermediate & Advanced SEO | | Rich_Coffman0 -
We 410'ed URLs to decrease URLs submitted and increase crawl rate, but dynamically generated sub URLs from pagination are showing as 404s. Should we 410 these sub URLs?
Hi everyone! We recently 410'ed some URLs to decrease the URLs submitted and hopefully increase our crawl rate. We had some dynamically generated sub-URLs for pagination that are shown as 404s in google. These sub-URLs were canonical to the main URLs and not included in our sitemap. Ex: We assumed that if we 410'ed example.com/url, then the dynamically generated example.com/url/page1 would also 410, but instead it 404’ed. Does it make sense to go through and 410 these dynamically generated sub-URLs or is it not worth it? Thanks in advice for your help! Jeff
Intermediate & Advanced SEO | | jeffchen0 -
Startpage and shop page shows the same thing, shall i set canonical url?
Our startpage http://siga-sverige.se/ and http://siga-sverige.se/butik/ shows the same woocommerce loop of all our products. Shall i set canonical url for http://siga-sverige.se/butik/ to http://siga-sverige.se/? Thanks! / Jonas
Intermediate & Advanced SEO | | knubbz0 -
What things should I consider if I am doing a 301 redirect on only 1 page/blog post?
I wrote a blog post on one of my websites and it got picked up by reddit and I got a bunch of nice backlinks and now that website got a nice boost overall, and especially that blog post page. I now wish I would have posted the article on a different website of mine. I would prefer if this other site was getting the traffic and the good backlinks that I've acquired. What are the pros and cons if I move the content over to my other website, and 301 redirect just that one article to the article location on my other website? The blog post I wrote almost instantly began ranking for certain terms in Google. Ideally I would like my other website to rank for those terms, but I realize there will be some differences as search engines look at the website as a whole and take many factors into consideration. I know there are tons of case studies and information about moving entire sites etc but I couldn't find much on this. Any advice, questions or comments would be greatly appreciated. Thanks,
Intermediate & Advanced SEO | | bradbowman
Brad0 -
Duplicate content within sections of a page but not full page duplicate content
Hi, I am working on a website redesign and the client offers several services and within those services some elements of the services crossover with one another. For example, they offer a service called Modelling and when you click onto that page several elements that build up that service are featured, so in this case 'mentoring'. Now mentoring is common to other services therefore will feature on other service pages. The page will feature a mixture of unique content to that service and small sections of duplicate content and I'm not sure how to treat this. One thing we have come up with is take the user through to a unique page to host all the content however some features do not warrant a page being created for this. Another idea is to have the feature pop up with inline content. Any thoughts/experience on this would be much appreciated.
Intermediate & Advanced SEO | | J_Sinclair0 -
Is there any SEO advantage to sharing links on twitter using google's url shortener goo.gl/
Hi is there any advantage to using <cite class="vurls">goo.gl/</cite> to shorten a URL for Twitter instead of other ones? I had a thought that <cite class="vurls">goo.gl/</cite> might allow google to track click throughs and hence judge popularity.
Intermediate & Advanced SEO | | S_Curtis0