Prevent indexing of dynamic content
-
Hi folks!
I discovered bit of an issue with a client's site. Primarily, the site consists of static html pages, however, within one page (a car photo gallery), a line of php coding:
dynamically generates a 100 or so pages comprising the photo gallery - all with the same page title and meta description. The photo gallery script resides in the /gallery folder, which I attempted to block via robots.txt - to no avail. My next step will be to include a:
within the head section of the html page, but I am wondering if this will stop the bots dead in their tracks or will they still be able to pick-up on the pages generated by the call to the php script residing a bit further down on the page?
Dino
-
Hello Steven,
Thank you for providing another perspective. However, all factors considered, I agree with Shane's approach on this one. The pages add very little merit to the site and exist primarily to provide the site users with eye-candy (e.g. photos of classic cars).
-
Just personally, I would still deindex or canonical them - they are just pages with a few images - so not of much value and unless all titles and descriptions are targeting varying keywords and content is added, they will canabalize eachother, and possibly even drag down the site due to 100's of pages of thin content....
So actually from an SEO perspective it probably IS better to deindex or canonical 3 - 5 or so years ago, maybe the advice would have been keep them and keyword target - but not in the age of content
(unless the images were optimized for image searches for sale able products (but I do not think it is)
-
Hi Dino,
I know this won't solve the immediate problem you asked for, but wouldn't it be better for your client's site (and for SEO) to alter the PHP so that the title and meta data description are replaced with variables that can also be dynamic, depending on whichever of the 100 or so pages gets created?
That way, rather than worrying about a robot seeing 100 pages as duplicate content, it could see 100 pages as 100 pages.
-
It depends on how the pages are being created (I would assume it is off of a template page)
So within the template of this dynamically created page you would place
But if this is the global template - you cannot do this as it will noindex every page which of course is bad.
If you want to PM me the URL of the page I can take a look at your code, and see what is going on and how to recitify, as right now i think we are talking about the same principles, but different words are being used.
It really is pretty straightforward. (what I am saying) The pages that you want to be not indexed DO NOT need a nofollow they need a meta noindex
But there are many variables, as if you have already robot.txt disallowed the directory, then no bot will go there to get the updated noindex directive....
If there is no way to add a meta noindex then you need to nofollow and put in for a manual removal
-
I completely understand and agree with all points you have conveyed. However, I am not certain as to the best approach to "noindex" the urls which are being created dynamically from within the static html page? Maybe I am making this more complex than it needs to be...
-
So it is the pages themselves that are dynamically created you want out of index, not the page the contains the links?
If this is so ---
noindex the pages that are created dynamically
Therein lies the problem. I did have the nofollow directive in place specifying the /gallery/ folder, but apparently, the bots still crawled it.
Nofollow does not remove from index, it only tells the bot not to pass authority, as it is still feasible that the bot will crawl the link, so without the noindex, nofollow is not the correct directive due to the page (even though nofollowed) is still being reached and indexed.
PS. also if you have the nofollow on the links, you may want to remove it, so the bots will go straight through to the page and grab the noindex directive, but if you wanted to try to not let any authority "evaporate" you can continue to nofollow, but you may need to request the dynamically generated pages (URLS) be removed using webmaster tools.
-
The goal is to have the page remain in the index, but not follow any dynamically generated links on the page. The nofollow directive (in place for months) has not done the job.
-
?
If a link is coming into the page, and you have Noindex, Nofollow - this would remove from index and prevent the following of any links -
This is NOT instant, and can take months to occur depending on depth of page, crawl schedule ect... (you can try to speed it up by using webmaster tools to remove the URL)
What is the goal You are attempting to achieve?
To get the page out of index, but still followed?
Or remain in index, but just not follow links on page?
?
-
Therein lies the problem. I did have the nofollow directive in place specifying the /gallery/ folder, but apparently, the bots still crawled it. I agree that the noindex removes the page, but I wasn't certain if it prevented crawling of the page, as I have read mixed opinions on this.
I just thought of something else... perhaps an external url is linking to this page - allowing it to be crawled. I am off to examine this perspective.
Thanks for your response!
-
noindex will only remove from Index and dissallow the act of indexing the specific page (or pages created off template) you place the tag in upon the next page crawl.
Bots will still follow the page, and follow any links that are readable as long as there is not a nofollow directive.
I am not sure I fully understand the situation, so I would not say this is my "reccomendation" but an answer to the specific question.....
but I am wondering if this will stop the bots dead in their tracks or will they still be able to pick-up on the pages generated
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Issue to index AMP pages
Hello In googe search console i have about 500 pages indexed for my websites https://horaire-priere.be/ and https://horarios-oracion.es/ but only 20 pages are indexed as AMP. Butmy site is amp only. I can't understand why google indexed the page but not in amp mode? Thank you in advanced
On-Page Optimization | | Zakirou0 -
Question about creating content pages for keywords
Good morning, We are trying to rank our India based company which provides the following services
On-Page Optimization | | harshal.khatavkar
Engineering Design Services
Architectural Design Services
MEP Design Services Our target audiences are in the US and UK. Offcource, we are targetting above services keywords on most of our main pages and created dedicated services pages too. But lately, we found out that we are ranking well for keywords like Outsourcing Engineering Design Services, Outsourcing Architectural Design Services, etc... which are actually very very good keywords in terms of closing the leads/inquiries as people are actually looking out for outsourcing but the search count for those keywords is low. (though we closed 2 inquiries from those keywords). These pages we created in past just to increase the content of the website. I really want to give it a try to target those keywords by creating more pages, blog posts, backlinks, etc... My question is if we create more and more pages around those keywords then will it affect the rankings of the pages which are already ranking for those keywords or will the new pages compete against those pages or the new pages will help to boost current pages? We can write good content and blog posts on the outsourcing topic but not sure if we should create new pages or increase the length of the existing pages. Can you guys please help with some directions on this as I really don't want to take the wrong route. Look forward! Regards0 -
Index or No Index (Panda Issue)
Hi, I believe our website has been penalized by the panda update. We have over 9000 pages and we are currently indexing around 4,000 of those pages. I believe that more than half of the pages indexes have either thin content. Should we stop indexing those pages until we have quality page content? That will leave us with very few pages being indexed by Google (Roughly 1,000 of our 9,000 pages have quality content). I am worried that we would hurt our organic traffic more by not indexing the pages than by indexing the pages for google to read. Any help would be greatly appreciated. Thanks, Jim Rodriguez
On-Page Optimization | | dustyabe0 -
Duplicate Content on our own website
Our website sells tickets for events. We also have an news articles section with information about events / artists / venues. From time to time we release a product page and a related news article on a separate page. Some of the content in the news article would be perfect for our product page. Essentially its our product page we want too rank. Would it harm our SEO if we had some of the same content on both of these pages?
On-Page Optimization | | Alexogilvie0 -
Duplicate content
Are images considered duplicate content too? Example:
On-Page Optimization | | BridalHotspot
I've got a size chart on each my lingerie pages. All written content is unique but I'm using the same chart for all those pages.0 -
Silo and content
I'm about to launch my site but I have a question regarding content and silo structure. If I don't have enough content to fill 4 subpages, could it be better to have only a content-keyword-rich landing page for a silo instead of multiple pages with poor content? Thank you!
On-Page Optimization | | mediodigital0 -
Does putting content in tabs devalue it at all?
Hello! Still very new to the SEO world and just trying to soak in as much information as I can. The site I work for took a substantial hit with the panda update, so we are looking into adding as much quality content as we can in the upcoming months. With our current site layout, space will quickly become an issue. Assuming the content is relevant and useful for the page, will putting the content into tabs be counter productive or devalue it at all?
On-Page Optimization | | davegtt0 -
Page Cache And Index
If you are browsing a site, what is the best way or programs to use to see if the page has been indexed and cached? Thanks
On-Page Optimization | | gregster10000