CMS dynamicly created pages indexed?
-
Hey Moz'erz,
Looking at the indexed pages of my clients eCommerce website I noticed that dynamically created pages are being indexed.
For example this page does not "exist" but is created by a drop down filter menu that sorts by product tag:
/collections/tools/TAG
I can only conclude that this page got indexed either through a backlink or once upon a time there was an internal link pointing to this URL and got indexed (currently there is not). Are either of these cases possibilities?
In either case before considering removal or any action I would of-course reference analytics to check for conversions, traffic and any backlinks for those "pages".
I believe at the end of the day is recommend a drop down filer that doesn't create new pages as the best solution.
Thoughts, comments and experience is greatly welcomed
-
Hey Dylan
Either of those are possibilities for Google finding and indexing a page like that. There could be many ways that happened - I've seen them spider "links" in a drop down depending on how it's implemented.
One thing you can do to check how, is looked at the text-only cache of the page (type cache:www.domain.com/page-name in your browser and click text only) - and look to see if the drop down items actually appear and clickable links. You can also try crawling the site with Screaming Frog and set the user-agent to GoogleBot and see if they got picked up.
If the filter is just for example re-sorting the list of items in a category, there is probably not a need to have this crawled or indexed, because it's just the same content in a different order.
If you do want to remove them from the index, you will want to add a meta noindex tag to the HTML, wait for them to drop out of the index, and then block crawling with robots.txt or nofollow the links that might be generated.
Hope that helps!
EDIT - I'd also check to be sure they are not showing up in your XML sitemap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have a site that has a 302 redirect loop on the home page (www.oncologynurseadvisor.com) i
i am trying to do an audit on it using screaming frog and the 302 stops it. My dev team says it is to discourage Non Human Traffic and that the bots will not see it. Is there any way around this or what can I tell the dev team that shows them it is not working as they state.
Web Design | | HayMktVT0 -
What are the downsides and/or challenges to putting page paths (www.example.com/pagepath) on a different server?
Hi, Our company is organized into three different segments and our development team recently needed to switch a portion of the business to subdomain because they wanted to move to a different server platform. We are now seeing the impact of moving this segment of the business to a subdomain on the main domain. SEO is hurting and our MOZ score has dropped significantly. One fix they are debating is moving everything back to one domain, but place segments of the business on different page paths and hosting specific paths on different servers. I.e. the main domain could be www.example.com hosted in one location and then www.example.com/segment1 would be hosted on a different server. They are hoping to accomplish this using some sort of proxy/caching redirection solution. The goal of this change would be to recapture our domain strength. Is this something that is a good option or no? If not, what are the challenges and issues you see arising from doing something like that as I don't know of any other site set up like this. Thanks in advance.
Web Design | | bradgreene0 -
Bing Indexation and handling of X-ROBOTS tag or AngularJS
Hi MozCommunity, I have been tearing my hair out trying to figure out why BING wont index a test site we're running. We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
Web Design | | AU-SEO
This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing. We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site. In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header. With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag. However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive. I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing. I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages. I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content. Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages. Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl. Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests? Thanks in advance for your assistance....0 -
Are these doorway pages or not? Concerned due to Panda 4.0
For a new site we're building, the Products team wants the header (let's call this Product-Header) to have links to every subsection of every section on every page. Since this is a bad idea, I want Product-Header to be coded in such a way that it doesn't appear in the code or the links are nofollow, noindex. I want to instead create static versions of these pages without the Product-Header. The homepage links to the static URL section pages, those main section pages link to static subsection pages, and so on. It's one nice silo. I am concerned though that Google won't like this due to these static pages are being created specifically for search engines. Users could click through to this static parallel site from the homepage, or they could use the dynamic URL site. This is similar to what etsy.com is doing where you can search Google for "mermaid bridal" and get this page https://www.etsy.com/market/mermaid_bridal but the dynamic version of the page does not show up. However you can search on etsy.com for " mermaid bridal" and get https://www.etsy.com/search?q=mermaid bridal&ship_to=US. Could these static versions that show up in search engines be seen as doorway pages? I know ebay.com got spanked for doorway pages and I don't want to do anything that would get this site penalized.
Web Design | | CFSSEO0 -
What seo benefit does setting up a photo gallery where each photo is a separate web page?
what seo benefit does setting up a photo gallery where each photo is a separate web page? My old SEO guy set up my photo gallery like that claiming that because each photo was a separate page, it added a big seo benefit and i never understood what he was talking about. Maybe alt text on the photo with key phrases in it pointing to my other pages to give my site a theme for google? I'm not really sure. He has since moved away and i am considering redoing the photo gallery to multiple images on one page to be more user friendly to my users. This photo gallery is 3 years old and the photos might have some page rank to them helping my site so i don't want to remove this gallery if there really is a benefit to it and it will hurt my site. I once removed four static page rank 3 pages from my site that weren't used for my site anymore and my rankings dropped 5 positions. Thoughts anyone? Thanks! Ron
Web Design | | Ron100 -
Our "home page" is behind a member wall, options?
So www.pch.com(portal) redirects to www.pch.com/unrecognized(landing page) if you are not registered with us and logged in. This means that the search engines are not logged in, so they see only our landing page. It used to be that there was no portal/home, on pch.com, that was just the landing page, but that changed about 6 months ago. We do rank for our brand terms, but my company would like to rank for terms like "sweepstakes." They DO understand why we don't, thankfully. They don't think SEO is magic voodoo. They get it. But they asked for options, as I have said that the portal on www.pch.com really is a good page to optimize for non-brand, core terms like sweepstakes....but only if the search engines can see it. I gave them these options, and they asked me to seek out more. So any thoughts would be good: 1. Best case scenario would be to abandon the landing page, just have the keyword rich portal page be the actual home page with no re-direct. (this won't happen, but I decided it needed to be first on my list). 2. Turn the portal into the home page (remove the redirect), but have the landing page overlay in a light box. This should, if I am not mistaken, be a best of both worlds situation, where the light box landing page would still have all of the value of the actual keyword rich portal page behind it. 3. If the landing page has to remain as it does now with the non-logged in redirect to it, change the URLs so that the landing page is www.pch.com and the portal becomes www.pch.com/members/ or something like that. Any other thoughts? Thanks! Kenn Gold Publishers Clearing House
Web Design | | Kenn_Gold0 -
Should my link href be www or go direct to page?
Hi, just wondering which is the best format for linking to pages. In my navigation at the moment i have links like; Car Repair Services Is this the recommended format or should it be; Car Repair Services Many thanks for any answers. Alex
Web Design | | SeoSheikh0 -
Are links from main page to inner pages will affect on ranking?
About 3 weeks ago I converted index.html to index.php. Both are 301 redirect to main url. Also I have about 70 links on main page pointing to internal pages. The Website is about 11 years old,and was on active link building . Is this conversion from html to php and also 70 links pointing to inner pages will affect on ranking?Since all links are passing juice to inner pages.
Web Design | | LosAngelesLimo0