Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Massive Amount of Pages Deindexed

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Massive Amount of Pages Deindexed

    Intermediate & Advanced SEO
    4
    12
    1587
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • D.J.Hanchett
      D.J.Hanchett last edited by

      On or about 12/1/17 a massive amount of my site's pages were deindexed. I have done the following:

      • Ensured all pages are "index,follow"
      • Ensured there are no manual penalites
      • Ensured the sitemap correlates to all the pages
      • Resubmitted to Google
      • ALL pages are gone from Bing as well

      In the new SC interface, there are 661 pages that are Excluded with 252 being "Crawled - currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling." What in the world does this mean and how the heck do I fix this. This is CRITICAL. Please help!

      The url is https://www.hkqpc.com

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing last edited by

        the report was run prior canonical directives

        Anytime remember to noindex your robots.txt

        https://yoast.com/x-robots-tag-play/

        There are cases in which the robots.txt file itself might show up in search results. By using an alteration of the previous method, you can prevent this from happening to your website:

         <filesmatch "robots.txt"="">Header set X-Robots-Tag "noindex"</filesmatch> 
        
        **And in Nginx:** 
        
        location = robots.txt {
            add_header  X-Robots-Tag "noindex";
        }
        
        1 Reply Last reply Reply Quote 1
        • D.J.Hanchett
          D.J.Hanchett last edited by

          Looking at the first report, "Redirect Chains"..  As I understand the table, these are correct..

          Column A is the page (source) with the redirecting link
          Column B is the link that is redirecting (http://www.hkqlaw.com)
          Column C shows 2 redirects happening
          Column I shows the first redirect (http://www.hkqlaw.com -> http://www.hkqpc.com) (non ssl version)
          Column N shows the second redirect (http://www.hkqpc.com -> https://www.hkqpc.com) (ssl version)

          The original link (hkqlaw.com) is a link in the footer of our news section so is common on those pages which is why it shows so often.  So, like I said, this appears to be correct.

          I added the canonical directives to the pages earlier so perhaps that report was run prior to me doing that?

          Again, thanks so much for your effort in helping me!

          1 Reply Last reply Reply Quote 0
          • D.J.Hanchett
            D.J.Hanchett last edited by

            Now I'm really baffled. I just ran Screaming Frog and don't see any of the redirects or other stats. Which software are you using that is showing this information? I'm trying to replicate it and figure out if there's something, somewhere else doing this.

            1 Reply Last reply Reply Quote 0
            • BlueprintMarketing
              BlueprintMarketing last edited by

              Wow, I got it

              your 301  redirecting a ton of URLs back to the homepage.

              • Redirect chains https://bseo.io/cZW0w0
              • internal URLs https://bseo.io/4sFqUk
              • insecure content https://bseo.io/YDDKGD
              • no canonical https://bseo.io/fWey1Q
              • crawl overview https://bseo.io/Zg6bpM
              • canonical errors https://bseo.io/YtTh7W
              1 Reply Last reply Reply Quote 0
              • D.J.Hanchett
                D.J.Hanchett last edited by

                Ok, canonical is set for each page (and I fixed the // issue).  I used x-robots header to noindex the robots.txt and sitemap.xml files, along with a few other extensions while I was at it.

                I'll get the secured cookie header set after this is resolved.  We don't store any sensitive data via cookies for this site so it's not of immediate concern but still one I'll address.

                EDIT:  The https://www.hkqpc.com/attorney/David-Saba.html/ page no longer exists which was the cause of the errors.  I've redirected that to the appropriate page.

                1 Reply Last reply Reply Quote 1
                • BlueprintMarketing
                  BlueprintMarketing last edited by

                  https://cryptoreport.websecurity.symantec.com/checker/

                  This server cannot be scanned for these vulnerabilities:HeartbleedServer scan unsuccessful. <a>See possible causes.</a>Poodle (TLS)Server scan unsuccessful. See possible causes.BEASTThis server is vulnerable to a BEAST attack. <a>More information.</a>

                  I am sorry I said your IP was  Network solutions when it was 1&1 I still strongly recommend changing hosting companies even though I am German and so is 1&1

                  DNS resolves www.hkqpc.com to 74.208.236.66

                  The SSL certificate used to load resources from https://www.hkqpc.com will be distrusted in M70. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.

                  Look: https://cl.ly/pCY5

                  Look: https://cl.ly/pAKa

                  symantec  SSL certificates are now owned by DigiCert

                  <big>https://www.digicert.com/help/</big>

                  https://www.dareboost.com/en/report/5a70b33e0cf28f017576367f

                  The Set-Cookie HTTP header can be configured with your Apache server. Make sure that the mod_headers module is enabled. Then, you can specify the header (in your .htaccess file, for example). Here is an example:  <ifmodule mod_headers.c=""># only for Apache > 2.2.4: Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure  # lower versions: Header set Set-Cookie HttpOnly;Secure</ifmodule>

                  1. robots.txt file inside of the SERPS big photo https://i.imgur.com/cJeDR9t.png
                  2. XML sitemap inside of SERPS should be no indexed big photo https://i.imgur.com/tlx5jc7.png

                  Double forward slashes after verdicts the same page without double forward slashes you need to add rel canonical tags zero canonical's on any page whatsoever.

                  • https://www.hkqpc.com/news/verdicts//hkq-attorneys-win-carbon-county-real-estate-case/
                  • https://www.hkqpc.com/news/verdicts/hkq-attorneys-win-carbon-county-real-estate-case/

                  The URLs above need a rel=canonical tag I have created an example below for you. For the page without the double forward slashes, and this tells Google the one you'd prefer to have indexed besides it keeps the query string pages and junk pages out of Google's index. Please see the resources below and add them to your website  because I do not know what type of CMS you're using I cannot recommend a plug-in to do it but if you were using something like WordPress it would be automatically done by something like Yoast WordPress SEO for the site that you are using it may be a wise move to move to something like WordPress it is a solid platform for a site that size and makes things a lot easier for you to implement change across the entire site quickly.

                  • https://moz.com/blog/complete-guide-to-rel-canonical-how-to-and-why-not
                  • https://yoast.com/rel-canonical/
                  • https://moz.com/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

                  You need to add a canonical

                  • Bigger photo of problem https://i.imgur.com/1qMMPSM.png
                  • this page https://www.hkqpc.com/attorney/David-Saba.html/
                  • Warning: Creating default object from empty value in /homepages/43/d238880598/htdocs/classes/class.attorneys.php on line 38
                  • Warning: Invalid argument supplied for foreach() in /homepages/43/d238880598/htdocs/headers/attorney.php on line 15
                  • ** FIx for this**
                  • https://stackoverflow.com/questions/14806959/how-to-fix-creating-default-object-from-empty-value-warning-in-php
                  • http://thisinterestsme.com/invalid-argument-supplied-for-foreach/

                  You have

                  Heartbleed Vulnerability

                  An unknown error occurred while scanning for the Heartbleed Bug.

                  1qMMPSM.png tlx5jc7.png cJeDR9t.png

                  1 Reply Last reply Reply Quote 1
                  • D.J.Hanchett
                    D.J.Hanchett last edited by

                    Thanks for the great feedback!  The hkqlaw.com url simply forwards (301) to hkqpc.com.  The IP address you have is for hkqlaw.com which is registered through Network Solutions, but hosting of hkqpc.com is on 1and1.com hosting.  Also, the timeout error you're getting is because there is no SSL cert for hkqlaw.com, again, it's just forwarded to hkqpc.com (which does have an SSL attached to it).  As far as SC, everything is setup to index hkqpc.com.

                    1 Reply Last reply Reply Quote 0
                    • BlueprintMarketing
                      BlueprintMarketing last edited by

                      Right now I cannot get that site to load on my browser, and when I used https://tools.pingdom.com it was unable to load as well you could be having some serious server problems, and that could be causing the issue although I was getting it to run through screaming frog which is surprising.

                      This is a zip file of your screen frog results this will show if there are any no index pages which I found none of it looks to me like you have a server issue. Zip file: http://bseo.io/BXYpZh

                      I checked your site for malware using https://sitecheck.sucuri.net/results/www.hkqlaw.com/ ( please understand this only check the homepage and a handful of others) and found none though when I checked your IP address I noticed a lot of ransomware information tied directly to your IP

                      https://ransomwaretracker.abuse.ch/ip/205.178.189.131/

                      Here is a large screenshot of when I tried to browse your website: https://i.imgur.com/OzcLhbx.png

                      Here is Pingdom ( remember to test on something outside of your local computer because you have caching and other things that could give you incorrect results.)

                      https://tools.pingdom.com/#!/bd6d52/https://www.hkqlaw.com/

                      in my experience network solutions, hosting is terrible I would strongly suggest doing two things.

                      Get a better hosting company for your site.

                      A good host that is not too expensive is and also managed is liquid Web, cloudways, rack space, pairnic, you can also build out your own system on non-managed hosting like Linode, digital ocean, AWS, Google cloud, Microsoft Azure if you want a high-quality, inexpensive manage host that offers more than one back and like the ones I've listed above https://www.cloudways.com/en/  will host anything and manage it, and you can use the backends provided before this.  If you want what I think is the best and price is not a big deal considering you're not running WordPress https://armor.com is my preferred hosting company. Otherwise, cloudways or liquid Web would be where I would host your site.

                      Considering you already have an IP address attached to ransomware and you're using hosting company that will not be beneficial to you in security terms. I would add a web application firewall/reverse proxy you can do that with https://sucuri.net/website-firewall/  https://incapsula.com  https://fastly.com and if you want most basic and least secure but better than what you have https://cloudflare.com

                      At the very least put Cloudflare on their but what I'm seeing is a severe problem coming from your web host and knowing that hosting company I would strongly advise you to move to a better host.

                      I hope this was of help,

                      Thomas

                      OzcLhbx.png

                      1 Reply Last reply Reply Quote 0
                      • TimHolmes
                        TimHolmes last edited by

                        Not sure if this is of help to you, I suppose it depends how many pages you are expecting to be indexed, but according to John Mu at Google - Google does not necessarily index all pages.

                        https://www.seroundtable.com/google-index-all-pages-20780.html

                        1 Reply Last reply Reply Quote 0
                        • D.J.Hanchett
                          D.J.Hanchett last edited by

                          Not recently. It migrated well over a year ago to HTTPS.

                          1 Reply Last reply Reply Quote 0
                          • ThompsonPaul
                            ThompsonPaul last edited by

                            First thing to confirm - did you recently migrate to HTTPS?

                            1 Reply Last reply Reply Quote 1
                            • 1 / 1
                            • First post
                              Last post

                            Got a burning SEO question?

                            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                            Start my free trial


                            Browse Questions

                            Explore more categories

                            • Moz Tools

                              Chat with the community about the Moz tools.

                            • SEO Tactics

                              Discuss the SEO process with fellow marketers

                            • Community

                              Discuss industry events, jobs, and news!

                            • Digital Marketing

                              Chat about tactics outside of SEO

                            • Research & Trends

                              Dive into research and trends in the search industry.

                            • Support

                              Connect on product support and feature requests.

                            • See all categories

                            Related Questions

                            • seoanalytics

                              Fresh page versus old page climbing up the rankings.

                              Hello, I have noticed that if publishe a webpage that google has never seen it ranks right away and usually in a descend position to start with (not great but descend). Usually top 30 to 50 and then over the months it slowly climbs up the rankings. However, if my page has been existing for let's say 3 years and I make changes to it, it takes much longer to climb up the rankings Has someone noticed that too ? and why is that ?

                              Intermediate & Advanced SEO | | seoanalytics
                              0
                            • Inevo

                              Why does Google rank a product page rather than a category page?

                              Hi, everybody In the Moz ranking tool for one of our client's (the client sells sport equipment) account, there is a trend where more and more of their landing pages are product pages instead of category pages. The optimal landing page for the term "sleeping bag" is of course the sleeping bag category page, but Google is sending them to a product page for a specific sleeping bag.. What could be the critical factors that makes the product page more relevant than the category page as the landing page?

                              Intermediate & Advanced SEO | | Inevo
                              0
                            • boostaman

                              How to check if the page is indexable for SEs?

                              Hi, I'm building the extension for Chrome, which should show me the status of the indexability of the page I'm on. So, I need to know all the methods to check if the page has the potential to be crawled and indexed by a Search Engines. I've come up with a few methods: Check the URL in robots.txt file (if it's not disallowed) Check page metas (if there are not noindex meta) Check if page is the same for unregistered users (for those pages only available for registered users of the site) Are there any more methods to check if a particular page is indexable (or not closed for indexation) by Search Engines? Thanks in advance!

                              Intermediate & Advanced SEO | | boostaman
                              0
                            • abhihan

                              Google indexing only 1 page out of 2 similar pages made for different cities

                              We have created two category pages, in which we are showing products which could be delivered in separate cities. Both pages are related to cake delivery in that city. But out of these two category pages only 1 got indexed in google and other has not. Its been around 1 month but still only Bangalore category page got indexed. We have submitted sitemap and google is not giving any crawl error. We have also submitted for indexing from "Fetch as google" option in webmasters. www.winni.in/c/4/cakes (Indexed - Bangalore page - http://www.winni.in/sitemap/sitemap_blr_cakes.xml) 2. http://www.winni.in/hyderabad/cakes/c/4 (Not indexed - Hyderabad page - http://www.winni.in/sitemap/sitemap_hyd_cakes.xml) I tried searching for "hyderabad site:www.winni.in" in google but there also http://www.winni.in/hyderabad/cakes/c/4 this link is not coming, instead of this only www.winni.in/c/4/cakes is coming. Can anyone please let me know what could be the possible issue with this?

                              Intermediate & Advanced SEO | | abhihan
                              0
                            • Philip-DiPatrizio

                              Putting "noindex" on a page that's in an iframe... what will that mean for the parent page?

                              If I've got a page that is being called in an iframe, on my homepage, and I don't want that called page to be indexed.... so I put a noindex tag on the called page (but not on the homepage) what might that mean for the homepage?  Nothing?  Will Google, Bing, Yahoo, or anyone else, potentially see that as a noindex tag on my homepage?

                              Intermediate & Advanced SEO | | Philip-DiPatrizio
                              0
                            • Bio-RadAbs

                              Dynamic pages - ecommerce product pages

                              Hi guys, Before I dive into my question, let me give you some background.. I manage an ecommerce site and we're got thousands of product pages. The pages contain dynamic blocks and information in these blocks are fed by another system. So in a nutshell, our product team enters the data in a software and boom, the information is generated in these page blocks. But that's not all, these pages then redirect to a duplicate version with a custom URL. This is cached and this is what the end user sees. This was done to speed up load, rather than the system generate a dynamic page on the fly, the cache page is loaded and the user sees it super fast. Another benefit happened as well, after going live with the cached pages, they started getting indexed and ranking in Google. The problem is that, the redirect to the duplicate cached page isn't a permanent one, it's a meta refresh, a 302 that happens in a second. So yeah, I've got 302s kicking about. The development team can set up 301 but then there won't be any caching, pages will just load dynamically. Google records pages that are cached but does it cache a dynamic page though? Without a cached page, I'm wondering if I would drop in traffic. The view source might just show a list of dynamic blocks, no content! How would you tackle this? I've already setup canonical tags on the cached pages but removing cache.. Thanks

                              Intermediate & Advanced SEO | | Bio-RadAbs
                              0
                            • HD_Leona

                              Blocking Pages Via Robots, Can Images On Those Pages Be Included In Image Search

                              Hi! I have pages within my forum where visitors can upload photos.  When they upload photos they provide a simple statement about the photo but no real information about the image,definitely not enough for the page to be deemed worthy of being indexed.  The industry however is one that really leans on images and having the images in Google Image search is important to us. The url structure is like such:  domain.com/community/photos/~username~/picture111111.aspx I wish to block the whole folder from Googlebot to prevent these low quality pages from being added to Google's main SERP results.  This would be something like this: User-agent: googlebot Disallow: /community/photos/ Can  I disallow Googlebot specifically rather than just using User-agent:  * which would then allow googlebot-image to pick up the photos?  I plan on configuring a way to add meaningful alt attributes and image names to assist in visibility, but the actual act of blocking the pages and getting the images picked up... Is this possible? Thanks! Leona

                              Intermediate & Advanced SEO | | HD_Leona
                              0
                            • Visually

                              Are there any negative effects to using a 301 redirect from a page to another internal page?

                              For example, from http://www.dog.com/toys to http://www.dog.com/chew-toys. In my situation, the main purpose of the 301 redirect is to replace the page with a new internal page that has a better optimized URL. This will be executed across multiple pages (about 20). None of these pages hold any search rankings but do carry a decent amount of page authority.

                              Intermediate & Advanced SEO | | Visually
                              0

                            Get started with Moz Pro!

                            Unlock the power of advanced SEO tools and data-driven insights.

                            Start my free trial
                            Products
                            • Moz Pro
                            • Moz Local
                            • Moz API
                            • Moz Data
                            • STAT
                            • Product Updates
                            Moz Solutions
                            • SMB Solutions
                            • Agency Solutions
                            • Enterprise Solutions
                            Free SEO Tools
                            • Domain Authority Checker
                            • Link Explorer
                            • Keyword Explorer
                            • Competitive Research
                            • Brand Authority Checker
                            • Local Citation Checker
                            • MozBar Extension
                            • MozCast
                            Resources
                            • Blog
                            • SEO Learning Center
                            • Help Hub
                            • Beginner's Guide to SEO
                            • How-to Guides
                            • Moz Academy
                            • API Docs
                            About Moz
                            • About
                            • Team
                            • Careers
                            • Contact
                            Why Moz
                            • Case Studies
                            • Testimonials
                            Get Involved
                            • Become an Affiliate
                            • MozCon
                            • Webinars
                            • Practical Marketer Series
                            • MozPod
                            Connect with us

                            Contact the Help team

                            Join our newsletter
                            Moz logo
                            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                            • Accessibility
                            • Terms of Use
                            • Privacy

                            Looks like your connection to Moz was lost, please wait while we try to reconnect.