undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Massive Amount of Pages Deindexed

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Massive Amount of Pages Deindexed

Intermediate & Advanced SEO
4
12
1.6k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • D.J.Hanchett
    D.J.Hanchett last edited by Jan 29, 2018, 8:42 PM

    On or about 12/1/17 a massive amount of my site's pages were deindexed. I have done the following:

    • Ensured all pages are "index,follow"
    • Ensured there are no manual penalites
    • Ensured the sitemap correlates to all the pages
    • Resubmitted to Google
    • ALL pages are gone from Bing as well

    In the new SC interface, there are 661 pages that are Excluded with 252 being "Crawled - currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling." What in the world does this mean and how the heck do I fix this. This is CRITICAL. Please help!

    The url is https://www.hkqpc.com

    1 Reply Last reply Reply Quote 0
    • BlueprintMarketing
      BlueprintMarketing last edited by Jan 31, 2018, 12:20 AM Jan 31, 2018, 12:20 AM

      the report was run prior canonical directives

      Anytime remember to noindex your robots.txt

      https://yoast.com/x-robots-tag-play/

      There are cases in which the robots.txt file itself might show up in search results. By using an alteration of the previous method, you can prevent this from happening to your website:

       <filesmatch "robots.txt"="">Header set X-Robots-Tag "noindex"</filesmatch> 
      
      **And in Nginx:** 
      
      location = robots.txt {
          add_header  X-Robots-Tag "noindex";
      }
      
      1 Reply Last reply Reply Quote 1
      • D.J.Hanchett
        D.J.Hanchett last edited by Jan 30, 2018, 9:12 PM Jan 30, 2018, 9:12 PM

        Looking at the first report, "Redirect Chains"..  As I understand the table, these are correct..

        Column A is the page (source) with the redirecting link
        Column B is the link that is redirecting (http://www.hkqlaw.com)
        Column C shows 2 redirects happening
        Column I shows the first redirect (http://www.hkqlaw.com -> http://www.hkqpc.com) (non ssl version)
        Column N shows the second redirect (http://www.hkqpc.com -> https://www.hkqpc.com) (ssl version)

        The original link (hkqlaw.com) is a link in the footer of our news section so is common on those pages which is why it shows so often.  So, like I said, this appears to be correct.

        I added the canonical directives to the pages earlier so perhaps that report was run prior to me doing that?

        Again, thanks so much for your effort in helping me!

        1 Reply Last reply Reply Quote 0
        • D.J.Hanchett
          D.J.Hanchett last edited by Jan 30, 2018, 8:52 PM Jan 30, 2018, 8:52 PM

          Now I'm really baffled. I just ran Screaming Frog and don't see any of the redirects or other stats. Which software are you using that is showing this information? I'm trying to replicate it and figure out if there's something, somewhere else doing this.

          1 Reply Last reply Reply Quote 0
          • BlueprintMarketing
            BlueprintMarketing last edited by Jan 30, 2018, 8:23 PM Jan 30, 2018, 8:23 PM

            Wow, I got it

            your 301  redirecting a ton of URLs back to the homepage.

            • Redirect chains https://bseo.io/cZW0w0
            • internal URLs https://bseo.io/4sFqUk
            • insecure content https://bseo.io/YDDKGD
            • no canonical https://bseo.io/fWey1Q
            • crawl overview https://bseo.io/Zg6bpM
            • canonical errors https://bseo.io/YtTh7W
            1 Reply Last reply Reply Quote 0
            • D.J.Hanchett
              D.J.Hanchett last edited by Jan 30, 2018, 6:23 PM Jan 30, 2018, 6:22 PM

              Ok, canonical is set for each page (and I fixed the // issue).  I used x-robots header to noindex the robots.txt and sitemap.xml files, along with a few other extensions while I was at it.

              I'll get the secured cookie header set after this is resolved.  We don't store any sensitive data via cookies for this site so it's not of immediate concern but still one I'll address.

              EDIT:  The https://www.hkqpc.com/attorney/David-Saba.html/ page no longer exists which was the cause of the errors.  I've redirected that to the appropriate page.

              1 Reply Last reply Reply Quote 1
              • BlueprintMarketing
                BlueprintMarketing last edited by Jan 31, 2018, 10:25 AM Jan 30, 2018, 5:14 PM

                https://cryptoreport.websecurity.symantec.com/checker/

                This server cannot be scanned for these vulnerabilities:HeartbleedServer scan unsuccessful. <a>See possible causes.</a>Poodle (TLS)Server scan unsuccessful. See possible causes.BEASTThis server is vulnerable to a BEAST attack. <a>More information.</a>

                I am sorry I said your IP was  Network solutions when it was 1&1 I still strongly recommend changing hosting companies even though I am German and so is 1&1

                DNS resolves www.hkqpc.com to 74.208.236.66

                The SSL certificate used to load resources from https://www.hkqpc.com will be distrusted in M70. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.

                Look: https://cl.ly/pCY5

                Look: https://cl.ly/pAKa

                symantec  SSL certificates are now owned by DigiCert

                <big>https://www.digicert.com/help/</big>

                https://www.dareboost.com/en/report/5a70b33e0cf28f017576367f

                The Set-Cookie HTTP header can be configured with your Apache server. Make sure that the mod_headers module is enabled. Then, you can specify the header (in your .htaccess file, for example). Here is an example:  <ifmodule mod_headers.c=""># only for Apache > 2.2.4: Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure  # lower versions: Header set Set-Cookie HttpOnly;Secure</ifmodule>

                1. robots.txt file inside of the SERPS big photo https://i.imgur.com/cJeDR9t.png
                2. XML sitemap inside of SERPS should be no indexed big photo https://i.imgur.com/tlx5jc7.png

                Double forward slashes after verdicts the same page without double forward slashes you need to add rel canonical tags zero canonical's on any page whatsoever.

                • https://www.hkqpc.com/news/verdicts//hkq-attorneys-win-carbon-county-real-estate-case/
                • https://www.hkqpc.com/news/verdicts/hkq-attorneys-win-carbon-county-real-estate-case/

                The URLs above need a rel=canonical tag I have created an example below for you. For the page without the double forward slashes, and this tells Google the one you'd prefer to have indexed besides it keeps the query string pages and junk pages out of Google's index. Please see the resources below and add them to your website  because I do not know what type of CMS you're using I cannot recommend a plug-in to do it but if you were using something like WordPress it would be automatically done by something like Yoast WordPress SEO for the site that you are using it may be a wise move to move to something like WordPress it is a solid platform for a site that size and makes things a lot easier for you to implement change across the entire site quickly.

                • https://moz.com/blog/complete-guide-to-rel-canonical-how-to-and-why-not
                • https://yoast.com/rel-canonical/
                • https://moz.com/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

                You need to add a canonical

                • Bigger photo of problem https://i.imgur.com/1qMMPSM.png
                • this page https://www.hkqpc.com/attorney/David-Saba.html/
                • Warning: Creating default object from empty value in /homepages/43/d238880598/htdocs/classes/class.attorneys.php on line 38
                • Warning: Invalid argument supplied for foreach() in /homepages/43/d238880598/htdocs/headers/attorney.php on line 15
                • ** FIx for this**
                • https://stackoverflow.com/questions/14806959/how-to-fix-creating-default-object-from-empty-value-warning-in-php
                • http://thisinterestsme.com/invalid-argument-supplied-for-foreach/

                You have

                Heartbleed Vulnerability

                An unknown error occurred while scanning for the Heartbleed Bug.

                1qMMPSM.png tlx5jc7.png cJeDR9t.png

                1 Reply Last reply Reply Quote 1
                • D.J.Hanchett
                  D.J.Hanchett last edited by Jan 30, 2018, 5:13 PM Jan 30, 2018, 5:13 PM

                  Thanks for the great feedback!  The hkqlaw.com url simply forwards (301) to hkqpc.com.  The IP address you have is for hkqlaw.com which is registered through Network Solutions, but hosting of hkqpc.com is on 1and1.com hosting.  Also, the timeout error you're getting is because there is no SSL cert for hkqlaw.com, again, it's just forwarded to hkqpc.com (which does have an SSL attached to it).  As far as SC, everything is setup to index hkqpc.com.

                  1 Reply Last reply Reply Quote 0
                  • BlueprintMarketing
                    BlueprintMarketing last edited by Jan 30, 2018, 3:26 PM Jan 30, 2018, 3:26 PM

                    Right now I cannot get that site to load on my browser, and when I used https://tools.pingdom.com it was unable to load as well you could be having some serious server problems, and that could be causing the issue although I was getting it to run through screaming frog which is surprising.

                    This is a zip file of your screen frog results this will show if there are any no index pages which I found none of it looks to me like you have a server issue. Zip file: http://bseo.io/BXYpZh

                    I checked your site for malware using https://sitecheck.sucuri.net/results/www.hkqlaw.com/ ( please understand this only check the homepage and a handful of others) and found none though when I checked your IP address I noticed a lot of ransomware information tied directly to your IP

                    https://ransomwaretracker.abuse.ch/ip/205.178.189.131/

                    Here is a large screenshot of when I tried to browse your website: https://i.imgur.com/OzcLhbx.png

                    Here is Pingdom ( remember to test on something outside of your local computer because you have caching and other things that could give you incorrect results.)

                    https://tools.pingdom.com/#!/bd6d52/https://www.hkqlaw.com/

                    in my experience network solutions, hosting is terrible I would strongly suggest doing two things.

                    Get a better hosting company for your site.

                    A good host that is not too expensive is and also managed is liquid Web, cloudways, rack space, pairnic, you can also build out your own system on non-managed hosting like Linode, digital ocean, AWS, Google cloud, Microsoft Azure if you want a high-quality, inexpensive manage host that offers more than one back and like the ones I've listed above https://www.cloudways.com/en/  will host anything and manage it, and you can use the backends provided before this.  If you want what I think is the best and price is not a big deal considering you're not running WordPress https://armor.com is my preferred hosting company. Otherwise, cloudways or liquid Web would be where I would host your site.

                    Considering you already have an IP address attached to ransomware and you're using hosting company that will not be beneficial to you in security terms. I would add a web application firewall/reverse proxy you can do that with https://sucuri.net/website-firewall/  https://incapsula.com  https://fastly.com and if you want most basic and least secure but better than what you have https://cloudflare.com

                    At the very least put Cloudflare on their but what I'm seeing is a severe problem coming from your web host and knowing that hosting company I would strongly advise you to move to a better host.

                    I hope this was of help,

                    Thomas

                    OzcLhbx.png

                    1 Reply Last reply Reply Quote 0
                    • TimHolmes
                      TimHolmes last edited by Jan 30, 2018, 11:53 AM Jan 30, 2018, 11:53 AM

                      Not sure if this is of help to you, I suppose it depends how many pages you are expecting to be indexed, but according to John Mu at Google - Google does not necessarily index all pages.

                      https://www.seroundtable.com/google-index-all-pages-20780.html

                      1 Reply Last reply Reply Quote 0
                      • D.J.Hanchett
                        D.J.Hanchett last edited by Jan 30, 2018, 8:08 AM Jan 30, 2018, 8:08 AM

                        Not recently. It migrated well over a year ago to HTTPS.

                        1 Reply Last reply Reply Quote 0
                        • ThompsonPaul
                          ThompsonPaul last edited by Jan 29, 2018, 11:42 PM Jan 29, 2018, 11:42 PM

                          First thing to confirm - did you recently migrate to HTTPS?

                          1 Reply Last reply Reply Quote 1
                          • 1 / 1
                          1 out of 12
                          • First post
                            1/12
                            Last post

                          Got a burning SEO question?

                          Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                          Start my free trial


                          Browse Questions

                          Explore more categories

                          • Moz Tools

                            Chat with the community about the Moz tools.

                          • SEO Tactics

                            Discuss the SEO process with fellow marketers

                          • Community

                            Discuss industry events, jobs, and news!

                          • Digital Marketing

                            Chat about tactics outside of SEO

                          • Research & Trends

                            Dive into research and trends in the search industry.

                          • Support

                            Connect on product support and feature requests.

                          • See all categories

                          Related Questions

                          • Caffeine_Marketing

                            Internal links to landing pages

                            Hi, we are in the process of building a new website and we have 12 different locations and for theses 12 locations we have landing pages with unique copy on the following: 1. Marketing...2 SEO....3. PPC....4. Web Design Therefor there are 48 landing pages.  The marketing pages are the most important ones to us in terms of traffic and priority.  My question is: 1. Should we put a dropdown of the are pages in the main header under locations that link to the area marketing pages? 2. What is the best way to link all the sub pages such as London Web Design?  Should these links just be coming off the London marketing page? or should we have a sitemap in the footer that lists every page? Thanks

                            Intermediate & Advanced SEO | Jul 30, 2020, 11:54 AM | Caffeine_Marketing
                            0
                          • yaelslater

                            Images on their own page?

                            Hi Mozers, We have images on their own separate pages that are then pulled onto content pages. Should the standalone pages be indexable? On the one hand, it seems good to have an image on it's own page, with it's own title. On the other hand, it may be better SEO for crawler to find the image on a content page dedicated to that topic. Unsure. Would appreciate any guidance! Yael

                            Intermediate & Advanced SEO | Oct 8, 2019, 10:39 AM | yaelslater
                            1
                          • Amor2005

                            Which of these examples are doorway pages?

                            Hi there, I am soon to launch a new platform/directory website, however, have a concern over doorway pages. I have read many articles on the difference between Doorway and Landing pages and do have a good understanding, however, am still very anxious that what I intend to do will be risking Google penalties. I have looked at other directory/platform websites and have noticed that a lot of them are still using doorway pages and are not getting penalised. So I was wondering if someone wouldn't mind kindly letting me know their opinion on which of the following examples are doorway pages and which are not so I can better understand what I can and cannot do? Example 1: When I Google 'piano lessons new york' and 'trumpet lessons new york' I get the following 'landing pages' in search: https://takelessons.com/new-york/piano-lessons https://takelessons.com/new-york/trumpet-lessons To me, the above pages are definitely doorway pages as they are very similar with content and text and are simply an intermediary step between the Google search and their listings pages for piano/trumpet teachers in New York. Is this correct? Example 2: When I Google 'piano lessons Sydney' I get presented with the following web page in search: http://www.musicteacher.com.au/directory/sydney-nsw/lessons/piano/ I would think that this is NOT a doorway page as the user has been taken directly to the search results page in the directory and the page doesn't seem to have been set up for the sole purpose of listing in search results for 'Piano Lessons in Sydney'. Example 3: When I Google 'pet minding Sydney' I get presented with the following two pages in search: https://www.madpaws.com.au/petsitters/Sydney-New-South-Wales?type=night&service=1&from=0&to=99&city=Sydney&state=New-South-Wales https://www.pawshake.com.au/petsitters/Sydney%252C%2520New%2520South%2520Wales%252C%2520Australia Like Example 2, I don't think these pages would be classified as doorway pages as they too direct to the search results page in the site directory instead of an intermediary page. What do you think? Thanks so much in advance for your expertise and help! Kind Regards, Adrian

                            Intermediate & Advanced SEO | Dec 22, 2016, 10:17 PM | Amor2005
                            0
                          • CostumeD

                            On 1 of our sites we have our Company name in the H1 on our other site we have the page title in our H1 - does anyone have any advise about the best information to have in the H1, H2 and Page Tile

                            We have 2 sites that have been set up slightly differently. On 1 site we have the Company name in the H1 and the product name in the page title and H2. On the other site we have the Product name in the H1 and no H2. Does anyone have any advise about the best information to have in the H1 and H2

                            Intermediate & Advanced SEO | Jul 8, 2015, 10:13 AM | CostumeD
                            0
                          • friendoffood

                            What if page exists for desktop but not mobile?

                            I have a domain (no subdomains) that serves up different dynamic content for mobile/desktop pages--each having the exact same page url, kind of a semi responsive design, and will be using "Vary: User-Agent" to give Google a heads up on this setup. However, some of the pages are only valid for mobile or only valid for desktop.  In the case of when a page is valid only for mobile (call it mysite.com/mobile-page-only ), Google Webmaster Tools is giving me a soft 404 error under Desktop, saying that the page does not exist, Apparently it is doing that because my program is actually redirecting the user/crawler to the home page.  It appears from the info about soft 404 errors that Google is saying since it "doesn't exist" I should give the user a 404 page--which I can make it customized and give the user an option to go to the home page, or choose links from a menu, etc.. My concern is that if I tell the desktop bot that mysite.com/mobile-page-only basically is a 404 error (ie doesn't exist), that it could mess up the mobile bot indexing for that page--since it definitely DOES exist for mobile users.. Does anyone here know for sure that Google will index a page for mobile that is a 404 not found for desktop and vice versa?  Obviously it is important to not remove something from an index in which it belongs, so whether Google is careful to differential the two is a very important issue.  Has anybody here dealt with this or seen anything from Google that addresses it?  Might one be better off leaving it as a soft 404 error? EDIT: also, what about Bing and Yahoo?  Can we assume they will handle it the same way? EDIT: closely related question--in a case like mine does Google need a separate sitemap for the valid mobile pages and valid desktop pages even though most links will be in both?  I can't tell from reading several q&a on this. Thanks, Ted

                            Intermediate & Advanced SEO | Dec 10, 2014, 8:10 PM | friendoffood
                            0
                          • WebServiceConsulting.com

                            NoIndexing Massive Pages all at once: Good or bad?

                            If you have a site with a few thousand high quality and authoritative pages, and tens of thousands with search results and tags pages with thin content, and noindex,follow the thin content pages all at once, will google see this is a good or bad thing? I am only trying to do what Google guidelines suggest, but since I have so many pages index on my site, will throwing the noindex tag on ~80% of thin content pages negatively impact my site?

                            Intermediate & Advanced SEO | Oct 1, 2013, 4:18 PM | WebServiceConsulting.com
                            0
                          • Prime85

                            Stuck on Page 2 - What Would You Do?!?

                            My site is : http://goo.gl/JgK1e My main keyword is : Plastic Bins i have been going back and forth between page 1 and 2 for this keyword and i was wondering if any of you could provide any guidance as to why i can't get on the top of page 1, and stay there... My site has been around for a while, we believe we have a great user experience, all unique, fresh content, and the lowest prices... I must be missing out on something major if I cannot get a steady page 1 ranking... Any thoughts? Thanks in advance...

                            Intermediate & Advanced SEO | Feb 29, 2012, 5:37 PM | Prime85
                            0
                          • EricPacifico

                            Should the sitemap include just menu pages or all pages site wide?

                            I have a Drupal site that utilizes Solr, with 10 menu pages and about 4,000 pages of content. Redoing a few things and we'll need to revamp the sitemap. Typically I'd jam all pages into a single sitemap and that's it, but post-Panda, should I do anything different?

                            Intermediate & Advanced SEO | Jul 14, 2011, 3:44 AM | EricPacifico
                            0

                          Get started with Moz Pro!

                          Unlock the power of advanced SEO tools and data-driven insights.

                          Start my free trial
                          Products
                          • Moz Pro
                          • Moz Local
                          • Moz API
                          • Moz Data
                          • STAT
                          • Product Updates
                          Moz Solutions
                          • SMB Solutions
                          • Agency Solutions
                          • Enterprise Solutions
                          Free SEO Tools
                          • Domain Authority Checker
                          • Link Explorer
                          • Keyword Explorer
                          • Competitive Research
                          • Brand Authority Checker
                          • Local Citation Checker
                          • MozBar Extension
                          • MozCast
                          Resources
                          • Blog
                          • SEO Learning Center
                          • Help Hub
                          • Beginner's Guide to SEO
                          • How-to Guides
                          • Moz Academy
                          • API Docs
                          About Moz
                          • About
                          • Team
                          • Careers
                          • Contact
                          Why Moz
                          • Case Studies
                          • Testimonials
                          Get Involved
                          • Become an Affiliate
                          • MozCon
                          • Webinars
                          • Practical Marketer Series
                          • MozPod
                          Connect with us

                          Contact the Help team

                          Join our newsletter
                          Moz logo
                          © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                          • Accessibility
                          • Terms of Use
                          • Privacy

                          Looks like your connection to Moz was lost, please wait while we try to reconnect.