Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. How Does Google's "index" find the location of pages in the "page directory" to return?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    How Does Google's "index" find the location of pages in the "page directory" to return?

    Technical SEO
    3
    9
    1765
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • reidsteven75
      reidsteven75 last edited by

      This is my understanding of how Google's search works, and I am unsure about one thing in specific:

      1. Google continuously crawls websites and stores each page it finds (let's call it "page directory")
      2. Google's "page directory" is a cache so it isn't the "live" version of the page
      3. Google has separate storage called "the index" which contains all the keywords searched.  These keywords in "the index" point to the pages in the "page directory" that contain the same keywords.
      4. When someone searches a keyword, that keyword is accessed in the "index" and returns all relevant pages in the "page directory"
      5. These returned pages are given ranks based on the algorithm

      The one part I'm unsure of is how Google's "index" knows the location of relevant pages in the "page directory".  The keyword entries in the "index" point to the "page directory" somehow. I'm thinking each page has a url in the "page directory", and the entries in the "index" contain these urls.   Since Google's "page directory" is a cache, would the urls be the same as the live website (and would the keywords in the "index" point to these urls)?

      For example if webpage is found at wwww.website.com/page1, would the "page directory" store this page under that url in Google's cache?

      The reason I want to discuss this is to know the effects of changing a pages url by understanding how the search process works better.

      1 Reply Last reply Reply Quote 0
      • reidsteven75
        reidsteven75 @cbielich last edited by

        Yeah that makes sense.  I also have a lot of experience with databases and the back ends of websites so I know your language.

        I'm wondering how Google correlates the url with the page entries then. Maybe each page entry would have a url field so Google knows the location of the live version to constantly update that entry in the "page directory" database?

        1 Reply Last reply Reply Quote 0
        • cbielich
          cbielich @reidsteven75 last edited by

          That is a question that no one here can answer. We cant speak for how Google does things internally.

          but.... as a web / database programmer for 14+ years let me tell you how its "generally" done

          Usually when you have to link to separate sets of data together (ie. database or tables) there is usually a unique_id created to link them which usually is never changed. So when a new record is created that record will live with that ID for its life, also known as a (unique identifier which tends to be an auto-incremented number that is dynamically generated and can not be repeated).

          Since records tend to be linked this way, any other fields that exist in the record (firstName, lastName, Url, blah blah) then can be changed without the original ID being disturbed.

          So to answer your question from my experience I would assume Google links from a unique identifier of some sort and not the URL directly.

          Hope I didn't lose you, its my favorite subject...but no one here speaks that language to much 🙂

          reidsteven75 1 Reply Last reply Reply Quote 1
          • reidsteven75
            reidsteven75 @TakeshiYoung last edited by

            That makes sense, thanks for getting back to me so fast!

            Perhaps you can help answer my next question.  I have a client who used to host his domain at "www.oldurl.com", and has migrated his website to "www.newurl.com".  He wants to use his old domain "www.oldurl.com", so he setup forwarding/masking so that when someone tries to access "www.oldurl.com" they are forwarded to "www.newurl.com" but the url shown to the user is "www.oldurl.com".

            My client want his old url "www.oldurl.com" to be ranked in Google, but from what I understand his new url will be ranked.  I know masking is really bad for SEO, and I want to educate my client as to why on the technical side.  I have read Google see's all the content as duplicate with masking.  Do you know the details as to why?

            1 Reply Last reply Reply Quote 0
            • reidsteven75
              reidsteven75 last edited by

              Hey Cesar,

              Thanks for the links!  Really useful info there.

              Unfortunately they I couldn't find the answer I was looking for so I'll be more specific in what I'm asking.

              From what I understand Google uses two database systems.   One contains keywords and the other contains cached pages.  How does a keyword entry point to a page entry?  Does it use a unique id number, or does it use the url that page is using in the "live" vesion on the web?

              cbielich 1 Reply Last reply Reply Quote 0
              • TakeshiYoung
                TakeshiYoung @reidsteven75 last edited by

                Just because you create a new page and delete the old one, Google won't know immediately about it. So if Google crawls the new page before it's had a chance to crawl the old one, then it will indeed consider the new page to be duplicate content. Then when it tries to crawl the old page, it will discover that it no longer exists. However, as long as links to the old page exist, it will continue to try to crawl that page. Eventually it may de-index the old page if it keeps returning an error.

                Bottom line, if you are moving content to a new URL, be sure to include a 301 redirect on the old page so that Google (and other search engines) know that the piece of content has moved. You can also do this with canonical tags, but 301s are more effective.

                reidsteven75 1 Reply Last reply Reply Quote 1
                • reidsteven75
                  reidsteven75 @TakeshiYoung last edited by

                  Thanks for the response and links Takeshi.  Maybe I can rephrase the question to be more clear. Let's say a piece of content (or page) is at the url "www.oldurl.com/page".  During a migration this same piece of content now at the url "www.newurl.com/page".   The "www.oldurl.com" doesn't exist anymore so there isn't duplicate content in the live web.

                  Would Google create a new entry in it's "page directory" (what is the industry standard name for this directory?) and give it the url "www.newurl.com/page"?

                  If it does create a new entry, would Google keep the old entry "www.oldurl.com/page" although the old url doesn't exist in the "live" web anymore?

                  TakeshiYoung 1 Reply Last reply Reply Quote 0
                  • cbielich
                    cbielich last edited by

                    Wow you just asked questions that would require about 10,000,000,000 answers 😉

                    Lets start here

                    1. Video from the man himself Mr. Matt Cutts - Matt Cutts (Works for Google)
                    2. Great Web 2.0 Page create from Google themself - (Google Them self)
                    3. Older but still relevant description about how "backlinks" affect PR - (Google Them self)
                    1 Reply Last reply Reply Quote 2
                    • TakeshiYoung
                      TakeshiYoung last edited by

                      This a pretty confusing question, and the terminology you use is different from industry standard. Check out these links for a quick overview of how Google works:

                      • http://www.google.com/insidesearch/howsearchworks/thestory/
                      • http://www.googleguide.com/google_works.html

                      If you are just worried about changing a page's url, just be sure to put in a 301 redirect from the old page to the new page. That way, even if Google has an older version of the page indexed, it will automatically redirect the user to the new page as well as help Google discover the new location of the page.

                      reidsteven75 1 Reply Last reply Reply Quote 1
                      • 1 / 1
                      • First post
                        Last post

                      Got a burning SEO question?

                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                      Start my free trial


                      Browse Questions

                      Explore more categories

                      • Moz Tools

                        Chat with the community about the Moz tools.

                      • SEO Tactics

                        Discuss the SEO process with fellow marketers

                      • Community

                        Discuss industry events, jobs, and news!

                      • Digital Marketing

                        Chat about tactics outside of SEO

                      • Research & Trends

                        Dive into research and trends in the search industry.

                      • Support

                        Connect on product support and feature requests.

                      • See all categories

                      Related Questions

                      • amanda5964

                        Should I "no-index" two exact pages on Google results?

                        noindex google search engine results seo expert

                        Hello everyone, I recently started a new wordpress website and created a static homepage. I noticed that on Google search results, there are two different URLs landing on same content page. I've attached an image to explain what I saw. Should I "no-index" the page url? Google url.JPG In this picture, the first result is the homepage and I try to rank for that page. The last result is landing on same content with different URL. So, should I no-index last result as shown in image?

                        Technical SEO | | amanda5964
                        0
                      • flo_seo

                        Not all images indexed in Google

                        Hi all, Recently, got an unusual issue with images in Google index. We have more than 1,500 images in our sitemap, but according to Search Console only 273 of those are indexed. If I check Google image search directly, I find more images in index, but still not all of them. For example this post has 28 images and only 17 are indexed in Google image. This is happening to other posts as well. Checked all possible reasons (missing alt, image as background, file size, fetch and render in Search Console), but none of these are relevant in our case. So, everything looks fine, but not all images are in index. Any ideas on this issue? Your feedback is much appreciated, thanks

                        Technical SEO | | flo_seo
                        1
                      • Rignite

                        Should I noindex my blog's tag, category, and author pages

                        Hi there, Is it a good idea to no index tag, category, and author pages on blogs? The tag pages sometimes have duplicate content.  And the category and author pages aren't really optimized for any search term. Just curious what others think. Thanks!

                        Technical SEO | | Rignite
                        0
                      • inlinear

                        Correct linking to the /index of a site and subfolders: what's the best practice? link to: domain.com/ or domain.com/index.html ?

                        Dear all, starting with my .htaccess file: RewriteEngine On
                        RewriteCond %{HTTP_HOST} ^www.inlinear.com$ [NC]
                        RewriteRule ^(.*)$ http://inlinear.com/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^./index.html 
                        RewriteRule ^(.)index.html$ http://inlinear.com/ [R=301,L] 1. I redirect all URL-requests with www. to the non www-version...
                        2. all requests with "index.html" will be redirected to "domain.com/" My questions are: A) When linking from a page to my frontpage (home) the best practice is?: "http://domain.com/" the best and NOT: "http://domain.com/index.php" B) When linking to the index of a subfolder "http://domain.com/products/index.php" I should link also to: "http://domain.com/products/" and not put also the index.php..., right? C) When I define the canonical ULR, should I also define it just: "http://domain.com/products/" or in this case I should link to the definite file: "http://domain.com/products**/index.php**" Is A) B) the best practice? and C) ? Thanks for all replies! 🙂
                        Holger

                        Technical SEO | | inlinear
                        0
                      • rhoadesjohn

                        The word 'shop' in a page title

                        I'm reworking most of the page titles on our site and I'm considering the use of the word 'Shop' before a product category. ex.  Shop 'keyword' | Brand Name As opposed to just using the keyword sans 'Shop.'  Some of the keywords are very generic, especially for a top level category page. Question: Is the word 'Shop' damaging my SEO efforts in any way?

                        Technical SEO | | rhoadesjohn
                        0
                      • CsmBill

                        How do I add "noindex" or "nofollow" to a link in Wordpress

                        It's been a while since I've SEOed a Wordpress site. How do I add "nofollow" or "noindex" to specific links? I highlight the anchor text in the text editor, I click the "link" button. I could have sworn that there used to be an option in the dialogue box that pops up.

                        Technical SEO | | CsmBill
                        0
                      • DHS_SH

                        Ecommerce website: Product page setup & SKU's

                        I manage an E-commerce website and we are looking to make some changes to our product pages to try and optimise them for search purposes and to try and improve the customer buying experience. This is where my head starts to hurt! Now, let's say I am selling a T shirt that comes in 4 sizes and 6 different colours. At the moment my website would have 24 products, each with pretty much the same content (maybe differing references to the colour & size). My idea is to change this and have 1 main product page for the T-shirt, but to have 24 product SKU's/variations that exist to give the exact product details. Some different ways I have been considering to do this: a) have drop-down fields on the product page that ask the customer to select their Tshirt size and colour. The image & price then changes on the page. b) All product 24 product SKUs sre listed under the main product with the 'Add to Cart' open next to each one. Each one would be clickable so a page it its own right. Would I need to set up a canonical links for each SKU that point to the top level product page? I'm obviously looking to minimise duplicate content but Im not exactly sure on how to set this up - its a big decision so I need to be 100% clear before signing off on anything. . Any other tips on how to do this or examples of good e-commerce websites that use product SKus well? Kind regards Tom

                        Technical SEO | | DHS_SH
                        0
                      • dreadmichael

                        How to block "print" pages from indexing

                        I have a fairly large FAQ section and every article has a "print" button. Unfortunately, this is creating a page for every article which is muddying up the index - especially on my own site using Google Custom Search. Can you recommend a way to block this from happening? Example Article: http://www.knottyboy.com/lore/idx.php/11/183/Maintenance-of-Mature-Locks-6-months-/article/How-do-I-get-sand-out-of-my-dreads.html Example "Print" page: http://www.knottyboy.com/lore/article.php?id=052&action=print

                        Technical SEO | | dreadmichael
                        0

                      Get started with Moz Pro!

                      Unlock the power of advanced SEO tools and data-driven insights.

                      Start my free trial
                      Products
                      • Moz Pro
                      • Moz Local
                      • Moz API
                      • Moz Data
                      • STAT
                      • Product Updates
                      Moz Solutions
                      • SMB Solutions
                      • Agency Solutions
                      • Enterprise Solutions
                      Free SEO Tools
                      • Domain Authority Checker
                      • Link Explorer
                      • Keyword Explorer
                      • Competitive Research
                      • Brand Authority Checker
                      • Local Citation Checker
                      • MozBar Extension
                      • MozCast
                      Resources
                      • Blog
                      • SEO Learning Center
                      • Help Hub
                      • Beginner's Guide to SEO
                      • How-to Guides
                      • Moz Academy
                      • API Docs
                      About Moz
                      • About
                      • Team
                      • Careers
                      • Contact
                      Why Moz
                      • Case Studies
                      • Testimonials
                      Get Involved
                      • Become an Affiliate
                      • MozCon
                      • Webinars
                      • Practical Marketer Series
                      • MozPod
                      Connect with us

                      Contact the Help team

                      Join our newsletter
                      Moz logo
                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                      • Accessibility
                      • Terms of Use
                      • Privacy

                      Looks like your connection to Moz was lost, please wait while we try to reconnect.