How much content does Google Crawl on your site?
-
Hi,
We've had a debate around the office where some people believe that Google only crawls the first 150-200 words on a page and some people believe that they priority content that is above the fold and other people believe that all content has the same priority. Can you help us?
Thanks,
Matt -
Google actually crawls 150kb, excluding css files, images, etc.
150kb is much more than 200 words, and the experiment suggested by Mr Bennett proves it.
-
They definitely crawl more than that, and it's easy to prove as well.
Pick a long page, such as the Wikipedia page about London. Choose a block of text from near the bottom of that page, I've selected this:
in the south-western suburb of Wimbledon.[252] Other key events are the annual mass-participation London Marathon which sees some 35,000 runners
If you search for that text you will see the Wikipedia page in the results. If they only crawled the first 200 words they wouldn't have been able to find that result.
Prioritising is harder to demonstrate (and probably also to define!). However it is generally believe that greater importance is given to text towards the top of the page. That is logical if you consider how the majority of documents are structured.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google ranking content for phrases that don't exist on-page
I am experiencing an issue with negative keywords, but the “negative” keyword in question isn’t truly negative and is required within the content – the problem is that Google is ranking pages for inaccurate phrases that don’t exist on the page. To explain, this product page (as one of many examples) - https://www.scamblermusic.com/albums/royalty-free-rock-music/ - is optimised for “Royalty free rock music” and it gets a Moz grade of 100. “Royalty free” is the most accurate description of the music (I optimised for “royalty free” instead of “royalty-free” (including a hyphen) because of improved search volume), and there is just one reference to the term “copyrighted” towards the foot of the page – this term is relevant because I need to make the point that the music is licensed, not sold, and the licensee pays for the right to use the music but does not own it (as it remains copyrighted). It turns out however that I appear to need to treat “copyrighted” almost as a negative term because Google isn’t accurately ranking the content. Despite excellent optimisation for “Royalty free rock music” and only one single reference of “copyrighted” within the copy, I am seeing this page (and other album genres) wrongly rank for the following search terms: “free rock music”
On-Page Optimization | | JCN-SBWD
“Copyright free rock music"
“Uncopyrighted rock music”
“Non copyrighted rock music” I understand that pages might rank for “free rock music” because it is part of the “Royalty free rock music” optimisation, what I can’t get my head around is why the page (and similar product pages) are ranking for “Copyright free”, “Uncopyrighted music” and “Non copyrighted music”. “Uncopyrighted” and “Non copyrighted” don’t exist anywhere within the copy or source code – why would Google consider it helpful to rank a page for a search term that doesn’t exist as a complete phrase within the content? By the same logic the page should also wrongly rank for “Skylark rock music” or “Pretzel rock music” as the words “Skylark” and “Pretzel” also feature just once within the content and therefore should generate completely inaccurate results too. To me this demonstrates just how poor Google is when it comes to understanding relevant content and optimization - it's taking part of an optimized term and combining it with just one other single-use word and then inappropriately ranking the page for that completely made up phrase. It’s one thing to misinterpret one reference of the term “copyrighted” and something else entirely to rank a page for completely made up terms such as “Uncopyrighted” and “Non copyrighted”. It almost makes me think that I’ve got a better chance of accurately ranking content if I buy a goat, shove a cigar up its backside, and sacrifice it in the name of the great god Google! Any advice (about wrongly attributed negative keywords, not goat sacrifice ) would be most welcome.0 -
Content above the Fold, or Below
Hi, I have an ecommerce site with several categories that I consider good landing pages. In order to get better search results I add content to these pages, usually above the fold, then after the content products are listed. Example:https://www.carburetor-parts.com/Carburetor-Kits_c_568.html I worry that customers get to the page and since they don't see the products above the fold, they move on. Should I be putting content in the footer instead of the header and if so how does that effect SEO? This has been bugging me for a long time. Thanks
On-Page Optimization | | MikeCarbs
Mike0 -
PDF Instructions come up in Crawl report as Duplicate Content
Hello, My ecommerce site has many PDF instruction pages that are being marked as duplicate content in the site crawl. Each page has a different title, and then a PDF displayed in an iframe with a link back to the previous page & to the category that the product is placed in. Should I add text to the pages to help differentiate them? I included a screenshot of the code that is on all the pages. Thanks! Justin 9tD9HMr
On-Page Optimization | | JustinBSLW0 -
Duplicate Content
I'm currently working on a site that sells appliances. Currently, there are thousands of "issues" with this site, many of them dealing with duplicate content. Now, the product pages can be viewed in "List" or "Grid" format. As Lists, they have very little in the way of content. My understanding is that the duplicate content arises from different URLs going to the same site. For instance, the site might have a different URL when told to display 9 items than when told to display 15. This could then be solved by inserting rel = canonical. Is there a way to take a site and get a list of all possible duplicates? This would be much easier than slogging through every iteration of the options and copying down the URLs. Also, is there anything I might be missing in terms of why there is duplicate content? Thank you.
On-Page Optimization | | David_Moceri0 -
404 errors in wordpress... Pages have never existed so why is google trying to crawl them?
I've just logged into webmaster tools and have over 100 404 errors. I'm running wordpress and I recently added child pages to 2 of my categories like so. www.mydomain.com/category1/lincolnshire www.mydomain.com/category1/cambridgeshire etc... The 404 errors though are for pages or categories I've never created though. I have over 20 root categories but decided to test adding child pages to only two of them. The 404 errors are for www.mydomain.com/category5/cambridgeshire .... It seem that gogle has tried to crawl these pages that don't exist. Can anyone explain what's going on? When I click 'linked from' in webmaster tools it's showing links from pages on my site that don't exist also.
On-Page Optimization | | SamCUK0 -
Duplicate Content
Part of a site I am working on, features many different bags in all thicknesses colors and sizes. I'm getting an error when some pages have different content like different thicknesses. The only differences between the pages are a single digit - but in trash bags that makes it a whole different product! I can't do a canonical because it's not the same. For example: http://www.plasticplace.net/index.php?file=productdetail&iprod_id=274 and http://www.plasticplace.net/index.php?file=productdetail&iprod_id=268 Any ideas?
On-Page Optimization | | EcomLkwd0 -
Crawl with cach problem
Hello, My Crawl results in Seomoz shows me that i have few thousands of 302 direct problem, bucause it was crawling links like http://www.sposae.com/abito-sposa-g2026-pr-347.html?action=buy_now and it would be redirected automatically to http://www.sposae.com/cookie_usage.php because of cookie not activated from the user. Now I'm wondering if this is an issue to be solved or just a minor thing without impact. Thanks
On-Page Optimization | | angelowei0 -
Silo and content
I'm about to launch my site but I have a question regarding content and silo structure. If I don't have enough content to fill 4 subpages, could it be better to have only a content-keyword-rich landing page for a silo instead of multiple pages with poor content? Thank you!
On-Page Optimization | | mediodigital0