What is the largest page size a searchbot will crawl?
-
When setting up pagination, what should we limit the page size to? When will a searchbot stop crawling a particular page?
-
I think the ideal size is below 500k, yet I feel Google will crawl even larger sized pages if the content provides value for the users.
Back in 2005 I remember Google had much tighter figures on these types of numbers yet in today's market it is a bit different, they seem to allow larger file sizes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages Crawl Per Day Gone Drasitcaly Down, is it google issue?
Hello Expert, In search console in Crawl Stats Pages Crawl per day going day by day i.e. from 4 lac pages per day now it is reduce upto 2 lac in last 15 days. So where is the issue? Where I am going wrong or it is issue from google end? Thanks!
Technical SEO | | Johny123450 -
Why does my site have so many crawl errors relating to the wordpress login / captcha page
Going through the crawl of my site, there were around 100 medium priority issues, such as title element too short, and duplicate page title, and around 80 high priority issues relating to duplicate page content - However every page listed with these issues was the site's wordpress login / captcha page. Does anyone know how to resolve this?
Technical SEO | | ZenyaS0 -
I have 3500 pages crawled by Google, - why is SEOMOZ only able to crawl 400 of these ?
I added my site almost two weeks ago to the PRO DashBoard, and so far only 404 pages has been crawled, - but I know for a fact that there is 3500 pages that should be crawled. Other search engines has no problem in crawling and indexing these pages, so what can be wrong here ?
Technical SEO | | haybob270 -
Will using tabs on my page for navigation hurt SEO
Currently we have tabs on some pages that are set so that the page shows all the content for search engines on one page, but the user sees only part of the content until they click a tab. We are unable to use title tags for the tabbed areas. We think this hurts us from an SEO standpoint. We are getting only 3 keyword where we could have 12 if they were individual pages. What is the thought on using tabs and is it okay? Has there been any case studies on this?
Technical SEO | | cayseo0 -
No crawl code for pages of helpful links vs. no follow code on each link?
Our college website has many "owners" who want pages of "helpful links" resulting in a large number of outbound links. If we add code to the pages to prevent them from being crawled, will that be just as effective as making every individual link no follow?
Technical SEO | | LAJN0 -
Does Google pass link juice a page receives if the URL parameter specifies content and has the Crawl setting in Webmaster Tools set to NO?
The page in question receives a lot of quality traffic but is only relevant to a small percent of my users. I want to keep the link juice received from this page but I do not want it to appear in the SERPs.
Technical SEO | | surveygizmo0 -
Will loading ads in an iframe increase page response time?
We are experiencing slow response time because our pages are ad heavy. If we load ads in an iframe, will the google bots, when indexing the page, count the time it takes for the contact within the iframe to load? Or is that load time separate from the total page response/load time?
Technical SEO | | kbbseo0 -
Will changing page extensions from .html to .php require a redirect?
Hi. We are launching a new website and our .html page extensions will be replaced with a .php page extension. Example: www.theideapeople.com/web_design.html (current url) www.theideapeople.com/web_design.php (new url) Will this require any special treatment to maintain the page SEO ranking? Does it make a difference if you use a .html or .php? Thank you for your help and insight! Jay
Technical SEO | | theideapeople0