Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
If someone was brand new at all of marketing, where would you send them to get educated?
If someone had no marketing education, where would you send them to get a thorough education covering all aspects of marketing/advertising? Would you say you have to go to school for that? Would you recommend an online course set from CMI or MarketingProfs? If so, which one? Thank you 🙂
Industry News | | Crystalline_151 -
Google Analytics (Not Provided) Count will Increase 100% by Oct 2014 ? - Your Advice ?
What will you Do if you cannnot find your Top Keyword in Google Analytics "not provided"
Industry News | | Esaky
Check here for more details: http://www.notprovidedcount.com/0 -
Effect of changes on the Content page on search ranking
Hi, I have a question related to my ranking on Google Search On the content pages of my website, there is a section where our content keeps on changing. Whenever a visitor enters new information, Old information will be removed from the page. Therefore our content page is dynamic. Does this make us difficult to rank for particular keyword / overall.
Industry News | | adiez12340 -
How do I ascertain how/why a site appears higher/lower in search results, based on different search terms?
The site in question is www.bullethq.comHere are the search terms I used, and what position Bullethq.com appears in the SERP's: <colgroup><col width="188"><col width="58"><col width="95"></colgroup>
Industry News | | PeterConnor
| Search-term | Position | Page returned |
| irish payroll online | 5 | Home Page |
| irish payroll online software | 20 | Home Page |
| online irish payroll software | 20 | Home Page |
| online payroll | 75 | (Blog post) | Could someone be so kind as to help me figure this out?0 -
What is triggering Google account suspensions?
Over the past 24 hours many of our clients have had their Google accounts suspended. The explanation has been: "After reviewing your profile, we determined that it has been used to impersonate another individual or mislead other users. This violates the Google+ User Content and Conduct Policy." We are NOT impersonating our clients, we have their permission. We are not misleading anyone, simply setting up profiles for our clients on Google+. This has not affected all of our clients, but a significant number of them. We cannot find a common variable between the clients that have been suspended, and those who have not. Some have had other Google+ profiles in the past, in another account, some have not. Some have been previously verified via SMS, others by phone. Some have posts in their profile, others have only the profile info filled out. Again, we are not trying to game Google, we are simply setting up authorship for them, with their permission. I have not seen much in the SEO community about this today, and this is NOT related to fake reviews. We do not partake in that kind of activity. We have written a post on the topic, and no matter how this shakes out, I think our take is solid. Authorship is changing the game, content is changing the game, trust is changing the game… and Google is getting serious about it. We have also seen this happen to our clients, to our competitors' clients, and to other marketing firms' clients, outside of our vertical. Does anyone know more about the topic, especially in regards to the suspensions over the past 24 hours?
Industry News | | Einstein-Industries0 -
Strange video site adding unwanted links.
Today I was checking our backlink profile and noticed a lot of strange back links coming from grosezinga.com apparently this site is some type of search engine for videos, and it somehow pulls videos from Youtube, and Daily Motion, and adds them to the grosezinga search engine. I have never placed a link on this site, I do not want a link there, and I never even knew it existed until today. We do have video tutorials on the web, and somehow this site has pulled them from our video page without our permission, and added them to this site. Anyone ever had this problem, should I ask for them to be removed? When I do a Google video search my url is showing up on about a hundred different videos that are not ours. I don't want to be accused of building unnatural links.
Industry News | | TinaGammon0 -
Subdomains seen as one site
Earlier this month Google announced that sub-domains are now
Industry News | | AlanMosley
be treated as one site. At first I thought this was good news as I like to use
sub-domains for separation of categories and the like. But what about links
from one sub-domain to the other, they uised to be external links now they are
internal links. If you don’t have many external links, I would say that the
cross sub-domain links would have been important, if you have a lot of external
links then the flow of link juice would be of more benefit. I think overall its
is a good thing. Does anyone have any opinions about this or know of any writings
on the subject since this announcement? http://googlewebmastercentral.blogspot.com/2011/08/reorganizing-internal-vs-external.html0 -
SEO sites were blasted by Panda
Just noticed that lots of SEO sites were blasted by Panda... http://trends.google.com/websites?q=webmasterworld.com&geo=all&date=2011&sort=0 http://trends.google.com/websites?q=seomoz.org&geo=all&date=2011&sort=0 http://trends.google.com/websites?q=digitalpoint.com&geo=all&date=2011&sort=0 http://trends.google.com/websites?q=forums.seochat.com&geo=all&date=2011&sort=0
Industry News | | EGOL3