Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Very odd behavior. Google is changing “%20” to “+” in my URLS
I just realized many of the links on my site are BROKEN when entered from a Google SERP. This didn’t used to be this way. I have no idea what’s going but I’m worried. It involves a folder of our site that has a space in it. Google is even displaying the proper “%20” in the SERP but when I click the link it replaces that “%20” with a + which breaks the link!! You can see this in action by typing in “brown jordan sheffield furniture” without quotes into Google. You’ll see our site come up first and displaying this link sheffieldfurniture.com/Other%20Furniture/BrownJordan.html but when you click it the link is broken! This is happening on many of our pages! Anyone know what in the world is going on?
Industry News | | SheffieldMarketing0 -
Is there a way to get a list (backlink profile) of all tiny url's that point to my site or a competitors site?
I have noticed that most all links you find in all the major back link profile tools such as OSE or GWM, etc... do not show tiny url's. If there is a service that shows all the tiny urls pointing to your site, can someone please share. It has already been proven that tiny url's do pass link juice, so with that being said... if there is no way to find all the tiny urls that point to a site, wouldn't it be a great strategy to create all my back links with tiny url's to mask my profile from competitors? Thanks!
Industry News | | johnd57890 -
Help seems like google is punishing our site and I don't understand why
I would love some helpMy anme is Suzanne Diamond our web site is WWW.thefutonshop.com we have been on line for years, 10 brick and mortar stores, had a strong ecommerce web site but in the past 4 months we have been dropping like a leaf. Organic search has been dropping since November 2012 but now total traffic seems to be slowing We are working on our SEO but I feel it has to be something more direct that we are being punished by google Perhaps it s obvious but not to me Can anyone help me please or give me a direction. suzanne@thefutonshop.com thanks in advance for any direction
Industry News | | FUTONSHOP0 -
Does Google still have a standard search result? How can I get it?
I have heard a lot from the experts that there are no "Standard" Google search results anymore. They said that most of the SERP's of Google that show up are customized/tailored for each individuals even if they are not logged-in using their Google Custom Search. My questions are, Is there still a way to retrieve the standard Google search result? How? Are these scripts will be helpful when searching on Google? *webhp?
Industry News | | RafaelRanada
*complete=0
*pws=0 watch?v=B8ofWFx525s B8ofWFx525s watch?v=B8ofWFx525s0 -
How to report rankings after the Google Venice update?
As a profesional agency we focus on traffic and conversions, but rankings are still a good KPI to please customers. Unfortunately rankings are not reliable anymore sinds the Google Venice update. My question is; "How can you still report about rankings, but without the risk that your customer sees total different results?" Software we use At the moment we use Rank Tracker from Link Assistant.
Industry News | | VanSoelen0 -
Google Releases Penguin Update 1.1
So what affect is this playing on you guys? http://searchengineland.com/google-pushes-first-penguin-algorithm-update-122518 It was not to nice to me... Why over the holiday Google, why!? -Chenzo
Industry News | | Chenzo0 -
How do you value a site that generates no income before it's sold?
Hi, I'm currently creating a number of sites which target the wedding market. Each site is localized and specific to a niche within the wedding market. The intention is to create the site, rank it in the top position of Google and then sell it to a local business within that niche and area. The problem that i am having is that I don't know how to place a value on each site. I have factors taken from Google (using Market Samurai) so I know the broad and phrase match traffic details as well as the Google PPC average prices. My initial thought was to use the Google PPC price and multiply that by the number of organic visitors the site would receive if it was ranked #1 on Google. I appreciate that the PPC figures may be higher than what an advertiser would actually pay per click so I conservatively halved the Google PPC figure given in Market Samurai. Is this a good way of valuing the site or do I need to look at the value per customer to the business that I am trying to sell the site to and work from a conversion factor based on the number of visitors that the site would receive. If the site received 500 visitors per month and 5% of those became customers multiplied by the average profit per customer - that would give a monthly value of the website. I am very unsure as to what the best approach is here and look forward to hearing any advice and ideas from fellow members. Thanks, Michael
Industry News | | XSMedia0 -
Anyone else seeing a new google search result look?
Just noticed my google search results page has changed. Anyone else seeing this? Screenshot http://cl.ly/2h0v2i2w191O32082U0T
Industry News | | JohnTurner791