Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What are your opinions on the Google News vs Spanish Government Issue ?
Greg Sterling said: "Governments across Europe are justifiably alarmed by the declining fortunes of their respective newspaper industries. However punitive or parasitic taxation measures targeting Google, masquerading as copyright protections, are not the answer." Do you agree?
Industry News | | Tintanus1 -
What shall I call my site?
With all the google changes and social media focus on seo, I am in a bit of a dilemma and any opinions welcome. My husband has a very small horseback riding company based in a tiny mountain town of Costa Rica and I cannot decide whether to call it by the company name or by a more generic name that is likely to be picked up by the search engines ( I do believe the importance of domain name is less than it used to be in terms of search with all the new google changes). The two names are www.horsetrekmonteverde.com (bear in mind that his customers come from the States and horse trekking is not commonly used) and www.horseridingcostarica.com One mentions Costa Rica, the other not as you can see. Also not many people have beard about Monteverde. However I am trying to do some strong branding! What would you do?
Industry News | | Llanero0 -
Google Analytics Tracking Miscount - Originating Around 5/24
Is anyone else experiencing an issue where it seems GA failed to track many of their visitors? This is why I think it's something on GA's end and not mine (and not natural): 1. No changes were made, site wasn't down, tracking code is correct and is "receiving data" 2. Referral, Direct and Organic traffic all dropped at about the same decline over the same period *3. Pinterest Analytics shows me being sent over 10 times more traffic every day than GA shows me actually receiving. In the screenshot, Pinterest shows 584 uniques while GA shows 43 for Sunday. I figure if something was broken, I wouldn't be showing ANY traffic and if it was natural, it wouldn't have happened on the same decline across all traffic mediums and Pinterest Analytics wouldn't be so far off. I saw a few people mentioning similar things on Google's forums and wanted to ask you guys if anyone noticed any issues? y8XAvwz.png pRsRAmK.png
Industry News | | zDucketz0 -
If I have a Google+ Business page, do I need a Google Places page as well?
It seems like the two are redundant? Any official word on this? I'm fairly OCD about things being tidy and I dont want to split my reviews / shares / etc between two profiles. Are they not the same thing? I searched for my company, both my plus business page and my places page came up. I attached a SS of the situation. placesvplus.png
Industry News | | jonnyholt1 -
Email marketing - ready to get serious about this
Ok, I have my PILE of business cards from Chamber of Commerce meetings and different networking events I have attended. I have 8-10 blogs that are worth sharing on my website - www.sawwebmarketing.com (I realize that's a pathetically low number) I am committing to and excited about blogging MUCH more starting today! I have a huge list of ideas. I am adding a signup to my site and putting up a pop up to collect addresses. I have 15-18 clients that are paying me for ongoing seo / Internet marketing (or simply had my team build a new site for them) AND AN ACCOUNT WITH AWEBER I HAVE USED ONCE! 🙂 Where do I start!? Any advice? I want to start building a LIST of subscribers and grow my list by sharing great advice, strategies and tips to help people put their website to work. A) Send one email to EVERYBODY (ongoing clients, past clients, contacts who subscribe, website visitors, blog subscribers) or customize my messages for different groups? B) Put whatever I want to share on my blog and share it via social media AND send an email to my list inviting them to check out what I posted so I get the most bang for my time? (LINKS BABY!) I feel like an idiot I haven't got on this sooner but truly ANY ADVICE from somebody who has been there and done it would be GREATLY APPRECIATED! 🙂 Thank you! Matthew http://sawwebmarketing.com/internet-marketing-blog/
Industry News | | Mrupp440 -
How can you change 'Google related searches'?
I am having problems for a few clients where google is suggesting negative related searches. I have had success getting Google suggestion (auto-complete) to suggest the keywords we want to be associated with the brand but the 'Searches Related to....' at the bottom of the search page is a big problem. Many of the terms are negative of sometimes defamatory and have absolutely not relevance to the brand. Even upon clicking on the related searches the pages Google displays are about other businesses. Can anyone suggest how to teach Google these term are irrelevant or any other suggestions as to how to remove them?
Industry News | | skehoe0 -
Anyone else know much about the Google Pirate penalty?
The Google 'Pirate' (no official name) seems to have gone largely undiscussed since it was launched last - Fri 10th August http://insidesearch.blogspot.co.uk/2012/08/an-update-to-our-search-algorithms.html. The idea of it is to ensure those 'Pirating' content or abusing trademarks e.g. fake ugg boot sites and file sharing sites do not appear higher in the search results than the genuine websites. Google is using DMCA take down requests for labeling sites as Pirate and demote their rankings, Im amazed not even seomoz has covered the subject yet as far as I can see, yet it is a hugely important new update, albeit affecting a relatively small number of sites now, and in some cases (at least one I know first hand) seemingly without justification (the example I know is not a file sharing, fake goods, trademark abusive site at all.) Google updating its search algorithm based on DMCA take down requests seems a bit strong - these are takedown requests, not legal proof that a site is infringing a trademark. A real weapon for negative SEO? Anyone else had experience of the pirate update or know much more about it? Outside Danny Sullivan I dont see many SEO folk covering it. Heres my own insights into it and what ive learned about what (only innocently) affected sites should do to appeal http://www.andy-maclean.net/the-google-pirate-dmca-guidance/
Industry News | | AndyMacLean0 -
Paste 'do a barrel roll' into Google - See what happens!!
Hi, Paste the words 'do a barrel roll' in Google and see what happens!
Industry News | | activitysuper0