Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Google not showing all search result even Google index more than 14k.. its showing 120 only
http://www.blackbowchauffeur.com.au/ any advice why Google not showing full search result.... Even its more than 14k indexed and show only 122 in search result... Same as Bing ... and Yahoo...
Industry News | | blackbowchauffeur0 -
Best ranking Magento sites?
What would be your top 10 Magento sites in terms of SEO and ranking? I'm looking for examples of Magneto sites that have been set up correctly and are ranking well. Especially recently launched sites that have managed to get themselves ranking well in competitive markets! Cheers
Industry News | | OnlineAssetPartners0 -
I need some help with google please
Hello i am trying to remove google partial penalization to my website since july 2012 I have removed link, used disallow tools, wrote letters Last comunicacion was a letter send on 12/2012 that they responded 3/2013 web in cuestion is www.propdental.com I have not answered yet because i am looking for a good profesional help they responded saying that i still have unnuturall links wich is pretty probably but i dont nkow wich ones remove. Site authority has dropped from 43 to 32 with all links removed traffic dropped from 48000 to 15.000 Please profesional contacts needed
Industry News | | maestrosonrisas0 -
Help seems like google is punishing our site and I don't understand why
I would love some helpMy anme is Suzanne Diamond our web site is WWW.thefutonshop.com we have been on line for years, 10 brick and mortar stores, had a strong ecommerce web site but in the past 4 months we have been dropping like a leaf. Organic search has been dropping since November 2012 but now total traffic seems to be slowing We are working on our SEO but I feel it has to be something more direct that we are being punished by google Perhaps it s obvious but not to me Can anyone help me please or give me a direction. suzanne@thefutonshop.com thanks in advance for any direction
Industry News | | FUTONSHOP0 -
Is it OK to discuss SEO for an adult site?
Hey guys, I run a few big tube sites and sales pages that contain adult material. I'm also new to these forums, so before I do anything that might be considered rude or innapropriate, I wanted to know what the stance was on asking for advice related to the SEO of my adult sites and/or posting links. I've read the Etiquette disclaimer, however, I don't actually intend to post images or anything of the sort, only ask for your opinions and preferred strategies. If its completely frowned upon, then I completely understand and will never bring the subject back up again. Cheers and Happy to be amongst you all!
Industry News | | CrakJason1 -
Google Search Quality Team - Commission Based Reviews
I have been busy this past week writing articles for various sources about the recent update on Google. A number of people contacted me about the analysis I was doing and the report. Some were members of the Google Search Quality Team. I knew manual reports were done before - but after the documents they showed me regarding the reports they do and the compensation for doing the reports - I am left in a state of being pretty shocked. May be I have been naive for all these years but I didn't realize that; Google outsourced the review and reconsideration requests to individual reviewers for a compensation Google's position in terms of checking qualification and experience of these "reviewers" was very insufficient at best, The three contacts I spoke to who had done reports had very little training or experience. I went through the GSQT REVIEWERS PDF (a very long and thorough document) that I was sent - with them. We went together through some sites I wanted them to review and their comments that came back were quite astounding to say the least and would have made many of you Mozzers laugh. Obviously I don't want to post said document online here.... BUT, I wanted to know if: a) any Mozzers had ever been part of such a group - the GSQT b) had any dealings with them - in terms of having your website reviewed and known about it. I knew about this group way back - like in 2005 or 2006 or sometime around then - I was told at time it was stopped and Google had stopped paying these sub contractor reviewers. Please don't get me wrong here... totally on board with manual reviews... I would just prefer them done by a trained team that possibly worked for either a professional company that maintain high quality review testing and standards - or for that matter GOOGLE employees that were trained. I just am a little unsure of them being done by individual subbies that get paid for the amount they do. What if that subbie has got some skin in the game for a particular keyword? What if their knowledge about certain aspects isn't up to par or not tested on a regular basis. This space is always changing and as you guys ./ girls on this forum know - it can change pretty quick. I just would want all websites to be judged fairly and equally by a group trained EQUALLY and to the same standards. I don't care if this is a G team or not - I just want it to be a team that is trained equally and trained continuously as opposed to paying outside people based on numbers of reviews done. When the livelihood of a small business is the balance I don't want a commission hungry toe rag with one years experience being the gate keeper for me or any of our clients. Carlos
Industry News | | CarlosFernandes0 -
Google guidlines 2011
Guys I have asked for the leaked SEO guidlines just to fine tune my SEO campaigns and it seems no one wanted to send it to me. Anyone can do it here, please?
Industry News | | SearchOfficeSpace230