How to Safely Scrape Google Results?
-
I've built a couple of small tools that I use personally, maybe 2 or 3 times per day.
Both tools scrape the top 10 results from Google and provide more details about each domain (like the SEOMoz Keyword Difficulty Tool).
Google seem to have banned my IP address for automated searches... can anyone tell me a safe way of scraping the google results? Is there a suitable API for this?
How do SEO Moz do this on such a huge scale?
-
As I doubt that the APIs have considerably improved since this blog post http://www.seomoz.org/blog/the-nasty-problem-with-scraping-results-from-the-engines, google scraping is still a big issue and necessary for our daily seo work.
Scraping savely can only work if you succeed in convincing Google that you're a "natural" user and not a scarping robot. How can you do that?
- Search with alternating IPs, from different locations using proxies from the countries where you'd like to scrape from
- don't send too many requests at once from the same source
Consider that, when requesting a URL, the browser sends various information elements to the server, containing, for example, your Operating System, browser version, referer, etc. - every element can and should be changed to virtually change your identity when executing a new search.
- change browsers, browser versions, operating system information, etc.
- take care when changing browser localization values (en-GB, en-US probably don't return the same results)
- have a good network of proxy servers ready to send the different requests with your different identities to
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Possible google sandbox issue? Organically ranking page 1 for our number 1 keyword, but page 5 sometimes 6 on google?
What are some things I can look into to figure out why google is ranking us on page 5 sometimes page 6, with some slight rank boosts to rank 36 from 48 but then falls right back. While Yahoo and Bing rank us page 1 consistently, without these big drops back. I use google search ( webmaster tools ) daily, fix 404s and make sure to fetch new content I create. Our site is within the Sandbox issue time frame, google 1st indexed the site about a year and a half ago, the site has been through various SEO service checks, and has had those issues fixed ( some bigcommerce won't allow, such as full sitewide ssl and a few other small factors ) but all the big stuff was handled or will be 100% handled after this redesign is complete. But just still seems we're stuck in the google hole again. We do use adwords, but no clear signs as to why we'd suffer such hardships with google ranking, only thing we don't have optimized in terms of on page optimization from moz is keyword in url, of which will be changing within the next month or two, as we're rolling our a new redesign with SEO 100% at the forefront, nice url paths, with keywords in their url, much more responsive site that uses less resources. But before we release this redesign, I'd like to find out what toe we stubbed of google's to give us such a ranking blackeye... We don't have that many backlinks, and I know these are a huge factor, however, building quality backlinks it's harder than walking on water at time and on the same level as spinning hay into gold. Any ideas community...
Competitive Research | | Deacyde0 -
Strange google search - help me mozzers
Hey Mozzers How Are You All? Hope You Are Doing Well. 🙂 I need your help. I have been having a very bad experience with Google. 😞 Do you know what is "Google Dance"? Somewhere I have heard about this. I don't have even clearance about Google Dance. BTW, the problem is that i have an adult video site about a porns tar. I was ranking at 3-6 position on a "particular keyword" on google. But about 20 days ago when i searched my "keyword term", I could not find it anywhere even on page 50 in Google search. 😄 My heartbeat was gone faster. As i was getting 25k visitors from google of that keyword. 🙂 Then i looked at my webmaster tools to check "manual spam action". There was no issue with spam or anything related to it. Then i checked my google analytics i was getting visitors from google but not from the 'particular keyword'. I was sure it was not "google hit or penalized". I could not find the exact reason why did it happen. I checked my sites everything seems to be fine. Then i did "fetch as google" and the strange thing was happened, my site pop up again in google on "that keyword". But this not the end. From that day my search keyword was getting vanish within 24 hours or more in google. And i do google fetch and it again comes to search results. 😄 I tried moz tools. no such issue. Though i cannot sort out the whole matter. What’s wrong going with google. Its an adult site, otherwise I would provide my site url and the keyword. By the way, I have attached some screenshots. I am waiting for your help. Thanks to all. 🙂 gyKIRVW
Competitive Research | | raja290 -
Google Keyword Tool Alternatives
Hey All, So ever since Google shut down the Keyword Tool we have used the keyword planner and a few other tools only to find they aren't quite the same, I was wondering what everyone is using now to get there keyword research and average traffic estimates ? We previous used keyword tool for phrase match and now not much offers phrase match or provides us with what seems as accurate results. We have tried Word Tracker after a numerous amount of research to find a alternative to the keyword tool, and found that the results are very different from the Google Keyword Planner, 1 noticeable result would be of "Web Design Melbourne", in Word Tracker it showed the average search of 9,900 (per month or year I'm not sure) and we compared that to the Google Keyword Planner and it showed 3,000. I have looked over a number of tools and found they weren't quite what we are looking for (Word Tracker, SEMRush, Google Keyword Planner) so I have turned to the MOZ Community. So my first question for everyone would be: A) what do you use for Keyword research and average search estimates now (any good tools with phrase match)? B) If you have used Word Tracker have you found accurate results in your keyword research? If you have found any tools that have proved beneficial and useful please let me know 🙂 (paid or free) Thank you
Competitive Research | | KBB_Digital
Jake Crone0 -
How much time takes Google to Index a page in the search?
Hello all, I have created a page and published it 2 days ago and it does not appear in the Google Search.How much time can usually/on average take to index a page by Google in order to appear in the search results of a web search? I have check the Google Webmaster tools and neither no urls have been reported to be blocked nor the content should be blocked. Thank you very much, Best regards, Antonio
Competitive Research | | aalcocer20030 -
Trying to understand the sporadic and not relevant search results from Google?
Trying to understand the sporadic search results from Google I have been searching for answers to my situation for two
Competitive Research | | rdominey
months now with little results. I have had numerous answers and various
solutions to the issue, and I have followed through with them trying to solve
this problem but the results are the same. Beginning in late April our SERP went down drastically. I
have checked and cleaned the website of any errors and issues that I can find with
no change in the results. I have submitted a request to Google to be
reconsidered and have been told now three times that we are not being manually
penalized. The last request I asked for
any specific reason that we were seeing these very unusual SERP results and got
no answer. The best way that I can explain the issues is with a specific example. These results are similar with other pages throughout the website: www.getyourphotosoncanvas.com/ If you search for a specific page Title you get some really strange results: Search for: Gallery Wrap vs Museum Wrap Photos on Canvas The results vary from day to day but for the most part remain very similar: ·** Black and White Photos on Canvas make Amazing
Wall Art** www.getyourphotosoncanvas.com/.../black-and-white-photos-on-can... Mar 20, 2012 – ... shots to Black and White. What a great way to display your photos on canvas. ... Gallery Wrap vs. Museum Wrap · Before & Afters
That WOW! #53 | PR: ·** Sitemap - Get Your Photos on Canvas** www.getyourphotosoncanvas.com/sitemap/ Use our Sitemap to help locate exactly what you are looking for and navigate directly to that ... FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #54 | PR: ·** Like Get Your Photos on Canvas on Facebook** www.getyourphotosoncanvas.com/like-us-on-facebook/ Like Us on Facebook and keep in touch with Special Offers and information about Get Your **...**FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #56 | PR: ·** [Help Get Your Photos on Canvas](http://www.getyourphotosoncanvas.com/help/)** www.getyourphotosoncanvas.com/help/ Our Help Pages have information about Get Your Photos on Canvas, our Gift Cards, What our ... FREE Enhancements · MORE Enhancements · Gallery Wrap vs. #62 | PR: ·** Free Convert to Black and White for Your Photo on
Canvas Prints.** www.getyourphotosoncanvas.com/free-black-and-white-adjustments/ ... OF CHARGE. We produce the very best quality Black and White Photos on Canvas in the business. ... Wrap vs. Museum Wrap · Before & Afters That WOW! #67 | PR ·** Large Format Photographic Canvas Wall Displays - Get Your Photos ...** www.getyourphotosoncanvas.com/wall-displays/ Photo on Canvas Wall Displays or Wall Groupings can be produced in a wide variety of Layouts including ... Wrap vs. Museum Wrap · Before & Afters That WOW! #73 | PR The target page does not show up in the top 350 search results yet all of the above pages are listed in the top 100. It is important to note that most of these pages do not even contain the keywords allery Wrap vs Museum Wrap or any variation of those keywords. Yet, they appear in the displayed results? If you search just Gallery Wrap vs Museum Wrap our title page appears as #2 on Google? ·** Gallery Wrap vs Museum Wrap Photos on Canvas** www.getyourphotosoncanvas.com/gallery-wraps-vs-museum-wraps/ Gallery Wrap vs Museum Wrap Canvas, two options for your photo on canvas prints. Find out the difference between Gallery Wrap and Museum Wrap. #2 | PR Just as a point of interest, both keyword phrases score A in on page results. I would be grateful for any ideas or assistance on what might be causing these results and of course guidance on what I need to do to correct the problem. Thank You.0 -
Is it valuable for a local business to build links into its Google Place?
G'Day All, Almost all of my clients are geo-based small service-based businesses. I've noticed during my research that the google places for our competitors in 3 separate niches (3 different clients) seem to be the dominating results for almost all relevant keyword terms. I'm curious to see if anyone has actively tried to increase the ranking of a google place by building links into it. Is this something that anyone else sees value to for a local small business? I would love to get some thoughts. And for that matter I'm also curious to see if anyone thinks there might be value to optimizing a Facebook Fan Page or Yelp Business page. They all seem to be key drivers of traffic our client websites so I'm wondering how difficult it is to make them rank as opposed to a website. Thanks!
Competitive Research | | blahblahblah20150 -
Sometime I just don't get Google rankings
We currently rate #10 on google.com.au on Modern Cloth Nappies and the #4 site is a dead link to a page http://www.modernclothnappies.org/ who's total content is: Index of / <address>Apache Server at www.modernclothnappies.org Port 80</address> <address></address> <address>They have been at that rank for a quiet a while and even the cached version is full of broken links.</address> <address></address> <address>It seems Google is quick to jump on low value sites or ones with duplicate content, but what about stale links and sites? Has anyone else had similar experiences of being out ranked by domant or dead sites?</address>
Competitive Research | | oznappies0 -
Can i have chance to rank higher than official website in google local domains ?
Q : can i have chance to rank higher than official website in google local domains ? for example : rank higher than microsoft,kaspersky,nokia etc... in google italy or google germany or any other local domain for google
Competitive Research | | activeacts0