Thoughts on scraping SERPs and APIs
-
I'm perplexed about a couple of things.
Google says that scraping keyword rankings is against their policy from what I've read. Bummer. We comprise a lot of reports and manual finding and entry was a pain. Enter Moz! We still manually check and compare, but it's nice having that tool. I'm confused now though about practices and getting SERPs in an automated way. Here are my questions
- Is it against policy to get SERPs from an automated method? If that is the case, isn't Moz breaking this policy with it's awesome keyword tracker?
- If it's not, and we wanted to grab that kind of data, how would we do it? Right now, Moz's API doesn't offer this data. I thought Raven Tools at one point offered this, but they don't now from what I've read. Are there any APIs out there that we can grab this data and do what we want with it? (let's day build our own dashboard)?
Thanks for any clarification and input!
-
You're much too kind Happy New Year!
-
Hi Rand. Anyone ever tell you you're awesome?
I just did. Thanks bud
-
Hi Boogily,
A) In terms of ToS, I'm not 100% clear. I know that Google has been strict about automated rank checking programs installed on a user's local machine, and they've enforced that a bit through technical and guidelines means. However, they've been unclear about companies like Moz, Searchmetrics, GetStat, AuthorityLabs, Conductor, Brightedge, Yield, RankAbove, and hundreds of others which provide this data in their software. I suspect they don't love it, but tolerate it and haven't taken action for over a decade now.
B) On the API front, check with GetStat and AuthorityLabs. I believe they both have rank data offerings from an API standpoint. Moz doesn't offer that and probably won't in the future, either.
Cheers!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Does anyone know why the MOZ SERP analysis has become so slow that it's not even useful?
Does anyone know why the MOZ SERP analysis has become so slow that it's not even useful?
Moz Pro | | PLEXI3 -
What is the difference between my pages' keyword rank and keyword-specific SERP?
I've been tasked with identifying high volume keywords for products our company sells and optimizing our copy and listings to rank higher on Google. The Keyword Explorer and On-Page Grader have been fantastic for this, but as I write my reports, i'm not sure whether SERP or Moz's keyword page rank would be more accurate for reporting results and movement. For instance, one of our product pages ranks #12 for the highest (relevant) volume keyword, but when looking at the SERP analysis for that keyword, our product page actually shows up at #7 down the list. I've tried naturally searching in Google in case it was an issue of private results based on my user data but it seems to give the same result. Am I conflating two different tools? Or is the Moz rank just an estimate?
Moz Pro | | BJake0 -
API for On Page tool
I'm looking for a tool similar to On Page Grader (Moz) or Focus Keyword (Yoast) with API. We are building out or internal CRM system. Even though none of these tools can replace manual on page analysis, it will be used as a metric and to catch human mistakes.
Moz Pro | | OscarSE0 -
All ranked pages on Googles SERP only links to home
I got a problem regarding my website called musik.dk, and hope you guys are able to help. I just got my first ranking results from Moz. My question is: All my keywords are linking to the home page, and not the artist page? For examples if I were to search on Rihanna on Google, then when musik.dk appears on the SERP, it only links to the home, and not musik.dk/rihanna.. This problem applies to all the given artists ranked on Googles SERP, it never shows the artist page itself, only links to home. Let me know if you need any information, and I gladly supply, this is kind of frustrating to me.. As a note: all pages besides the home has a PA of 1, and it doesn't really seem to change. UT3OBHr kVp7z38
Moz Pro | | Morten_Hjort0 -
What search engines does SEOMoz pull the SERP data from on the keyword analysis report?
I did a Google search and I noticed that some of the rankings that pulled up did not match the SEOmoz top 1-10 competitors. I was sure not to include personal results. I was completely signed out of my Google account. Another one of the SEO tools showed the same result as when I did the Google search, completely different from SEOmoz's report. I am curious to know where the data comes from, so I can accurately compare/ contrast the varying results.
Moz Pro | | JMFieldMarketing0 -
API Limit
I have some problems with API Access. All time i send more than 10 requests per second i got HttpError() {\n "status" : "503",\n "error_message" : "This request exceeds the limit allowed by your current plan. To increase your request limit, see: http://www.seomoz.org/api/pricing"\n}. My Plan is Low Volume and continually got this 503 error. How to solve this?
Moz Pro | | Predicta0 -
Rankings Different in Serps from Report
My site shows up on page one for many search queries, but I'm "not in the top 50" for those same terms according to the rankings report. Having troubleshooted the problem, it may be a canonical issue. I did not include www when I originally set up my reports and it seems that my site shows up in search results with www. However; if you type in the url without the www, it will automatically forward to the page with www. I did this intentionally in order not to get penalized for duplicate content pages. So here's my question: If I've got one page redirected to the other, shouldn't the rankings report also follow that redirect? If so, then it should show the correct results, right? Thanks, Dino
Moz Pro | | Dino640 -
Why can't we access metrics in the API that are available in the mozbar?
For example, the Chrome/Firefox mozbar uses peid (Root Domain External Links) but the api doesn't allow this. I've done a little bit of reverse engeniering and it looks like the accessID for the mozbar (when logged in as a pro member) is "pro-xxxx". Does this mean that each pro account has its own accessid/secret key? If so, when will pro members have access to that? I've created a tool that I use for deciding which expired domains to buy and it uses all of the free metrics - but I find myself having to do one extra step to get the peid (navigating to ose). It doesn't make much sense that the metric is available to me when I'm using the mozbar logged in as a pro member (which I pay for), but not apart of the API.
Moz Pro | | SeanStewart811