Thoughts on scraping SERPs and APIs
-
I'm perplexed about a couple of things.
Google says that scraping keyword rankings is against their policy from what I've read. Bummer. We comprise a lot of reports and manual finding and entry was a pain. Enter Moz! We still manually check and compare, but it's nice having that tool. I'm confused now though about practices and getting SERPs in an automated way. Here are my questions
- Is it against policy to get SERPs from an automated method? If that is the case, isn't Moz breaking this policy with it's awesome keyword tracker?
- If it's not, and we wanted to grab that kind of data, how would we do it? Right now, Moz's API doesn't offer this data. I thought Raven Tools at one point offered this, but they don't now from what I've read. Are there any APIs out there that we can grab this data and do what we want with it? (let's day build our own dashboard)?
Thanks for any clarification and input!
-
You're much too kind
Happy New Year!
-
Hi Rand. Anyone ever tell you you're awesome?
I just did. Thanks bud
-
Hi Boogily,
A) In terms of ToS, I'm not 100% clear. I know that Google has been strict about automated rank checking programs installed on a user's local machine, and they've enforced that a bit through technical and guidelines means. However, they've been unclear about companies like Moz, Searchmetrics, GetStat, AuthorityLabs, Conductor, Brightedge, Yield, RankAbove, and hundreds of others which provide this data in their software. I suspect they don't love it, but tolerate it and haven't taken action for over a decade now.
B) On the API front, check with GetStat and AuthorityLabs. I believe they both have rank data offerings from an API standpoint. Moz doesn't offer that and probably won't in the future, either.
Cheers!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are the metrics in Moz's SERP analysis relevant in my area?
Our site (https://loco2.com/) specialises in train journey planning at ticket booking, so we want to improve our ranking when people search Google for a specific journey they want to make (eg "London to Barcelona by train"). We have a set of about 400 Guide pages (eg London to Barcelona) which we want people to find when they search for those journeys. I've been trying to find out why some of those pages do better than others in search, and what our competitors are doing better. I was hoping that the SERP analysis and Keyword Reports tool could help me with this, so I ran analyses for a set of the keyword phrases we want to rank for, but I was surprised to find that most of the metrics in both the basic and full reports don't seem to relate to how high in the search result a page ranks. The screengrab attached shows the results for "Brussels to Paris train", which are fairly typical - many of the top-ranking pages have fairly low scores for most metrics, and the metrics don't seem to have much effect on how high they are in the search results. ( I assume N/A means the Page Authority is very low, or is it not calculated for some other reason?) I looked into this a bit further by downloading full reports for 11 keyword searches, and comparing the average scores for each metric with the rank, but I still couldn't see any effects of the metrics on the ranks. If anything, pages with higher Page Authority and Domain Authority were further down in search rankings. Does this mean that link metrics aren't very important for the kinds of pages we're looking at? This makes sense to me, because if I'm searching for tickets from London to Barcelona I want to find a very specific results page which is unlikely to have many links, but I keep reading that link metrics are one of the most important factors in SEO. Am I misunderstanding something? wJ9uUcG
Moz Pro | | MargotLoco20 -
I cant generate my Mozscape API
I have tried to generate my Mozscape API numerous times after waiting at least 20 mins between. Any help would be great.
Moz Pro | | MLMLEADZONE1 -
Difference between SEOmoz Pro and Mozscape API
Hi, I probably put this question in a wrong category, so I'd like to ask again. I am new in SEOmoz and writing this from Japan.
Moz Pro | | flaminGoGo
I have couple of questions and really appriciate if you can help me to understand SEOmoz. better. First of all, what is the difference between SEOmoz Pro and Mozscape API? I signed up for 30days free trial and now continue being $99 PROmember.
Is that correct that rate limit is one request every 2 seconds with PRO? and also want to know about free API that I found after signed up for PRO.
Is there any restrictions on free API except for rate limit is one request every 10 seconds? The main reason I signed up for SEOmoz was to use the API link metrics, and because of the language I got really confused now.0 -
SERP Changes depending on ISP
If I search Google for my site using my mobile internet provider I get different results from my main internet. Before anyone suggests it, it's nothing to do with saved searches as I'm clearing cookies and I've tried on multiple different platforms. I can plug my iPad and use my wireless, it's my normal rankings, but as soon as I switch to my mobile provider for internet, I'm 3rd instead of 2nd on the SERPs. Any ideas?
Moz Pro | | awdgareth0 -
Why my seomoz keyword ranking are different to google SERPs
I use Cute Rank to check my keywords ranking in Google, but after last update seomoz is showing some of keyword ranked much higher than Cute Rank. I checked them manually and they are on positions showed by Cute Rank and not by Seomoz. Can someone explain me why is that?
Moz Pro | | misterdam0 -
Top 10 SERP Competition PA
I really like the Keyword Difficulty Tool. I'm looking to get into some affiliate marketing and I'm trying to assess a good niche site to create. I have a couple questions about assessing the results of this tool: 1. When looking to compete for top 10 SERPs are Page Authority & Domain Authority closely related? because a page could have a 1 PA and an 80 DA. 2. What individual PA ranking is considered to be surpass-able? below 40? 30? 3. What difficulty percent would you consider worth competing for? or a waste of time? I guess I'm looking for some breaking points here. Thanks!
Moz Pro | | AndySolo0 -
Report: keyword difficulty verses SERPS ranking
Hi there, I've been using the PRO tools for a few months now, and find them great for revealing information for my clients, and planning our SEO approach. One report I've put together has been really helpful for a client, but I built it manually and it took ages. It lists, in a spreadsheet, keywords side-by-side with keyword difficulty, target URL, and SERPs result. Sorting by keyword difficulty or SERPS position has helped us to prioritise which pages to hit first. But it took a while to generate the report, and I'm wondering if anyone has a better approach, or if this kind of report can be generated automatically: http://www.cicada-online.com/app/images/documents/product-page-SEO-blank.pdf Thanks Mozzers!
Moz Pro | | nedwells0