Thoughts on scraping SERPs and APIs
-
I'm perplexed about a couple of things.
Google says that scraping keyword rankings is against their policy from what I've read. Bummer. We comprise a lot of reports and manual finding and entry was a pain. Enter Moz! We still manually check and compare, but it's nice having that tool. I'm confused now though about practices and getting SERPs in an automated way. Here are my questions
- Is it against policy to get SERPs from an automated method? If that is the case, isn't Moz breaking this policy with it's awesome keyword tracker?
- If it's not, and we wanted to grab that kind of data, how would we do it? Right now, Moz's API doesn't offer this data. I thought Raven Tools at one point offered this, but they don't now from what I've read. Are there any APIs out there that we can grab this data and do what we want with it? (let's day build our own dashboard)?
Thanks for any clarification and input!
-
You're much too kind Happy New Year!
-
Hi Rand. Anyone ever tell you you're awesome?
I just did. Thanks bud
-
Hi Boogily,
A) In terms of ToS, I'm not 100% clear. I know that Google has been strict about automated rank checking programs installed on a user's local machine, and they've enforced that a bit through technical and guidelines means. However, they've been unclear about companies like Moz, Searchmetrics, GetStat, AuthorityLabs, Conductor, Brightedge, Yield, RankAbove, and hundreds of others which provide this data in their software. I suspect they don't love it, but tolerate it and haven't taken action for over a decade now.
B) On the API front, check with GetStat and AuthorityLabs. I believe they both have rank data offerings from an API standpoint. Moz doesn't offer that and probably won't in the future, either.
Cheers!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why isnt my site in top 10 SERPs?
Our website, https://www.tradewindfinance.com, hasnt improved ranking for keyword " international trade finance companies." we average around the 10th page of Google SERPs. Our DA, PA, and links are higher thank the top 10 competitors. What are we doing wrong or what can I do to increase rank?
Moz Pro | | solution.advisor0 -
Are the metrics in Moz's SERP analysis relevant in my area?
Our site (https://loco2.com/) specialises in train journey planning at ticket booking, so we want to improve our ranking when people search Google for a specific journey they want to make (eg "London to Barcelona by train"). We have a set of about 400 Guide pages (eg London to Barcelona) which we want people to find when they search for those journeys. I've been trying to find out why some of those pages do better than others in search, and what our competitors are doing better. I was hoping that the SERP analysis and Keyword Reports tool could help me with this, so I ran analyses for a set of the keyword phrases we want to rank for, but I was surprised to find that most of the metrics in both the basic and full reports don't seem to relate to how high in the search result a page ranks. The screengrab attached shows the results for "Brussels to Paris train", which are fairly typical - many of the top-ranking pages have fairly low scores for most metrics, and the metrics don't seem to have much effect on how high they are in the search results. ( I assume N/A means the Page Authority is very low, or is it not calculated for some other reason?) I looked into this a bit further by downloading full reports for 11 keyword searches, and comparing the average scores for each metric with the rank, but I still couldn't see any effects of the metrics on the ranks. If anything, pages with higher Page Authority and Domain Authority were further down in search rankings. Does this mean that link metrics aren't very important for the kinds of pages we're looking at? This makes sense to me, because if I'm searching for tickets from London to Barcelona I want to find a very specific results page which is unlikely to have many links, but I keep reading that link metrics are one of the most important factors in SEO. Am I misunderstanding something? wJ9uUcG
Moz Pro | | MargotLoco20 -
Does Live Chat affect SERPS
For the last week we have hosted Live Chat on www.freedomltd.com desktop and tablet. The latest MOZ ranking reports shows 4 increase, 86 decrease and 170 unchanged keywords. Could Live Chat have anything to do with this?
Moz Pro | | henandstag0 -
All ranked pages on Googles SERP only links to home
I got a problem regarding my website called musik.dk, and hope you guys are able to help. I just got my first ranking results from Moz. My question is: All my keywords are linking to the home page, and not the artist page? For examples if I were to search on Rihanna on Google, then when musik.dk appears on the SERP, it only links to the home, and not musik.dk/rihanna.. This problem applies to all the given artists ranked on Googles SERP, it never shows the artist page itself, only links to home. Let me know if you need any information, and I gladly supply, this is kind of frustrating to me.. As a note: all pages besides the home has a PA of 1, and it doesn't really seem to change. UT3OBHr kVp7z38
Moz Pro | | Morten_Hjort0 -
I'm shocked! KEYWORD SERPs: GA avg. position vs. SEOmoz
SEOmoz has us our position for a particular keyword as "Not in Top 50". The same keyword is listed in GA with an average position of. 3.0. What am I missing here? When I do a search, I most definitely am not seeing us listed as #3 for this keyword, and tend to believe the SEOmoz number, but why is it so wildly different in GA? Should I have hope? There is another keyword that SEOmoz has us at #4 and it isn't even on the GA list. Can anyone explain this to me? Many thanks.
Moz Pro | | gfiedel0 -
SERP Changes depending on ISP
If I search Google for my site using my mobile internet provider I get different results from my main internet. Before anyone suggests it, it's nothing to do with saved searches as I'm clearing cookies and I've tried on multiple different platforms. I can plug my iPad and use my wireless, it's my normal rankings, but as soon as I switch to my mobile provider for internet, I'm 3rd instead of 2nd on the SERPs. Any ideas?
Moz Pro | | awdgareth0 -
Related Backlinks by Keyword SERP results
Is there no tool to search a keyword and find related backlinks among the top10,20,30 SERP results?
Moz Pro | | FirePowered0 -
Which Google API do you use to creat a rank checking tool?
I can't seem to work this one out. Basically I want to make a very simple rank checker tool something similar to http://www.mikes-marketing-tools.com/ranking-reports/ where a user simply type in a domain and a keyword and in return is shown the position within the results. The problem is I can’t work out which API's I need in order to build it, does anyone know what I need to poke about with? It must work with Google and ideally with Bing and the rest
Moz Pro | | therealmarkhall0