Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Exporting Google and Bing Search Results
-
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages.
Cheers
-
There is a chrome extension called "scrape similar" that is useful for doing small batches of stuff like this. However it does have a couple of limitations in that you have to view each page & google will not show you all pages of a large domain. However it is quite easy and effective for sites with under 1000 pages.
https://chrome.google.com/webstore/detail/mbigbapnjcgaffohmbkdlecaccepngjd
The process can be sped up using other tools. I use tool that is designed for black hat forum/comment spamming to do SERP scrapes like that. Even if I did such spamming (I don't), I don't actually think this is a very good tool to do it with. However it is rather good at scraping results from google. However, again you are limited to how many results Google/Bing choose to show you.
If you need a bigger list then log files might be the way to do. You can get a list of all crawled URLs for any particular agent (including the likes of googlebot) from your server logs. Some hosts limit the size of these, so it might be worth checking before you start. However the data does get collected. The downside here of course is that you need access to the logs.
Of course crawled is not the same as indexed. Once you have that list you might need a further step to see which is indexed. Possibly cross-referencing it against google analytics landing pages or querying the google cache for that page (SEOtools for Excel from Biels Bosma is good for this).
Similarly, if you have a definitive list of the URLs on site you could start with that list and query which are cached.
Harder than it seems isn't it? Hopefully one of those methods will put you on the right track.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Can Dollarpesa Website Achieve a High Google or Bing Ranking with a Domain Authority of DA 50+ on Moz?
Can Dollarpesa Website Achieve a High Google or Bing Ranking with a Domain Authority of DA 50+ on Moz? How to Attain Domain Authority of DA 50+ on Moz? everyone please help me
Moz Pro | | dollarpesa0 -
Domain still not being found in search
Hi guys, I've been handed a client who needs some seo work. I've tweaked one of their pages to focus on a chosen keywords about 4 months back but still the site is not even visible using the new Domain Analysis tool from moz and it still won't rank at all for the keywords. Am I missing something here? Is there something blocking the SERP from listing the website? I've ran a site: search on Google and it returns 283 results on the website. It's puzzling me as there clearly is something stopping it from being ranked. The domain name in question is: https://cloud9inecommunications.co.uk Thanks in advance.
Moz Pro | | Easigrass1 -
Competitor getting External Links from search.aol.com
Recently, I noticed that one of the competitors I track within my Moz campaign received about 12 new inbound links. As a result, there DA jumped about 10 points. I reviewed these new external links and was surprised to see that they are all "search.aol.com/aol/search?query= ..." with Link Anchor Text that is good for the industry we compete in. Can anyone tell me why these are being counted as "Inbound Links". It just doesn't seem right. Is this some sort of black hat seo tactic?
Moz Pro | | itvisionsinc0 -
GWMT / Search Analytics VS OpenSiteExplorer
Just had the experience of using OSE data to show what we call "linkrot" to a client -- only to find that GWMT / Search Analytics shows no such thing. Fortunately the client is an old friend and no face was lost, but it was dicey there for a bit as I have come to rely on and reference OSE again and again and again, OSE showed Domain Authority dropping by about 1/3 in the last 12 months, presumably due to old links getting broken, linking sites changing their architecture etc. And of course, ranking is tanking, as you would expect. But Google shows many more (and much more spammy looking!) backlinks. Has anyone had any experience benchmarking the 2 data sets of backlinks against each other? Dr Pete?
Moz Pro | | seo_plus
Does one update more frequently than another? Do you trust one more than another?? If so, why?? Thanks!0 -
Screaming frog, Xenu, Moz giving wrong results
Hello guys and gals, This is a very odd one, I've a client's website and most of the crawlers I'm using are giving me weird/ wrong results. For now lets focus on screaming frog, when I crawl the site it will list e.g. meta titles as missing (not all of them though), however going into the site the title is not missing, and Google seems to be indexing the site fine. The robots.txt are not affecting the site (I've also tried changing the user agent). The other odd thing is SF gives a 200 code but as a status tells me "connection refused" even though it's giving me data. I'm unable to share the clients site, has any one else seen this very odd issue? And solutions for it? Many thanks in advanced for any help,
Moz Pro | | GPainter0 -
SEO Yoast data export
Just thought I would give something back. (Is this the right place!) I use Wordpress with the excellent SEO Yoast plugin. I needed a way of extracting the focus keywords that I have entered onto my pages along with the url for use on the SEOmoz On-page Optimisation tool. So I created GetYoastData which outputs to the browser the required data (and a bit more) that can be saved into an csv (Excel) file. Hope you find it useful - Yes it's not polished and yes it might output a blank line now and again but it's fairly useful. http://deanandrews.uk/get-yoast-seo-data/
Moz Pro | | DeanAndrews0 -
What Is he Difference Between Google UK and Google Great Britain
When setting up a campaign for keywords and selected search engines. I am targeting a UK market so obviously Google.co.uk is my main target. However when I set up the campaign i select Google as the SE but it asks me if I want United Kingdom or Great Britain. I selected United Kingdom and it has been returning 0 results for rankings even though I can physically see them in the SERPS. Does anyone know exactly what is the difference between the UK and GB options. I assume I should of specified GB but not sure. Any help would be appreciated. Thanks
Moz Pro | | hanv0 -
Google SERP count 100
I'm not finding a clear solution to getting 100 results in the Google SERP. I'm currently using Firefox as I use Chrome for being logged into Google (Gmail, Docs, etc.) I tried the Grease Monkey script, no luck. I tried appending the url with &num=100 (looks like that may have been killed in 2010 which may explain why the GM script isn't working). And it doesn't look like any toolbar, SEOMoz included has that functionality, any ideas or places to look? In addition, does anyone have a good SERP numbering tool? If I do a search and the first page, including Local & Organic has 15 results, when I click over to the 2nd page it starts back at #11, kind of defeats the purpose of using a numbering tool 😉 Any help would be appreciated. Thanks much!
Moz Pro | | nsauser0