Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Capital - Antitrust Conspiracy
I think we all have heard about Thumbtack breaking the rules w/ badges. Getting deindexed, then getting a 100M injection from Google capital and having the penalties removed: https://techcrunch.com/2014/08/20/service-marketplace-thumbtack-raises-100m-round-led-by-google-capital/ Our primary competitor is a different marketplace backed by Google Capital. Does anyone know of any low frequency products (reliant on SEO) backed Google Capital that has not won out within search? (i.e. is there any hope of competing against a low frequency marketplace after they have Google Capital backing?)
Search Behavior | | MarketGrowth0 -
My website disappears off google!
So this might be kinda of a weird question... Every morning and night I check the ranking of a website that I am building.. The ranking has gone up a lot the last two months. It went from the fifth page to now the second page. I have a issue where some days I check Google my website is completely gone! I go through every page for my keyword and it's not there! After a couple of days of frustration I check again and all of a sudden it is there but now at a higher ranking... I went through the code to make sure there's a not a not follow code in the robots.txt page... Btw another weird thing is so then I look up my website on a google out of country like google.sg and I'm ranking first page like number 5 but again disappeared off google usa. Literally driving my crazy.. does anyone know why this could be? Btw the first time it disappeared I went into webmasters and sent a request because I thought I got penalized but they responded they could not find any spam and I was NOT penalized...
Search Behavior | | BecCan0 -
Getting People to Leave Product Reviews, When they've reached our site in search of reviews
We have a product review website that has been showing a steady increase in organic traffic. However we seem to be having a disconnect between what happens when a visitor arrives. We essentially want a visitor coming to our website in search of a review, to LEAVE a review for another product they may have used... Any suggestions on ways we might encourage this? We are kind of newbies at this!
Search Behavior | | Sarah_Frantz0 -
Noticed Bing UK and Yahoo UK are almost exactly the same ranking results?
I've run a few keyword ranking updates today and I've noticed that Bing UK and Yahoo UK results are almost exactly the same? This is unusual, the results for these two engines rarely tally with one another. Is anyone seeing the same?
Search Behavior | | MiroAsh0 -
From #2 TO Not Found in top 100 Results, WTF?
So my website was #2 for a targeted keyword on google. (6 months ago: # 😎 (3 months ago: #6) (1 week ago: #2). As you can see, it was moving up nicely, now it's not found on the top 100 search results. Only appears #2/(B) in the local search section. Could that be the reason? Websites that appear on the local search no longer appears in the SERP?
Search Behavior | | Droidman860 -
Dating Blog Posts & How Fast Google Picks up on New Pages
I had until a few months ago included the original post date of a new blog post on the site. I then removed it and none of my results in Google now include the blog post date, although for some (for articles written about events) Google includes the date of the event where you would usually see the post date. Since I did this, it seems like new blog posts are taking longer to rank on Google, some results are ranking well, and others declined relative to what I would have previously expected. What's the best thing to be doing? To include a date (considering a lot of my content is not time-relevant) or to keep it as it is now? The second thing, is I often go through and update my articles with new information and re-post it in my rss feed etc - ie the date becomes new again. How does Google treat this? Any ideas or comments would be great! Thanks
Search Behavior | | ben10001 -
Pro/Cons of JUST using Google Analytics
I find myself in a situation where I am having to defend why purely relying on just Google Analytics for site data i.e. exit pages, keywords, bounce rate etc... is not good practise. So I thought/wondered if other people have come up against this. What are the Pro and Cons for ONLY using Google Analytics for your site data....
Search Behavior | | JohnW-UK0 -
Has anyone yet seen any Search or User benefits from implementing any Schema tags (as in Schema.org) ? Thanks
Schema tags have been around for a while, am really interested in knowing of any noticeable benefits seen by anyone who has had experience of implementing Schema tags.
Search Behavior | | SimonCullum0