Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Index Issue - Indexing pages that don't exhist
Hi All, I have noticed a weird issue when performing a search on Google to show me all the pages it is indexing of our site. site:www.one2create.co.uk It brings up most of our website pages but then is also brings up a few HTTPS urls (our site has not been converted to HTTPS yet) but also the URL path, Title, and Meta Description are from one of our clients websites (an Automotive Job site). When clicked they take you to a generic 404 server error page, not our branded 404 page. The site that it has taken the url, title and meta description from is on a different server completely so I don't see how it has even managed to get that information and linked it to our site? Has anyone seen anything like this before? And what is the best way to fix it? We have asked Google to re-index the site but still no luck.
Search Behavior | | Jvickery0 -
What is the best way to learn google analytics?
Hi I'm trying to get my head around google analytics as a first time user. Can any suggest the best place to start after I have taken a look through the whole thing, how do I go about managing and tracking performances on my website? eg: Registrations/sign ups, conversions all that sort of thing Some advice will be much appreciated Thanks
Search Behavior | | edward-may0 -
Store locator searches online and offline store visits
Hi All, What is the accepted % for visitors that conduct a store locator search online actually visiting that particular store for the nutrition industry? (For example if 50 people used the store locator search on the website , we can confidently say that 25 of them actually went to that store) Any input or reference to studies/white papers on this topic would be appreciated. Cheers, SEO5..
Search Behavior | | SEO5Team0 -
Why do all traffic curves show a "saw tooth" pattern in Google Analytics
Greetings Mozzers This sounds like a dumb question, but it's bothering me just the same and I would like to think what the fine community of SEO experts at MOZ thinks I have taken a look at the traffic curve for about 20 different sites in very different industries (home improvement, wedding party supplies, pet feeding systems) and it seems to me that they all show a saw tooth pattern, more or less like the one below.
Search Behavior | | Masoko-T0 -
Looking for something better than Googles in page analytics
Buongiorno from 12 degrees C wetherby UK 🙂 I'm interested in measuring content engagagement, things like what gets clicked on mouse tracking beahaviour etc. Google Analytics in page analytics does a good job but is there another product able to guage user interactive behaviour with content (Yes ive added event tracking). Thanks in advance,
Search Behavior | | Nightwing
David1 -
Huge difference in search results depending on the country?
I have some keywords ranking in pages 2 and 3 in the US but for instance, in Canada, the same keywords rank somewhere between pages 15 and 25. When I check my US rankings, I use a Windows VPS located in the US. When I check my rankings in Canada, I use a computer physically located in Canada. Basically, about 80% of my rankings have gone away for countries other than the US. I have not set any geographical target for my website in GWT. How come there is such a big difference in search results across different countries?
Search Behavior | | sbrault740 -
Google Is Displaying An Alt image tag as my homepage's page title
Google is randomly displaying the alt image tag as the page title for my homepage. It happens when you search for the brand name, but the page title appears as "BrandName Logo" (obviously not "BrandName"). Has anybody seen this happen before?
Search Behavior | | MichaelWeisbaum0 -
Google Location - Taking Away Our National Reach?
Hey, I was just noticing that we achieve #2 ranking on Google for one of our customers for one of their primary keyword phrases. But then I noticed the traffic analytics were not matching what we should expect from that keyword phrase. Then I noticed, in using "Chrome's Incognito Window", that our location was automatically selected for our main geographical city area. I then went and changed that location from Denver, to San Diego & Also Chicago, just to see what would happen, and I noticed we instantly dropped from #2 to #7 when changing our location. I don't know what my question is, but I guess I feel like that is preventing us from achieving the results we need to sell ecommerce products. Is there any info on this or suggestions anyone has on how to tackle this issue? It feels like Google is pulling the rug out from underneath our feet and trying to spread rankings more to localized areas, rather than offering someone the opportunity to capitalize on good rankings for a national audience. I understand why they would do it, and I don't say I disagree. But it just seems to affect our work as SEO's doesn't it? Since we can't be as effective for customers that have a global audience instead of strictly a localized one. I'm curious to see what people have to say about this issue. Thanks!
Search Behavior | | JerDoggMckoy0