Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Viewing search results for 'When searching in google we find our site in the first position but when some others search it is seen on the second page 1 st position why is this happening?'
Viewing search results for 'When searching in google we find our site in the first position but when some others search it is seen on the second page 1 st position why is this happening?'
Search Behavior | | Alyaauditors0 -
Google Analytics Tagging
Hi. I'm trying to figure out a solution to three questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. In Google Acquisition > Overview > Referral, the clients website is showing up as a referring domain, both the TLD and the subdomain. My understanding is that it should actually be reporting under the "Direct" channel. How do I correct this? Is it just a matter of updating the Direct channel to include those domains? Or do I need to update the settings? The domain's www. http: all 301 redirect to their https://domain.com and https://subdomain.domain.com. Within settings it has been specified as www.domain.com and URL is http:// - also noticed that Bot Filtering has not been checked, assuming this could mess up the analytic data if not define? Do you know? 3. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
Google Index Issue - Indexing pages that don't exhist
Hi All, I have noticed a weird issue when performing a search on Google to show me all the pages it is indexing of our site. site:www.one2create.co.uk It brings up most of our website pages but then is also brings up a few HTTPS urls (our site has not been converted to HTTPS yet) but also the URL path, Title, and Meta Description are from one of our clients websites (an Automotive Job site). When clicked they take you to a generic 404 server error page, not our branded 404 page. The site that it has taken the url, title and meta description from is on a different server completely so I don't see how it has even managed to get that information and linked it to our site? Has anyone seen anything like this before? And what is the best way to fix it? We have asked Google to re-index the site but still no luck.
Search Behavior | | Jvickery0 -
How do I know if I have been hit with a Google penalty
How can you tell if you have been hit by the Panda update as I have a Page rank 3 Homepage, several 100 other pages but no page rank is flowing through to these pages they are all zero? Thanks
Search Behavior | | webguru20140 -
Google Analytics Segments - Can't find location
Hi all, I am trying to set up a Google Analytics segment with the location as 'Kurdistan'. I have tried all location groupings (city, region etc) and am unable to locate it. Anyone else not able to locate it? Or does anyone know why I can't? Thanks,
Search Behavior | | Davinia22
Davinia0 -
Using Google Analytics to See What Time of Day Visitors View My Site
Hi folks, My company has Google Analytics setup for all of our websites, but I am a bit stumped on something. Now, this may not be possible, but am I able to see what time of day visitors most frequently view my blog? I would like to optimize blog post publishing for when I know we have in influx of visitors, yet I cannot find this information on GA. Any input would be much appreciated. Regards,
Search Behavior | | Instabill
Meghan0 -
How to target local searches?
How do I start to target a national audiences? I currently get most of my hits from The US and Australia? But I see no reason why I don't get vies in Canada and The UK.
Search Behavior | | Stefan-Thorpe0