Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google not giving ranking to the intended page of my website.
Hello friends, This is my very first question, I hope I will explain my issue correctly. I have created two pages related to SSC CGL keyword on my website: https://www.ibtindia.com/ssc-cgl-notification-exam-date 2) https://www.ibtindia.com/ssc-cgl-apply-online I want to target the keyword SSC CGL Apply Online on the 2nd URL but Google is only considering the 1st page for all the "apply" related keywords as well. Can anyone suggest to me how to get the second page in rankings for its intended keywords? I will be very thankful for this help. I tried adding image but its showing dummy. Please you can check it op GOOGLE SERP Second page
Search Behavior | | namitathakur0 -
What would the US traffic increase be for a website YoY if all Google SERP rankings remained the same?
This question has come up a few times with some of our clients and I've spent some time researching this question, but I can't find an answer online so hopefully, someone at MOZ has this data available to them with all the data they collect. The data points that would be needed to answer this question off the top of my head: Increase in the # of Google Searches in the US YoY The decrease in CTR for organic results "10 blue links" which take a searcher off of Google YoY, as Google continues to keep more searchers on Google.com with rich snippets, increased AdWords prominence, AdWords extensions, etc I'm sure this greatly varies per industry, but an average for all industries is all that is needed to answer this client question. Many thanks in advance and I've included a video which hopefully helps to better explain the search "plus/minus" that we can expect to see as SEOs in 2018. WF1yLlJC6LetnpbD3
Search Behavior | | WebpageFX1 -
Google Analytics Tagging
Hi. I'm trying to figure out a solution to three questions one of my current clients has asked me in regards to Google Analytics tagging, and I'm unsure how to respond. Can anyone help? See below the questions, 1. In Google Acquisition > Overview, their paid media is reporting as "Other". They do not run any Google paid ads. They only run Facebook paid ads. Is there a way to update the source so that it says "Paid" versus "Other" within the default channel? The current solution was advised to create a channel group that the client has to then tick on overtime they want to see this data with the correct labeling. They would prefer to see it in the default. Is it just a matter of going into the *default channel, choosing the "Paid" option, and then specifying the source/medium that contains Facebook, CPC, or referral to be categorized under this channel? Or is it something else? *Aware that changes to the Default Channel are permanent changes and will change how new traffic is classified. 2. In Google Acquisition > Overview > Referral, the clients website is showing up as a referring domain, both the TLD and the subdomain. My understanding is that it should actually be reporting under the "Direct" channel. How do I correct this? Is it just a matter of updating the Direct channel to include those domains? Or do I need to update the settings? The domain's www. http: all 301 redirect to their https://domain.com and https://subdomain.domain.com. Within settings it has been specified as www.domain.com and URL is http:// - also noticed that Bot Filtering has not been checked, assuming this could mess up the analytic data if not define? Do you know? 3. Audience segmentation > The client wants to be able to define it's audience by shopping intent and informational intent. Is there a clear way to do this, for example, by keywords used, e.g. buy, product name, entry (shopping intent), versus e.g. non-purchase intent, entry to the blog, length of time on site (info intent). Would be happy to have a conversation about the last question, since I'm conscious that there are probably multiple ways to define this - thanks. To the group, thank you for readying my questions and helping me with these solutions - your time is appreciated and valued. Sincerely, Amanda
Search Behavior | | AmandaValle.Digital0 -
Google keyword planner shows low search volumes
Hello, There are some terms that you'd expect to see a lot of results. In fact, in the old days of Keyword Tool, I recall seeing thousands of results for a keyword like "anderson cooper". Today, I see a small amount as a monthly average. Am I doing something wrong? All I really want to do is find search volumes on particular keywords either globally or local to a country. And it's proving soo hard! 😞 Screen_Shot_2013_11_26_at_11_13_20_PM.png
Search Behavior | | mhamilton0 -
Industry Benchmarks for Google Analytics
Hey Mozites! I was wondering if anyone has come across any good resources for industry benchmarks relating to google analytics? (Social stats would be great as well, but Google Analytics is my main concern) Specifically, I have a client that is a bank and I would love to be able to compare their stats (traffic, unique visitors, bounce rate, time on site, conversion rates...) to other regional banks. Any ideas? Thanks for reading and for your help!
Search Behavior | | iBecC0 -
Google Analytics Search Engine Optimisation Report
Hello, Quick question. How much data should be available within Google Analytics within the Search Engine Optimisation Reports? I was always of the impression it was 6 months, however the data available as of this date only extends back to January 1st 2013. Thanks,
Search Behavior | | HelloAlba0 -
How to best figure out synonymous search phrases
I'm curious about synonymous key phrases and competitive key term analysis. I have a few methods for conducting research to find what phrases Google considers synonymous but they are very time consuming. What are some ways to figure this out? An example would be livestock and cattle. Are there any good ways to figure out user intent related key phrases? For example in our niche if someone is looking to purchase something Google often delivers pages with title tags that include the term supplies. Would it be advantageous to optimize for these types of key phrases if the competition is less or should I assume that Google groups exact match and synonym and intent match together when calculating relevancy. Eg I can choose to target livestock or cattle on page. Livestock gets a quarter the search volume cattle gets but has half the competition. Will I be competing against the cattle sites too? If so should I just bite the bullet and target cattle?
Search Behavior | | Punkaj0 -
Keyword Search Historical data?
Is there a way to track the history of keyword search over time. I.E. the frequency of the Google search of the phrase 'unpasteurized milk' from 2006-2012 to see trending patterns.
Search Behavior | | preventionaid0