Export Google Search Results
-
I would like to export a list of pages indexed by Google and other search engines. According to our site search with google, we have over 300,000 pages indexed. Anyway to export a list of all of these pages?
Tools like ScreamingFrog (which I own a copy of) can crawl your site but does not tell you the pages indexed by a search engine... I've tried using tools that will let you export each page of the results. However, this won't work for a 300K page website.
Thanks for your help!
-
Hi Shani! Do these responses help to answer your question or are you looking for more information? If you're good to go, please mark this as answered. Thanks!
-
Thank you Thomas and Logan! Thomas, GWT puts a cap at 999 pages, but it is also the search terms used to get to our site as opposed to the pages that are indexed. Logan, it appears your answer is accurate as I have been researching this for awhile with no luck.
-
There's no tool that will handle it in that kind of volume. I know of one tool called SERP Scraper that will do 100 URLs, but that's no good when you're evaluating 300k. I'm fairly certain Google makes it impossible for anyone to build this tool, as useful as it would be, it would exist already if this weren't the case.
-
I'm not aware of any tools that would be able to achieve what you need. However, would GWT not be able to help you here?
Search Analytics section would allow you to download all the pages that are indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Massive unexplained organic traffic drop; disappeared from Google
Hi there,
Search Behavior | | katelynroberts
Our site has experienced a huge organic traffic drop, specifically from Google. The drop occurred on Feb 19 and I've got no clue why it happened. We have not made any significant changes to the website and it doesn't look like there was an algorithm update last week. We don't have any Google penalties or indexing issues noted, and the drop isn't specific to any particular segment/region/keyword. What am I missing? Any advice or insight is super duper appreciated. Our site is a Wordpress/WooCommerice e-commerce site with a blog and long-standing #1 ranks for keywords related to our main product offering. Screen Shot 2024-02-26 at 3.12.25 PM.png
Screen Shot 2024-02-26 at 3.07.52 PM.png0 -
How does Google treat significant content changes to web pages and how should I flag them as such?
I have several pages (~30) that I have plans to overhaul. The URLs will be identical and the theme of the content will be the same (still talking about the same widgets, using the same language) but I will be adding a lot more useful information for users, specifically including things that I think will help with my fairly high bounce rate on these pages. I believe the changes will be significant enough for Google to notice, I was wondering if it goes "this is basically a new page now, I will treat it as such and rank accordingly" or does it go "well this content was rubbish last time I checked so it is probably still not great". My second question is, is there a way I can get Google to specifically crawl a page it already knows about with fresh eyes? I know in the Search Console I can ask Google to index new pages, and I've experimented with if I can ask it to crawl a page I know Google knows (it allows me to) but I couldn't see any evidence of it doing anything with that index. Some background The reason I'm doing this is because I noticed when these pages first ranked, they did very well (almost all first / second page for the terms I wanted). After about two weeks I've noticed them sliding down. It doesn't look like the competition is getting any better so my running theory is they ranked well to begin with because they are well linked internally and the content is good/relevant and one of the main things negatively impacting me (that google couldn't know at the time) is bounce rate.
Search Behavior | | tosbourn0 -
Setting up Google Analytics
Hello, I will try to explain what I want and I am not sure if that is possible. Firstly I want to set up my Google Analytics that I would know where people come from and how many of them click on Google Ads. For example:
Search Behavior | | FCRMediaLietuva
Facebook > 2% of them click on ads.
Google > 3% of them click on ads.
Direct > 6% of them click on ads.Or whatever the results are. Secondly I would like to take people coming from facebook and divide them into age groups. For example: 100 ad clicks from facebook.
25-34: 9 clicks (9% of total facebook clicks, 2% of total ad clicks)
18-24: 7 clicks (.....
35-44: ... and so on Thirdly I would like to know which group use the adblocker the most.
Any other stuff that would be interesting if working with adsense? I believe that this would help me working with my target audience the best on facebook. Should this be set as goals? Sorry I am not an expert of Google Analytics 🙂 Thanks for the help.0 -
Google PageRank and google keyword ranks
Hi, I manage around 13 Clients for SEO and this last two weeks i have been having problems with google keyword ranking some of my clients went down almost 40 positions for some keywords other 15 other stayed the same and some just went down 1 or 5 positions. Somebody know what is happening with google? is it shuffling around positions of people? Any advice or how you do your SEO I'm willing to work together and share experiences and tactics on SEO to people that also share with me in order to build more strategies on this amazing market. Thanks.
Search Behavior | | computernc0 -
I need Help with Google!!!!
I am trying to have my picture on the first page just like SEOmoz when someone search just the name, I know have something to do with google plus, but I am so new doing that no luck or probably I am doing wrong, I have been looking in the internet but I haven't found anything. Is there anyone who can write a tutorial and post here. Maybe is already done and I don't know where. Please see the picture attached so you understand better what I want to do Again I want appear just like this but with my company http://www.sombras.co.uk/images/pic.jpg pic.jpg
Search Behavior | | teksyte0 -
Recovering from a Hack: How long until Google reindexes changes?
In a previous post I made, I was able to determine that one of my sites; http://pokeronamac.com/ was hacked and was feeding spam perscription drug content to search engines, then redirecting to another site when clicked on Google. I then contacted my web host, and, after they did a scan of our files, they determined that something within the wp-includes directory was compromised and malicious. They removed the file, though they weren't able to determine the source of the attack, or how they god in (should we be scared?). Anyway, its been several days now ~5 and if I do a site search the spam pages still show up, but the redirect is no longer working. At this point, I am at a standstill, because i'm loosing traffic on my site by about 90%, and google hasn't sent us any warnings of malaware or the like. I know I was recommended against this before, but should I attempt to submit a reconsideration request, or should I just wait it out? Thanks for your help, Zach
Search Behavior | | Zachary_Russell0 -
Visitors via Android App in Google Analytics
Hey there Mozzers! For one of our clients, we've developed an application on both Android and iOS. In the app, there is a link to the regular website. Can I see these visitors from the IOS/Android app in Google Analytics? Can I find them via Refferals? Thanks a lot in advance! Partouter
Search Behavior | | Partouter0 -
Number of searches for specific keywords
Hi everyone, I am new to SEOmoz and SEO but learning fast. Is there a way within SEOmoz to find out how many people are searching for specific keywords? It is a question I ask myself, "how many people are using the keywords I think I would use to find my products or services"? I've seen other SEO professionals produce reports showing how many searches happen for certain keywords. Thanks James
Search Behavior | | avecsys0