Ways to analyze a 1M rows dataset of search queries
-
Hi,
I have this large dataset, about 1 million search queries with visits, bounce rate and a few other metrics. I'm trying to explore this data to find keyword "buckets" (such as include product name, location name, transactional objective, informational, etc.), as well as explore the density of certain keywords (keywords as in instances of a single word amongst all queries)
My idea was to use Excel and a macro to split all queries in separate words (also clearing punctuation and uppercase/lowercase), then storing this word in a new worksheet, adding to another column the visit counts from the row where the word was extracted (as to give a sense of weight). Before adding the word to the new worksheet, the script will look if the word already existed, if so it would just add the current value of visits to the existing visit counts etc.
In the end it will create sort of a "dictionary" of all the keywords in all search queries ranked by weight (= visits from search query including this keyword)
This would help me get started I believe, because I can't segment and analyze 1M raw search queries...
My issue is: this VBA has been running on my (fast) PC for the last 24hr and it doesn't seem to get to an end. Obviously excel+VBA is not the best way to do text mining and manipulation in such a large dataset (although it's just a 30mb file)
What would you do if you had this dataset and would like to mine the text/semantic as I am doing? Any idea of tools? process?
I'm considering dumping this data into a MySQL db and doing the processing through PHP (the only backend language I'm versed in), and getting the "summified" data stored into another table, which I'll then be able to export to a Excel for analysis. But I'm afraid that I'll be facing memory limit issues and such...
In the meantime, I'm definitely interested into knowing what you guys would do if you had this data and wanted to simply start exploring its constituencies
Thanks!
-
Yeah, Access can process any number of rows. It's Microsoft's database program. You can upload data, and then create queries. They have a design view where you can construct queries in a WYSIWYG fashion, or if you want, you can write your own SQL.
-
Thanks a lot John!
I'm going to try this out tonight!
So, I assume, Access won't have the same processing limitations with 1 million rows, will it?
Once I'll be done with the "discovery phase" I'm going through with this keyword list, I'll definitely use Advanced filters (in Excel) as you recommend to understand keyword groups in details
-
I had a similar problem going through my search query reports. If you're already familiar with VB you could do this with a Microsoft Access database rather than setting up a MySQL one w/PHP. I've been working on creating an Access database that I can import my data into, and have it spit out all sorts of useful info (for example negative keywords and placements), but it's only in its early stages right now.
If you just want to see it for a few terms and don't mind doing it one at a time, in the past I've filtered data like this in Excel without VB using advanced filters. I found that using advanced filters rather than VB sped up the process quite a bit; I'd imagine because it's an innate Excel function. Using 4 filters you can match whole words in the queries. For example, to find queries with "blah", you'd set a filter for "blah", "* blah", "blah " and " blah *". Then you can use the Subtotal command to do calculations over the visible rows and calculate the data.
More about advanced filters: http://office.microsoft.com/en-us/excel-help/filter-by-using-advanced-criteria-HP005200178.aspx
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Console Linking
Hey everyone! Quick question; there have been problems in the past with some of our accounts' Search Console integration becoming unlinked from Google Analytics without us knowing about it. I did some digging into Google Analytics to see if there was a way to create an alert for search queries dropping off, but there doesn't seem to be any options for Search Console alerts like there are for other integrations like Google Ads. Has anyone discovered a way to have an alert or some type of notification set up for when a Search Console integration is unlinked in Google Analytics or if search queries drop off significantly? I'd appreciate the help!
Reporting & Analytics | | ReunionMarketing0 -
What's more accurate? GA queries data or Moz/SEMRush keyword data for rankings
What do you guys think? What's more accurate? GA queries data or Moz/SEMRush keyword data for rankings? Any thoughts appreciated.
Reporting & Analytics | | znotes0 -
UTM source errors in google search console
Dear Friends, I need help with UTM source and UTM medium errors. There are 300 such errors on my site which is affecting the site i think, The URL appended at the end is utm_source=rss&utm_medium=rss&utm_campaign= How do i resolve this? Please help me with it.Thanks ccEpFDn.png ccEpFDn.png
Reporting & Analytics | | marketing910 -
Google Search Console - Why is my average mobile position better than my average desktop position?
I'm wondering why my average mobile position is much better than my average desktop position. I'm wondering if Google is comparing the same queries for both mobile vs desktop or if they're only showing me the top ranked for each type of search. Is it example 1 or 2? Example 1: Desktop may have 5,000 ranking queries that average to 21.6
Reporting & Analytics | | Pauly_Gigs
Mobile may have 1,500 ranking queries that average to 8.5 OR Example 2: example.com has 5,000 total ranking keywords, those queries' average ranking in a desktop search 21.6 and mobile search 8.5. I'm curious to know exactly what I'm seeing in Google's Search Console. https://08875344305734164866.googlegroups.com/attach/777ae98664ed418f/Mobile%20VS%20Desktop.png?part=0.1&view=1&vt=ANaJVrEHOjRLlPH43i00NnC8PxaG3ct7bsHum_TWnUoa7xVamCpRp8jrvRQJL-gz4n7Q0otqKcKxcAJA5z1VySs2naQU_Zy5tDps6bJhUSZsLRQq4uU-tJQ0 -
Best way to measure local search keyword rank
I have several clients that have a regional or local presence and want to track their rankings for various keywords, but only in the areas they are located in. What are the best ways to track keyword rank in specific locations or regions of the US?
Reporting & Analytics | | TheURLdr0 -
No Query parameter for site search
Hi Guys, I have enable site search for analytics a number of times. But this time it's the first time I came across a search with no query parameters. example.com/search/item/searchterm What is the most simple way to approach this? thank you!
Reporting & Analytics | | GetApp0 -
Rank #1 for a 110,000/month query search, but barely any traffic?
Hi guys, As it says in the title, we've recently reached the absolute #1 position for a certain key phrase in the travel industry which the Google Keyword Tool tells me averages 110,000 local (165,000 global) searches a month... however we have received barely any traffic at all over the past TWO months for it and I'm trying my best to determine why. We've checked on multiple different devices with all forms of personalisation off, different browsers, 3G connections as opposed to office Wi-FI etc. and it still returns us as the #1 rank. Meta descriptions and title tags are pretty much pristine if I don't say so myself, however what should be a very lucrative key phrase is currently returning little to no traffic results. Has anyone had experience in a similar situation to this? Any possible causes that I might be missing? Would greatly appreciate any help. Thanks.
Reporting & Analytics | | ExperienceOz0 -
Google Secure Search Announcement: How do you think this will play out?
So, I read this post on the Google Blog and then this commentary about it. Wondering what some of you veterans think about how this will impact results. I always remain logged out for queries and use several different software tools to check when I run an analysis. Since social influences are becoming a bigger factor, do you think this is going to create some issues in data? What are your thoughts?
Reporting & Analytics | | TheARKlady0