Screaming Frog - What are your "go to" tasks you use it for?
-
So, I have just purchased screaming frog because I have some specific tasks that need completing. However, looking at Screaming Frog generally, there is so much information I was wondering for those who use it, what are the top key tasks you use it for. I mean what are your "go to" things you like to check, that perhaps are not covered by the Moz Crawl reports.
Just looking for things I perhaps hadn't thought about, that this might be useful for.
-
Ha ha, I know! It's like giving the developers a little present all wrapped up with a bow...here's the problem, and here's where to fix it
-
Allie,
That's a great example use-case. After my audits, clients are like "you found thousands of internal redirects and 404s - where are they?"
I'm like - hold on I have a spreadsheet of that!
-
I love Screaming Frog! One use case I've used recently is using it to find internal 404 errors prior-to and immediately-after a major site redesign.
After running a crawl, go to Bulk Export > Response Code > Client error (4xx) Inlinks and download the report. It shows the offending URL and the URL referring to it, which makes it easier to update the bad link.
I also have this page bookmarked, and it's my go-to guide:
-
It's one of the best tools so I feel like I use it "for everything." But some includes:
-
Title / meta duplication & finding parameters on ecomm stores
-
Title length & meta desc length
-
Removing meta keywords fields
-
Finding errant pages (anything but 200, 301, 302, or 404 status code)
-
Large sitemap export (most tools do "up to 500 pages." Useless.)
-
Bulk export of external links (what ARE we linking to??)
-
Quickly opening a page in Wayback Machine or Google cache
-
Finding pages without Analytics, as was mentioned.
I use Screaming Frog for tons of other things. Finding the AJAX escaped frag URL, identifying pages with 2 titles, 2 canonicals, 2 H1 tags, etc. Even seeing www & non-www versions live, links to pages that shouldn't be linked and http vs https.
Very cool tool - useful for pretty much everything! haha
-
-
That's awesome. Thanks. Will take a look at all those things this week.
-
I use SF religiously for all the audit work I do. I run a sample crawl (using Googlebot as the crawler) to check for all the standard stuff and go further.
My standard evaluation with SF includes:
- Redirect / dead end internal linking
- Redirect / dead end "external" links that point to site assets housed on CDN servers.
- URL hierarchical structure
- Internal linking to both http and https that can reinforce duplicate content conflicts
- Page Title/H1 topical focus relevance and quality
- Confusion from improperly "nofollowing" important pages (meta robots)
- Conflicts between meta robots and canonical tags
- Slow page response times
- Bloated HTML or image file sizes
- Thin content issues (word count)
- Multiple instances of tags that should only have one instance (H1 headline tags, meta robots tags, canonical tags)
-
That crawl path report is pretty cool, and it led me to the redirect chain report, which I have a few issues to resolve with that with a few multiple redirects on some old links. Fantastic stuff.
-
I am a big fan of Screaming frog myself. Apart from the real basic stuff (checking H1, titles,...etc) it's also useful to check if all your pages contain your analytics tag and to check the size of the images on the site (these things Moz can't do).
It's also extremely useful when you're changing the url structure to check if all the redirects are properly implemented.
Sometimes you get loops in your site, especially if you use relative rather than absolute links on your site - Screaming Frog has an extremely helpful feature: just click on the url and select "crawl path report" - which generates an xls which shows the page where the problem originates
It's also very convenient that you can configure the spider to ignore robots.txt / nofollow / noindex when you are test a site in a pre-production environment. Idem for the possibility to use regex to filter some of the url's while crawling (especially useful for big sites if the they aren't using canonicals or noindex where they should use it)
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to deal with rel=canonical when using POST parameters
Hi there,
On-Page Optimization | | mjk26
I currently have a number of URLs throughout my site of the form: https://www.concerthotels.com/venue-hotels/o2-academy-islington-hotels/256133#checkin_4-21-2024&checkout_4-22-2024&rooms_1&guests_2&artistid_15878:256133 This sends the user through to a page showing hotels near the O2 Academy Islington. Once the page loads, my code looks at the parameters specified in the # part of the URL, and uses them to fill in a form, before submitting the form as a POST. This basically reloads the page, but checks the availability of the hotels first, and therefore returns slightly different content to the "canonical" version of this page (which simply lists the hotels before any availability checks done). Until now, I've marked the page that has had availability checks as noindex,follow. But because the form was submitted with POST parameters, the URL looks exactly like the canonical one. So the two URLs are identical, but due to POST parameters, the content is slightly different. Does that make sense? My question is, should both versions of this page be marked as index,follow? Thanks
Mike0 -
Will including "Contact Me" form degrade Google page ranking?
I have a content-rich page about one of my offerings. Will Google knock down the ranking if I include a contact me reply form on the page vs including a link to a standalone reply page? My concern is that including the form will cause Google to downgrade the page as being "too commercial".
On-Page Optimization | | Lysarden0 -
Linking back to the homepage im trying to rank - Using exact match anchor? Linking from footer?
Hello, Our site is an basically advertisements / listings website. Structure is as following <last 200="" adds=""> (homepage trying to rank) < category 1 > < category 1 > < category 1 > < category 1 > < category 1 ></last> My question - each of the categories links back within the menu back to homepage. The link text currently is last 200 adds. Can i use exact match anchor? Or should i use just last 200 ads? The issue is that one of my categorys (category 1) has already the exact match anchor im trying to rank for. So i can not use the same to link back to homepage. Im worried that google does not see any exact keyword anchor texts back to homepage hence will rank my homepage with lower strenght for that keyword . Im also worried that the category 1 page might now compete with the main homepage for this word (even tho at the moment category itself does not rank for this keyword) Can i link from footer back to homepage with an alternative keyword then to give some "context" to google more? Would this be spamming?
On-Page Optimization | | advertisingcloud0 -
MOZ Pro Has Not Helped Yet. What is Going On?
A month ago I started using MOZ and I improved my website significantly. I reduced the Crawl Issues to few of them, improved the sitemap, url, etc. Unfortunately, my organic search has not improved at all. At the same time one of my competitors has the same Domain Authority, near the same Page Authorizaty, relatively similar social media performance, slightly higher backlinks, same blog content, etc. BUT, they are better ranked by Alexa.com, Similarweb.com and appear on google in top 10 for many of the keywords I am not even in top50. What am I doing wrong?
On-Page Optimization | | Threeding.com0 -
Proper Use and Interpretation of new Query/Page report
When I'm in WMT/Search Console - I start a process of looking at all of the data initially unfiltered Then I select a query. Let's say its a top query for starters and I filter my results by that top query (exactly) With the filter on, I flip over to Pages and I get about a dozen results. When I look at this list, I get the normal variety of output: impressions, clicks, CTR, avg. position One thing that seems a bit odd to me is that most of the average positions for each of the URLs displayed is about the same. Say they range from 1.0 to 1.3. Does this mean that Google is displaying the dozen or so URLs to different people and generally in the 1st or 2nd position. Does this mean that my dozen or so pages are all competing with each other for the same query? On one hand, if all of my dozen pages displayed most of the time in the SERP all at the same time, I would see this as a good thing in that I would be 'owning' the SERP for my particular query. On the other hand, I'm concerned that the keyword I'm trying to optimize a particular page for is being partially distributed to less optimized pages. The main target page is shown the most (good) and it has about a 15x better CTR (also good). But all together, the other 11 pages are taking in around 40% of impressions and get a far lower CTR (bad). Am I interpreting this data correctly? Is WMT showing me what pages a particular query sends traffic to? Is there any way to extract the keywords that a particular page receives? When I reset my query and then start by selecting a specific page (exact match) and then select queries - is this showing my the search queries that drove traffic to that page? Is there a 'best practices' process to try to target a keyword to a specific page so that it gets more than the 60% of impressions I'm seeing now? Obviously I don't want to do a canonical because each keyword goes to many different pages and each page receives a different mix of keywords. I would think there would be a different technique when your page has an average position off of page 1.
On-Page Optimization | | ExploreConsulting0 -
Duplicate content - "Same" profile-information
Hi, I own a casting website with lots of profiles. Some of these profiles only typed in their firstname, email and age, when they registered on the site, and they haven't added more information ever since. From Crawl Diagnostics, I can see that there is "lots" of these profiles, which looks exactly the same (only showing age and firstname), allthought they are not the same. I could add which day the profile were created on the site, to maybe avoid these "duplications". The email will always be hidden. Or, how big an issue is this? Crawl Diagnostics tells me, that there is around 200 of these, and they are "marked" as High Priority. Any ideas on what to do? /Kasper
On-Page Optimization | | KasperGJ0 -
Too Many on page links! Will "NoFollow" for navigation help?
I am getting to many on page links ( for all my pages). Here is my website: http://www.websterpowerproducts.co.uk I think it is to do with the the navigation bar down the right hand side. I don't really want to get ride of this as it offers users a way of getting where they want without lots of clicking. I was wondering if adding a "NoFollow" tag to each of they links would stop the link juice getting diluted by the navigation bar. Many Thanks
On-Page Optimization | | WebsterPowerTools0 -
Should I include a "|" for better page title SEO results?
I have seen many sites that include the "|" in page titles and was wondering if there is some SEO value in the practice. Example: Product Name | Company Name Instead of: Product Name by Company Name I have not seen any value in it myself other than a good way to avoid stop words. I wanted to make sure. Currently I have the "by" included in the page titles.
On-Page Optimization | | JedHenning0