Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Intent and Relevance
Hello SEO gurus 🤓 I’m looking for the most efficient ways to analyze the search intent and relevance of competitors who are ranking for the keywords we’re targeting. While I know Google excels at assessing search intent and relevance, I’m interested in learning how we can evaluate these factors as metrics for our competitors. The goal is to understand their strategy better and find ways to outrank them. Do you have any tools or methodologies that you recommend for assessing competitor content to determine its alignment with search intent and user needs/relevance? I’d love to hear your thoughts and suggestions on this!
Competitive Research | Nov 1, 2024, 8:37 PM | Cricket931 -
How to set up a competitor URL with a language slug for a campaign
Hello, I am trying to set up a competitor with language slug for my (subfolder) website with a language slug. Let's say my website is something like: websiteholding.com/de/website
Competitive Research | Mar 15, 2024, 1:52 PM | Siir
My competitor is: competitor.com/de When I go to Campaign Settings > Comptetitor Sites > type in competitor.com/de > Hit Save Competitor > Then it shows the saved competitor without the language slug as competitor.com I am not sure if this is a correct method of tracking since for my DE website I would like to track the DE page of the competitor, not their global page. Please correct me if I am wrong and help me out on possible solutions? I am quite new to SEO & Moz , so any help on the topic would be appreciated.0 -
What Exactly is traffic lift?
Under the keyword gap analysis under the competitive analysis report, there is a mention of traffic lift and it provides a number. Does the number represent the number of users or the number traffic to expect for that term? Or is it the positions?
Competitive Research | Aug 12, 2022, 4:55 AM | SEOBrandBoca1 -
What is the relationship between inbound links and linking domains?
In doing competitor research, there's a specific competitor that has fewer linking domains, but dramatically more inbound links. How do these two relate to each other? I do have a higher domain authority and more internal followed links with them having more internal nofollow and external (both follow and nofollow) links. I'm trying to understand how they can have 3.7k linking domains and 4.8 million inbound links while I have 4.1k linking domains and only 601k inbound links. What really defines an inbound link and what's there relationship between inbound links and linking domains?
Competitive Research | Sep 7, 2022, 7:40 AM | grayloon0 -
Interal linking
Re: I'm stuck by internal linking. What structure should a football website follow? Silo or Topic Cluster?
Competitive Research | Jul 5, 2024, 5:41 PM | thanhdung0906
I need advice on my website. My website: https://tipbongda247.net/
I hope there are answers!
Thanks0 -
How do you select which keywords to push in SEO?
Hi Guys Selecting the right keywords that a website can realistically rank for is a key to gain top rankings relatively quick. I am just curious to hear how you guys do it (the methodology) when selecting which keywords to push? I mean you need to check the competition for each keyword as well so how to check this quickly to see what we realistically can rank for? Cheers John
Competitive Research | Apr 19, 2024, 6:52 AM | igniterman751 -
MOZ vs Ahrefs vs SEMRush vs Spyfu and so on
Hello, fellas. I've been trying to find some type of comparison analysis for online SEO related tools and I couldn't find any, which are fresh and overview more than three. I'm interested to see if anybody saw good comparison reviews of such tools, as well as your own thoughts (back your thoughts up with more than just "I like it", please 🙂 The tools I'm interested in comparing are: MOZ Ahrefs Majestic Spyfu SEMRush Raven Wordstream whatever else you use or have in mind
Competitive Research | Jun 10, 2019, 8:43 PM | DmitriiK2 -
Traffic Data for Competitor Subdomain
Does anyone know of a way to get monthly visitor data for a subdomain that I do not manage? I would normally use SEMRush or Compete, but they only provide domain level data.
Competitive Research | Mar 12, 2014, 11:12 AM | dsinger0