Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
What's a good way to get started with competitive research?
Hi all, SEO noob here. I'm doing an audit for a firm that makes specialized accounting software. It's a relatively new firm, with a barebones website. My client has identified three direct business competitors. In addition, I see indirect competitors (such as product reviews) on the SERP for a relevant keyword phrase. I want to provide actionable advice for my client. What information should I present? I'd like to help my client understand: Why my client's competitors are outperforming them on the SERPs What my client needs to do to overtake their competitors What information should I present to my client? Thanks, all.
Competitive Research | | AndyKubrin0 -
Missing tab in Competitor research Template
The template provided in https://moz.com/blog/competitor-analysis-for-seo is missing the 'Audit Data' Tab. Can you please update it or share the Original Template
Competitive Research | | AnkitaDwivedi3 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0 -
Comparing competitor folder path
Hi All I've just take on a new client and need to be able to compare their performance with a competitor's sub-folder site. The competitor is http://thebestof.co.uk/local/brightonandhove/ I've tried to add that to the competition structure in my campaign but can't use "Path" info. What do you guys do in these types of cases. BTW - TheBestOf.co.uk is a franchise with a /local/CityNames across the UK
Competitive Research | | stevecounsell0 -
What if your content is way better than competitors?
Do you have any suggestions if you are being outranked by competitors even though your page has way better content and better links?
Competitive Research | | Charlessipe0 -
Find competitors based on keywords
Are there any tools where I can enter say 5 keywords and it will look at the top 10 sites for each of those keywords then return which sites show up the most.
Competitive Research | | eyeflow0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2