Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
What's a good way to get started with competitive research?
Hi all, SEO noob here. I'm doing an audit for a firm that makes specialized accounting software. It's a relatively new firm, with a barebones website. My client has identified three direct business competitors. In addition, I see indirect competitors (such as product reviews) on the SERP for a relevant keyword phrase. I want to provide actionable advice for my client. What information should I present? I'd like to help my client understand: Why my client's competitors are outperforming them on the SERPs What my client needs to do to overtake their competitors What information should I present to my client? Thanks, all.
Competitive Research | | AndyKubrin0 -
What tools can give reliable competitor traffic data?
Could you give details of reliable tools that can measure and provide competitors' traffic data? Google Adplanner used to provide such information until June of last year, and while it's numbers didn't match Google Analytics data, the numbers were reasonably close. The metrics we are most interested in are visits, page views and unique visitors. We have looked at sites such as Compete -- but feel it is far too inaccurate/ skewed with US data only; also, would prefer to not rely on Alexa traffic stats. Thanks in advance.
Competitive Research | | ontarget-media0 -
Moz Rank Moz Trust and Authority higher than my competitor but still getting outranked
Hi There, I'm new to SeoMoz but ive been using it for the past month and I've been using the tools to try to optimize my site but for some reason even though my metrics are showing that my rank and authority are rising past my competitor they still rank higher in the searches. my website is the site on the far left with the domain authority @27. the competitor that keeps outranking me is the third from the left with a domain authority of 12. If anyone can advise as to why I'm being outranked that would be very helpful. I'm stumped here. Thanks guys! eYtMRr9.png
Competitive Research | | SamRaskin1 -
Which top 10 organic results are my competitors?
Hi all, This is a question about whether the top 10 ORGANIC Google results for a broad match is your competition, or whether it's the top 10 for a phrase match, or an exact match. I'm a newbie here but not a newbie to the world of SEO. I hope to be able answer just as many questions as I ask 🙂 QUESTION: If a customer comes to me and says, "hey, who's my (organic) competition for wedding present?" and I want to use Google to get the top 10 organic results, do I use a broad, phrase, or exact match? It seems many people think an exact match is the way to go but I think they were more referring to Adwords results / competition. I'm not trying to determine search volume for Adwords or even the search volume for organic results... I'm only interested in the top 10 competitors in the organic results. No one types in "wedding present" (with inverted commas" when doing a search in Google, so surely to see who ranks organically for wedding present I'd want to simply type in wedding present (no inverted commas, aka broad match). I understand all the concepts about how Google results cary whether you're logged in, etc, etc so I don't want to get distracted by that. And I know there's a bunch of tools we can use like the SEOMoz Keyword Analysis Tool. But I just want to know specifically what people would use (broad, phrase or exact) to look at the top 10 organic competitors are when doing a manual search in Google.
Competitive Research | | eatyourveggies1 -
Anyone have experience with "Link Research Tools" ?
I'm looking into Linkresearchtools.com for competitive backlink analysis. Already have SEOmoz (obviously) and Majestic. Anyone have any experience with the toolset? If so, what does it do well, what do you like, and where does it fall down? Cheers
Competitive Research | | BedeFahey0 -
Competitor Ranking High has 2 Domains, But Duplicate Website ?
I was using OSE and noticed all the backlinks to one of our competitors is there other domain name, which is the EXACT SAME website. You can enter both url's and they display the same content. They are not useing any canonical tags either. Why are they not penalized for duplicate content? And for using there own website for backlinks ? We try to do everything right, but still cannot beat them. Any thoughts on this?
Competitive Research | | hfranz0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2