Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I can see competitors ranking for certain long-tail keywords but cannot find them on web pages. What am I missing?
Hi there. I'm pretty new to SEO and I've been doing a fair bit of training but there is one aspect I have yet to grasp. When I carry out keyword research, I get all these results and I understand the metrics. What I'm not getting is, when a competitor is ranking highly for say "where can I buy fresh turkeys", I assume that that phrase must appear somewhere on the page, but it doesn't. I realise I'm just not thinking about this in the right way. Can anyone offer clarification, please? Kind regards, Bruce
Competitive Research | | BruceBarbour0 -
Competitors' paid search analysis tool
Hi, What tool will do the following: will tell me paid keywords our competitors are using. Ideally, I could narrow down by country. Thanks. Katarina
Competitive Research | | Katarina-Borovska0 -
Competitor Keywords
Is there any way to find out what keywords my competitors are ranking for? I would like ideas for blogs, website content, etc. to drive traffic and am curious as to what topics my competitors are focusing on. I am able to see my competitors' rankings for keywords which I am currently tracking but I would like to see keywords they are ranking for that I am not tracking if possible. Any suggestions? Josh
Competitive Research | | jbeals0 -
Competitor reporting
We are in the process of creating customer profiles, conducting in-depth keyword/competitor research, fleshing out a content strategy, addressing old content, meta data etc, and researching sites (particular industry news sites) relevant to our industry who could be sources of authoritative backlinks. Once we receive approval from the powers that be, we'll start putting our strategy in motion. However, we are looking at SEO efforts as a long-term plan the entire organization understands is necessary. If our strategy is sound, I would think we'll start seeing results and some competitors will begin to take notice (and action.) So here is my question: How easy is it for a competitor to "report" a small/medium size business as breaking Google's rules i.e. could a competitor claim we are scraping their content (or content off of any site for that matter) and anonymously email Google? I think the answer to that question is "yes." How closely would Google look into the claim? We have zero plans on doing anything shady or underhanded but I was wondering if it is a simple process for a competitor to impact our efforts this way? Thanks for any insight.
Competitive Research | | SEOSponge0 -
Fast Multiple Keyword research
Does anyone know of a good tool to drop a list of keywords into and get competitive and search volume results from? Don't want to have to enter them one by one to the Google KWT.
Competitive Research | | AESEO0 -
Which top 10 organic results are my competitors?
Hi all, This is a question about whether the top 10 ORGANIC Google results for a broad match is your competition, or whether it's the top 10 for a phrase match, or an exact match. I'm a newbie here but not a newbie to the world of SEO. I hope to be able answer just as many questions as I ask 🙂 QUESTION: If a customer comes to me and says, "hey, who's my (organic) competition for wedding present?" and I want to use Google to get the top 10 organic results, do I use a broad, phrase, or exact match? It seems many people think an exact match is the way to go but I think they were more referring to Adwords results / competition. I'm not trying to determine search volume for Adwords or even the search volume for organic results... I'm only interested in the top 10 competitors in the organic results. No one types in "wedding present" (with inverted commas" when doing a search in Google, so surely to see who ranks organically for wedding present I'd want to simply type in wedding present (no inverted commas, aka broad match). I understand all the concepts about how Google results cary whether you're logged in, etc, etc so I don't want to get distracted by that. And I know there's a bunch of tools we can use like the SEOMoz Keyword Analysis Tool. But I just want to know specifically what people would use (broad, phrase or exact) to look at the top 10 organic competitors are when doing a manual search in Google.
Competitive Research | | eatyourveggies1 -
Keyword Research Tool Problem
Hi, I had been in SEO for around 2 to 3 years, I am looking for your suggestion if there is any best keyword research tool. I tried almost most of the keyword research tool (both paid or free) but I found the following problem: -Worktracker: I think it is too difficult to use and understand as a non technical person. I always wonder the data accuracy because when the same keyword appear in Google for few thousand traffic, it only appear in Wordtracker for few dozen or hundred of search. Also I don't think they suggest any long trail keywords in my view. The long trail keyword you can also find in free keyword search tool. -Google Keywords Tool- Good, but it can only show 100 results. I cannot get a better pictures of related or niches keywords. I always find some competitive keywords can rank easily but some non competitive keywords usually very difficult to rank in some case. It will depend on how you interpret the data results. -Keyword Discovery and Wordstream-Although I really want to subscribe, but they are a way too expensive (you have to subscribe for 12 months). I check with their free keywords tool but only shown very limited results. Although there are some other like spyfu.com or keywords spy...... Is there any one can suggest me which is the best tool to start with keyword search for the long trail and niche keywords ? or is there any way or trick to find the best keywords ? Thank you. Tom
Competitive Research | | Stevejobs20110 -
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period?
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period via either Organic / Paid means? Thanks
Competitive Research | | jaycfc1230070