Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is our website ranked lower but beats most competitors in full SERP report?
We are analysing why our website ranks so low (currently position 8 in serp). We beat our competitors in most areas. Also we produce by far the most useful content. Do not buy links or do any other malpractice. We have been for a long time ranked in top 3 (well mostly 1-2) positions. In the last year we have seen a decline to position 8-10 and we are not sure why this is the case. Can anyone suggest what we should be focusing on? We are clueless. All the practices we used to "know" now seem obsolete. m8qIeyC,Kf1LLei#0
Competitive Research | | urkeman0 -
Which top 10 organic results are my competitors?
Hi all, This is a question about whether the top 10 ORGANIC Google results for a broad match is your competition, or whether it's the top 10 for a phrase match, or an exact match. I'm a newbie here but not a newbie to the world of SEO. I hope to be able answer just as many questions as I ask 🙂 QUESTION: If a customer comes to me and says, "hey, who's my (organic) competition for wedding present?" and I want to use Google to get the top 10 organic results, do I use a broad, phrase, or exact match? It seems many people think an exact match is the way to go but I think they were more referring to Adwords results / competition. I'm not trying to determine search volume for Adwords or even the search volume for organic results... I'm only interested in the top 10 competitors in the organic results. No one types in "wedding present" (with inverted commas" when doing a search in Google, so surely to see who ranks organically for wedding present I'd want to simply type in wedding present (no inverted commas, aka broad match). I understand all the concepts about how Google results cary whether you're logged in, etc, etc so I don't want to get distracted by that. And I know there's a bunch of tools we can use like the SEOMoz Keyword Analysis Tool. But I just want to know specifically what people would use (broad, phrase or exact) to look at the top 10 organic competitors are when doing a manual search in Google.
Competitive Research | | eatyourveggies1 -
Much better authority and content than some competitors, why do they appear first?
On august we have lost the top positions for some keywords. Our website authority is 62 (google page rank 5), and some quite small competitors appear first now (some of them have authority 22, google page rank 3!). Our only big change was the DNS server, from godaddy to dnsimple.com, which seems to be some milisseconds slower. Every other metric is in our favor: from on-page repports to site speed. Sometimes by a factor of 3 or 4. What can be happening? We have also realized that, in private browsing, rankings will change a little bit in different cities, even if they are quite close to each other.
Competitive Research | | PauloSilveira0 -
Competitor's Ranking's Reports are Showing Blank Values for Rankings
When selecting competitors to see how they rank for specific terms in the ranking's report page here, it loads the page normally, but without any actual rankings. Where values should be, it lists all values as our competition not ranking whatsoever (for all competitors). I'm not sure if there is something we're doing wrong, or perhaps we just need to wait for a refresh, but if anybody could help out, that would be wonderful!
Competitive Research | | ACLens0 -
Your favorite/most comprehensive guide to keyword research?
what's your favorite guide to keyword research? in lieu of having a favorite, what is the most comprehensive guide you've come across?
Competitive Research | | Mozzin0 -
Competior Research & Analyse
Hi, I like to know the inbound marketing sources of my competitors, and like to know how many traffic are receiving from Organic Ranking and Adwords or Search Engine Paid Marketing. ANd it would be nice to know the social media traffic too. I am searching, but i see here compete.com, but at the moment it's too expensive for me, like to ask you if you can find me to find any software cheaper than compete. Thanks.
Competitive Research | | leadsprofi0 -
Competitor Ranking High has 2 Domains, But Duplicate Website ?
I was using OSE and noticed all the backlinks to one of our competitors is there other domain name, which is the EXACT SAME website. You can enter both url's and they display the same content. They are not useing any canonical tags either. Why are they not penalized for duplicate content? And for using there own website for backlinks ? We try to do everything right, but still cannot beat them. Any thoughts on this?
Competitive Research | | hfranz0 -
My client has shown me a similar site, though not a competitor. He wants to know what sites they are linked from that give them such a good Google rank for certain kewords. Can SEOMoz tell me this?
When using google.com.au and searching for "travel to france", www.frenchtravel.com.au is the 3rd organic result. (the 1st two are not travel businesses, they are non profit travel guides) My client, who runs www.visituk.com.au, an Australian site that organises tours of the UK, said "so we just need to add these sort of words to the site?" I said, yes, but it doesn't end there. The real task is to have a link to your site on other sites surrounded with the words "travel" and "UK". He asked if he could see a list of the sites the french site was being referred by relevant to the search phrase. Is there an SEOmoz tool for this? Or is there another way I can generate that list? Thanks Simon
Competitive Research | | electrik0