Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitors ranking in top three with worse SEO
Ok I am at a loss. Our client is trying to rank for keywords like "dental assistant programs utah", but they are still on the bottom of page 1/top of page 2. The guys in the top 3 have terrible SEO. No links, newer domains, etc. I looked into it extensively and learned about the "honeymoon period" people talk about but we are long past that. I've looked through webmaster tools to see if there are any issues, and I can't find a thing. Anyone have an idea of what we could be doing wrong that is making us be put down so far on the page? Before I got here the SEO guys were doing older tactics which I think was starting to hurt them with penguin and what not so I tried my best to delete the crap content and clean everything up on the website. thanks for all your help! I really appreciate it.
Competitive Research | | boyackp0 -
Traffic Data for Competitor Subdomain
Does anyone know of a way to get monthly visitor data for a subdomain that I do not manage? I would normally use SEMRush or Compete, but they only provide domain level data.
Competitive Research | | dsinger0 -
Much better authority and content than some competitors, why do they appear first?
On august we have lost the top positions for some keywords. Our website authority is 62 (google page rank 5), and some quite small competitors appear first now (some of them have authority 22, google page rank 3!). Our only big change was the DNS server, from godaddy to dnsimple.com, which seems to be some milisseconds slower. Every other metric is in our favor: from on-page repports to site speed. Sometimes by a factor of 3 or 4. What can be happening? We have also realized that, in private browsing, rankings will change a little bit in different cities, even if they are quite close to each other.
Competitive Research | | PauloSilveira0 -
Tracking competitors in search engines
Hi Mozzers, I'm using Advanced Web Rankings (AWR) to track my site and a number of competitors in Google - seeing how rankings and brand visibility change from week to week. I didn't set this up from scratch and I'm worrying with all of the recent algorithm changes that I might well be tracking the wrong competitors. Is there a tool or methodology I can use to find the biggest players in the market? I'm in the travel market so there's lots of choice and I track the large sites but want to be sure I'm aware of smaller/mid-sized sites gaining visibility without me tracking them via a platform like AWR. Many thanks!
Competitive Research | | panini0 -
Find competitors based on keywords
Are there any tools where I can enter say 5 keywords and it will look at the top 10 sites for each of those keywords then return which sites show up the most.
Competitive Research | | eyeflow0 -
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period?
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period via either Organic / Paid means? Thanks
Competitive Research | | jaycfc1230070 -
Whats the best way to see why your competitor is out ranking you?
I am trying to rank in the number on position for Part Time CFO. Currently we are in 2nd and 3rd. We have way more quality inbound links then the site in first position, more content etc. How would you go about investigating why they are ahead of us?
Competitive Research | | b2bcfo0 -
My client has shown me a similar site, though not a competitor. He wants to know what sites they are linked from that give them such a good Google rank for certain kewords. Can SEOMoz tell me this?
When using google.com.au and searching for "travel to france", www.frenchtravel.com.au is the 3rd organic result. (the 1st two are not travel businesses, they are non profit travel guides) My client, who runs www.visituk.com.au, an Australian site that organises tours of the UK, said "so we just need to add these sort of words to the site?" I said, yes, but it doesn't end there. The real task is to have a link to your site on other sites surrounded with the words "travel" and "UK". He asked if he could see a list of the sites the french site was being referred by relevant to the search phrase. Is there an SEOmoz tool for this? Or is there another way I can generate that list? Thanks Simon
Competitive Research | | electrik0