Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitor Ranks Top Keywords Without Backlinks
One of my competitors is ranking very well for many different competitive keywords (1k+ searches per month). I'm trying to figure out how in the world he is ranking so well. I've signed up for MozPro and looked at his back-links. He has 1 branded site-wide back-link from a decent blog. He also has 1 contextual back-link from a decent blog. Other than these 2 back-links, the rest are garbage links unlikely to even count for anything (he has maybe 12 of these low quality back-links). My website on the other hand has more than 15 back-links from different (high quality) websites and does not rank anywhere near this competitor. This leads me to believe that either MozPro back-link reporting is inadequate or there is foul-play on the part of my competitor. As far as on-page SEO is considered, his website is far inferior. Therefore, I highly doubt this would play a role. What are some reasonable approaches I can take to better understand the cause of this discrepancy. Clearly the back-link reporting has not revealed any answers.
Competitive Research | | poke10 -
Competitor Keywords
Is there any way to find out what keywords my competitors are ranking for? I would like ideas for blogs, website content, etc. to drive traffic and am curious as to what topics my competitors are focusing on. I am able to see my competitors' rankings for keywords which I am currently tracking but I would like to see keywords they are ranking for that I am not tracking if possible. Any suggestions? Josh
Competitive Research | | jbeals0 -
Methods for estimating competitor website traffic from natural search
Hi all, I'm currently working on some competitor analysis and estimating website traffic based on rankings and estimated search volume for approximately 500 keywords. I'm then estimating based on click through rate from Webmaster tools. I think this will give me a relatively accurate estimation but was wondering how everyone else does this? Any other methods out there? Thanks, Elias
Competitive Research | | A_Q0 -
Subdomain Research Tool
Does anybody know of a research tool that can track the amount of subdomains on a root domain? Maybe there is a way to manipulate a Google search to display the different subdomains that are indexed?
Competitive Research | | iSTORM-New-Media0 -
Ecommerce seo - competitors are using spammy links to rank HELP
After setting up SEO Moz and whilst waiting for it to crawl my site i have checked out my competitors who are ranking for the keywords i am going after. It appears from what i can see in OSE that they are ranking with back links from spammy looking sites and lots of exact match anchor text. I have tried to match a few links with what they have currently without getting myself into anything too spammy but i really don't want to follow how they have done it exactly. The niche i am in is pretty boring but the rewards are great but i am finding creating content and distributing it correctly difficult which is why i need some help. My site is roughly 9 months old and the product descriptions are hand written trying to give as much info as possible with a hint of personality so no cookie cutter back of the box descriptions. i have matched a few of the links from competitors which weren't too spammy and as relevant as i could get to my niche, i have also created articles for some article directories - all of which isn't really getting me very far and i am running out of ideas on how to create new content & the types of new content to help me get back links naturally. If any of you friendly guys could offer me some assistance in setting up a basic seo campaign which would help me target my keywords i would be very much in your debt
Competitive Research | | GarethEJones0 -
Is there a SEO Moz Competitor Analasys tool that will show me how many Vistors their site receives?
I'm trying to find out how much traffic a competitor gets. Is there an SEOmoz tool for this?
Competitive Research | | starkSEO0 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0 -
Tool for finding what keywords a competitor ranks for?
Does anyone know of any good tools that display what keywords a competitor ranks for? I have many competitors that I know get a lot of traffic, but I'm not entirely sure where the traffic comes from so it would be nice to plug in their url and get a general overview of what keywords they rank for and what positions.
Competitive Research | | shawn810