Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do we find keyword opportunities - but not just against 3 competitors...
Hello everyone! How can I find out what might be a keyword opportunity against a competitor but make sure it's still a realistic opportunity in SERPS generally?* After all, we are not just competing with 2 or 3 competitors. We are competing against everyone! Perhaps there is a part of the keyword research process that helps? I can't help but think there's a wasted effort to just beat a couple of competitors unless you can rank well generally - especially if it's a competitive sector. (*I'm not in the position of making lots of pages as part of a long-tail keyword/ content strategy so I need to focus on just a few and make them count) Thanks in advance. Mp
Competitive Research | | mark_seomoz10 -
Ahrefs Much Higher Than Competitors But Competitors Doing Significantly More Traffic?
Hey guys, I'm baffled by checking my Ahrefs ratings lately. It looks like our rating just went up considerably... from over 1 million to now around 500k. One of my main competitors, however, has a much lower Ahrefs rank (over 3 million) but he does a considerable more amount of traffic than we do... how is this? My domain rating is higher than his by a lot. Very curious how this works.
Competitive Research | | jfishe19880 -
Competitor Ranks Top Keywords Without Backlinks
One of my competitors is ranking very well for many different competitive keywords (1k+ searches per month). I'm trying to figure out how in the world he is ranking so well. I've signed up for MozPro and looked at his back-links. He has 1 branded site-wide back-link from a decent blog. He also has 1 contextual back-link from a decent blog. Other than these 2 back-links, the rest are garbage links unlikely to even count for anything (he has maybe 12 of these low quality back-links). My website on the other hand has more than 15 back-links from different (high quality) websites and does not rank anywhere near this competitor. This leads me to believe that either MozPro back-link reporting is inadequate or there is foul-play on the part of my competitor. As far as on-page SEO is considered, his website is far inferior. Therefore, I highly doubt this would play a role. What are some reasonable approaches I can take to better understand the cause of this discrepancy. Clearly the back-link reporting has not revealed any answers.
Competitive Research | | poke10 -
Why competitors rank with no content & filters
We're investing a lot in on-page content and creating categories/sub categories to target relevant keywords. Although it seems competitors with high domain authorities are clearly dominating in the SERPs and the pages have NO content. They are literally filters that create landing pages. For example when you search "Kaydian Beds", Debenhams rank below the manufacturer with just a filtered page, this page has not been optimised with content to target the keywords, only the META title changes depending on the chosen filter. Link: http://www.debenhams.com/furniture/beds/kaydian Our link: https://kontenta.co.uk/brands/kaydian-beds.html This is very frustrating as quality content should beat pages with little or no content, we will spend some time and get some inbound links to the page to try boost the rankings but does anyone have any similar experiences with anything like this?
Competitive Research | | Jseddon921 -
Would sharing the same IP address with competitors in the same market hurt SEO?
I work for the Chevy dealership in New York who is trying to rank against other Chevy dealerships in the same area. All the dealerships, including my client, are using the same Chevy endorsed CMS (Cobalt). I just noticed that all of these competitors, including my client, are using the SAME IP address. Would it be beneficial to SEO if we were to ditch the Cobalt platform and choose another one that gave us a unique IP address? Have any of you run into this before? Any help would be appreciated.
Competitive Research | | tjkirgin0 -
Much better authority and content than some competitors, why do they appear first?
On august we have lost the top positions for some keywords. Our website authority is 62 (google page rank 5), and some quite small competitors appear first now (some of them have authority 22, google page rank 3!). Our only big change was the DNS server, from godaddy to dnsimple.com, which seems to be some milisseconds slower. Every other metric is in our favor: from on-page repports to site speed. Sometimes by a factor of 3 or 4. What can be happening? We have also realized that, in private browsing, rankings will change a little bit in different cities, even if they are quite close to each other.
Competitive Research | | PauloSilveira0 -
Why OSE cant show competitor backlinks , please could you help?
Today i am searching for backlinks in OSE but it cant go to result pages, can anyone help me why OSE cant work today.
Competitive Research | | conversiontactics0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2