Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to set up a competitor URL with a language slug for a campaign
Hello, I am trying to set up a competitor with language slug for my (subfolder) website with a language slug. Let's say my website is something like: websiteholding.com/de/website
Competitive Research | | Siir
My competitor is: competitor.com/de When I go to Campaign Settings > Comptetitor Sites > type in competitor.com/de > Hit Save Competitor > Then it shows the saved competitor without the language slug as competitor.com I am not sure if this is a correct method of tracking since for my DE website I would like to track the DE page of the competitor, not their global page. Please correct me if I am wrong and help me out on possible solutions? I am quite new to SEO & Moz , so any help on the topic would be appreciated.0 -
Why our competitor has bigger DA?
I just compare our website www.simpletranslation.com with one of our competitor www.straker.com.au and their DA is much better than ours even we have more external backlinks. Can somebody explain me why? We worked really hard on new content on our blog and did few guest posts and instead of see some results on increasing of our DA our DA went down in last 2 months and I have no idea why.
Competitive Research | | Lukas-ST0 -
Why is our website ranked lower but beats most competitors in full SERP report?
We are analysing why our website ranks so low (currently position 8 in serp). We beat our competitors in most areas. Also we produce by far the most useful content. Do not buy links or do any other malpractice. We have been for a long time ranked in top 3 (well mostly 1-2) positions. In the last year we have seen a decline to position 8-10 and we are not sure why this is the case. Can anyone suggest what we should be focusing on? We are clueless. All the practices we used to "know" now seem obsolete. m8qIeyC,Kf1LLei#0
Competitive Research | | urkeman0 -
What tools can give reliable competitor traffic data?
Could you give details of reliable tools that can measure and provide competitors' traffic data? Google Adplanner used to provide such information until June of last year, and while it's numbers didn't match Google Analytics data, the numbers were reasonably close. The metrics we are most interested in are visits, page views and unique visitors. We have looked at sites such as Compete -- but feel it is far too inaccurate/ skewed with US data only; also, would prefer to not rely on Alexa traffic stats. Thanks in advance.
Competitive Research | | ontarget-media0 -
Moz Rank Moz Trust and Authority higher than my competitor but still getting outranked
Hi There, I'm new to SeoMoz but ive been using it for the past month and I've been using the tools to try to optimize my site but for some reason even though my metrics are showing that my rank and authority are rising past my competitor they still rank higher in the searches. my website is the site on the far left with the domain authority @27. the competitor that keeps outranking me is the third from the left with a domain authority of 12. If anyone can advise as to why I'm being outranked that would be very helpful. I'm stumped here. Thanks guys! eYtMRr9.png
Competitive Research | | SamRaskin1 -
Competitor reporting
We are in the process of creating customer profiles, conducting in-depth keyword/competitor research, fleshing out a content strategy, addressing old content, meta data etc, and researching sites (particular industry news sites) relevant to our industry who could be sources of authoritative backlinks. Once we receive approval from the powers that be, we'll start putting our strategy in motion. However, we are looking at SEO efforts as a long-term plan the entire organization understands is necessary. If our strategy is sound, I would think we'll start seeing results and some competitors will begin to take notice (and action.) So here is my question: How easy is it for a competitor to "report" a small/medium size business as breaking Google's rules i.e. could a competitor claim we are scraping their content (or content off of any site for that matter) and anonymously email Google? I think the answer to that question is "yes." How closely would Google look into the claim? We have zero plans on doing anything shady or underhanded but I was wondering if it is a simple process for a competitor to impact our efforts this way? Thanks for any insight.
Competitive Research | | SEOSponge0 -
Is there a SEO Moz Competitor Analasys tool that will show me how many Vistors their site receives?
I'm trying to find out how much traffic a competitor gets. Is there an SEOmoz tool for this?
Competitive Research | | starkSEO0 -
Anyone have experience with "Link Research Tools" ?
I'm looking into Linkresearchtools.com for competitive backlink analysis. Already have SEOmoz (obviously) and Majestic. Anyone have any experience with the toolset? If so, what does it do well, what do you like, and where does it fall down? Cheers
Competitive Research | | BedeFahey0