Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to set up a competitor URL with a language slug for a campaign
Hello, I am trying to set up a competitor with language slug for my (subfolder) website with a language slug. Let's say my website is something like: websiteholding.com/de/website
Competitive Research | | Siir
My competitor is: competitor.com/de When I go to Campaign Settings > Comptetitor Sites > type in competitor.com/de > Hit Save Competitor > Then it shows the saved competitor without the language slug as competitor.com I am not sure if this is a correct method of tracking since for my DE website I would like to track the DE page of the competitor, not their global page. Please correct me if I am wrong and help me out on possible solutions? I am quite new to SEO & Moz , so any help on the topic would be appreciated.0 -
Visibility Score Against Top Search Competitors
Top Search Competitors
Competitive Research | | PhoebeK
The competitors that compete for the same keywords as this domain, ranked by visibility. Other websites that compete with this domain for the same keywords, ranked by visibility. Visibility (What does the visibility rating mean on this report)?
For example: If you have a visibility of 8.33 on this report and your competitor has a rating of 6.21 or 4.64, is your visibility better than theirs?0 -
How does my competitor website jump from a natural ranking of 10 to 1 in a matter of days?
OK this has really got me baffled, and I really hope the MOZ community can give me the answers I am seeking.... My statements and queries below are my own opinions and thoughts. I am not an experienced SEOer and I am learning all the time, so if I have written something incorrectly, please feel free to correct me 🙂 I have been told that natural rankings take time, and I have read with the recent algorithm changes from Google, that purchased links do not have an effect like they used to. I have also read somewhere that google has clamped down on bogus article writing (MOZ please clarify) But what I am seeking some clarification on is, how does my competitors website, that has had a natural rank of #1 in the past, suddenly drop off; for a lengthy period of time, then suddenly re-appear at #10 on page #1, then some three/four days later, hit #1? I have read that natural rankings are purely natural. I.e. a domain owner cannot pay google to boost their natural ranking. But is this really true? Why is it when I speak to a close colleague of mine in the same industry, he mentions that his maps listing has dropped off, and after making a couple of phone calls to his SEO guys in India, maybe a day or two later, magically his maps re-appear? I really am starting to question is a "natural ranking" really "natural?" I have heard that some of these SEO agencies do have some tools that dramatically effect the natural ranking of a domain. What are these tools?
Competitive Research | | Giorgio680 -
What keywords & phrases are my competitors targeting?
I wanted to compare our keyword focus, to my competitors. What is the best way to audit what type of content / keywords - my competitors are using?
Competitive Research | | jwochna0 -
Keyword difficulty/research question
Wondering if I could get some opinions from the fellow moz users' I have a website which I which to rank for the term 'evening dress'.As you can imagine it is a pretty difficult term with a score of 62% (the term gets 301,000 broad matches and 27,000 exact matches a month). As much as I would like to target this term I feel that my domain is not strong enough (DA 39) to match the competition. Therefore, would a better strategy be to target long tail keywords which also contain the primary keyword, ie black evening dress evening dress hire cheap evening dress buy evening dress online please note that these were just examples, I haven't researched a comprehensive long tail list. Would targeting these long tail keywords mean that a) I should be able to rank for them faster and thus receive more traffic, sooner, and b) build up links to the page which I ultimately want to rank for evening dresses with numerous backlinks containing the keyword evening dress. The trade off with doing this is that I would need to seo one page for all the long tail keywords to gain the maximum benefit for the 'money' keyword. Does this sound like a sensible approach to both ranking for big money term and also getting traffic sooner rather than later? Thanks Carl
Competitive Research | | Grumpy_Carl0 -
How to track competitor who uses another website on the url?
Hi everyone, The thing is pretty simple: I have a competitor who uses a different format of url, very similar to this: www.example.com - but he started to use a thing like this: example.mtv.com The problem is that every analysis that I make tell me something about mtv.com instead of the site I want. The url is redirected. Sorry about my english, but I think it is very clear. I want to know how to track something like this, because even in competitor analysis I'm getting info about the big site, and not the one I need. Thank you very much.
Competitive Research | | bluehelmet0 -
How do i improve my site's google ranking against my competitors?
the searches that i would like to improve my ranking on are
Competitive Research | | oswaldjr
st kitts real estate
nevis real estate
st kitts and nevis real estate. my website is www.sknlistings.com0 -
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period?
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period via either Organic / Paid means? Thanks
Competitive Research | | jaycfc1230070