Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to set up a competitor URL with a language slug for a campaign
Hello, I am trying to set up a competitor with language slug for my (subfolder) website with a language slug. Let's say my website is something like: websiteholding.com/de/website
Competitive Research | | Siir
My competitor is: competitor.com/de When I go to Campaign Settings > Comptetitor Sites > type in competitor.com/de > Hit Save Competitor > Then it shows the saved competitor without the language slug as competitor.com I am not sure if this is a correct method of tracking since for my DE website I would like to track the DE page of the competitor, not their global page. Please correct me if I am wrong and help me out on possible solutions? I am quite new to SEO & Moz , so any help on the topic would be appreciated.0 -
Is it allowed to buy an old website with external links en redirect it to your own site??
Hello, We noticed at MOZ that one of our competitor's websites has increased the number of external backlinks from 5k to 25k within a month. Upon investigating the links, we found that the competitor had acquired an old, unknown website with completely different content and external links in that month. The competitor then redirected that purchased website to their own website, instantly gaining 20k external links. This seems to be against Google's guidelines as it is an extremely unnatural way of link building and should be penalized by search engines. However, the website's Domain Authority (DA) has increased by 10 points in that month, and its rankings have greatly improved on Google. So it appears that acquiring an old website with many external links unrelated to your own website is highly profitable. We try to obtain links honestly, but we cannot reach 25k links naturally. Is there any way to report these practices to Google? Does anyone know how to do this?
Competitive Research | | Femamedia0 -
What is the relationship between inbound links and linking domains?
In doing competitor research, there's a specific competitor that has fewer linking domains, but dramatically more inbound links. How do these two relate to each other? I do have a higher domain authority and more internal followed links with them having more internal nofollow and external (both follow and nofollow) links. I'm trying to understand how they can have 3.7k linking domains and 4.8 million inbound links while I have 4.1k linking domains and only 601k inbound links. What really defines an inbound link and what's there relationship between inbound links and linking domains?
Competitive Research | | grayloon0 -
Rank online sex shop
I have been trying for some time to take the online store to the front page, but it is very difficult to go up in google searches for the keyword ( sex shop ).
Competitive Research | | Constantin2021
Can anyone help me with some tips. Thank you.
https://www.voluptas.ro/0 -
How do you select which keywords to push in SEO?
Hi Guys Selecting the right keywords that a website can realistically rank for is a key to gain top rankings relatively quick. I am just curious to hear how you guys do it (the methodology) when selecting which keywords to push? I mean you need to check the competition for each keyword as well so how to check this quickly to see what we realistically can rank for? Cheers John
Competitive Research | | igniterman751 -
MOZ vs Ahrefs vs SEMRush vs Spyfu and so on
Hello, fellas. I've been trying to find some type of comparison analysis for online SEO related tools and I couldn't find any, which are fresh and overview more than three. I'm interested to see if anybody saw good comparison reviews of such tools, as well as your own thoughts (back your thoughts up with more than just "I like it", please 🙂 The tools I'm interested in comparing are: MOZ Ahrefs Majestic Spyfu SEMRush Raven Wordstream whatever else you use or have in mind
Competitive Research | | DmitriiK2 -
Traffic Data for Competitor Subdomain
Does anyone know of a way to get monthly visitor data for a subdomain that I do not manage? I would normally use SEMRush or Compete, but they only provide domain level data.
Competitive Research | | dsinger0 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0