Can't crawl website with Screaming frog... what is wrong?
-
Hello all - I've just been trying to crawl a site with Screaming Frog and can't get beyond the homepage - have done the usual stuff (turn off JS and so on) and no problems there with nav and so on- the site's other pages have indexed in Google btw.
Now I'm wondering whether there's a problem with this robots.txt file, which I think may be auto-generated by Joomla (I'm not familiar with Joomla...) - are there any issues here? [just checked... and there isn't!]
If the Joomla site is installed within a folder such as at
e.g. www.example.com/joomla/ the robots.txt file MUST be
moved to the site root at e.g. www.example.com/robots.txt
AND the joomla folder name MUST be prefixed to the disallowed
path, e.g. the Disallow rule for the /administrator/ folder
MUST be changed to read Disallow: /joomla/administrator/
For more information about the robots.txt standard, see:
http://www.robotstxt.org/orig.html
For syntax checking, see:
http://tool.motoricerca.info/robots-checker.phtml
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/ -
For anyone wondering; The answer above by Ecommerce Site (odd name btw) works - 21-Nov-2016.
-
This is the best I could find to so someone who had a similar problem with Joomla-
"In the premium version you can slow down the crawl rate under 'speed' in the configuration. In the free lite version, you can crawl the site and then right click on any URLs with a 403 response and press 're-spider'. The server will generally then allow you to crawl these pages (and return a 200 ok response) as you're not requesting too many at once, so you might have to re-spider them individually."
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can’t put a finger on, what is causing 12 year domain, SEO optimized and decent link profile to rank lower than other less superior domains.
Can’t put a finger on, what is causing 12 year domain, SEO optimized and decent link profile to rank lower than other less superior domains. I have dissected the site and link, content, etc profile using ahrefs tools, still no luck, and unfortunately they do not have a community to ask anyone opinion. Hoping someone on Moz will be able to provide me with a secondary opinion or something I obviously missing here. Looking for any constructive feedback/professional opinion with fresh look on what maybe the cause of our down rankings and what may be a cause of it. Any feedback is very much appreciated. Search Term: 3030 aventura condos / One of our link samples (SE Position #6): https://goo.gl/FbYj4V Competing Domains (SE Position #1): https://goo.gl/fLPKX5 Competing Domains (SE Position #2): https://goo.gl/GqXGse
Intermediate & Advanced SEO | | Im_Jake0 -
EComm Sites that Don't Display Pricing
I've got a client that only shows pricing if a user is logged in - they're B2B and only sell at a wholesale level. The site is massive, has been around for about a decade, and has had an active SEO campaign for years. They've been losing ground on top ranked keywords, primarily in the 1-2 spots, rest of the first page remains strong and actually improves regularly.My hunch is that Google recognizes the inability for anyone to make a purchase on the site. As a result, they're realizing that the searcher intent doesn't match the actions that can be taken on the site and are bumping them down. Has anyone seen a similar situation or have any evidence to suggest my hunch is correct?
Intermediate & Advanced SEO | | LoganRay0 -
One of my Friend's website Domain Authority is Reducing? What could be the reason?
Hello Guys, One of my friend's website domain authority is decreasing since they have moved their domain from HTTP to https.
Intermediate & Advanced SEO | | Max_
There is another problem that his blog is on subfolder with HTTP.
So, can you guys please tell me how to fix this issue and also it's losing some of the rankings like 2-5 positions down. Here is website URL: myfitfuel.in/
here is the blog URL: myfitfuel.in/mffblog/0 -
Would you redirect Website A to Website B, when Website B is in the middle of a HTTP=>HTTPS migration?
Hey guys, I'm curious on your thoughts around this scenario... Website A: 35,000 monthly pageviews 1,000 pages 375 root linking domains currently HTTPS focused on one topic weak rankings for competitive keywords Website B: 3M monthly pageviews 32,500 pages 3,500 root linking domains started HTTP to HTTPS migration 1 week ago. 1/3 of pages indexed as HTTPS. focused on many topics strong rankings for competitive keywords Requirement: I want to have a reliable read on how Website A's keyword rankings change after redirecting it's pages to Website A. This post-migration analysis will be used as a basis to assess the risk of redirecting another website we own that is similar to Website A into Website B. My question: Would you wait until most of the pages on Website B are indexed as HTTPS before doing a 301 of Website A to Website B? Please back up your answer with reasons why or why not 🙂
Intermediate & Advanced SEO | | jeremycabral0 -
SEO Adjustments Where Content Isn't Front And Centre...
So I am wondering what people think for a SEO strategy for sites where (1) the interaction is a one-off event and (2) content is not often shared or something that people want. Specificially regarding two sites this applies to: Site 1 is basically a mortgage site. So customers interact with the site once and then most likely never again once their mortgage is sorted. Mortgages aren't great content pieces and customers don't really read a lot of the content - it's part of the reason loan officers/mortgage professionals exist... Site 2 is also for a one off purchase but it's an embarrassing problem that nobody would share content for because they don't want people to know that they sought help for this. This also makes getting backlinks hard. Also it is a one off purchase, never to be made again... Am interested in how people would adapt their SEO strategies to these circumstances - where content development and promotion is limited...
Intermediate & Advanced SEO | | GTAMP0 -
Want to merge high ranking niche websites into a new mega site, but don't want to lose authority from old top level pages
I have a few older websites that SERP well, and I am considering merging some or all of them into a new related website that I will be launching regardless. My old websites display real estate listings and not much else. Each website is devoted to showing homes for sale in a specific neighborhood. The domains are all in the form of Neighborhood1CityHomes.com, Neighborhood2CityHomes.com, etc. These sites SERP well for searches like "Neighborhood1 City homes for sale" and also "Neighborhood1 City real estate" where some or all of the query is in the domain name. Google simply points to the top of the domain although each site has a few interior pages that are rarely used. There is next to zero backlinking to the old domains, but each links to the other with anchor text like "Neighborhood1 Cityname real estate". That's pretty much the extent of the link profile. The new website will be a more comprehensive search portal where many neighborhoods and cities can be searched. The domain name is a nonsense word .com not related to actual key words. The structure will be like newdomain.com/cityname/neighborhood-name/ where the neighborhood real estate listings are that would replace the old websites, and I'd 301 the old sites to the appropriate internal directories of the new site. The content on the old websites is all on the home page of each, at least the content for searches that matter to me and rank well, and I read an article suggesting that Google assigns additional authority for top level pages (can I link to that here?). I'd be 301-ing each old domain from a top level to a 3rd level interior page like www. newdomain/cityname/neighborhood1/. The new site is better than the old sites by a wide margin, especially on mobile, but I don't want to lose all my top positions for some tough phrases. I'm not running analytics on the old sites in question, but each of the old sites has extensive past history with AdWords (which I don't run any more). So in theory Google knows these old sites are good quality.
Intermediate & Advanced SEO | | Gogogomez0 -
Web developer won't 301 redirect to my new website....what can I do?
I want to come away from our third party web developer and use a new website with different web developers, however the web developer wont 301 redirect the old url's to the new ones. Is he required to do this by law? Is there away of doing the 301 redirects without him? Any advice would be much appreciated. Many thanks,Tom
Intermediate & Advanced SEO | | CoGri0 -
Re-Direct Users But Don't Affect Googlebot
This is a fairly technical question... I have a site which has 4 subdomains, all targeting a specific language. The brand owners don't want German users to see the prices on the French sub domain and are forcing users into a re-direct to the relevant subddomain, based on their IP address. If a user comes from a different country, (ie the US) they are forced on the UK sub domain. The client is insistent on keeping control of who sees what (I know that's a debate in it's own right), but these re-directs we're implementing to make that happen, are really making it difficult to get all the subdomains indexed as I think googlebot is also getting re-directed and is failing to do it's job. Is there are a way of re-directing users, but not Googlebot?
Intermediate & Advanced SEO | | eventurerob0