Why can no tool crawl this site?
-
I am trying to perform a crawl analysis on a client's website at https://www.bravosolution.com
I have tried to crawl it with IIS for SEO, Sreaming Frog and Xenu and not one of them makes it further than the home page of the site. There is nothing I can see in the robots.txt that is blocking these agents.
As far as I can see, Google is able to crawl the site although they have noticed a significant drop in organic traffic.
Any advise would be very welcome
Regards
Danny
-
I would look into finding a method to redirect via your server rather than with javascript. This will ensure that bots can properly crawl your site.
I would also add hreflang tags which should help Google with the multiple language versions of the site.
Also in the short term you may want to do something like add a link or a delayed meta refresh just in case someone either has javascript disabled or is using script blocking extensions. This will make sure they at least see something instead of a blank page.
-
Really helpful and much appreciated - many thanks!
Danny
-
Yes that's what I said CleverPhD, I just couldn't type that fast today.
Only joking Thanks for expanding on the subject.
-
To expand on Dean's point.
If you look at the source code on https://www.bravosolution.com/ you get a bunch of JavaScript (shown below). It is basically looking at the users location and the sending them to the appropriate version of your website based on country. This is why here in the US we are sent to https://www.bravosolution.com/cms/us
Many spiders/tools (and Googlebot was not really good at this until recently) are not good at (or do not do any) crawling and executing on JavaScript so they get stuck when they hit your home page.
If you want to evaluate any of your localized sites, just run those URLs through various tools like screaming frog etc. You would then ask, "Well, how do I know that my main https://www.bravosolution.com is working properly for SEO?". I don't have as much background in how to optimize for international SEO, but you can do a several things to start with.
-
Google anything having to do with Aleyda Solis and International SEO. She posts a lot of stuff here at Moz and is pretty sharp on this stuff. There may be a more appropriate way to redirect international clients from your main page that how you are executing.
-
Run your home page through Google Webmaster Tools under Crawl > Fetch as Google. See what the page looks like
-
Double check your robots.txt to make sure you are not blocking any folders that would contain a JavaScript library. Based on the code below, I do not see you referencing any external libraries, but if you are dependent on JS to send Google, it would be worth having your developer check things
-
As with everything on what to do, it all depends. If all of your local country sites are independently ranked and successful, this main website may nor may not be doing you any favors currently if it is just a pass through with no domain authority to start with. Spend time on step #1 to see if there is anything else worth doing.
Cheers!
name="description" />
-
-
Yes, it should redirect you to the correct country version based on your IP. But I still can't crawl the site from the home page
-
Cheers Bryan - much appreciated. It's driving me crazy!
-
Hi Danny,
Have you looked at the site via http://web-sniffer.net/
It would appear that the home page is just a JavaScript redirect.
I was redirected to https://www.bravosolution.com/cms/us which then could seen via Sreaming Frog.
The reason for my (default) redirect is given by web-sniffer as:
DEFAULT CORPORATE if ( path == '' ) { path = '/cms/us
-
Interesting. I verified the robots file and tried running through screaming frog... nothing. I' will dig into this with my dev team to try and get you an answer asap.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Launch of improved site
Hi, Just want to ask you guys if i have missed something in my planning. We have done a migration from Ithemes Exchange to woocommerce. The complete migration are done on our dev server. It has an exakt setup as our live one. My plan is to change our live version with a backup from our migrated and finished site from our dev site. All of our product links will be intact with accept from some that we have combined in to new ones, the ones that are changed has been redirected with a 301. Will this way of launching our site effect our ranking/seo in some way? Thankful for any thoughts about this one! // Jonas
Technical SEO | | knubbz0 -
Can You Use More Then One Google Local Rich Snippet on a single site/ on a single page.
I am currently working on a website for a business that has multiple office locations. As I am trying to target all four locations I was wondering if it is okay to have more then one Local Rich Snippet on a single page. (For example they list all four locations and addresses within their footer and I was wondering if I could make these local rich snippets). What about having more then one on a single website. For example if a company has multiple offices located in several different cities and have set up individual contact pages for these cities, can each page have it's own Local Rich Snippet? Will Google look at these multiple "local rich snippets" as spaming or will they recognize the multiple locations and count it towards their local seo?
Technical SEO | | webdesignbarrie1 -
Can the Hosting location of image files have a negative effect if on the developers own media server rather than on client site server ?
Hi Can the Hosting location of image files have a negative effect if on the developers own media server as opposed to on the actual websites server ? In the case i'm looking at the image files are hosted on a totally separate server (a media subdomain of the developers site server) from the subject sites dedicated server. Will engines still attribute the properties of files hosted in this manner to the main website (such as file name or should they really be on the subject sites server own media folder ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Why my site is not indexing in google
In google webmaster i have updated my sitemap in Mar 6th..There is around 22000 links..But google fetched only 5300 links for long time...
Technical SEO | | Rajesh.Chandran
I waited for 1 month till no improvement in google index..So apr6th we have uploaded new sitemap (1200 links totally)..,But only 4 links indexed in google ..
why google not indexing my urls? Is this affect our ranking in SERP? How many links are advisable to submit in sitemap for a website?0 -
Why does my site rank so badly
its my turn to ask the interminable question why does my site rank so badly? site is: marriagerecords.org.uk. it was #1 for 'marriage records' on google for about 6 months. then it was 5th to 10th for about 2 months. now it is nowhere for this phrase and anything else, none of the pages I have written rank for anything. I have spent hours upon hours researching original content and I have got some great backlinks from sites like wrexham.gov.uk and somerset.gov.uk (some dont show in opensiteexplorer yet). im guessing im over-optimizing something but i'd love some concrete fixes if anyone could suggest any. thanks, tom
Technical SEO | | lethal0r0 -
Crawling and indexing content
If a page element (div, e.g.) is initially hidden and shown only by a hover descriptor or Javascript call, will Google crawl and index it’s content?
Technical SEO | | Mont0 -
Whats the best tools for site architecture
Look for tools that can visualise a sites architecture (idealy automated). Also looking for tools that can visualise internal linking sturures
Technical SEO | | Motionlab0