Welcome to the Q&A Forum

Moz can't crawl his root domain (i tested using page grader) or any other URLs on his site,

So while the special chars might not help, i dont think thats the issue here :). Atleast, not the only one.

https://moz.com/researchtools/on-page-grader?keyword=preguntas&url=https%3A%2F%2Fwww.respuestasparaelhombre.com

Its possible your SSL is the issue, do you know if its an SNI cert? Moz crawler doesnt yet support SNI and a lot of recently issued certs are SNI ones...

Also, if your using Cloudflare, those certs are SNI as well.

http://serverfault.com/questions/506177/how-can-i-detect-if-a-server-is-using-sni-for-https

All reference i can find to this error is firewall issues. You will probably need to check with your hosting providers that they arnt blocking moz.

a similar post made last year:

http://moz.com/community/q/moz-tools-are-returning-url-is-inaccessible

DA on a couple of my accounts sank by 2 ~ 3 in the last update, maybe a minor update to the metrics triggered something...

A few more links about this, that i was able to dig out:

http://live.theverge.com/sundar-pichai-google-mwc-2015/

and the full thing:

http://www.mobileworldlive.com/mwl-extra-sundar-pichai-svp-of-products-at-google

Hi James,

Thanks, I know it's not a 1 day fix, it's just somewhat frustrating to find that Moz doesn't yet support technology that is fast becoming standard, and thats been around for years. I look forwards to hearing when Moz does have an ETA for this! The Crawler is the main reason I created a Moz account, so i can't emphasize enough how much of an effect not having a long term solution at least announced soon is going to have on my decision to remain a Moz subscriber in the long term.

If it makes any difference, turns out Bing doesn't support it either, so something else to remember as more people start seeing crawler errors / lack of rankings that they may not understand!

http://webmasters.stackexchange.com/questions/77267/bing-and-lack-of-sni-support

And thank you for confirming the issue :).

Examining the server logs yourself probably wont help your understanding of the issue unless you know what your looking at specifically. On the Yahoo note, i have found Slurp to be really bad in the past, but no legitimate bot should be able to bring down a properly configured web server, especially an 'enterprise-level' one.

I would check your .htaccess and apache settings for bad redirects (or web.conf if on windows) before considering banning the bot. Other things to check would be website code or if a bot hits a massive and horribly optimised Database Query for example, that could bring the server down.

Ask IT exactly what the bot did that caused the server to go down, they should atleast be able to tell you that. If not then they need to run load tests against the website itself to try and reproduce the scenario and thus debug the issue, if indeed there is one.

Tl;dr :- Normally bad config or code / queries are to blame for this kind of thing. I'd review that before blocking a bot that crawls hundreds of thousands of other sites without issue.

The main problem here is that i can't disable SNI unless i pay for a business account ($200 USD per website, per month), except for the enterprise account everything else is Pro which is more than enough for my clients (no way i could justify $200 a month just so moz can crawl).

SNI is a technology thats been supported since IE7 and is now standard in all browsers, so still not supporting it now is just a little insane....

But as i said, this is going to be effecting everyone using SSL under cloudflare (which is probably everyone using cloudflare) so could you guys please make this a priority? its not just an edge case issue.

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

Since im here im going to respond to this with a couple of things.

"While Cloudflare offers many optimization services and DDoS protection, many of them break Magento and Wordpress functionality."

I've not had any issue with this. Typically if something does break in WP or Magneto its pretty simple to fix. Personally, i dont run WP sites without cloudflare, its too much of a security issue.

" Implementing caching and Google PageSpeed with Nginx or Apache shows much better performance,"

You should do this anyway....

" Worse though, Cloudflare doesn't allow you to use your own SSL on any plan under $200/moth"

Sure it does, i run 40 + sites on standard Comodo SSL certs on Cloudflare pro without issue.

"If you're a Pagespeed/YSlow nut, they you'll really be upset to learn Cloudflare adds a security cookie to everything, so there is no way to have serve content on a cookie-free domain when using Cloudflare DNS"

Thats a fair point, though perhaps not such an issue in most situations.

"Customer service is also slow and mostly automated, so don't expect great service on unpaid plans."

Again i dont really find this. Most of my accounts are CF Pro but for those that arn't, i dont notice any difference in support level. I know Business and Enterprise plans get faster support but for $200 + a month, i'd expect that anyway.

You may find that you have to wrap the code that gets called when Ajax fires in something to catch the user agent. I.e. if your making an Ajax request to a php script in order to return data, you could wrap that php code in something like this (please excuse the Sudo code):

if(in_array($_SERVER['HTTP_USER_AGENT'], $knownagents){

//known webspider, or blocked agent, return nothing.

return "";

} else {

//not a known spider so continue.

}

?>

Thats very generalised but you get the idea. I put a short list together in JSON format a while back, you can find it here if its of any use: https://www.source-control.co.uk/knownspiders/spiders.php

PM me if you need any more specific help than that with development, hopefully someone else will have a slightly easier way of dealing with this though heh

Nothing much i can add to Martijin's response, you might want to run a scan with Screaming Frog or Xenu to triple check, but 301's should be fine.

Ok, so, my view on this.

In response to livecam's comment, __VIEWSTATE (the code he was refering too) is a base64 encoded form field used in ASP.net to hold data. Its probably not malicous in this instance. see this: http://stackoverflow.com/questions/1350216/what-does-the-viewstate-hold

For me, when i search "prom dress shop" in an incognito chrome window, i dont see either entry on the front page of google, though i expect this is because im searching from the UK.

Reviewing the pages specifically, i can make a couple of suggestions.

1. Check your web.conf file, your main domain may have been hacked and this adjusted to send only search engine to um.org (to hide the hack)
2. it may be that um.org has used Black Hat SEO technique's to massivly raise its profile, this will be short term as google will slap them with loads of penalties pretty quickly.
3. Check your web server specifically for viruses etc. Being an ASP.net site, you'll be hosted on a windows server, running IIS. It will be just as prone to viruses as your windows PC at home (without the proper protection).

If you would like a hand to check your site code specifically, drop me a PM and we can see what we can do. Otherwise, if you have in house developers, they should be able to take a look.

Hi Jarno,

A couple of things to look at, did your site structure change when you switched to responsive? How has the responsiveness been done? (Poor implementation might cause you problems if google can no longer follow links correctly).

Are you testing while logged into a google account? Or logged out in an incognito window in Chrome? (The later will give you a much cleaner result as google adjust search results per user based on viewing habits). Or do you see these changes in Moz specifically?

Something to remember, mobile responsiveness has absolutally no known effect on desktop searches, only those carried out on mobile devices and so won't be a ranking factor for a large number of your keyword searches.

I would look at what going responsive changed though. If you like, PM me a link to the website (or link it here) and I’ll take a look to see if i can spot anything with the implementation.

Hi, Sorry i missed your response,

I think in general terms rather then me listing everything here, you should follow Patrick's advice as a start and come back once your done with all that as you will have a much improved base for further adjustment.

SEO is not a quick process, it can take months to get a good foundation if not longer, so keep that in mind when your making changes.

"We went from Drupal to WordPress"

That seems like a very backwards move to me, that should be the other way around (My personal opinion anyway)

On the 404s though, any URL changes will cause problems, you will need to add in redirects to deal with those.

On a drupal specific note, if your old site was using the 'URL Redirect' module, you may have had a number of redirects in place in the database, these wouldn't have been in your .htaccess file and will no longer work after moving to wordpress.

I dont have a huge amount to add to this, other than in my experiance a lot of agencies tend to be 2 or 3 years behind in general practises, especially the smaller ones.

"The times of looking to rank numero uno for one term are long gone." is completely correct though, so be very careful about messing up other key words etc.

A couple of improvements i can suggest straight off:

You need to redirect all non-www traffic to www. or vice versa. Currently so far as google can see, you have 2 different websites, one on http://bassilimousineservice.com/ and one on http://www.bassilimousineservice.com/

Your URL structure needs to be improved,for example: http://www.bassilimousineservice.com/index.php/services/rates.html

Should be something like: http://www.bassilimousineservice.com/services/rates

Thats a good point actually, being UK based, i tend to forget States in the US is how you're likely to do location redirects (that and i read Colorado as Canada... my bad).

The short answer i think is not directly. You could use the users IP address and something called mode_geoip in order to handle the redirects, it takes a bit more configuration and setup though.

Im seeing lots about redirecting on country, not all that much on redirecting on state, but a good starting point is perhaps this Stackoverflow thread. It's probably something you'll need to ask a developer to do.

http://stackoverflow.com/questions/9838344/how-to-redirect-domain-according-to-country-ip-address

There are a number of solutions, perhaps the easiest and best solution is to add Cloudflare to your site, and take a look at this blog:

http://netjunky.net/redirect-web-traffic-based-on-country-origin/

(this assumes your website runs on PHP)

No worries Did you make a decision in the end?

Just to add to this, if your subdomain has more than /blog on it, and you only want to block /blog, change Dirk's robots.txt to:

User-agent: Googlebot
Disallow: /blog

or to block more than just google:

User-agent:*
Disallow: /blog

Redirect 301 /green/red.cgi/blue/ http://www.newsite.com/summary-page/

is the old Redirect format, assuming your server is up to date (ish) you should be using something like:

RewriteRule ^/green/red.cgi/blue(.*)$ http://www.newsite.com/summary-page/ [L,R=301]

Looks like you've been hacked then,

You'll need to clean out your code and content and make sure wordpress and all of its plugins are the latest version. Unfortunatly there is no 'Safe' way to recover from this without reverting to a backup from before the hack.

https://codex.wordpress.org/FAQ_My_site_was_hacked

EDIT::
I deleted my other response because i fail at reading :P, sorry about that

What Patrick Delehanty has said is completely correct. You need to edit your pages, menu's etc in order to remove or fix the links that have 404's if they are part of your website.

If the links are external websites linking to you, you need to setup redirects in your .htaccess file for them. The later is considerably more technical so if your not confident with it, ask your developer to do it for you.

If your not sure which it is, could you provide is with an example of a link that shows as a 404?

You can also use Xenu to check the links on your site (not the ones linking to it), i find it works better than the Broken Link Checker plugin.

Actually a good question. I havent stopped to look at this one before, following OSE for me the links are similar but definitly not the same.

I cant help with this one, but i'd love to know the answer!

Got to love that referal spam!

I've not seen this one yet in my reports, kind of disappointing really :(.

So far as i can see, you should have no problem with this.

It looks like they want to hide the subdomain from indexes so as long as robots.txt. is properly configured, you should have no issue at all.

to be honest i havent had any good experiances of them. I agree though, that was a load of rubbish.

If you do revert to a backup, make SURE you update everything bnefore making the site live.

With wordpress, you must keep it up to date, that thing is a security minefield.

Are you getting errors?

If you just dont see any difference, that will be because filters dont apply retrospectivly to data. They will only work for new data after the filter is added. So it might take some time before you see the changes.

My Take on this:

-my client's site has a .com domain where most others have .co.uk

Google does rank regionally, .com is great, .co.uk would probably be better for UK specific listings. (301 from .com .co.uk would be your best bet)

-most sites on the first page have either jewellery, wholesale or both in their domain names

This is less of a thing than it used to be. Google seems to be starting to favour brand identity over direct keyword rich domains (EMD) - http://moz.com/blog/are-exact-match-domains-in-decline

-relevance of linking istes could be playing a part here. My client has links from wholesale sites but few from jewellery-related sites

Probably not going to be a penalty from this, but relevance might make a difference to just how much extra juice those links provide.

-within my client's link profile there are very few, if any exact anchor text links for the term 'wholesale jewellery'

Probably some effect from this, how much though im not sure (Exact anchor text isnt something i've had to look into before)

-legacy of a penalty could be making ranking progress that much more difficult?

Depends how long ago this was. So long as whatever caused the penalty is fixed, this is probably not an issue now.

Hi shr109,

I've sent an email over so you have my address. Please let me know if it doesnt come through, we're recovering from a couple of email issues this end (infected web server in the same IP Subnet as our email server got us blacklisted), it might have ended up in spam!

Thanks,

As with any backlinks you dont want, if you cant request removal you should disavow them in GWMT

Hmm i cant see anything funny going on, this is essentually what googlebot should see when viewing the page. If you'd been hacked i would expect to see the wrong description here.

http://www.evolvedwebsites.com.au/googlebot/view.php?url=protectachild.com&keywords=evolved+websites%2C+website&submit=VIEW+AS+GOOGLEBOT

If the site has been live a week, you might just be in a bit of a blip. Best thing i can suggest is request a reindex / new crawl in GWMT and wait it out. Google can take 3 ~ 4 weeks to update listings properly when meta information changes (in my experiance atleast)

Ah, completely missed the attachement.

I'll poke about and see if i can spot anything

Can you provide exactly what your seeing? (as in the full meta description that you think is incorrect and what it should be?)

This is what i see when viewing markup on your homepage, but with the above info i have no idea if thats correct or not

Im not sure your really in the right place for that kind of question (its not really SEO related, more technical web development)

To point you in the right direction though, stackoverflow is a good resource for development questions:

http://stackoverflow.com/questions/14520036/how-to-detect-user-is-on-a-mobile-device-and-offer-app

I think your pretty much spot on. Google -can- crawl queries but they wont rank very well at all.

Your best bet will be to change: (just reading into this that looks like a 'get all jobs' query)

http://www.prospect-health.com/Jobs/?st=0&o3=973&s=1&o4=1215&sortdir=desc&displayinstance=Advanced Search_Site1&pagesize=50000&page=1&o1=255&sortby=CreationDate&o2=260&ij=0

to just

http://www.prospect-health.com/Jobs

There are loads of ways to remove the query string and keep the functionality, depending on the software powering the site, personally i'd fix up the search to POST data so that it keeps the url clean and add in the appropriate routes to create the path.

I have some experience of job search sites (having worked on a couple of the largest in the UK) and breaking URLS down something like this seems to work best. (depending on the data you have obviously)

<domain>/jobs/<location>/</location></domain>

You could also take a look at how other job sites structure their URLS, (monster.co.uk, targetjobs.co.uk, jobsite.co.uk etc)

Let me know if you need a hand and i'll see if i can be more specific. (you'll have to tell me what its running on though)

EDIT:: You should force lower-case on urls as well. Caps wont effect google but they arn't user friendly (miss types etc)

Glad to hear thats all fixed! Though i will say thats a very slow response time for any development / hosting company typically i would expect a maximum response time of 8 hours.. We try to keep it under 2 heh.

But yes, glad thats working for you now

In that case that would probably be the best place to start. If you want any evidence for a missing A record, heres a DNS checking tool (it currently throws an error because it can't find an A record).

MXToolBox

Let me know what they say

Ok, so the file is working, thats a good start!

Looks like we need to go back a step in the request process then. Do you have access to the DNS settings for the http://atp-instrumentation.co.uk domain? If so, please could you check if there is an A record set for it? (looks to be registered through Civica UK Ltd - Whois Report)

What i suspect might be the case is that you're missing an A record for atp-instrumentation.co.uk but that there is one set for the www.atp-instrumentation.co.uk.

I've run a couple of tests against the domain DNS and i get nothing back for the non-www address, which is what suggests that we're not even making it as far as your servers.

To set the A record, you'll be looking for something in your control panel for 'DNS settings' or maybe 'Host Records', you should see in there ether an option to select A record settings, or perhaps a dropdown with things like A, AAA, CNAME etc. You need to:

• select 'A'
• In the domain box type: atp-instrumentation.co.uk
• In the IP box, type: 82.118.110.42

Hopefully that makes sense. If your at all unsure, let me know and i'll do what i can to help more specifically. Domain control panels are so different for each provider its difficult to provide direct instructions without knowing what your panel looks like

Just a note here, the Redirects do seem to be working, so it looks like the .htaccess file IS being loaded.

Please check anyway, its possible that the redirects are comming from another location if they have been set elseware as well. From a (very quick) look, the file seems to be formatted correctly so no obvious reason for the www redirects to not be working...

Hmm ok,

before i do anything else, we need to make sure that the .htaccess file is actually being loaded. To do that, we need to break it for a second.

at the top of the htaccess file, put something like (specifics dont matter here)

THISisInValid666

What we're trying to do is put some invalid text into the htaccess file so that the site breaks when it loads. The idea being that we can confirm that the .htaccess file is actually being used. So if you put that in and the site throws a 500 error (when navigating to it with or without www) we can confirm that changes we make should work.

If the site continues to load without issue then we know that the .htaccess file isnt in use so we need to look at server configureation directly (specifically the AllowOverrides settings)

Once you have confirmed if it does / doesnt break, remove the line again

oh and just a note, no actual redirection is happening, the plugin is simply adding a query string to the end of the URL when the page loads.

I don't believe so, or at least any effect should be minimal. a couple of links on the subject:

http://webmasters.stackexchange.com/questions/15393/is-a-url-with-a-query-string-better-or-worse-for-seo-then-one-without-one

http://moz.com/community/q/link-building-does-query-string-matter

The general consensus seems to be that so long as you're using rel=canonical your fine.

Personally i would suggest removing the silly query string as I've said above, as if nothing else it doesn't look very nice for users. (in the past we've had clients say users are asking if they've been hacked etc.. as an example)

We have a couple of major law firms as clients who do exactly this, run one website for each sector that they operate in and then have a central corporate site.

At the end of the day Google wont penalize you so long as the content is unique.

You're correct, you can make it a little more generic though, without seeing all of your .htaccess file, try this:

Replace:

RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]

With:

RewriteCond %{HTTP_HOST} !^www. [NC]
RewriteRule ^ http%{ENV:protossl}://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

This is what could be called a wildcard redirect in that a direct copy paste should work for you with no need to edit. (you dont have to manually add in the correct domain name)

What it does:

• First it checks to see if the requested url has a www in it
• if it does -not- it then runs the rule, otherwise it ignores it.
• The rule first checks for http or https. Then adds in the www. followed by the domain and tld, finally adding the URI (/somepage/page for example).
• the L in square brakets means do not process anything else in the htaccess file
• the R=301 means that it will be a 301 (perminant) redirect.

If that still doesnt work for you, paste up your full .htaccess file, or you can send it to me directly if you'd rather and i'll take another look

As has been suggested you can request the removal of pages in GWMT and you should keep any wordpress site and plugins up to date.

To add to this, you might want to look at something like Cloudflare as an extra layer to protect your clients site. We've been using it for a year now and its made a massive difference, both to performance and security.