Welcome to the Q&A Forum

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

I think your pretty much spot on. Google -can- crawl queries but they wont rank very well at all.

Your best bet will be to change: (just reading into this that looks like a 'get all jobs' query)

http://www.prospect-health.com/Jobs/?st=0&o3=973&s=1&o4=1215&sortdir=desc&displayinstance=Advanced Search_Site1&pagesize=50000&page=1&o1=255&sortby=CreationDate&o2=260&ij=0

to just

http://www.prospect-health.com/Jobs

There are loads of ways to remove the query string and keep the functionality, depending on the software powering the site, personally i'd fix up the search to POST data so that it keeps the url clean and add in the appropriate routes to create the path.

I have some experience of job search sites (having worked on a couple of the largest in the UK) and breaking URLS down something like this seems to work best. (depending on the data you have obviously)

<domain>/jobs/<location>/</location></domain>

You could also take a look at how other job sites structure their URLS, (monster.co.uk, targetjobs.co.uk, jobsite.co.uk etc)

Let me know if you need a hand and i'll see if i can be more specific. (you'll have to tell me what its running on though)

EDIT:: You should force lower-case on urls as well. Caps wont effect google but they arn't user friendly (miss types etc)

DA on a couple of my accounts sank by 2 ~ 3 in the last update, maybe a minor update to the metrics triggered something...

I don't believe so, or at least any effect should be minimal. a couple of links on the subject:

http://webmasters.stackexchange.com/questions/15393/is-a-url-with-a-query-string-better-or-worse-for-seo-then-one-without-one

http://moz.com/community/q/link-building-does-query-string-matter

The general consensus seems to be that so long as you're using rel=canonical your fine.

Personally i would suggest removing the silly query string as I've said above, as if nothing else it doesn't look very nice for users. (in the past we've had clients say users are asking if they've been hacked etc.. as an example)

Hi Jarno,

A couple of things to look at, did your site structure change when you switched to responsive? How has the responsiveness been done? (Poor implementation might cause you problems if google can no longer follow links correctly).

Are you testing while logged into a google account? Or logged out in an incognito window in Chrome? (The later will give you a much cleaner result as google adjust search results per user based on viewing habits). Or do you see these changes in Moz specifically?

Something to remember, mobile responsiveness has absolutally no known effect on desktop searches, only those carried out on mobile devices and so won't be a ranking factor for a large number of your keyword searches.

I would look at what going responsive changed though. If you like, PM me a link to the website (or link it here) and I’ll take a look to see if i can spot anything with the implementation.

You're correct, you can make it a little more generic though, without seeing all of your .htaccess file, try this:

Replace:

RewriteCond %{HTTP_HOST} ^companyname.co.uk [NC]
RewriteRule ^(.*)$ http://www.companyname.co.uk/$1 [L,R=301]

With:

RewriteCond %{HTTP_HOST} !^www. [NC]
RewriteRule ^ http%{ENV:protossl}://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

This is what could be called a wildcard redirect in that a direct copy paste should work for you with no need to edit. (you dont have to manually add in the correct domain name)

What it does:

• First it checks to see if the requested url has a www in it
• if it does -not- it then runs the rule, otherwise it ignores it.
• The rule first checks for http or https. Then adds in the www. followed by the domain and tld, finally adding the URI (/somepage/page for example).
• the L in square brakets means do not process anything else in the htaccess file
• the R=301 means that it will be a 301 (perminant) redirect.

If that still doesnt work for you, paste up your full .htaccess file, or you can send it to me directly if you'd rather and i'll take another look

"We went from Drupal to WordPress"

That seems like a very backwards move to me, that should be the other way around (My personal opinion anyway)

On the 404s though, any URL changes will cause problems, you will need to add in redirects to deal with those.

On a drupal specific note, if your old site was using the 'URL Redirect' module, you may have had a number of redirects in place in the database, these wouldn't have been in your .htaccess file and will no longer work after moving to wordpress.

We have a couple of major law firms as clients who do exactly this, run one website for each sector that they operate in and then have a central corporate site.

At the end of the day Google wont penalize you so long as the content is unique.

Hmm i cant see anything funny going on, this is essentually what googlebot should see when viewing the page. If you'd been hacked i would expect to see the wrong description here.

http://www.evolvedwebsites.com.au/googlebot/view.php?url=protectachild.com&keywords=evolved+websites%2C+website&submit=VIEW+AS+GOOGLEBOT

If the site has been live a week, you might just be in a bit of a blip. Best thing i can suggest is request a reindex / new crawl in GWMT and wait it out. Google can take 3 ~ 4 weeks to update listings properly when meta information changes (in my experiance atleast)

A couple of improvements i can suggest straight off:

You need to redirect all non-www traffic to www. or vice versa. Currently so far as google can see, you have 2 different websites, one on http://bassilimousineservice.com/ and one on http://www.bassilimousineservice.com/

Your URL structure needs to be improved,for example: http://www.bassilimousineservice.com/index.php/services/rates.html

Should be something like: http://www.bassilimousineservice.com/services/rates

Nothing much i can add to Martijin's response, you might want to run a scan with Screaming Frog or Xenu to triple check, but 301's should be fine.

Glad to hear thats all fixed! Though i will say thats a very slow response time for any development / hosting company typically i would expect a maximum response time of 8 hours.. We try to keep it under 2 heh.

But yes, glad thats working for you now

That looks like the ShareThis plugin's doing. Have a look at the issue thread here:

https://wordpress.org/support/topic/code-being-added-to-the-end-of-my-urls

You or your developer needs to add

<code>doNotHash:true,
doNotCopy:true,
hashAddressBar:false</code>

To your site header.

Are you getting errors?

If you just dont see any difference, that will be because filters dont apply retrospectivly to data. They will only work for new data after the filter is added. So it might take some time before you see the changes.

Im not sure your really in the right place for that kind of question (its not really SEO related, more technical web development)

To point you in the right direction though, stackoverflow is a good resource for development questions:

http://stackoverflow.com/questions/14520036/how-to-detect-user-is-on-a-mobile-device-and-offer-app

Hi shr109,

I've sent an email over so you have my address. Please let me know if it doesnt come through, we're recovering from a couple of email issues this end (infected web server in the same IP Subnet as our email server got us blacklisted), it might have ended up in spam!

Thanks,

Just to add to this, if your subdomain has more than /blog on it, and you only want to block /blog, change Dirk's robots.txt to:

User-agent: Googlebot
Disallow: /blog

or to block more than just google:

User-agent:*
Disallow: /blog

Hmm ok,

before i do anything else, we need to make sure that the .htaccess file is actually being loaded. To do that, we need to break it for a second.

at the top of the htaccess file, put something like (specifics dont matter here)

THISisInValid666

What we're trying to do is put some invalid text into the htaccess file so that the site breaks when it loads. The idea being that we can confirm that the .htaccess file is actually being used. So if you put that in and the site throws a 500 error (when navigating to it with or without www) we can confirm that changes we make should work.

If the site continues to load without issue then we know that the .htaccess file isnt in use so we need to look at server configureation directly (specifically the AllowOverrides settings)

Once you have confirmed if it does / doesnt break, remove the line again

to be honest i havent had any good experiances of them. I agree though, that was a load of rubbish.

Just a note here, the Redirects do seem to be working, so it looks like the .htaccess file IS being loaded.

Please check anyway, its possible that the redirects are comming from another location if they have been set elseware as well. From a (very quick) look, the file seems to be formatted correctly so no obvious reason for the www redirects to not be working...

Ok, so, my view on this.

In response to livecam's comment, __VIEWSTATE (the code he was refering too) is a base64 encoded form field used in ASP.net to hold data. Its probably not malicous in this instance. see this: http://stackoverflow.com/questions/1350216/what-does-the-viewstate-hold

For me, when i search "prom dress shop" in an incognito chrome window, i dont see either entry on the front page of google, though i expect this is because im searching from the UK.

Reviewing the pages specifically, i can make a couple of suggestions.

1. Check your web.conf file, your main domain may have been hacked and this adjusted to send only search engine to um.org (to hide the hack)
2. it may be that um.org has used Black Hat SEO technique's to massivly raise its profile, this will be short term as google will slap them with loads of penalties pretty quickly.
3. Check your web server specifically for viruses etc. Being an ASP.net site, you'll be hosted on a windows server, running IIS. It will be just as prone to viruses as your windows PC at home (without the proper protection).

If you would like a hand to check your site code specifically, drop me a PM and we can see what we can do. Otherwise, if you have in house developers, they should be able to take a look.

Ok, so the file is working, thats a good start!

Looks like we need to go back a step in the request process then. Do you have access to the DNS settings for the http://atp-instrumentation.co.uk domain? If so, please could you check if there is an A record set for it? (looks to be registered through Civica UK Ltd - Whois Report)

What i suspect might be the case is that you're missing an A record for atp-instrumentation.co.uk but that there is one set for the www.atp-instrumentation.co.uk.

I've run a couple of tests against the domain DNS and i get nothing back for the non-www address, which is what suggests that we're not even making it as far as your servers.

To set the A record, you'll be looking for something in your control panel for 'DNS settings' or maybe 'Host Records', you should see in there ether an option to select A record settings, or perhaps a dropdown with things like A, AAA, CNAME etc. You need to:

• select 'A'
• In the domain box type: atp-instrumentation.co.uk
• In the IP box, type: 82.118.110.42

Hopefully that makes sense. If your at all unsure, let me know and i'll do what i can to help more specifically. Domain control panels are so different for each provider its difficult to provide direct instructions without knowing what your panel looks like

You may find that you have to wrap the code that gets called when Ajax fires in something to catch the user agent. I.e. if your making an Ajax request to a php script in order to return data, you could wrap that php code in something like this (please excuse the Sudo code):

if(in_array($_SERVER['HTTP_USER_AGENT'], $knownagents){

//known webspider, or blocked agent, return nothing.

return "";

} else {

//not a known spider so continue.

}

?>

Thats very generalised but you get the idea. I put a short list together in JSON format a while back, you can find it here if its of any use: https://www.source-control.co.uk/knownspiders/spiders.php

PM me if you need any more specific help than that with development, hopefully someone else will have a slightly easier way of dealing with this though heh

In that case that would probably be the best place to start. If you want any evidence for a missing A record, heres a DNS checking tool (it currently throws an error because it can't find an A record).

MXToolBox

Let me know what they say

Redirect 301 /green/red.cgi/blue/ http://www.newsite.com/summary-page/

is the old Redirect format, assuming your server is up to date (ish) you should be using something like:

RewriteRule ^/green/red.cgi/blue(.*)$ http://www.newsite.com/summary-page/ [L,R=301]

So far as i can see, you should have no problem with this.

It looks like they want to hide the subdomain from indexes so as long as robots.txt. is properly configured, you should have no issue at all.

Got to love that referal spam!

I've not seen this one yet in my reports, kind of disappointing really :(.