Sorry, yes, it should crawl the links - they used to do that.

But you can prove it to yourself, by doing what I said - and then report back.

Yes and you can test it by creating a page that is linked from nowhere else and then check your logs or analytics

It could be that the older wordpress had a setting that this new version has decided to ignore. This is typical of programmers!

The next possibility is to look in the database, but the options part of the database is hard to read.

Another idea is to look in the code of the the theme and hack it, so it is permanently index, follow or just remove that altogether.

Maybe someone else has a better idea?

Alan

Yes, I would have them indexed in that case too.

I think it is the categories that are noindex.

I think this is an 'All in one SEO pack' adjustable feature.

In the  setup for that, look for a checkbox:

"use noindex for categories"

uncheck that if it is checked.

If that isn't it, I don't know the answer

Thank you Mark

Nice looking site!

Your front page is index, follow.

Index pages are noindex, follow

Final pages are index, follow

I do something very close to this on my site.

Often, index pages are useless to searchers, because the index page changes so quickly that by the time the info gets into a search result, the information is no longer on that page, and the searcher will either just click away, cursing you and your site, or they will go looking through a few index pages and then curse you when they can't find what they wanted.

So I agree with the way you're doing that - if it is the case that the content changes quickly. If the index pages are just collectors of groups of items, then index, follow would be better, provided that you have enough text on the page to make it worthwhile.

As to how to make that happen, it isn't obvious.

I need to upgrade some of my sites to 3.5.

It could be that you have a plugin or a "custom field" that sets the index follow.

I suggest you edit a post and a page and scroll down to see if you have a field that handles it, such as "robotsmeta" that is set to "noindex,follow" for those pages

Hello Mark.

Please send me a bitly shortened link to your website so I can see what you are seeing

It probably isn't your robots file.

First try this.

In the Admin section, you should see "Settings" on the left navigation

Click that and you should see "Privacy"

Click that and you should see two radio buttons

<label for="blog-public">I would like my blog to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers</label>

<label for="blog-norobots">I would like to block search engines, but allow normal visitors</label>

Obviously, choose the top one and save it.

Then, refresh your front page or inner pages and look in the code to see if it still says noindex

If you have a cache, you will need to flush it.

Your sitemap should only list pages that actually exist.

If you delete some pages, then you need to rebuild the sitemap.

Ditto if you delete them and redirect.

Google is always lagging, so if you delete 10 pages and then update the sitemap, even if google downloads the sitemap immediately, they will still be running crawls on the old map, and they may be crawling the now-missing pages, but haven't shown the failures in your WMT yet.

If you update your sitemap quickly, it is possible they will never crawl the missing pages and get a 404 or 301.

(but of course, there could be other sites pointing to the now-missing pages, and the 404s will show up elsewhere as missing)

I am always checking, adding, deleting and redirecting pages, and I update the current sitemap every hour and all the others are rebuilt at midnight every night. I usually do deletions just before midnight if I can, to minimize the time the sitemap is out of sync.

Are there any site quality expert around?

Someone with actual experience and not guesses?

This is important to me, I may only have a few days to get this right.

I switched back to the original account, so ads are showing now, but I need to fix and switch back.

The big problem is there is no way to tell what the google testers saw, that they didn't like, or what they thought they saw.

Google policy doesn't include external ads in the count of 3.

I have removed the leaderboard, in case that was the problem, but if that was the problem, why didn't they say anything about that leaderboard over the past 5 years, since the google adsense team advised using it?

Where did you see screaming bananas?

Thanks Jarno.

This is an online newspaper, with 3 editors and 200+ writers/journalists and other contributors. We also have contracts with the major press release companies to process and display their press releases.

Obviously, it is not a chat site. It does not and never has encouraged anyone to click ads. It obviously doesn't sell anything illegal.

The stories have two google ads, one in the top left body and one skyscraper.

There is also one skyscraper ad below the fold.

The press releases (for business readers) have the same two google ads, plus a leaderboard ad (I could remove that, but these were placements advised by google a few years ago)

They also have a single skyscraper ad below the fold and a Chitika ad under the story.

I'm not going over to webmaster world - I'm a long-term member of SEOMoz for a reason - there are many highly knowledgeable people here. I just hope they haven't all gone on vacation this week

Cameron, I always recommend businesses link to their press releases. Doing so provides value in different ways. First, if there are no links and no references to it, it basically has no value, Second, you get a little bit of third party validation from it. Third, you can get visitors from it and brand recognition. I know of releases that we published in 2005 that still get hundreds or thousands of visitors each month, because they have incoming links. Don't base all your decisions on what immediate link juice you might get from an action you take, because it may hurt you some time in the future. And yes, prlog is good. I have some I will send there this week.

There has been a lot of discussion about press releases.

You can't only do things that you consider give you pagerank power.

(well you can, but that's not the real world)

If press releases were dead, why do the big PR companies still charge $300+ for a press release and issue 1000 to 2000 of them every day?

Press releases work. They get your name out there, and if other sites pick them up and publish them, you get access to their readers. If a journalist or blogger picks it up, you get more coverage.

Yes, my site publishes press releases, but thats not why I'm  saying this. I'm telling you because it works. If it didn't work, I would stop. I have press releases that get hundreds of readers every day, so if you have a story to tell that is worth reading, you could get those readers.

Honesty pays, I think. Google knows pretty much what you did. The engine guesses, but when a live body checks on your request, it's not likely you will fool them. They know which sites sell paid links. If someone else also paid for links from that site and they already did a reinclusion request and said they bought links from that site, you paint a target on yourself if you don't own up to it - unless you really didn't buy the links.

You could wait up to a month for any response, going on past history. Be sure you're disclosing everything you know about in one request. Don't think you can keep sending them in because, from what I've read, they will eventually stop listening.

I can't help you with protocol, other than to say spell it all out because they can't read your mind and expecting them to guess probably won't work in your favor.

Hi EugeneF

The problem with tel: is that most browsers don't know what to do with that, so they see it as a URL relative to your site, and when you try http://www.google.com/tel:1231231234  there is no such thing and of course, you get a 404 error.

So here is another gotcha that we will all have to cater for in our robots files and .htaccess files.

What you need to do, to handle that, is to detect that you are dealing with a smart browser that understands what to do with it, and only display those anchor links to those browsers.

The upside, of course, is that either robots or hopefully, real people are clicking on your phone links.

The downside is that if they are real people and they get a 404 error, you are giving them a bad surfing experience.

Hello Maarten

Those RewriteCond entries are cumulative and it looks like there are missing commands.

Who edited that file last, and what did they change?

The way conditionals work is you set a condition, Then you add a command, then a line break You can add more than one condition and it acts as AND

This page has what look like too many conditions and not enough commands -but it could be ok

Try adding a blank line between the rule entries and the Cond entries (but not between the Cond and the Rule entries)

Here is what to do to test anything like this: Save a copy of the .htaccess Then start editing it Delete everything below ##Start rewrites See if that fixes it. If not, the problem is above or if that fixes it, the problem is below Keep cutting the file in half or adding half until you discover the problem line

It is harder with all those conditionals, I suspect it is the lower block that is the problem

So remove those Cond entries from the bottom up

Your sitemap should only list pages that actually exist.

If you delete some pages, then you need to rebuild the sitemap.

Ditto if you delete them and redirect.

Google is always lagging, so if you delete 10 pages and then update the sitemap, even if google downloads the sitemap immediately, they will still be running crawls on the old map, and they may be crawling the now-missing pages, but haven't shown the failures in your WMT yet.

If you update your sitemap quickly, it is possible they will never crawl the missing pages and get a 404 or 301.

(but of course, there could be other sites pointing to the now-missing pages, and the 404s will show up elsewhere as missing)

I am always checking, adding, deleting and redirecting pages, and I update the current sitemap every hour and all the others are rebuilt at midnight every night. I usually do deletions just before midnight if I can, to minimize the time the sitemap is out of sync.

Sorry, yes, it should crawl the links - they used to do that.

But you can prove it to yourself, by doing what I said - and then report back.