How a google bot sees your site
-
So I have stumbled across various websites like this:
http://www.smart-it-consulting.com/internet/google/googlebot-spoofer/
The concept here is to be able to view your site as a googlebot sees it. However, the results are a little puzzling. Google is reading the text on my page but not the title tags according to the results. Are websites like this accurate OR does Google not read title tags and H1 tags anymore?
Also on a slighly related note. I noticed the results show the navigation bar is being read first by google, is this bad and should the navigation bar be optimized for keywords as well? If it did, it would read a bit funny and the "humans" would be confused.
-
You need to pull this forum question. That link redirects to a spammy site about "Freeing Syria."
-
The instructions are near the bottom of the page:
In order to use Fetch as Googlebot, you'll need to have added and verified your site in Webmaster Tools. Then, follow these instructions:
- On the Webmaster Tools Home page, click the site you want.
- On the Dashboard, under Diagnostics, click Fetch as Googlebot.
- In the text box, type the path to the page you want to check.
- In the dropdown list, select the type of fetch you want. To see what our web crawler Googlebot sees, select Web. To see what our mobile crawler Googlebot-Mobile sees, select cHTML (this is used mainly for Japanese web sites) or Mobile XHTML/WML.
- Click Fetch.
Once googlebot has fetched your page you'll have a "success" link that you can click on to see what Googlebot saw.
This will be the header, including the server response code and then the html that googlebot received.
What this doesn't tell you is how this was interpreted by Google, of course this is where SEOMoz's on-page reports and crawl stats can help detect errors and way your can improve your on-page optimisation.
-
Thanks, but even under that Fetch as Googlebot link you posted, I don't see how to get an accurate tool of how Google views your site.
-
These sites are good for a quick scan of the contextual formatting of a website, but not for really telling how Google (or any specific search engine) sees your site. That specific one you linked to is horrible.
Google does see title, H1, other headers, meta description and most elements of your site. A more accurate way to see how google sees your site would be to:
- See how the page looks in the index. Type "site:myspecificurl.com" into google for the page you want to see and google will just return the results of what it has in its index. That is how google sees your site. If your site/pages are not in the index, get them in (#2 below).
- Verify through Google Webmaster Tools. In the webmaster tools you can see what pages of your site are being indexed/crawled through google, and you can also request specific pages to be crawled again if you need. This combined with an xml sitemap will usually get pages indexed pretty quick, and then you can verify with the same methodology as i mentioned above.
- Use the SEOmoz pro toolset here and set up a campaign and the tool will tell you if you are missing any title tags or other important on-page elements. the seomoz "bot" crawls similar to google, so that should give you a feel for how it works.
-
A lot of these sites are badly coded garbage. I would ignore these sites.
-
I don't know how far I'd trust such third party sites. Take a look at Google Webmaster Tools. There's a Fetch as Googlebot tool under diagnostics.
Here's the Google help about it::
Google certainly does read title and heading tags!
As far as the navigation bar goes - always think about humans first. Sometimes you can improve relevance by avoiding generic names. Avoid generic terms like "Articles" for instance and replace it with something that better describes the content behind the click "nutrition guide" or "food facts"...
The fact that your navigation is being read first isn't a problem - it's a convention that is hardly going to be penalised by a search engine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows: rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-crawler-104@moz.com) Useragent from the bot IP address: 54.226.73.52 Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com Physical location: United States flag United States, VA, Ashburn We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place. Any clue on what may be going on here?
Technical SEO | | eTundra0 -
Trying to get google to know my site is a magazine site is this wrong
Hi, i have put a line to describe what my site is at the top of my site and i want to know if this is wrong or not. We have dropped frok being number one in google for lifestyle magazine to now number seven. Before we had to redo our site we were number one and then we dropepd to around number four when we finished the site and now we are number seven and i need to try and get back up there. To help google know we are a lifestyle magazine i have put a line at the top of the site and i want to know if this looks out of place and if i should take it down. i need advice on how to get google to know we are a lifestyle magazine and get back in the top five of google my site is www.in2town.co.uk any help would be great
Technical SEO | | ClaireH-1848860 -
How long does it take for Google to index a new site and has anyone experienced serious fluctuations in SERP within 2 weeks after launch?
Hi guys, I have recently launched my ecommerce jewellery site - www.luxuryfinejewellery.com - and noticed some serious swings in SERP over the last couple of weeks. From ranking No 2, 3 and 4 for the keyword 'luxury fine jewellery' on Google.com, the homepage periodically disappears from the Top 50 altogether. I thought it was the Sandbox, as I recently purchased the domain name, within the last 6 weeks, however the fact that it does rank on the 1st page some of the time is a mystery. Has anyone also experienced this? Could you provide some advice on what to expect until the the rankings settle. Thanks in advance, Satbir
Technical SEO | | deluxebydesign0 -
Google Has Indexed Most of My Site, why won't Bing?
We've got 600K+ pages indexed by Google and have submitted our same sitemap.xml's to Bing, but have only seen 100-200 pages get indexed by Bing. Is this fairly typical? Is there anything further we can do to increase indexation on Bing?
Technical SEO | | jamesti0 -
Google Penalty?
Hi, I have recently been asked to help www.mycanvas.ie I have a feeling they have a google penalty. All their Google Keywords have literally dropped out of the Google SERP but they are still shown on Yahoo SERP. I recently did a site:www.mycanvas.ie and the pages are still in google index. The only thing that comes to mind is that the site owner submitted to 380 web directories over a period of 2 months with http://www.directorymaximizer.com/ do you think this could be causing the problem with google? Advise and suggestions are welcomed, thank you.
Technical SEO | | Socialdude0 -
Site revision
our site has complete redesign including site architecture, page url and page content (except domain). It looks like a new site. The old site has been indexed about thirty thousand results by google. now what should i do first?
Technical SEO | | jallenyang0 -
What pages of my site does Google rank as the most important?
If I type site:youtube.com into Google, are the results listed by what Google considers to be the most important pages of the site? If I change my sitemap should this order change? Thanks!
Technical SEO | | Seaward-Group0 -
Site just will not be reincluded in Google's Index
I asked a question about this site (www.cookinggames.com.au) some time ago http://www.seomoz.org/qa/view/38488/site-indexing-google-doesnt-like-it and had some very helpful answers which were great. However I'm still no further ahead. I have added some more content, submitted a new XML sitemap, removed the 'lorem ipsum...' Now it seems that even Bing have ditched the site too. The number 1 result in Australia for the search term 'cooking games' is now this one - http://www.cookinggames.net.au/ which surely is not so much better to deserve a #1 spot whilst my site is deindexed? I have just had another reconsideration request 'denied' and am absolutely out of ideas/. If anyone can help suggest what I need to do... or even suggest how I can get feedback from the search engines what's wring that would be fantastic. Thank you David
Technical SEO | | OzDave0