Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
-
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens when we change redirects to pass linkjuice to different pages from backlinks? Google's stand?
Hi Moz community, We have employed different pages (topics) at same URLs for years. This has brought different backlinks to same page which has led to non relevancy of backlinks. Now we are planning to redirect some URLs which may improve or drop rankings of certain pages. If we roll back the redirects in case of ranking drop, will there be any negative impact from Google? Does Google notice anything about redirect changes beside just passing pagerank from backlinks? Thanks
Algorithm Updates | | vtmoz0 -
Google SERPs changes
Hi I wonder if anyone knew of any changes to the Google SERPs appearance in August 2015? We dropped in over a thousand visits to the homepage on brand so I wanted to find out why. Also, our DA went from 36 to 34 - does Google panda affect domain authority at all? Thank you
Algorithm Updates | | BeckyKey0 -
Any important change in SERPs between Nov 17th and Nov 20th?
I've noticed important changes in visibility for some websites, between Nov 17th and Nov 20th. Also some of the webs that monitor SERPs have detected similar stuff (Including Mozcast). Do you know if an important change in SERPs took place during those days?
Algorithm Updates | | emerlo0 -
Google Latest Algorithmic Change about Https & Mobile Friendliness
How effective did it prove for anyone with the latest algorithmic change google search engine made for being mobile friendly and using https (valid ssl certificate). I see a good change being made under the ecommerce category for sites being used for online shopping. Let me know if anyone observes a major difference.
Algorithm Updates | | mozexpone0 -
Adding the link masking directory to robots.txt?
Hey guys, Just want to know if you have any experience with this. Is it worthwhile blocking search engines from following the link masking directory.. (what i mean by this is the directory that holds the link redirectors to an affiliate site: example:
Algorithm Updates | | irdeto
mydomain.com/go/thislink goes to
amazon.com/affiliatelink I want to know if blocking the 'go' directory from getting crawled in robots.txt is a good idea or a bad idea? I am not using wordpress but rather a custom built php site where i need to manually decide on these things. i want to specifically know if this in any way violates guidelines for google. it doesn't change the custom experience because they know exactly where they will end up if they click on the link. any advice would be much appreciated.0 -
Site titles / descriptions change - Google Algo Change ?
Hello, During the weekend 4 of our sites automatically changed their search titles and descriptions at the same time.
Algorithm Updates | | lordish
They are not picking up the real pages: Title, Description. Our ranks are dropping because of this. can you please tell if it happened to you as well or if you recognize a problem here? sites:
http://www.robinhoodbingo.com
http://www.gossipbingo.com
http://www.moonbingo.com in the attached examples:
for the kws searched - the results show different titles and descriptions. results for these pages:
moon bingo - http://www.moonbingo.com
mobile bingo - http://www.robinhoodbingo.com/skin/mobile.php rhMzURw.png 2tRL5dZ.png0 -
Title changed in local pack, unchanged in local plus?!
Google seems to have pulled the title from the homepage and put that as the title in the local pack in the SERP for my targeted keyword. The local plus page title remains unchanged. Any way to influence this back to the way it was? The local plus title looks much better in results (even though it's just the brand name (which is the same as the domain name) and not the city + industry).
Algorithm Updates | | Mozzin1 -
Does this mean that exact keyword phrase anchor text is not the dominating ranking factor anymore for serps?http://insidesearch.blogspot.com/2011/11/ten-recent-algorithm-changes.html
Does this mean that exact keyword phrase anchor text is not the dominating ranking factor anymore for serps? http://insidesearch.blogspot.com/2011/11/ten-recent-algorithm-changes.html If so what is the new most important factor?
Algorithm Updates | | AndrewSEO0