Googlebot soon to be executing javascript - Should I change my robots.txt?
-
This question came to mind as I was pursuing an unrelated issue and reviewing a site's robots/txt file.
Currently this is a line item in the file:
Disallow: https://* According to a recent post in the Google Webmasters Central Blog: [http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better") Googlebot is getting much closer to being able to properly render javascript. Pardon some ignorance on my part because I am not a developer, but wouldn't this require Googlebot be able to execute javascript? If so, I am concerned that disallowing Googlebot from the https:// versions of our pages could interfere with crawling and indexation because as soon as an end-user clicks the "checkout" button on our view cart page, everything on the site flips to https:// - If this were disallowed then would Googlebot stop crawling at that point and simply leave because all pages were now https:// ??? Or am I just waaayyyy over thinking it?...wouldn't be the first time! Thanks all! [](http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html "Understanding Web Pages Better")
-
Excellent answer. Thanks so much Doug. I really appreciate it! Adding a "nofollow" attribute to the Checkout button is a good suggestion and should be fairly easy to implement. I realize that internal nofollows are not normally recommended, but in this instance, may not be a bad idea.
-
Hi Dana,
When you click on the checkout button - what's the mechanism for taking people to the https:// site. Is it just that the checkout link uses https:// in it's link? Is there some javascript wizardry you're particularly concerned about?
Even though googlebot follows this one link to the https version of the cart, it will still have all the other links on the previous page queued up to follow (non-https) so I don't think this will stop the crawl at that point. It would be a nightmare if googlebot stopped crawling hte entire site everytime it went down a rabbit hole!
That's not to say that you wouldn't want to consider no-following your checkout button. I'm sure neither you, nor google want to the innards of the cart pages to be indexed? There's probably other pages you'd rather Googlebot spent it's time finding right?
My take on the Google blog about understanding Javascript is that the aim is to try and do a better job discovering content that might be hidden by Javascript/Ajax. It's a problem for google when the raw html that they're crawling doesn't accurately reflect the content that is displayed in front of a real visitor.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does personalization that changes meta data display in SERPs impact SEO?
My company has been rolling out personalization at the page level across our site using behavior paths embedding content from cross pathed pages as well as customer journey mapping. The dynamically generated content doesn’t change the URLs. In the SERPs I’m seeing that our title tags and meta descriptions also seem to be dynamically generated even though we have these elements crafted. The way our elements are crafted: Title tag: descriptive Keyword rich phrase | Brand Meta description: Keyword rich, grammatically correct description tied to title tag and page content for consistency. I search a specific URL: Title tag display: Keyword rich phrase | Brand – Brand Meta description display: Random content pulled from the page I search a phrase that includes Brand + keywords in the URL: Title tag display: Title tag we crafted Meta description display: Meta description we crafted I search a phrase that includes Brand + keywords in the title tag: Title tag display: Title tag we crafted Meta description display: Random content pulled from the page Does Google crawl the page and digest the title tag and meta description we crafted? Or is Google going to ding us for having the brand twice, exceeding the length of the title tag, etc.? I have been searching the interwebs, forums and the cosmos, but the only information I’m finding is related to the fact that URLs are changing and how that would impact SEO. That’s not the case for us. Thoughts on how all this is impacting our SEO efforts?
Algorithm Updates | | NStarJM0 -
Any important change in SERPs between Nov 17th and Nov 20th?
I've noticed important changes in visibility for some websites, between Nov 17th and Nov 20th. Also some of the webs that monitor SERPs have detected similar stuff (Including Mozcast). Do you know if an important change in SERPs took place during those days?
Algorithm Updates | | emerlo0 -
How to keep damage low on Google after the change of URL's
Hi Peeps, Hope someone can shed a light on this and show a guidance if possible. We are going to move our sites to shopify and shopify's URL's cannot be customized to match exactly like our current URLs. What steps do I need to take so google knows the URL's are changed. Domain will be the same. Thank you in advanced.
Algorithm Updates | | cemalcebi0 -
Will Ranking Reports be Affected with the new Google Changes?
For example: Raven stopped use of scraped Google, SEMRush data on Jan. 2 Raven stopped offering unauthorized Google SERP rankings and keyword data (a.k.a. scraped Google data) on Jan. 2, 2013. The change included the retirement of the SERP Tracker and the elimination of SEMRush data from the Raven platform. Raven has released new SEO performance reports that make it easy to show clients the impact of campaigns to improve organic traffic. Raven will continue to upgrade reports through the year. We thank the many customers who continue their business with Raven. More details about the SEO performance reports and other recent releases are available Is SEOMoz protected in some way? Or will you have to give up rankings reports too?
Algorithm Updates | | MSWD0 -
SEO updates and rank changes
We have been updating page titles and meta descriptions for a client (not changing ANY links and the content we are replacing is "fluff," no major keywords or any relevant information) yet in the past few weeks, rankings have plummeted. I used the SEOMoz grader to check and make sure we have the keywords in there, in the right places for the updated page source info, and we're getting A's yet for those same keywords, the website is nowhere to be found. For example for the phrase "organic t shirts," we get an A for this page: http://greenpromotionalitems.com/organic-t-shirts.htm but when searching organic t shirts, no Green Promotional Items... Ideas?
Algorithm Updates | | laidlawseo0 -
Are you seeing changes in your sites today? Panda 2.2?
I've heard rumblings of some Panda sites recovering in the last few days and wondered if the talked about Panda 2.2 has been rolled out. My own site (which actually had a significant boost after Panda) has seen a significant increase in traffic today (started about noon EST yesterday) and a nice increase in Adsense revenue as well. How are your sites doing?
Algorithm Updates | | MarieHaynes1 -
Rankings changing based on location within a country... normal?
I recently had a satellite office across the country come to me and say that they couldn't find us on Google, based on a number of keywords they were searching on. I thought that isn't right... I know we rank for those terms. So, I did a search here, and there we were for those very terms, and ranking quite nicely. Sooo, what's going on there? I know there are variations from Google.com to Google.ca in terms of ranking. But within Google.ca I've not seen this before. Can anyone shed some light on that?
Algorithm Updates | | atcosl0 -
Changing Wordpress Permalink Structure, 301s, and Possibility of Rank Loss?
I have to change the permalink structure in wordpress, as using /%postname%/ in conjunction with a couple thousand pages triggers verbose rewrite rules, which further triggers about 5,000 requests per page load. The permalink structure must change as wordpress development probably won't change this in the near future. Now, changing the permalink structure worries me quite a bit, as about 25% of my traffic is attributed to my blog posts -- the rest is covered through CMS-like-use of pages (75%). blog posts will change permalink/url structure, pages won't The website is very respected in my niche and has quite a few links going to most of my posts and pages, as well as the homepage I've noticed in the last year that anything I post starts ranking on page 1 of Google for very competitive kws in 1-3 days, often with top 3 rankings PR4 / decent Alexa / Moz ranks not too shabby either / quality content / decent social media linking (mainly Facebook) / no penalties I provided the factors as to not gloat, but rather to get the best answer from those who have fairly established websites and perhaps had to change their URLs and noticed some or no changes to their rankings. How long of a hit am I going to take / how much my posts might drop down in SERPs if I change the permalink structure, properly 301 them, and implement all changes in one swoop? Info for WordPress users Benefits of changing the permalink structure to /%post_id%/%postname%/ -- for example -- include: way faster load times, not having 5,000 requests per page load, avoiding verbose rewrite rules trigger, finally modify the site without worrying about crashing the website and using a local server to make changes on thousands of pages (the database backups, the ritual of changing the settings in the local database, changing the post/page, saving the local database, loading the locally saved db on live server, and crossing fingers and pray it works -- just takes so darn long.) Ahh..yes, huge time saver. ** this issue occurs when using WP as a CMS with several hundred pages + and using the /%postname%/ or /%category%//%postname%/ or /somethingstatic/%postname%/ -- IF USING the date based way /%year%/%postname%/ or /%post_id%/%postname%/ you should be fine.
Algorithm Updates | | pepsimoz0