How to disallow google and roger?
-
Hey Guys and girls,
i have a question, i want to disallow all robots from accessing a certain root link:
Get rid of bots
User-agent: *
Disallow: /index.php?_a=login&redir=/index.php?_a=tellafriend%26productId=*
Will this make the bots not to access any web link that has the prefix you see before the asterisk? And at least google and roger will get away by reading "user-agent: *"? I know this isn't the standard proceedure but if it works for google and seomoz bot we are good.
-
In short, yes that will work.
To be clear, you can have multiple links which all lead to the same target page. The target of that link could still be indexed if there are other unblocked links which point to it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing site subdomains from Google search
Hi everyone, I hope you are having a good week? My website has several subdomains that I had shut down some time back and pages on these subdomains are still appearing in the Google search result pages. I want all the URLs from these subdomains to stop appearing in the Google search result pages and I was hoping to see if anyone can help me with this. The subdomains are no longer under my control as I don't have web hosting for these sites (so these subdomain sites just show a default hosting server page). Because of this, I cannot verify these in search console and submit a url/site removal request to Google. In total, there are about 70 pages from these subdomains showing up in Google at the moment and I'm concerned in case these pages have any negative impacts on my SEO. Thanks for taking the time to read my post.
Technical SEO | | QuantumWeb620 -
Google Ecommerce Alerts
I recently started getting email notifications from Google re: new products on our websites. I am subscribed to Google alerts. Can anyone shed some light on this?
Technical SEO | | AMHC0 -
Google webmaster and analytics access to others
Hi, Google webmaster and analytics access to others with restricted and user access is this ok and secure? With this access can anyone tamper anything? thanks
Technical SEO | | mtthompsons0 -
Home page deindexed by google
when I search my website on google by site:www.mydomain.com I have found my domain with www has been de-indexed by google, but when I search site:mydomain.com, my home page--**mydomain.com **show up on the search results without www, put it simple, google only index my domain without www, I wonder how to make my domain with www being indexed, and how to prevent this problem occure again.
Technical SEO | | semer0 -
Google Webmaster Sitemap *pending*
Hey guys, I've noticed that my sitemap has been "pending" for quite some time in Google Webmaster tools. This leads me to believe that Google is not indexing my site. Could someone help me and point me to what I'm doing wrong? My site is The Tech Block
Technical SEO | | ttb0 -
Google sitemap just for a part of site?
Hi, I am about reorganize (content and seo-wise) a part of a larger site and I wondered if it is possible to use a Google sitemap just for some but not all pages of a site? Does anyone know if this has any impact on pages that are not included in the sitemap? Thanks
Technical SEO | | haest0 -
Unit # No Longer Showing On Google Places
We've noticed that the unit # is no longer showing on a clients Places profile page. Any thoughts on why and if relevant to rankings? Places page - https://skitch.com/kyegrace/8fimr/vuppie-real-estate-team or http://maps.google.ca/maps/place?hl=en&qscrl=1&nord=1&rlz=1T4GGNI_en-GBCA461CA461&gs_upl=&ion=1&bav=on.2,or.r_gc.r_pw.r_cp.,cf.osb&biw=1366&bih=641&wrapid=tlif133037839755610&um=1&ie=UTF-8&q=realtor&fb=1&gl=ca&hq=realtor&hnear=0x548673f143a94fb3:0xbb9196ea9b81f38b,Vancouver,+BC&cid=5594900399034912659&ei=o_ZLT4XHM6ObiQKZnbCdDw&sa=X&oi=local_result&ct=placepage-link&resnum=8&ved=0CIMBEOIJMAc and as it appears in the backend https://skitch.com/kyegrace/8fimk/google-places-analytics Any insight greatly appreciated!
Technical SEO | | kyegrace0 -
Google indexing page with description
Hello, We rank fairly high for a lot of terms but Google is not indexing our descriptions properly. An example is with "arnold schwarzenegger net worth". http://www.google.ca/search?q=arnold+schwarzenegger+net+worth&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a When we add content, we throw up a placeholder page first. The content gets added with no body content and the page only contains the net worth amount of the celebrity. We then go back through and re-add the descriptions and profile bio shortly after. Will that affect how the pages are getting indexed and is there a way we can get Google to go back to the page and try to index the description so it doesn't just appear as a straight link? Thanks, Alex
Technical SEO | | Anti-Alex0