Block web archieve/way back machine
-
Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database.
Any ideas on how to do this?
Cheers,
Superpak -
You can block Wayback Machine from crawling and creating a record of your site by adding the following to your Robots.txt file:
User-agent: ia_archiver
Disallow: /This will not only stop new records from being created but also stop people viewing what had previously been indexed by Wayback Machine.
More information about this can be found here: https://archive.org/about/exclude.php
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google penalise in the way described in this article?
In an interesting article from January on content cannibalisation: https://ninjaoutreach.com/content-cannibalization-avoid/ there is the following paragraph: "When the same keyword is used across a number of pages of a single website, Google’s spiders automatically get directed to a page with low-grade quality which in turn results in the low ranking of all the pages on the website." Is this true? The suggestion here is that they automatically get directed there as a form of penalty. This seems like quite an extraordinary claim! Can anyone verify?
Intermediate & Advanced SEO | | Ad-Rank0 -
Will have /index in my url hurt?
I am trying to setup permalinks on a wordpress blog that is installed on iis. I can't update the web.config file so I have to make every page /index/pagetitle. as shown here-http://codex.wordpress.org/Using_Permalinks#PATHINFO:_.22Almost_Pretty.22 How much of a difference is there between no /index and having the /index in there?
Intermediate & Advanced SEO | | EcommerceSite0 -
Domain.com/keyword1.keyword2.html vs doamin.com/keyword1-keyword2.html
I was doing some research and saw this url structure in a website that was not ranking well and can't help but wonder was the url structure part of the problem as well it looks like this with a period between keywords. domain.com/keyword1.keyword2.html and was wondering if that is acceptable for search engines as opposed to the normal dashes like this expample ... domain.com/keyword1-keyword2-keyword3.html I have never noticed a period to separate words in a url before. Anyone have any experience with this ? Is this going to hurt possible rankings ? Thank you in advance, Joe
Intermediate & Advanced SEO | | jlane91 -
Google: How to See URLs Blocked by Robots?
Google Webmaster Tools says we have 17K out of 34K URLs that are blocked by our Robots.txt file. How can I see the URLs that are being blocked? Here's our Robots.txt file. User-agent: * Disallow: /swish.cgi Disallow: /demo Disallow: /reviews/review.php/new/ Disallow: /cgi-audiobooksonline/sb/order.cgi Disallow: /cgi-audiobooksonline/sb/productsearch.cgi Disallow: /cgi-audiobooksonline/sb/billing.cgi Disallow: /cgi-audiobooksonline/sb/inv.cgi Disallow: /cgi-audiobooksonline/sb/new_options.cgi Disallow: /cgi-audiobooksonline/sb/registration.cgi Disallow: /cgi-audiobooksonline/sb/tellfriend.cgi Disallow: /*?gdftrk Sitemap: http://www.audiobooksonline.com/google-sitemap.xml
Intermediate & Advanced SEO | | lbohen0 -
Zip Code Blocks the Search Engines!
I have a site where when you visit the product pages, it asks for your zip code. This is obviously blocking the bots from crawling the site. I know you can basically tell the bots how to ignore the zip code feature but I am not exactly sure how to do this. Any help would be appreciated
Intermediate & Advanced SEO | | lhawk0 -
/%category%/%postname%/ Permalink structure
Mostly everyone seems to agree that /%category%/%postname%/ is the best blog structure. I'm thinking of changing my structure to that because now it's structured by date which is bad. But almost all of my posts are assigned to more than one category. Won't this create duplicate pages?
Intermediate & Advanced SEO | | UnderRugSwept0 -
What would you pick? Species/Breed or Topic
If you'd like to take a look, the site under quesiton is http://ArkAnimals.Com. At the moment I am considering doing landing pages by topics and not by the type of animals. I will be blending both wild and domestic animals but how to best do this is confusing since so much has changed over the years. My competitors are focusing on animal types mainly and competition is fierce. Also the site attracts by three main topics not specific animals--so I want to be a bit unique which is why I am considering a topic driven focus. What would you recommend? Background This site has been online since 1994 and on its own domain for a long while. However, over time it has suffered from a lot of things--different designers, expansion, movement of content to niche sites and bad seo. LOL Once everything was on one site with sub directories. Then, it expanded and my online advisors recommended moving topics off into their own niche sites. So, I did that. Ugh. Now, much of that content is being integrated back as I am undergoing an intense revamp (the last one was a disaster). There are a few presenting problems that I could use your perspective and expertise--since I am too close to it. Problems for Needing Your Input The site is over 2600 pages with many in html and others in php.What is the best practice? Moving the remaining html pages over into php? Some of the pages that were not active have a redirect to the blog. I plan on doing page to page 301 redirects once I dig in--unless you have a better idea. There are a lot of well established links to some of the pages. How many topics are too many? I have a wide variety of content. First, the magazine format covered about six topics. Later, I began covering more pet related items and did a lot of different news summaries to keep it fresh. I want to dump the short outdated pages as many of them have obsolete links or are too short to add any value. Or should I update if they help with the seo rather than continue to let them dilute the site? Landing page or blog? Which is better, an index landing page or blog? At the moment the blog appears on the main index for freshness and the site attracts traffic for specific topics not animal breeds or species. I want to move the site from an educational site to serving as a main funnel for potential clients driving them to get on a list or to a niche site for sales related to the particular topic/training of interest. What your take on this if you were to tackle it? Any input would be greatly appreciated. My audience includes those who are pet owners, novice trainers, and animal lovers with no critter sense.
Intermediate & Advanced SEO | | TheARKlady0 -
Block Google Sitelinks for DSEO?
I am trying to manage DSEO for a client. The question is: would blocking a page listing from my client's Google Sitelinks cause that blocked sitelink page to be independently listed in the rankings and therefore potentially stuff a negative listing further down the rankings? Or would the blocked sitelink not show up at all in the SERPs
Intermediate & Advanced SEO | | bcmull0