Block web archieve/way back machine
-
Hi i want to block web archive/wayback machine from indexing my site and creating a record of it on their database.
Any ideas on how to do this?
Cheers,
Superpak -
You can block Wayback Machine from crawling and creating a record of your site by adding the following to your Robots.txt file:
User-agent: ia_archiver
Disallow: /This will not only stop new records from being created but also stop people viewing what had previously been indexed by Wayback Machine.
More information about this can be found here: https://archive.org/about/exclude.php
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does link position matter in the content/html code
My question is that if I have several links going to different landing pages will the one at the top of the content pass more value than ones at the bottom. Assuming that there are not more than 1 of the same link in the content. The ultimate question is whether or not link position in the content/html code make a difference if it passes more value. This question comes in response to this whiteboard Friday https://www.youtube.com/watch?v=xAH762AqUTU Rand talks about how if there are 2 links going to the same URL from the same content page then google will only inherit the value of the anchor text from the first link on the page and not the both of them. Meaning that google will treat that second link as if it doesn’t exist. There are lots of resources that shows this was true but there isn’t much content newer than 2010 that say this is still true, We all know that things have changed a lot since then Does that make sense?
Intermediate & Advanced SEO | | 97th_Floor0 -
SEO Class/Course/Certification
Hello, I'm located in the US and I'm looking for **good **SEO classes or courses. Something that provides certification when completed, if possible. This is something that my employer wants to do. I know MOZ offers a lot of information here and has a course as well, but I'm looking for something that is in person. I tend to learn better that way. I see some things pop up when I search, but I believe most of it is just junk. I am looking for something that would basically go from intermediate to advanced. Thank you!
Intermediate & Advanced SEO | | slecinc0 -
Qucikest and most effective way to improve DA and PA?
Hello Moz, In your experience what is the quickest and most effective way to improve a websites DA and PA? Any additional tips and side notes would be greatly appreciated. For example how hard is it to actually get a DA of 60+ - what would that require usually as an absolute minimum?
Intermediate & Advanced SEO | | xdunningx0 -
How to speed indexing of web pages after website overhaul.
We have recently overhauled our website and that has meant new urls as we moved from asp to php. we also moved from http to https. The website (https://) has 694 urls submitted through site map with 679 indexed in sitemap of google search console. As we look through the google search console analytics we notice that google index section / index status it says: https://www.xyz.com version - index status 2
Intermediate & Advanced SEO | | Direct_Ram
www.xyz.com version - index status 37
xyz.com version - index status 8 how can we get more pages to be indexed or found by google sooner rather than later as we have lost major traffic. thanks for your help in advance0 -
Canonical and Rel=next/prev Implementation
Hi, I have an ecommerce site that allows users to view numerous pages and sort by a number of options on categories. I've read numerous posts around my issue but am still a little confused on what is best practice with regards to the canonical tag and rel=next and prev. Below is an example of the various page/sort by URL's: Paginated URL: http://www.example.co.uk/category/subcategory.html?p=3 Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price Paginated & Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price&p=3 It is not viable for us to use a canonical tag to the view all page as some of the categories contain a large number of products and therefore would not have the best load speeds. Is it best to use the below structure when it comes to the canonical tag and rel=next and prev? Paginated URL: http://www.example.co.uk/category/subcategory.html?p=3 Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price Paginated & Sort by URL: http://www.example.co.uk/category/subcategory.html?dir=desc&order=price&p=3 http://www.example.co.uk/category/subcategory.html?dir=desc&order=price&p=2" /> Thanks
Intermediate & Advanced SEO | | GrappleAgency0 -
Am I on the right way ? any suggestion please ?
Hi : Now it's 3 month from starting seo my website by myself ( my website is like prchecker.info that give users one online service " My both primary keywords have 450.000 and 100.000 **exact usa search , **when I start my goal is to rank my both keywords on second page during the first year , now and after 3 month after creating few quality backlinks ( guest posting and comments on relevant topic on forum ) my both keywords are ranked on 3rd and fifth page. Any suggestion to create quality backlinks that might help me ? should I continue with guest posting ?
Intermediate & Advanced SEO | | Khaledmoalla0 -
Lost positions, how to get them back?
Ok, like 1 year ago my site was doing good in google search engine. But I decided to change CMS and design of the site. Did some mistakes like, changed site links structure (when I notice it later I changed it back), then some of the pages were lost because I simply didn't need them on the new cms and there were no feature of keeping them. Site has a lot of backlinks and was strong in rankings. It's been almost a year or maybe already a year and I can't get even to top100 for those keywords... But I was in (top10-20) Site is still getting links, ranking well on yahoo and baidu, but google, just won't let my main keywords to come back. I rank for some that doesn't make sense to me and I even wasn't working on them.
Intermediate & Advanced SEO | | onmaso0