How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Product URL
Hey Mozzers, Nice quick and simple one for you. Which of these 2 options is better for SEO and userbility and why domain.co.uk/productname.html
On-Page Optimization | | ATP
domain.co.uk/shop/category/product.html The top one stops stops any funky problems with magento making 2 paths to the same product but the second option feels more natural and helpful to the user. I feel both a valid but I would like some opinions please0 -
How important are clean URLs?
Just wanting to understand the importance of clean URLs in regards to SEO effectiveness. Currently, we have URLs for a site that reads as follows: http://www.interhampers.com.au/c/90/Corporate Gift Hampers Should we look into modifying this so that the URL does not have % or figures?
On-Page Optimization | | Gavo1 -
SOS - I have done a terrible mistake: How can I make it up?
Two weeks ago i changed the urls on our website, without redirecting the old ones. This has led to a dramatic drop in ranking. What should i do: redirect the old pages and keep the new ones Change the urls back to the original ones Which of these to methods will result in best ranking? Maia
On-Page Optimization | | MaiaHaaland0 -
Sorry, but that URL is inaccessible
Our site is all on the https protocol so every time I use the on-page grader it tells me the link is unavailable. What can we do? When I use the http protocol (which is 301 redirected to the https) it still gives me the same message.
On-Page Optimization | | whiskeyfl1 -
How do I remove a Canonical URL Tag?
Some of my report cards say I have too many canonical URL tags. However, there is no information no how to delete one. Can someone give me a link or explain? Thanks.
On-Page Optimization | | dealblogger0 -
Login webpage blocked by robots
Hi, the SEOMOZ crawl diagnostics shows that this page: www.tarifakitesurfcamp.com/wp-login.php is blocked (noindex, nofollow) Is there any problem with that?
On-Page Optimization | | juanmiguelcr0 -
Url question for seo
Would it be beneficial to use url that is a match for my keyword to help with seo, then have my currently url forward to that one so I don't have to change any marketing materials? I was one of the feedback that I got when doing the on page keyword optimization tool on seo moz. Thanks J
On-Page Optimization | | fertilityhealth0 -
How to use good keyword URL to help main site
Hi. I'm a long time ecommerce guy and starting a third business. The main site URL is the name of the new business but I also purchased a .com URL that is our #1 keyword to target. So I need to know the best strategy to use the keyword url for helping with getting a top ranking for that keyword. I'm curious if I can or should build out the keyword URL site for the search engines and use a 301 redirect. Can you get top ranking for a site that just redirects? Anyway, I guess you get my question. This keyword gets a ton of perfectly targeted traffic so seems like a goldmine if I work it right. Thanks very much.
On-Page Optimization | | jimmyseo1