How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Product names are defacto identical - How can I modify Title?
Hi! Our site http://www.metecoparts.gr/ help our customers search our inventory of cars for used parts. The way the titles of product pages are structured is: Car Parts – Make Model Year – Brand Name. So for instance this page http://www.metecoparts.gr/car/4074-alfa-romeo-156-1997 have the title "Car Parts Alfa Romeo 156 1997 – Meteco SA" (I have translated Greek words to English) So as you can imagine we end up with a bunch of duplicate titles pages. I’m trying to diversify the titles but the problem is that there aren’t many product identifiers other than body configuration, car color, fuel, engine capacity and engine code. But only color and year of production is partly unique as most Alfa Romeo 156 are 4 door saloons, with 1600cc petrol engine. The only thing that is unique for every product page is the page ID. So should I use it in the page title? Its the only unique identificator but on ther other hand it has no use for our customers. Or is there any other way of having unique titles that I am missing? Please note that in our industry on-page optimization is crucial since there is no social sharing and much link building (who wants to share an old car that is being sold for parts?) Thank you
On-Page Optimization | | Johnlock10 -
Dynamic URL Parameters + Woocommerce create 404 errors
Hi Guys,
On-Page Optimization | | jeeyer
Our latest Moz crawl shows a lot of 404-errors for pages that create dynamical links for users? (I guess it are dynamic links, not sure). Situation: On a page which shows products from brand X users can use the pagination icons on the bottom, or click on: View: / 24/48/All.
When a user clicks 48 the end of the link will be /?show_products=48
I think there were some pages that could show 48 products but do not exist anymore (because products are sold out for example), and that's why they show 404's and Moz reports them. How do I deal with these 404-errors? I can't set a 301-redirect because it depends on how many products are shown (it changes every time).
Should I just ignore these kind of 404-errors? Or what is the best way to handle this situation?0 -
Can I target one keyword with 2~3 pages?
Since my website is targeting a very specific field, there are not many widely searched keywords. So I'm thinking of targeting one keyword with 2~3 pages. 1. I've read Neil Patel's blog post on how to create dual rankings to make your search listings stand out on Google. So I assume it's okay to target one keyword with several pages. (http://www.quicksprout.com/2012/07/30/4-steps-to-making-your-search-listings-stand-out-on-google/ Step #2 Create Dual Rankings) 2. But I've also read things on Keyword Cannibalism saying that if you target one keyword with several pages, they will compete with each other, and Google will get confused. I'm wondering, is it okay to target one keyword with 2~3 pages? And is there any smart way to do it ? Thanks.
On-Page Optimization | | joony0 -
Changing URL Structure From Flat to Pyramid Theme
Hello Mozzers, I have an on-page SEO question regarding URL structure. A few months back we hired a full-time SEO person who is working on-page right now and she really wants us to completely re-due our URL restructure from a flat to pyramid style (example below). Current URL structure / page title is: Dog training Collars - K9electronics.com
On-Page Optimization | | k9byron
http://www.k9electronics.com/dog-training-collars/ Small Dog training Collars - K9electronics.com
http://www.k9electronics.com/small-dog-training-collars/ Einstein ET-300TS Mini Dog Training Collar - K9electronics.com
http://www.k9electronics.com/einstein-et-300ts-mini-remote-dog-trainer.html Suggested URL structure / page title change: Quality Dog Training Collars - Lowest Price Guarantee - K9electronics
http://www.k9electronics.com/dog-training-collars/ Dog Training Collars - Small Dog - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/ Einstein ET-300TS Mini - Dog Training Collars - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/einstein-et-300ts-mini I guess you could say we are the poster-boys for Google penalties and have received just about every penalty in the book. Panda, penguin algo penalties and a partial manual action for unnatural links. Several months ago we removed 1000's of in-bound links and had our manual action lifted a few months back and are now hovering around the top / mid 2nd page for all our big terms ...we used to be top 3 for everything. As we were removing bad links, we also completely redesigned the site and removed lots of categories and products and 95% of all our old, low quality content and replaced it with new, high quality content. The site was really slooooow, so we optimized it and moved it to a big dedicated server and tripled page load time. Added rich snippets, Google authorship, increased our FB and other social presences and much more ... I had also considered this URL structure change during the redesign because I had heard and read that it was good to do, but it required redirecting practically all our URL's which I know can hurt the site even more so then it already has been ... Our SEO says that as it sits now, our pages are competing with each other and really seems to think this is going to improve our rankings a lot ...after several weeks. My question is, at this stage in the game, is it really going to help a lot and give us more benefit compared to the 301 redirect link juice loss? Any comments and/or suggestions are very much appreciated!1 -
Disqus Comments or IntenseDebate for eCommerce. Can It HELP???
Anybody using any of those? Do you have example of eCommerce store using it? Will it help ranking? Any thing I should know about those ''Comments'' plug-in? Thank you, BigBlaze
On-Page Optimization | | BigBlaze2050 -
Canonical URL tags help I am not sure what this is
I am trying to get an A grade on my webpage and this is one of the critical steps canonical URL tags I cant find much information as to what this even is never mind fixing it. Thanks I am a total newbe at this any advice is appreciated
On-Page Optimization | | gemfirez0 -
Replacing "_" with "-" in url, results in new url?
We ran SEOmoz's "On-Page Optimization" tool on a url which contains the character "_". According to the tool: "Characters which are less commonly used in URLs may cause problems with accessibility, interpretation and ranking in search engines. It is considered a best practice to stick to standard URL structures to avoid potential problems." "Rewrite the URL to contain only standard characters." Therefore we will rewrite the url, replacing "_" with "-". Will search engines consider the "-" url a different one? Do we need to 301 the old url to the new one? Thanks for your help!
On-Page Optimization | | gerardoH0 -
URL Extensions (with or without??!!)
Hello, SEOers~ Today I have a question about URL extensions. Which one is more search engine friendly between URL with extensions and without extensions? e.g. URL with extension : www.example.com/tv/lcd.jsp URL without extension : www.example.com/tv/lcd I heard that URL without extensions is in trend considering user experience. User experience is also important but I would like to know from SEO perspective. Please people~ Help me out with this~! Thanks.
On-Page Optimization | | Artience0