How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query for paginated URLs - Shopify
Hi there, /collections/living-room-furniture?page=2
On-Page Optimization | | williamhuynh
/collections/living-room-furniture?page=3
/collections/living-room-furniture?page=4 Is that ok to make all the above paginated URLs canonicalised with their main category /collections/living-room-furniture Also, does it needs to be noindex, follow as well? Please advice, thank you!1 -
Is it better to shorten my existing url to use only keyword after domain with a 301 redirect from existing url
I have a long existing URL that has included my key word but the url has about 5 additional words in the text ( eg url would have " /super widgets in stock at the widget store " as url text after domain. keywords is super widget The URL was at the top of search results for my keyword for many years until recently. Is it better to shorten my url text to now use only my keyword " /super-widgets " after the domain with a 301 direct from my existing url to optimize it Thanks
On-Page Optimization | | mrkingsley2 -
Optimizing a URL/menu structure
Hi Mozzers, I'm working on Content Strategy at my job, and I'm close to making some recommendations on short/long-term direction. While I'm there, I want to tackle the URL/menu structure (correct term?), which is a bit of a mess as pages have been created without any consideration for it over time. For ease, let's just say we have 3 main subdirectories of the site (Section A-C), and let's also say that section A also has 3 important subdirectories. From a UX perspective at least, we want a page to look like: example.com/sectionA/subsectionAA/page1 but currently it's example.com/page1 We have dozens and dozens of these examples. To complicate matters a little further, Sections B and C have been earmarked to be consolidated into a new section (D), as they're currently confusing and overlapping, and create roadblocks in user journeys. So a page that is, say: example.com/sectionB/page2 may well want to be: example.com/sectionD/subsectionDA/page2 I'm comfortable enough with technically doing this, as I'm experienced enough in Drupal and have an agency on hand too, BUT - I don't know if there are any SEO pitfalls I need to be wary of when I'm doing this, beyond resubmitting sitemaps, and the trickle-down effects of redirects. Any advice, wise forum? thanks!
On-Page Optimization | | joberts0 -
Url permalink structure question!
Hello! I just read the "15 SEO Best Practices for Structuring URLs" but I have still a question: My A version bellow "20 accessoires" has no meaning in french. If I add "voyage" (version B), then is it considered as spammy? I mean the "voyage" keyword repetition? A) http://www.lytchee.com/preparer-son-voyage/20-accessoires/ B) http://www.lytchee.com/preparer-son-voyage/20-accessoires-voyage Thanks for advices! Sylvain
On-Page Optimization | | lytcheetv0 -
What word should I use in my URL for my blog
Should I use the word "blog" in my sub folder as in : http://www.mybusiness.com/blog or should I use http://www.mybusiness.com/news. Is there a difference for when my site is crawled. I understand that a blog works a little differently. Can someone explain the basics?
On-Page Optimization | | graemesanderson0 -
Does anyone use Genesis Framework? If so can a newbie use it and a few other questions
Hi, So as I search the wonderful land of the internet, I see this Genesis framework brought up quite a bit. I have researched it for a few weeks, but it seems like it uses hooks instead of shortcodes. So I am curious if anyone has used it? And if so what your thoughts are about it? I am a COMPLETE newbie here, so hooks look scary. I am sure with time they will seem like second nature. They claim it has airtight security. So if you have used this framework, how is this any different than an updated stock wordpress site? I understand that vulnerabilities may be in plugins and such, but if it is really airtight, that seems great. Any thoughts are appreciated as I just want the best user experience. So many people use this framework, yet my site gets if I'm lucky 1000 views each month. It is a basic site to let people know we exist. So its not like I have a popular blog with 50,000 pageviews each month. But... going into the future, I want a pleasant and consistent user experience. Maybe a wordpress theme is all you need. Maybe a framework is more for developers. Any thoughts are greatly appreciated. Chris
On-Page Optimization | | asbchris0 -
Language for URLs on new international websites
We are due to launch our new site and it will be targeting an international market. We have setup these new sites in the following way www.website.com/fr/content-goes-here www.website.com/es/content-goes-here This has been done in conjunction with setting up the parameters in GWT and making sure it is optimised for the language itself, and that countries search engine. But our web dev team have said that the URLs at the moment will be in english and not the native language, so if you were on the french version of our site you would see the url in english and not french. Will this negatively affect the site for SEO, and who else would think it would be negative from a usability perspective? Any help is appreciated.
On-Page Optimization | | RyanCrawf19840 -
Schema business address - can this affect me in the national rankings
Looking at using Schema microdata on our new website, we currently rank quite decently in Google UK. If I was to add schema mark up for our business address could this effect us in Google UK rankings wise i.e. our rankings for national search (for want of a better word!) suffer but our local rankings increase?
On-Page Optimization | | Jon-C0