How can i block the below URLs
-
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.?
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/
http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/
My robots.txt file is -
User-agent: * Disallow: /wp-admin/
-
robots.txt:
Disallow:
/wp-admin/
/wp-content/plugins/
Then, deindex them as mentioned in the first response.
-
It's also worth remembering that blocking a URL in a robots.txt file does not automatically mean that the URL will be deindexed. The robots.txt file will prevent robots, such as the Googlebot, from seeing or accessing the file and that's all. Now obviously, if a robot repeatedly tries to access a file and gets denied it will eventually stop trying to, which is what leads to the file being deindexed (the same principle applies to 404 and 410 errors).
Therefore, if you want a quicker and more definite deindexing solution, you should use the explicit noindex robot command, as recommended above. This will tell any visiting robot to not index it straight away, which will reduce the number of revisits and drop the page from the index faster.
-
Take a look over here - http://www.robotstxt.org/robotstxt.html and we can't forget the Moz version - http://moz.com/learn/seo/robotstxt
Alternatively you can just add meta noindex on the pages with with the added bonus of letting link juice flow better as well as being a bit more stern to robots ( recommend the noindex tag!)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query for paginated URLs - Shopify
Hi there, /collections/living-room-furniture?page=2
On-Page Optimization | | williamhuynh
/collections/living-room-furniture?page=3
/collections/living-room-furniture?page=4 Is that ok to make all the above paginated URLs canonicalised with their main category /collections/living-room-furniture Also, does it needs to be noindex, follow as well? Please advice, thank you!1 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
How can i make my sub-domain bring in more traffic?
we have a social networking site that has user-generated content and users have to log in before they can access the platform. We are in the process of creating indexable pages that google bots can pick up. These pages will sit in front of the log in pages and will be open for guest access. Can you advise on the best way of optimising these pages for higher ranking and traffic.
On-Page Optimization | | seoworx1230 -
Which URL Structure is best for New Website?
Hello All, I have one client and he want to develop his website but the he want the URL structure for his website page like below: http://www.example.com/category/xxproduct.html But I have suggest him below URL Structure http://www.example.com/category/xx-product.html So Can you people please suggest me that from above which URL structure is better from SEO side as well as from Visual side..? Is using - is better to separate the words in URL?
On-Page Optimization | | jemindesai0 -
Duplicate URL for homepage
Hi Gurus, Thank you for reading this question My site is developed in Classic ASP How can i make sure the homepage is not duplicated for http://www.partyrama.co.uk/ http://www.partyrama.co.uk/default.asp http://partyrama.co.uk/ http://partyrama.co.uk/default.asp Regards Sri
On-Page Optimization | | partyrama0 -
Can I place H1 tag anywhere on page
Hello, For those of you who use Magento you will know it is not SEO friendly. When you create a category or product, the name of the product or category then becomes the H1 tag. We sell mens business shirts. For example we have a product called 'White poplin classic fit' this is also the H1 tag, nobody is ever going to search for that term so I have had my developer create a new attribute that allows me to keep the product name as it is and let's me create a new SEO friendly H1 tag, for example 'White business shirt' However, placing 'White business shirt' on the page to be visible by the visitor does not look good on the page. My question is. Can I place the H1 tag anywhere on the page? I have some tabs like below. I am thinking of add a tab in between delivery and returns called more info and placing more seo keywords including the H1 tag in this tab. Will this be OK or will this be seen as black hat technique?
On-Page Optimization | | mullsey0 -
Long URL in listing job portal
Hello I have job portal and I am listing job offers by: regions, position, sector, language skill For example, when user searchs job in Bratislava, Programmer, information technology, english. My URL is www.presbium.sk/bratislava/pragrammer/information-technology/english Title:Job in Bratislava, Programmer, Information technology, english I know, that URL is too long and no good for SEO. Is it solution? When user chooses max. 2 items for example: www.presbium.sk/bratislava/programmer/ than I put in every next links atribute nofollow and google will index only pages with max. 2 items: www.presbium.sk/bratislava/ or www.presbium.sk/programmer/ or www.presbium.sk/bratislava/programmer/ but not www.presbium.sk/bratislava/protrammer/english/, because english link has atribut nofollow. And I want to ask, what is the best solution for SEO when I am listing job offers by regions, position, sector, language skill and I have than long URL and Title www.presbium.sk/bratislava/pragrammer/information-technology/english Title:Job in Bratislava, Programmer, Information technology, english
On-Page Optimization | | PeterSEO0 -
New sitelinks - can we control the number?
A quick question on Google's new sitelink format. When searching for our brand name (Confetti) Google returns 8 sitelinks for our site. When searching for our domain (confetti.co.uk) Google returns the maximum number of 12 sitelinks. Is there a quick way (Webmaster Tools for example) to increase the number of sitelinks for our brand name to 12? Thanks,
On-Page Optimization | | Confetti_Wedding0