Files blocked in robot.txt and seo
-
I use joomla and I have blocked the following in my robots.txt is there anything that is bad for seo ?
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/
Disallow: /mailto:myemail@myemail.com/
Disallow: /javascript:void(0)
Disallow: /.pdf
-
What you have there is just blocking rootdomain.com/javascript:void(0). Googlebot can execute and index JavaScript; you should not block it without a good reason. I'd let it read the JavaScript and see the submenus.
-
Thank you and is blocking javascript bad ? ( I was thinking about submenus )
-
If you don't want pages in those Disallowed directories to be indexed, then you're doing fine. These pages won't be able to be crawled, so, they won't be likely to appear in search results for any search engines.
The last three entries look fishy to me. I'd need to know what types of URLs you're trying to block to fix them. For the last one, if you're looking to block all pdfs on your site, the syntax would be Disallow: /*.pdf.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Worldpress multisite for SEO
hello, guys, at the moment we have 3 websites, basically, the websites have the same content and appearance. we got UK website, New Zealand web site and USA website for the different business purpose. I have some questions about multisite for SEO, with similar content, will it harm website ranking? if it is bad, what should we do to deal with multisite? thank you
Intermediate & Advanced SEO | | kelvinbongcn850 -
Pixel tags impact on SEO?
I was asked by our IT if switching to a tag management company that removes the pixels from our site and is replaced by a javascript would have a negative impact on SEO. I have not been able to find anything that discusses this: Does anyone have experience with this? Has it caused any issues? Has it caused any issues? How do crawlers see pixel data, and what do they do with it?
Intermediate & Advanced SEO | | Shawn_Huber0 -
Should I use meta noindex and robots.txt disallow?
Hi, we have an alternate "list view" version of every one of our search results pages The list view has its own URL, indicated by a URL parameter I'm concerned about wasting our crawl budget on all these list view pages, which effectively doubles the amount of pages that need crawling When they were first launched, I had the noindex meta tag be placed on all list view pages, but I'm concerned that they are still being crawled Should I therefore go ahead and also apply a robots.txt disallow on that parameter to ensure that no crawling occurs? Or, will Googlebot/Bingbot also stop crawling that page over time? I assume that noindex still means "crawl"... Thanks 🙂
Intermediate & Advanced SEO | | ntcma0 -
Map with usability and SEO purpose
For my client I need to add some structure to its pages. The deepest pages are about restaurants and are sorted per city and then per province as a larger silo. I want to do this: Homepage > Provinces > Cities > Restaurant page This structure is optimal, but I as a usability freak I prefer making the experience cool for the users. I want to add interactive pictures that are cool for the user and hopefully are readable for the google bots, I want to do it like this: The homepage shows a map of my country that has the twelve provinces outlined, that light up when you hover over them. Then when you click a province you get to the province page. On the province page you see a large image of the province and see all cities where there are restaurants, when you hover over a city it grows a little and when you click it you arrive at the city page, at that page you will find a list of all restaurants that are available in the city. What I need to know is, is it possible for google to see these pictures as a nice site structure? Or do I need to add the ugly footer links and have pages with lists of links...? And what is the smartest way to structure this, flash?
Intermediate & Advanced SEO | | Lebron270 -
Pop Up Advertisement - Bad for SEO?
So i have been working with a company running their SEO for close to two years now. Since i started to engage with them they have always used a very simple pop up for the first time an end user visits their website (via javascript and cookies). The pop up simply ask them if they would like to download a solutions brochure from their website. So as far as pop ups go, it is at least relevant. The client loves this pop up, i do not. For a while we have always held spots #1-3 for a lot of our keywords but we have started to drop to lower on the first page. So i have been researching to see if some of the new algorithm changes are targeting sites with this type of functionality. If i have some data i could definitely get them to remove it. So the question is, do pop-ups hurt your organic ranking? Thanks for the input! Kyle
Intermediate & Advanced SEO | | kchandler1 -
SEO Strategy for Microsite
I am working on a project to build a microsite of sorts that will represent a joint program between two large organizations with established web presences and strong domains. Each of the organizations has dedicated sections on their sites speaking to the program, but the leadership has decided the joint program deserves it's own site with dedicated content. The two larger sites perform very well for SEO, and I don't necessarily want to jeopordize thir rankings by delivering content that competes directly with them. So I am doing some keyword research to find some opportunities that will alllow me to use the new site to target keywords not yet being captialized by the larger sites. My grand scheme is to have the three sites targeting the broadest array of keywords possible, thus maximizing exposure and avoiding competition. Here is the rub: the content between the three sites will be different but very similar, and there will be plenty of cross linking, especially from the existing sites to the new site, as we grow the brand of the joint program. I'm curious to here some expert opinions on what the puitfalls of the strategy are and what are some of the things I can do to avoid falling in the black hat category - I recognize that proliferating sites around a single topic and cross linking them is black hat. The organizations simply want to build a brand around a joint program and we are striggling to do that without a dedicated website.
Intermediate & Advanced SEO | | AmyLB0 -
A very basic seo question
Sorry, been a long day and wanted a second opinion on this please.... I am developing an affiliate store which will have dozens of products in each category. We will not be indexing the product pages themselves as they are all duplicate content. The plan is to have just the first page of the category results indexed as this will have unique content about the products in that section. The later pagnated pages (ie pages 2,3,4,5 etc) will have 12 products on each but no unique content. Would the best advice be to add a canonical tag to all pages in the 'chairs' category pointing to the page with the first 12 results and the descriptions? This would ensure that the visitors are able to browse many pages of product but google won't index products 13 and onwards. Am I right in my thinkings? A supplemental question. What is the best way to block google from indexing/crawling 90,000 product listings which are pulled direct from the merchant so are not unique in the least. I have previous played with banning google from the product folder but it reports health issues in webmaster tools. Would the best route be a no index tag on all the product pages and to no follow all the products in the category listings? Many thanks Carl
Intermediate & Advanced SEO | | Grumpy_Carl0 -
Advertising links hurt SEO?
I'm working with a publisher who said that having DFA links on his site will hurt his SEO. He is taking my link and pointing it back to his site and then to mine. Does that sound right to you?
Intermediate & Advanced SEO | | GFTMarketer0