Robots.txt Help
-
I need help to create robots.txt file.
Please let me know what to add in the file. any real example or working example.?
-
Michael, from what i can tell, your website is built using WordPress. We typically recommend installing the Yoast SEO plugin and using that--which will help with your robots.txt file. If you need more information, take a look here: https://yoast.com/wordpress-robots-txt-example/
Generally, most of your site won't need to be disallowed in the robots.txt file, unless you're using tags and categories on your site. Yoast typically helps disallow the proper directories that you need to disallow.
One thing that you need to be aware of is the fact that you don't want to disallow your .CSS or .JS files on your site, many of the themes nowadays will put those files in your wp-admin folder--which by default typically gets disallowed.
-
This is the site I used to really get a good understanding of how to create a robots.txt file: http://www.robotstxt.org/
-
A very basic robots.txt file would look something like the below
User-agent: *
Sitemap: http://www.yourwebsite.com/sitemap.xml
Disallow: http://www.yourwebsite.com/url-you-dont-want-indexed
Disallow: http://www.yourwebsite.com/another-url-you-dont-want-indexedHope that helps
-
Include sitemaps. Disallow: Pages that you don't want indexed: search pages, login pages, core admin files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help with structure for optimizing Photography Website SEO
Hey guys , I am building a photography website and currently I have it setup the following way for my image galleries : https://ricfrancophotography.com/portfolio/norway-landscape-photography/#!gallery6618-6577 This provides me with an individual link for each of the images in my photography gallery but as a gallery these have obviously no content and I figured the best way would be to add the images I want to work my SEO for to individual blog posts . So here is what I did so far : - Added a link to the caption of each image inside the lightbox that is linking to the individual blog post - In order to not break the navigation I made the post with the content for each image open in a modal popup (it changes the link in the top bar but once closed it goes back to the gallery) . - I made the image inside the post link back to the fullsize image in the lightbox gallery when clicked instead of linking to the .jpg in wp-content/uploads. Now, I have some questions regarding whether this is a good practice in terms of SEO and if the fact of having duplicate images or this structure is going to hurt my SEO any way . Although both images are in different urls they ultimately link to each other this way : Blog Image --> Gallery Image url --> wp-content/uploads/file.jpg Is there a better approach for this ? Thanks
Intermediate & Advanced SEO | | ricfranco0 -
HELP: Why do I have a 61% score for "% of total links, external + follow"?
Firstly, I understand what this percentage is. It's the ratio of external links that are "follow" -> compared to the links that are "no-follow". Four questions: This is definitely not accurate! I have loads of no-follow links Does anyone have ideas or techniques to add more healthy no-follow links? Am I completely misunderstanding this? Will this high score negatively affect my ranking? I could definitely use some help. Thanks so much in advance. I don't think my website address should help, but if you need it for context, it's estatediamondjewely.com.
Intermediate & Advanced SEO | | SamCitron0 -
301 redirection help needed!
Hi all, So if we used to have a domain (let's say olddomain.com) and we had a new site created at newdomain.com how do we properly setup redirects page to page. Caveat, the urls have changed so for instance the old page oldomain.com/service is now newdomain.com/our-services on the new site. Do we need to have hosting on the old site? Do we need to setup individual 301s for each page corresponding to the new page? Just looking for the easiest way to do this CORRECTLY. Thanks, Ricky
Intermediate & Advanced SEO | | RickyShockley3 -
Ratings Snippets Gone? ( Help! )
Hello We had good traffic from ratings ( stars ) . I have added Offer details in the rich snippets in various currencies - the snippet testing tool likes it , but for some reason the stars on my site have completely dissapeared and been gone for almost a week. I need the offer information in there for google shopping automatic updates and google told me that it's implemented correctly for the shopping part.. but I really don't know what to do about this. Any ideas why would be really appreciated. http://www.return2health.net/yeast-imbalance/threelac-candida-defence/ Thanks 🙂
Intermediate & Advanced SEO | | s_EOgi_Bear0 -
Looking for help with my website
Hi does any one know of a good seo company that will get results, i.e., fix site issues and get the site improving in the serps.
Intermediate & Advanced SEO | | Taiger0 -
Blog Not Ranking Well at All in Search Engines, Need Help!
Hi Mozers, Need some help on a CMS I've been working with over the last year. The CMS is built by a team of guys here in Washington State. Basically, I'm having issues with clients content on the blog system not getting ranking correctly at all. Here's a few problems I've noticed: Could you confirm and scale these problems based upon being, "not a problem" "a problem" and "critical must fix" 1. The title tag is pulling from the title of the article which is also automatically generating a URL with underscores instead of dashes. Is having a duplicate URL, Title, and Title tag spammy looking to search engines? Are underscores on long URL's confusing google? Where shorter one's are fine (i.e. domain/i_pad/
Intermediate & Advanced SEO | | Keith-Eneix
(i.e.http://www.ductvacnw.com/blog/archives/2013/05/20/5_reasons_to_hire_a_professional_to_clean_your_air_ducts_and_vents), 2. The CMS is resolving all URL's with a canonical instead of a 301 redirect (I've told webmaster tools which preferred url should be indexed). Does using a canonical over a 301 redirect cause any confusion with Google? Is one better practice then the other? 3. The H1 tags on the blog pull from "blog category" instead of the title of the blog post. Is this is a problem? 4. The URl's are quite long with the added "archives/2013/05/20/5". Does this cause problems by pushing the main target keyword further away from the domain name? 5. I'm also noticing the blog post is actually not part of the breadcrumbs where we normally would expect that to populate after the blog category name, Problem? These are some of the things I've noticed and need clarification on. If you see anything else please let me know?0 -
How can I improve my rankings in Google with help of seomoz
Hey guys, I have joined seomoz today and set up campaigns for my sites. I got reports about keyword rankings, errors, notices etc. But I am still confused about how to use seomoz in order to improve my rankings. My point is does seomoz provides any services for improving position in google or simply seomoz provides only reporting? These reports are good but my ultimate goal to join seomoz is to improve my rankings for my each website and each post. Please help. BJ
Intermediate & Advanced SEO | | intmktcom0 -
Help! My Domain Authority keeps dropping! What do I do?
Hey! I just noticed my Domain Authority keeps dropping? What's happening? What do I do to get it better. I'm scared and dont know the next move to make to get this site better. Help please! Thanks! http://www.moondoggieinc.com Kristy O
Intermediate & Advanced SEO | | KristyO1