The use of robots.txt
-
-
Thank you Martijn. It helps indeed.
-
Hi Daniela,
I can confirm that it won't be any problem if you don't have a robots.txt file if you don't want to block any pages. For myself I find it more useful to still have a robots.txt file in there which allows search engines to crawl the complete site. But that's just my personal opinion.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt Tester - syntax not understood
I've looked in the robots.txt Tester and I can see 3 warnings: There is a 'syntax not understood' warning for each of these. XML Sitemaps:
Technical SEO | | JamesHancocks1
https://www.pkeducation.co.uk/post-sitemap.xml
https://www.pkeducation.co.uk/sitemap_index.xml How do I fix or reformat these to remove the warnings? Many thanks in advance.
Jim0 -
Create Page Titles from H1 using Yoast?
I'm working on a site that has 280 blog posts that have either been migrated from an old CMS site or created on the Dev version of the new WordPress site. We've written 280 unique meta descriptions so they don't truncate but it there a quick way I can export the current H1s and then import them into Yoast so they are set as the Page Titles? I've written unique Page Titles and meta descriptions for all the Service and Products page and just want a way to speed up the blog posts as their H1s are really good and what I would use as Page Titles anyway. Any help, greatly appreciated!
Technical SEO | | Marketing_Today0 -
What Metadata should one use multi country directory
Currently this is what applies throughout the site. property="og:locale" content="en_GB" /> How would one set this for properties in Italy or Spain for example? (The language is all in English) Regards Tai
Technical SEO | | Taiger0 -
Am I using pagination markups correctly?
Hey Mozzers! I am receiving duplicate title tag errors from Search Console on paginated pages (blog.com/chlorine, blog.com/chlorine-2, blog.com/chlorine-3). I do not currently have a view all page. If I were to create one, would I add all the content from chlorine-2 and chlorine-3 to the blog.com/chlorine page? Then use the rel=cononical on chlorine-2 and chlorine-3 to blog.com/chlorine? If I move forward without the view all page, I could implement the next/prev HTML markups but can I do this without dev help? I am currently using the Yoast SEO plugin and do not see the option. Would I use the text editor to add the markups directly before the content? I think I have a grasp on this, but this will be my first time implementing and I want to double check first! Thanks!
Technical SEO | | localwork0 -
Should you use the keyword for your page in an image?
Hi there! I am currently working on building up the SEO ranking on a page using a specific keyword - dresses. Within this page, we have an online image library full of dresses which are then added to multiple pages determined by brand, colour, type etc for people to search. I am adding hundreds of images all of dresses - I wanted to know if I name these images using the keyword: for example 'dresses_1, 'dresses_2' - will that have a knock on effect on page I am trying to build up and optimise for the keyword 'dresses'??? Any help is appreciated.
Technical SEO | | Jaybeamer0 -
Google is indexing blocked content in robots.txt
Hi,Google is indexing some URLs that i don't want to be indexed and also is indexing the same URLs with https. This URLs are blocked in the file robots.txt.I've tried to block this URLs through Google WebmasterTools but Google doesn't let me do it because this URL are httpsThe file robots.txt is correct so, what can i do to avoid this content to be indexed?
Technical SEO | | elisainteractive0 -
Does anyone use inspect firebug for their site
Hi, i am using wordpress for the first time, i normally use joomla, but now i have been recommend to use inspect firebug to help me sort problems on my site which includes the title of the home page, www.cheapflightsgatwick.com being called - lanzarote reviews - cheap flights gatwick but it should say cheap flights reviews magazine The problem i have with firebug is understanding what it is telling me and where things are, i would like to know if anyone uses this product and if so how easy or hard have they found using it
Technical SEO | | ClaireH-1848860 -
Meta-robots Nofollow
I don't understand Meta-robots Nofollow. Wordpress has my homepage set to this according to SEOMoz tool. Is this really bad?
Technical SEO | | hopkinspat1