Can we listed URL on Website sitemap page which are blocked by Robots.txt
-
Hi,
I need your help here.
I have a website, and few pages are created for country specific. (www.example.com/uk).
I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page)
I really appreciate your help.
Thanks,
Nilay
-
if the content is of benefit to the user then include them in your navigation. Why are you blocking them in the first place, duplicate content?
-
Hi Zora,
But the pages which i have blocked are only visible in specific country. Addition, i have blocked theme also, so you think it's good to put those url on website or link them in website?
-
Hi Zora,
Thanks for your time.
-
Hi Jarno,
Thanks for your time.
Should i put URLs anywhere in my websites which are being excluded by Robot.txt?
Thanks,
Nilay
-
Hi Nilay,
I actually did this yesterday by accident.
I recommend you remove the blocked pages from your XML sitemap, otherwise Google will display a "warning" after you submit it.As far as your HTML sitemap, it does not really matter.
I think you are okay to keep the links there. -
Nilay,
if you have blocked them from your robots.txt but you do enclude them in your sitemap xml or html then they will be indexed unless you enclode a meta robots in it with a noindex tag. If that tag is not in the pages and you enclude that page in your sitemap Google will feel it as important content in list it in the SERPs.
Hope this helps
Regards
Jarno
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disadvantages of Migrating Website to New URL
Hi There, I am currently struggling with the ranking of my website. No matter how many initiatives I try (backlinking, blog commenting, social posting, etc.) I can't seem to make any progression in Google Search. I've done competitive metrics through Open Site Explorer and can't seem to really find the reason why my site is not ranking as well as my competitors. The only one possible glaring element I've thought about is my website URL. This company is in the heating and cooling industry and majority of my competitors have either "heating" or "cooling" or both in their website URL's but mine does not. Does anyone have any thoughts or recommendations on if changing my URL and then redirecting my current URL would be a step in the right direction help me to climb the rankings in Google Search? Thanks!
On-Page Optimization | | MainstreamMktg0 -
Why my website some pages is not index in google ?
Hi, I have submitted my pages in Google fetch for consideration tool but they are not indexed yet in the Google search. Additionally, there is also no error shown by the Google.
On-Page Optimization | | seo.kishore890 -
Product URL
Hey Mozzers, Nice quick and simple one for you. Which of these 2 options is better for SEO and userbility and why domain.co.uk/productname.html
On-Page Optimization | | ATP
domain.co.uk/shop/category/product.html The top one stops stops any funky problems with magento making 2 paths to the same product but the second option feels more natural and helpful to the user. I feel both a valid but I would like some opinions please0 -
Moving from Bigcommerce to Woocommerce on WP. Should we redirect size pages into one page?
We are moving from Bigcommerce to Woocommerce on WP. On Bigcommerce, due to some bizarre reasoning the previous developer had 3 separate URLS for the same product in different sizes - S, M and L. Now we plan to have one product page where the sizes can be selected and 301 redirect the 3 urls to the new one. Is this advisable? Or should we just have 3 separate pages. OR should we have one of the sizes pages as the new page and then redirect the other 2 to this one? I ask this because the site has a LOT of ranking power and we do not want to jeopardise that.
On-Page Optimization | | MashBonigala0 -
Keyword Appearing on Home Page - Moz Page Grader
Hi Today I entered www.partydomain.co.uk through the Moz Page Grader and found that the Home Page is Ranked B. I noticed that an Area we could improve on is the amount of times we are using our main keyword "Fancy Dress" on the home page. Please can you take a look at www.partydomain.co.uk and scroll to the bottom of the page were the tabs are containing losts of content. I am thinking about removing all of thoose Tabs. Our Competitors dont have any content as such on the home page and are ranking higher than Party Domain for "fancy dress" What do you think ? remove all the tabs to be like the others that rank better? Or cut the text right down ? Thanks Adam
On-Page Optimization | | AMG1000 -
Home Page Keywords not Ranking and Assigned to Inside Pages
Hi, thank you for taking the time to read this. We have a few websites with the same problem. I will use http://www.prepared-meals.com as an example: The home page was ranking on page one for keyword "Prepared Meals". The site is about 6 months old. We use the Moz page optimizer on all pages of our websites to score an A rating. Recently we found the home page is no longer showing up in search results and the keyword "prepared meals" now points to an inside page that is not relevant: http://www.prepared-meals.com/Senior-Meals/Moms-Meals-Reviews.html this page shows up for Prepared Meals around page 15 in Google results. We have read keywords in the URL might be the issue, even though the page optimizer in MOZ says to do that. We are wondering if this is the issue or there is some other problem we are not aware of. Again, thank you for you for your time. -Craig
On-Page Optimization | | CraigSWD0 -
In-page Optimization check list....
Hey, Mozzers I have a question about optimizing a web-page. I was just having a conversation with our web developer about optimizing our website and which changes would make the most difference in affecting the SERPs as far as on-page optimization goes. I was explaining to her we wanted to optimize our pages in the following levels (ranked most important to least important): <title>: key word laden </p> <p>2) <H1> : key word in <strong>bold</strong></p> <p>3) page content: keyword laden as well</p> <p>The idea was to have three or four layers of keywords on the page. I work for a real estate brokerage and the context was the actual page of the property listing. So in this case, the keywords would be the address of the property. She explained to me that the the H1 tag is more important than the title tag.</p> <p>Does this ring true with you guys?</p></title>
On-Page Optimization | | AubbiefromAubenRealty0 -
Where to Put Content For Product Pages - How To Structure Website?
Currently we have 300+ products. We do not have a CMS or Ecommerce site at the time being for certain reasons. Currently our site is set up with content on almost every single page. The main catagory page, explains everything on the main page, then our products page has a lot of text too. But right now, it seems as if our main pages are only ranking. In the near future I will be using a cms and purchasing a template. I noticed most Ecommerce style websites have just the product with the name and price, then when they click on the product it brings you to that page with a brief product description and some photos. My question is, does each page need content? Or can just the product page itself have content? For example, say we have a link to SHOES. Then the shoes page displays dress, casual and athletic. Then the athletic page brings you to a page with, running, tennis, cross training shoes, and so forth. Is it best to write content on this main catagory page? If so, how much? Or should we focus on putting content on the actual page of the individual product? Along with pictures and specifications? I know Content is Key and we are doing pretty well at that, however, I am starting to wondering if we have to much content or too similar content. What is the best structure to try and recieve GREAT organic rankings?
On-Page Optimization | | hfranz0