Robots blocked by pages webmasters tools
-
- a mistake made in software. How can I solve the problem quickly? help me.
-
Ok,
Updated.
I need to wait 1 week then
Thank You, Gaston
-
That sitemap line place it at the end of the file.
like this:User-agent: *
Disallow:Sitemap: https://site.com/sitemap.xml
Best luck!
-
-
its better to set it as I've discribed before:
User-agent: *
Disallow:It would take over a week, depending in scale of the website.
-
-
Hello there!
Have you identified all pages that you want to be blocked?
If you just do not want any page to be blocked, just edit you robots.txt as:User-agent: *
Disallow:For more information, check these resources:
Cómo bloquear URLs con robots.txt - Google Webmasters
About /robots.txt - robotstxt.orgBest Luck!
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should my back links go to home page or internal pages
Right now we rank on page 2 for many KWs, so should i now focus my attention on getting links to my home page to build domain authority or continue to direct links to the internal pages for specific KWs? I am about to write some articles for several good ranking sites and want to know whether to link my company name (same as domain name) or KW to the home page or use individual KWs to the internal pages - I am only allowed one link per article to my site. Thanks Ash
Intermediate & Advanced SEO | | AshShep10 -
GWT URL Removal Tool Risky to Use for Duplicate Pages?
I was planning to remove lots of URL's via GWT that are highly duplicate alike pages (similar pages exist on other websites across the web). However, this Google article had me a bit concerned: https://support.google.com/webmasters/answer/1269119?hl=en I already have "noindex, follow" on the pages I want to remove from the index, but Google seems to take ages to remove pages from index, which appear to drag down unique content pages from my site.
Intermediate & Advanced SEO | | khi50 -
Page is noindex
Hi, We set pages with this and i can see in the view source of the page <meta name="robots" content="noindex"/> We had a new page posted in the site and its indexed by Google but now the new post is visible on a page thats shows partial data which we noindexed as above because its duplicate data and search engines dont have to see it But its still crawling Any ideas?
Intermediate & Advanced SEO | | mtthompsons0 -
Www vs. non-www differences in crawl errors in Webmaster tools...
Hey All, I have been working on an eCommerce site for a while that to no avail, continues to make me want to hang myself. To make things worth the developers just do not understand SEO and it seems every change they make just messes up work we've already done. Job security I guess. Anywho,most recently we realized they had some major sitemap issues as almost 3000 pages were submitted by only 20 or so were indexed. Well, they updated the sitemap and although all the pages are properly indexing, I now have 5000+ "not found" crawl errors in the non-www version of WMT and almost none in the www version of the WMT account. Anyone have insight as to why this would be?
Intermediate & Advanced SEO | | RossFruin0 -
What should I block with a robots.txt file?
Hi Mozzers, We're having a hard time getting our site indexed, and I have a feeling my dev team may be blocking too much of our site via our robots.txt file. They say they have disallowed php and smarty files. Is there any harm in allowing these pages? Thanks!
Intermediate & Advanced SEO | | Travis-W1 -
404 in google webmaster tool
I have redesigned my website with new web address over 6 months ago and in the google webmaster tools it still shows my old urls with a reponse code 404 and still crawls those pages. How do I make sure they don't appear anymore in the webmaster tool and don't get crawled anymore ? or should I do a re-direct ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Penalized for "Unnatural Links" on Webmaster Tools
Has anyone ever logged in to Google Webmaster tools and seen a message about them seeing unnatural links (as a warning) Our homepage lost all its rankings. I will submit a reconsideration request. We don't engage in link buying practices (some directories, thats all.) Any feedback, please? Thanks
Intermediate & Advanced SEO | | PaulDylan0 -
Does a page on a site with high domain authority build page authority easier? i.e. less inbound links?
Is this also why people build backlinks to their BBB profiles, Yellowpages Profiles, etc. i.e. why do people build backlinks to other pages that link to them? Wouldn't it be more beneficial to just build that backlink directly to your target?
Intermediate & Advanced SEO | | adriandg0