Get a list of robots.txt blocked URL and tell Google to crawl and index it.
-
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list.
My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches,
One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file.
I need urgent recommendation as I do not want to see drop in my traffic any more.
-
"changing the lastmod of those pages to today".
How can I make these changes?
Right now the news is that Resubmitted the Sitemap and no warnings this time.
-
I imagine that since you've got a robots txt error you'll probably ended closing a whole directory to bots which you wanted to be indexed. You can easily spot the directory and resubmit a sitemap to google changing the lastmod of those pages to today and the priority to 1 but only of those pages.
If you still receive warnings it may be due to errors in your sitemap. You're probably including some directory you don't want. You can try it in GWT putting in the box at the bottom the url you want to maintain in the index and then trying to see if some urls are being blocked by your robots.
If you want you can post here your robots and the URIs you want to be indexed without knowing the domain so that won't be public. Hope this may help you
-
Ok Resubmitted it.but even with updated file it gives a lot of errors.I think it takes some time.20,016 warnings
I have not added no index attribute in my header region.It was all messy stuff with robots.txt file.It means that with site showing up in SERP the rank will probably be the same or it was deranked?
-
Go into GWMT and resubmit sitemap.xml files (with the URLs you want indexed) for recrawling and Google will digest the sitemaps again, instead of waiting for Googlebot to come around on their own, you are requesting it to come around, also include those new sitemap files in your robots.txt file.
-
In Google Webmaster Tools, go to Health -> Fetch As Google. Then add the previously blocked URL and click Fetch. Once you've done that, refresh the page and click "Submit to index". That should get Google indexing those pages again.
Getting external links to your pages also helps get pages crawled & indexed, so it may be worth submitting your pages to social bookmarking sites, or get other types of backlinks to your previously blocked pages if possible.
-
Since you fixed your robots.txt file you should be good to. It will probably take a few days for Google to recrawl your site and update the index with the URLs they are now allow to crawl.
Blocked URLs can still show up in SERPs if you haven't defined the no-index attribute in your section.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google analytics
hello there, While Adding My website in Google Analytics,It is not Showing India Country in list. What i can do to add My account in it?? Have requested Google, But Got No response.any body can help please . Thanx in advance,
Reporting & Analytics | | iepl5
Falguni0 -
Google analytics question
Ok so in my traffic sources break down I have 3 sections: direct, organic, and referral. My question is under the referral tab I have recently noticed a new traffic source, my own website.... How is this possible? My top referring site is my own website.... Is this considered direct traffic or how is this being traced?
Reporting & Analytics | | jameswalkerson0 -
Using Regex for Goals in Google Analytics
I’ve got a website with 20 different forms; and I’d like to track all 20 form completions with one Goal in Google Analytics. The completion-page URLs follow the pattern below (the website uses Drupal CMS) Form One: www.acme.com/node/2990/done?sid=651 Form Two: www.acme.com /node/2991/done?sid=785 Form Three: www.acme.com /node/2992/done?sid=1021 Form Four: www.acme.com /node/2993/done?sid=459 I believe that there is a way I can use regex so Google Analytics will track if any of these completion URLs appears. Looking for guidance how to format the regex statement.
Reporting & Analytics | | TopFloor0 -
Google Analytics Report throws up Google as a referrer
Good morning from Wet & Windy 12 degrees C wetherby UK... Using Google analytics I've noticed in the traffic sources refferer subsection some traffic is categorized as originating from Google. Whats puzzling me is.... I know a huge amount of traddic stems from Google but as the below screenshot illustrates only 21 visitors come from Google: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-refferal-sources-top-levelcopy.jpg And when i drill down some are coming from Google mobile 😞 http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-referral-sourcescopy.jpg Is traffic categorised as Google referrer down to Google hiding searches via ssl as explained here: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-referral-sourcescopy.jpg Any insights welcome 🙂
Reporting & Analytics | | Nightwing0 -
Exporting Keywords in Google Analytics
In the old Google Analytics, I would simply add &limit=30000 in the URL. Does anyone know how to export all keywords from the new Google Analytics?
Reporting & Analytics | | TommySwanson520 -
How do I best segment tablets on Google Analytics
I would like to find a way to best segment out my tablet traffic to measure performance; however I'm finding that there are road blocks. It doesn't seem that device operating systems or screen resolutions have clear cut differences in the tablet/mobile versions. Has anyone here found a good way to create a "tablet" segment in Google Analytics? Right now I'm having to lean on solely the ipad traffic to get indicators of tablet performance. Thanks!
Reporting & Analytics | | lvstrickland0 -
Not ranking in Google.com
I put up my last site on the web since a month. So far I have been optimizing mainly to Hungary and I got used to that my content was indexed in a day and if content was good it sometimes appeared in the first two pages in a couple of days. Now with my new site I am targeting google.com. I put it up since a week, sent the sitemap to google, it was intresting for mee te see that even the pages to get into the web index needed two days. Seomoz says my site is all right besides some duplicate content issue i will solve soon. So it is past a week and even if I copy a complete sentence from the beginning of my home page and paste it into google my site does not appear. I also purchased couple of backlins but they have not appeared so far as well. Is that really this slow? Am I to impatient? Or should there be something else problem I should be looking for? Thanks for any feedback
Reporting & Analytics | | sesertin0 -
Google Docs Paranoia
Recently, there has been a lot of great information on SEOmoz about using Google Apps, Docs, etc. However, I suffer from Google paranoia (the fear that Google can see what's going on in my Google products). Is this fear unreasonable? Should I be concerned about using Google Apps, docs, analytics, etc. for SEO data (including keyword position tracking, back link analysis, etc)?
Reporting & Analytics | | Gyi0