Google is Still Blocking Pages Unblocked 1 Month ago in Robots
-
I manage a large site over 200K indexed pages. We recently added a new vertical to the site that was 20K pages. We initially blocked the pages using Robots.txt while we were developing/testing. We unblocked the pages 1 month ago. The pages are still not indexed at this point. 1 page will show up in the index with an omitted results link. Upon clicking the link you can see the remaining un-indexed pages. Looking for some suggestions. Thanks.
-
Thanks. I fetched both the main page and made a slight tweak to the robots and resubmitted last night. It looks like it is making a bit of progress. There is nothing else blocking the pages. We did add a new sitemap when we first launched the pages with no warnings. I did notice yesterday that we had approx 268 broken links that went to 404 pages in WMT in this specific sub-folder. The discovery of the broken link-404 pages by Google seems to be around the same time Google stopped crawling this section of the site. We took care of the broken links this morning. Thanks for the help!
-
Hi,
Fetch the main page(s) with "Fetch as Google" under the Crawl section in Webmaster tools - then submit to the index.
You are sure that there are no other elements blocking the indexing of the page (like meta tag or X-robots tags in the header?
Also fetch the new robots.txt file - to be sure that Google notices that it has changed.
Did you add a sitemap for this new section - does it show any notifications/warnings in WMT?
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page disappears from Google search results
Hi, I recently encountered a very strange problem.
Technical SEO | | JoelssonMedia
One of the pages I published in my website ranked very well for a couple of days on top 5, then after a couple of days, the page completely vanished, no matter how direct I search for it, does not appear on the results, I check GSC, everything seems to be normal, but when checking Google analytics, I find it strange that there is no data on the page since it disappeared and it also does not show up on the 'active pages' section no matter how many different computers i keep it open. I have checked to page 9, and used a couple of keyword tools and it appears nowhere! It didn't have any back links, but it was unique and high quality. I have checked on the page does still exist and it is still readable. Has this ´happened to anyone before? Any thoughts would be gratefully received.0 -
What to do when Google automatically replaces the page title.
Hello Friends, Might you all are aware of the scenario when Google auto generates the snippets for search results. But nowadays I am seeing some changes like google is showing some specific words in the last of search results title for every page of my website. It looks Google is treating those words as the brand name. I have tried many things to solve this but unfortunately, nothing works for this. Does anyone see the same changes? Can anybody help me out with this or suggest me the reasons behind this.
Technical SEO | | Shalusingh1 -
Google Listing Brand Name as Page Title rather than actual set Title
Any search result including our website is displaying our search result title like this: "Brand Name: Partial Page Title" instead of what it should be "Page Title" (as an example). Where is the "Brand Name:" coming from? We've verified we don't have any code that could cause this, and we're blocking in robots.txt directory listings being used for our search result title/meta. This isn't happening for our competitors. Any ideas on why this is, and if it's something we can control?
Technical SEO | | Closetstogo0 -
Robots txt. in page with 301 redirect
We currently have a a series of help pages that we would like to disallow from our robots txt. The thing is that these help pages are located in our old website, which now has a 301 redirect to current site. Which is the proper way to go around? 1- Add the pages we want to disallow to the robots.txt of the new website? 2- Break the redirect momentarily and add the pages to the robots.txt of the old one? Thanks
Technical SEO | | Kilgray0 -
Page not indexed but still has a PageRank, how?
http://www.optiproerp.com/products.aspx page is not indexed in Google but still has a PageRank of 1. How? Regards
Technical SEO | | IM_Learner0 -
Google Places Page Changes
We had a client(dentist) hire another marketing firm(without our knowledge) and due to some Google page changes they made, their website lost a #1 ranking, was disassociated with the places page and was placed at result #10 below all the local results. We quickly made some changes and were able to bring them up to #2 within a few days and restore their Google page after about a week, but the tracking/forwarding phone number the marketing company was using shows up on the page despite attempts to contact Google through updating the business in places management as well as submit the phone number as incorrect while providing the correct phone number. And because the client fired that marketing company, the phone number will no longer be active in a few days. Of course this is very important for a dental office. Has anyone else had problems with the speed and updating Google Places/Plus pages for businesses? What's the most efficient way to make changes like this?
Technical SEO | | tvinson0 -
How many times robots.txt gets visited by crawlers, especially Google?
Hi, Do you know if there's any way to track how often robots.txt file has been crawled? I know we can check when is the latest downloaded from webmaster tool, but I actually want to know if they download every time crawlers visit any page on the site (e.g. hundreds of thousands of times every day), or less. thanks...
Technical SEO | | linklater0 -
Organic traffic still down 9 months after redesign
The Good: We redesigned our nature travel website (www.ietravel.com) in Drupal. Overall, it's a great improvement in look and usability. Also, we are ranking for more relevant search terms (the SEO was managed by an agency before, and there were a lot of junk terms in their campaigns that weren't converting). The Bad: Organic search referrals have consistently been down 10-20% year-over-year each and every month. The Ugly: I am trying to dig in and figure out why this is happening, and I'm at a loss. We are aggressively publishing to our blog 5 days a week, and I've built many keyword-focused landing pages. Here's what I do know in terms of things that could be problems which I've seen in Webmaster Tools and SEOmoz tools. I have a lot of files restricted by robots.txt - 1,337 of them. Many have to be that way by design because they are nodes generated by web forms (w/ private user data). The rest are "Dates & Rates" pages - I restricted them because for each destination they are very similar in content. Wondering now if that was a mistake. For example, http://www.ietravel.com/central-south-america/galapagos-islands/dates-rates We have duplicate title tags on 462 pages. The Lightbox Module that was installed for our photo galleries was a disaster. I am researching a more SEO-friendly solution, but that solution is a month or more away. We have 31 duplicate meta descriptions. My question is, could these errors be THAT significantly impacting our rankings? I should note that according to Google Analytics, Referral traffic & Direct traffic is also year-over-year every month since the redesign. I don't understand the Referral part especially, since we took great pains to put in many 301 redirects. There are no 404s or non-indexable pages showing up in Webmaster Tools either. If anyone has any suggestions for problem areas or red flags I should investigate, please let me know. Really, any thoughts are appreciated. Best, Carlton
Technical SEO | | csmithal1