Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?
-
Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC
-
You could also look at using the meta robots = noindex tag on /search/ pages, rather than just blocking it in robots.txt, as this will remove existing URLs from the index.
-
Glad to help
-
Thanks a lot Dan!
-
That is a good recommendation but ultimately search engines will make a final decision on crawl frequency. Take a look at your 'Crawl Stats' on GWTs and this will give you an idea of how often your site is crawled.
-
Is the time issue related in crawl frequency of the URLs in my sitemap?
Thanks Dan, appreciate it.
-
You will probably need to wait a little longer - it depends how often your site usually gets crawled and indexed.
However, robots.txt does not always stop search engines from indexing your pages. It will stop them crawling a page on your site but it tells them that they can still index that page. If they find links from external sites then the URL may still appear in the SERP.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google still listing pages from old domain after 2 change requests
Good Morning I put forward the following question in December 2014 https://moz.com/community/q/google-still-listing-old-domain as pages from our old domain www.fhr-net.co.uk were still indexed in Google. We have submitted two change request in WMT, the most recent was over 6 months ago yet the old pages are still being indexed and we can't see why that would be Any advice would be appreciated
Technical SEO | | Ham19790 -
Not showing the right results in Google.de
Hi Moz, I have a question concerning Vintykids.com.
Technical SEO | | B.Great
The site comes in four languages: German on vintykids.com/de Dutch on vintykids.com/nl English on vintykids.com/en French on vintykids.com/fr The German language gives a problem. In Google.de (German Google) the site is completely indexed in German but we also see results in Dutch. So when you do a search in Google.de on their brandname (Vintykids) we see results in Dutch on vintykids.com.
We think we have set the meta's right in the German language on vintykids.com/de and we also managed some links to Vintykids.com/de from good quality and relevant German sites. What can we do further to get vintykids.com/de ranking in Google.de on the brandname? Thank you.0 -
Self proclaimed 5 stars showing in organic search??
Hi there - I've just come across this site that has ratings showing in the organic listings, yet - I can't see any ratings for it online - other than a text listing on the site's own home page? The site in question has a facebook page with no followers, and a twitter account with the same, no google plus account, no google local maps listing. I've only ever seen "actual" google reviews and other citations being combined, averaged - then shown in search as an aggregated number. Is it just me - or does Google accept these self rated reviews on face value??!! See images attached... The URL of the site is here: http://www.home-securitysystems.com.au The search phrase they rank no.1 for is "Alarm System Melbourne" - which is pretty decent. Any ideas? Cheers, Dave oi86 oi8o
Technical SEO | | daveyboy0 -
Some competitors have a thumbnail in Google search results
I've noticed that a few of my top competitors have a small photo (thumbnail) next to their listing. I'm sure it's not a coincidence that they are ranked top for the search phrase too. Is this really a help and how can it be done? Many thanks, Iain.
Technical SEO | | iainmoran0 -
Can you have a /sitemap.xml and /sitemap.html on the same site?
Thanks in advance for any responses; we really appreciate the expertise of the SEOmoz community! My question: Since the file extensions are different, can a site have both a /sitemap.xml and /sitemap.html both siting at the root domain? For example, we've already put the html sitemap in place here: https://www.pioneermilitaryloans.com/sitemap Now, we're considering adding an XML sitemap. I know standard practice is to load it at the root (www.example.com/sitemap.xml), but am wondering if this will cause conflicts. I've been unable to find this topic addressed anywhere, or any real-life examples of sites currently doing this. What do you think?
Technical SEO | | PioneerServices0 -
Why isn't Google pushing my Schema data to the search results page
I believe we have it set up right. I'm noticing all my competitors schema data is showing up which is really giving them a leg up on us. We have a high ranking website so I'm just not sure why it's now showing up. Here is an example URL http://www.airgundepot.com/3576w.html I've used the Google webmaster tools tester and it all looks fine. Any ideas? Thanks in advance.
Technical SEO | | AirgunDepot0 -
My organic search results are down 16% since the Penguin update 4/24
Penguin has affected my search results down 16% When I look at my SEOmoz scan the only problem I see is "too many on page links" The problem is that my blog for each month is considered one page-eg august 2007 I wrote many blogs-the total on page links was 106-but that included all the blogs that were written in a month. The other problem area is duplicate content. I thought Penguin was after "link farming" which I do not do. Any advice how I can correct this? Brooke
Technical SEO | | wianno1680 -
Do we need to manually submit a sitemap every time, or can we host it on our site as /sitemap and Google will see & crawl it?
I realized we don't have a sitemap in place, so we're going to get one built. Once we do, I'll submit it manually to Google via Webmaster tools. However, we have a very dynamic site with content constantly being added. Will I need to keep manually re-submitting the sitemap to Google? Or could we have the continually updating sitemap live on our site at /sitemap and the crawlers will just pick it up from there? I noticed this is what SEOmoz does at http://www.seomoz.org/sitemap.
Technical SEO | | askotzko0