Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?
-
Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC
-
You could also look at using the meta robots = noindex tag on /search/ pages, rather than just blocking it in robots.txt, as this will remove existing URLs from the index.
-
Glad to help
-
Thanks a lot Dan!
-
That is a good recommendation but ultimately search engines will make a final decision on crawl frequency. Take a look at your 'Crawl Stats' on GWTs and this will give you an idea of how often your site is crawled.
-
Is the time issue related in crawl frequency of the URLs in my sitemap?
Thanks Dan, appreciate it.
-
You will probably need to wait a little longer - it depends how often your site usually gets crawled and indexed.
However, robots.txt does not always stop search engines from indexing your pages. It will stop them crawling a page on your site but it tells them that they can still index that page. If they find links from external sites then the URL may still appear in the SERP.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Google Search Console Still Reporting Errors After Fixes
Hello, I'm working on a website that was too bloated with content. We deleted many pages and set up redirects to newer pages. We also resolved an unreasonable amount of 400 errors on the site. I also removed several ancient sitemaps that listed content deleted years ago that Google was crawling. According to Moz and Screaming Frog, these errors have been resolved. We've submitted the fixes for validation in GSC, but the validation repeatedly fails. What could be going on here? How can we resolve these error in GSC.
Technical SEO | | tif-swedensky0 -
I submitted Sitemaps from AIO SEO to google search console, if I now delete the AIO plugin, do my sitemaps become invalid?
I use Yoast as SEO for my new Wordpress website https://www.satisfiedshoes.com/, however I couldn't get the sitemaps with Yoast as it was giving me error 404, and regardless of what I tried, it wasn't working. So I then got the All In One SEO while still having Yoast installed, I easily got the AIO sitemaps and then submitted them successfully to the Google search console. My question is that now I got the sitemaps on Google, since I'd rather use Yoast, If I want to delete AIO, will the sidemaps given to Google become invalid? There is no point keeping both SEO plugins active right? Thank You
Technical SEO | | iamzain160 -
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
Google search console image indexing issue
Google search console tells that only '58 out of the 3553' images in the images sitemap are indexed. But if I search "site:example.com" in Google images there seem to be lots of images. There are no errors in the sitemap and I am still getting reasonable number of image search hits daily. Are the webmaster tools stats for images indexed accurate? When I click on the Sitemap Errors & Index Errors this is what i get - Error details: No errors found. https://www.screencast.com/t/pqL62pIc
Technical SEO | | 21centuryweb0 -
Duplicate pages with "/" and without "/"
I seem to have duplicate pages like the examples below: https://example.com https://example.com/ This is happening on 3 pages and I'm not sure why or how to fix it. The first (https://example.com) is what I want and is what I have all my canonicals set too, but that doesn't seem to be doing anything. I've also setup 301 redirects for each page with "/" to be redirected to the page without it. Doing this didn't seem to fix anything as when I use the (https://example.com/) URL it doesn't redirect to (https://example.com) like it's supposed to. This issue has been going on for some time, so any help would be much appreciated. I'm using Squarespace as the design/hosting site.
Technical SEO | | granitemountain0 -
Will Google still ignore the second instance of anchor text on a page if it has an H2 tag on it?
We have a page set up that has anchor text with header tags. There is an instance where the same anchor text is on the page twice linking to the same page, and I know that Google will ignore the second instance. But in the second instance it also had an H2 tag (which I removed and put it on the first instance of anchor text even though it's smaller). Is this good practice?
Technical SEO | | AliMac260 -
Image Search / sudden drop in traffic
One of our sites in Germany had a very sudden drop in traffic (starting Oct. 7th). The site gets most of it's organic traffic from Image Search. Checking in Search Console revealed that search volume for keywords increased in that period our average position is stable our click rate dropped dramatically (we double checked - searching the keyword in "anonymous mode" still showed our results for main keywords in top image positions (first 2 rows)). As an example (see attached screencopy) - keyword had clickrate of 1% (average) - dan dropped to 0.06% while the position remained stable. Germany is still using the "old" version of image search (unlike the rest of the world) - which gives the site preview rather than just the image slider when you click on a result in image search. Our first thought that this was changed - but it seems that it didn't change. Ideas what might cause this dramatic drop in click%? There have been no major technical modifications on the site for the last 2 months. thanks, Dirk GjlV8CW.jpg
Technical SEO | | DirkC0 -
Unit # No Longer Showing On Google Places
We've noticed that the unit # is no longer showing on a clients Places profile page. Any thoughts on why and if relevant to rankings? Places page - https://skitch.com/kyegrace/8fimr/vuppie-real-estate-team or http://maps.google.ca/maps/place?hl=en&qscrl=1&nord=1&rlz=1T4GGNI_en-GBCA461CA461&gs_upl=&ion=1&bav=on.2,or.r_gc.r_pw.r_cp.,cf.osb&biw=1366&bih=641&wrapid=tlif133037839755610&um=1&ie=UTF-8&q=realtor&fb=1&gl=ca&hq=realtor&hnear=0x548673f143a94fb3:0xbb9196ea9b81f38b,Vancouver,+BC&cid=5594900399034912659&ei=o_ZLT4XHM6ObiQKZnbCdDw&sa=X&oi=local_result&ct=placepage-link&resnum=8&ved=0CIMBEOIJMAc and as it appears in the backend https://skitch.com/kyegrace/8fimk/google-places-analytics Any insight greatly appreciated!
Technical SEO | | kyegrace0