Have a Robots.txt Issue
-
I have a robots.txt file error that is causing me loads of headaches and is making my website fall off the SE grid. on MOZ and other sites its saying that I blocked all websites from finding it. Could it be as simple as I created a new website and forgot to re-create a robots.txt file for the new site or it was trying to find the old one? I just created a new one.
Google's website still shows in the search console that there are severe health issues found in the property and that it is the robots.txt is blocking important pages. Does this take time to refresh? Is there something I'm missing that someone here in the MOZ community could help me with?
-
Hi primemediaconsultants!
Did this get cleared up?
-
You not always have to do this, if you would go to domain.com/robots.txt then it should be removed maybe already. If that's the case you should be starting to see an increase in the number of pages crawled in Google Search Console.
-
This seems very helpful as I did remove it, and fetch as google, but i'm a complete novice. How do you clear server cache?
-
What does your robots.txt file contain? (or share the link)
Try removing it, clearing server cache and fetching as google again.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Same product in different categories and duplicate content issues
Hi,I have some questions related to duplicate content on e-commerce websites. 1)If a single product goes to multiple categories (eg. A black elegant dress could be listed in two categories like "black dresses" and "elegant dresses") is it considered duplicate content even if the product url is unique? e.g www.website.com/black-dresses/black-elegant-dress duplicated> same content from two different paths www.website.com/elegant-dresses/black-elegant-dress duplicated> same content from two different paths www.website.com/black-elegant-dress unique url > this is the way my products urls look like Does google perceive this as duplicated content? The path to the content is only one, so it shouldn't be seen as duplicated content, though the product is repeated in different categories.This is the most important concern I actually have. It is a small thing but if I set this wrong all website would be affected and thus penalised, so I need to know how I can handle it. 2- I am using wordpress + woocommerce. The website is built with categories and subcategories. When I create a product in the product page backend is it advisable to select just the lowest subcategory or is it better to select both main category and subcategory in which the product belongs? I usually select the subcategory alone. Looking forward to your reply and suggestions. thanks
Intermediate & Advanced SEO | | cinzia091 -
Robots.txt Disallowed Pages and Still Indexed
Alright, I am pretty sure I know the answer is "Nothing more I can do here." but I just wanted to double check. It relates to the robots.txt file and that pesky "A description for this result is not available because of this site's robots.txt". Typically people want the URL indexed and the normal Meta Description to be displayed but I don't want the link there at all. I purposefully am trying to robots that stuff outta there.
Intermediate & Advanced SEO | | DRSearchEngOpt
My question is, has anybody tried to get a page taken out of the Index and had this happen; URL still there but pesky robots.txt message for meta description? Were you able to get the URL to no longer show up or did you just live with this? Thanks folks, you are always great!0 -
Client wants to show 2 different types of content based on cookie usage - potential cloaking issue?
Hi, A client of mine has compliance issues in their industry and has to show two different types of content to visitors: domain.com/customer-a/about-us domain.com/customer-b/about-us Next year, they have to increase that to three different types of customer. Rather than creating a third section (customer-c), because it's very similar to one of the types of customers already (customer-b), their web development agency is suggesting changing the content based on cookies, so if a user has indentified themselves as customer-b, they'll be shown /customer-b/, but if they've identified themselves as customer-c, they'll see a different version of /customer-b/ - in other words, the URL won't change, but the content on the page will change, based on their cookie selection. I'm uneasy about this from an SEO POV because: Google will only be able to see one version (/customer-b/ presumably), so it might miss out on indexing valuable /customer-c/ content, It makes sense to separate them into three URL paths so that Google can index them all, It feels like a form of cloaking - i.e. Google only sees one version, when two versions are actually available. I've done some research but everything I'm seeing is saying that it's fine, that it's not a form of cloaking. I can't find any examples specific to this situation though. Any input/advice would be appreciated. Note: The content isn't shown differently based on geography - i.e. these three customers would be within one country (e.g. the UK), which means that hreflang/geo-targeting won't be a workaround unfortunately.
Intermediate & Advanced SEO | | steviephil0 -
Google adwords destination link issue
In google adwords I set destination link like this http://www.abcd.com/ and like this http/:www.abcd.com so how google adwords react on both this? And how it will show data in adwords? Thanks! Dev
Intermediate & Advanced SEO | | devdan0 -
Multi Language Redirect Issues
Hello everyone, this is my first post and as so let me first say, Thank you! The SEO Moz community and SeoMozPro have been giving a great help in making my workflow simpler and richer. Lately I've been reading and learning a lot about indexation, in the process I have been making several improvements to some websites, but there is one particular that I am not able to understand. I am writing this post to ask for your help on an issue related to this website: www.dengun.com We are a Web Agency based in Portugal and most our clients are from Portugal. We have an English version and a Portuguese version of the website. It is setup like this: www.dengun.com/en www.dengun.com/pt When the user hits www.dengun.com it redirects to /en or /pt acording to the browser language. The HTTP status code is 302, i was reading in SEOMoz that this is bad because it's not passing rank to the other pages. Will a 301 redirecting to /en and /pt according to the browsers language? What is the best solution? Thanks.
Intermediate & Advanced SEO | | PedroSaiote0 -
Duplicate Page Title/Content Issues on Product Review Submission Pages
Hi Everyone, I'm very green to SEO. I have a Volusion-based storefront and recently decided to dedicate more time and effort into improving my online presence. Admittedly, I'm mostly a lurker in the Q&A forum but I couldn't find any pre-existing info regarding my situation. It could be out there. But again, I'm a noob... So, in my recent SEOmoz report I noticed that over 1,000 Duplicate Content Errors and Duplicate Page Title Errors have been found since my last crawl. I can see that every error is tied to a product in my inventory - specifically each product page has an option to write a review. It looks like the subsequent page where a visitor can fill out their review is the stem of the problem. All of my products are shown to have the same issue: Duplicate Page Title - Review:New Duplicate Page Content - the form is already partially filled out with the corresponding product My first question - It makes sense that a page containing a submission form would have the same title and content. But why is it being indexed, or crawled (or both for that matter) under every parameter in which it could be accessed (product A, B, C, etc)? My second question (an obvious one) - What can I do to begin to resolve this? As far as I know, I haven't touched this option included in Volusion other than to simply implement it. If I'm missing any key information, please point me in the right direction and I'll respond with any additional relevant information on my end. Many thanks in advance!
Intermediate & Advanced SEO | | DakotahW0 -
Is my Company Blog Causing issues?
We have a company blog. IE companydomain.com/blog/. Much of the content is unrelated to what we do. So why do we do it? Cover local topics that our customers may be interested in and searching for. Get them in the back door so to speak. Link bait, social sharing etc Develop a marketing culture that's engaged always learning. Customers get to know the people behind the company. Here's my Pickle If i sub domain or separate the site entirely, i won't benefit from all the juice we're generating. If i keep the status quo, since much of the content isn't directly related to my "category", logic tells me that i could be diluting my website in Googles prying eyes. Feedback PLEASE.
Intermediate & Advanced SEO | | schmeetz0 -
Randomly Displayed Text: Hidden text issue?
I want to add some script to my site so that a given page publishes a different paragraph of text every time the page loads. Something like randomly displayed testimonials (but with more text). So, when you look at the page source, you would see all the text (e.g testimonial-1, testimonial-2, etc.), but the user would only see one paragraph randomly. Would this be considered hidden text (one code for search engine, one for use)? Is there a safe number of words you can do this with without setting off red flags? I appreciate the help.
Intermediate & Advanced SEO | | inhouseseo0