Have a Robots.txt Issue
-
I have a robots.txt file error that is causing me loads of headaches and is making my website fall off the SE grid. on MOZ and other sites its saying that I blocked all websites from finding it. Could it be as simple as I created a new website and forgot to re-create a robots.txt file for the new site or it was trying to find the old one? I just created a new one.
Google's website still shows in the search console that there are severe health issues found in the property and that it is the robots.txt is blocking important pages. Does this take time to refresh? Is there something I'm missing that someone here in the MOZ community could help me with?
-
Hi primemediaconsultants!
Did this get cleared up?
-
You not always have to do this, if you would go to domain.com/robots.txt then it should be removed maybe already. If that's the case you should be starting to see an increase in the number of pages crawled in Google Search Console.
-
This seems very helpful as I did remove it, and fetch as google, but i'm a complete novice. How do you clear server cache?
-
What does your robots.txt file contain? (or share the link)
Try removing it, clearing server cache and fetching as google again.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple h1 tags on this html 5 page a issue?
Hi Guys, I have a html5 page located here: https://tinyurl.com/yc6s3xs2 I know from some online discussions having multiple h1 tags on HTML 5 pages like this, shouldn't be an issue. Any thoughts on this? Cheers,
Intermediate & Advanced SEO | | bridhard80 -
Robots.txt wildcards - the devs had a disagreement - which is correct?
Hi – the lead website developer was assuming that this wildcard: Disallow: /shirts/?* would block URLs including a ? within this directory, and all the subdirectories of this directory that included a “?” The second developer suggested that this wildcard would only block URLs featuring a ? that come immediately after /shirts/ - for example: /shirts?minprice=10&maxprice=20 BUT argued that this robots.txt directive would not block URLS featuring a ? in sub directories - e.g. /shirts/blue?mprice=100&maxp=20 So which of the developers is correct? Beyond that, I assumed that the ? should feature a * on each side of it – for example - /? - to work as intended above? Am I correct in assuming that?
Intermediate & Advanced SEO | | McTaggart0 -
Weird Indexation Issue
On this webpage, we have an interactive graphic that allows users to click a navigational element and learn more about an anatomical part of the knee or a knee malady. For example, a user could click "Articular Cartilage" and they will land on this page: http://www.neocartimplant.com/knee-anatomy-maladies/anatomy/articular-cartilage The weird thing is whether you perform a Google Search for the above URL or for a string of text on that URL (i.e. "Articular cartilage is hyaline cartilage (as opposed to menisci, which consists of fibrocartilage) on the articular surfaces, or the ends, of bones. This thin, smooth tissue lines both joint surfaces where the bones come together to form the knee. ") the following page ranks: http://www.neocartimplant.com/anatmal/knee-anatomy-maladies/anatomy/articular-cartilage.php I have two questions: 1 - Any idea on how the Googlebot is getting to that page?
Intermediate & Advanced SEO | | davidangotti
2 - How should I get the Googlebot to index the correct page (http://www.neocartimplant.com/knee-anatomy-maladies/anatomy/articular-cartilage)? Thanks in advance for your help!0 -
Replatforming possible issue with Submitting URLS
We are replatforming an ecommerce site and will need to change 90% of the urls. Many current urls contain uppercase characters and the new system forces all lowercase. We are concerned with submitting the urls all at once to google it might look like spam, receive some sort of penalty or negatively affect organic search. In the last month this site received 428k unique visitors, 3.2 mil page views and has about 10k urls. They are a top 3 competitor in their vertical. We are certainly planning to do all 301 redirects. What can we do additionally to reduce the risk of penalties here?
Intermediate & Advanced SEO | | RocketWeb0 -
Silo This! Siloing issue with KW targets and multiple categories
I am having a difficult time determining how to silo the content for this website (douwnpour). The issue I am having is that as I see it there are several different top-level keyword targets to put at the top of the silos, however due to the nature of the products they fit in almost every one of the top-level categories. For instance our main keyword term is "Audio Books" (and derivatives thereof). but we also want to target "Audiobook Downloads" and "Books on CD". Due to the nature of the products, almost every product would fit in all 3 categories. It gets even worse when you consider normal book taxonomy. The normal breakdown would be from audiobooks>Fiction(or Nonfiction). Now each product also belongs to one of these categories, as well as "download", "CD", and "Audiobook". And still worse, our navigation menus link every page on the site back to all of these categories (except audiobooks, as we don't really have a landing page for that besides the home page, which is lacking in optimized content, but is linked from every page on the site.) So, I am finding siloing, or developing a cross-linking plan that makes sense very difficult. It's much easier at the lower levels, but at the top things become muddy. Throw in the idea that we may eventually get e-books as well, and it gets even muddier. I have some ideas of how to deal with some of this, such as having the site navigation put in an i frame, instituting basic breadcrumbs, and building landing pages, but I'm open to any advice or ideas that might help, especially with the top level taxonomy structure. TIA!
Intermediate & Advanced SEO | | DownPour0 -
Using WP All Import csv import plugin for wordpress to daily update products on large ecommerce site. Category naming and other issues.
We have just got an automated solution working to upload about 4000 products daily to our site. We get a CSV file from the wholesalers server each day and the way they have named products and categories is not ideal. Although most of the products remain the same (don't need to be over written) Some will go out of stock or prices may change etc. Problem is we have no control over the csv file so we need to keep the catagories they have given us. Might be able to create new catgories and have products listed under multiple categories? If anyone has used wp all import or has knoledge in this area please let me know. I have plenty more questions but this should start the ball rolling! Thanks in advance mozzers
Intermediate & Advanced SEO | | weebro0 -
Google changing the meta title for the homepage causing branding issues
A client of mine." Ross X Bute" current meta title is "Luxury designer clothing | Womens designer clothing" for the homepage. If i search for luxury designer clothing it will show the full meta title for the homepage. however if i search for the brand name.... "Ross & Bute" will show instead of the meta title. Whats the problem? Well my client a few month ago has decided to re brand the business to have a "X" to show instead of the "And". The rest of the site is branded with an "X" rather than "And" The URL www.rossandbute.com, so you can understand where google is getting this assumption from. Is there anyway to change this so it reads the the meta title in the SERPs? Thanks
Intermediate & Advanced SEO | | Martin_Harris0 -
Old pages still crawled by SE returning 404s. Better to put 301 or block with robots.txt ?
Hello guys, A client of ours has thousand of pages returning 404 visibile on googl webmaster tools. These are all old pages which don't exist anymore but Google keeps on detecting them. These pages belong to sections of the site which don't exist anymore. They are not linked externally and didn't provide much value even when they existed What do u suggest us to do: (a) do nothing (b) redirect all these URL/folders to the homepage through a 301 (c) block these pages through the robots.txt. Are we inappropriately using part of the crawling budget set by Search Engines by not doing anything ? thx
Intermediate & Advanced SEO | | H-FARM0