How to fix and issue with robot.txt ?
-
I am receiving the following error message through webmaster tools
http://www.sourcemarketingdirect.com/: Googlebot can't access your site
Oct 26, 2012
Over the last 24 hours, Googlebot encountered 35 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.The site has dropped out of Google search.
-
Hi Stacey
What plugins do you have running - any caching plugins such the W3 Total Cache plugin?
Are you able to access your servers error logs to see if you can see anything there?
-
Thanks for your answer.
I have received this message from Google
**http://www.sourcemarketingdirect.com/ **using the Meta tag method (less than a minute ago). Your site's home page returns a status of 500 (Internal server error) instead of 200 (OK)
It looks like the permalink structure has changed but I'm not sure how.
-
I've seen several people ask this very same question over the last week in different forums. I am wondering if the major outages with hurricane Sandy have affected several hosts or DNS's.
Your robots.txt looks fine to me.
I'm guessing that you will completely recover once Google has a chance to fully crawl the site again.
-
just a quick check you have got wordpress visible to search engines set in the admin area? if not it will be set to disallow googlebot to crawl it.
it is in admin - options - privacy and select appropriate box - default is no index, no follow.
-
Thanks Matt.
There is no robots.txt as far as I can see. Is there a plugin I can use for wordpress?
The site was down for 2 days last month while hte original host transfered the site over to me.
Right now a site search says their are 13 pages indexed.
Just concerned that this site has always ranked number 1 for a company name search and now they are not on the first 10 pages in Google.
-
have you made sure your robots.txt is loading in your browser by adding robots.txt after your domain same as a normal page and can you see contents? has your site been down in this period? have you changed the contents of the file just before this issue? are you sure googlebot hasnt come back since that date - whats your analytics say? do an index site: search for your domain to see if it is in google.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dead end pages are really an issue?
Hi all, We have many pages which are help guides to our features. These pages do not have anymore outgoing links (internal / external). We haven't linked as these are already 4th level pages and specific about particular topic. So these are technically dead end pages. Do these pages really hurt us? We need to link to some other pages? Thanks
Web Design | | vtmoz0 -
Question Concerning HTML5/CSS Templates & Google Mobility Issues
Hi all, Looking for some kind of solution for a responsive update for a site and I am wondering if there are any templates (not Wordpress) that are both great SEO wise and would also pass muster with the impending Google update for responsiveness? I was looking at things like Canvas and Porto ( http://themeforest.net/popular_item/by_category?category=site-templates ) but can't find any discussion on whether or not these things have been addressed with any of these templates. If any of you have suggestions or other places to look for something that could possibly fit the bill (even if temporarily) I would be very appreciative. Thank you so much in advance!
Web Design | | Pixelwik1 -
HELP! IE secure page display issue on new live site
For some reason IE 7, 8, & 9 do not display the following page: https://www.jwsuretybonds.com/protools.htm All they show is the Norton seal. It shows properly in all other browsers without issue (including IE 10+), but the earlier versions flash the page for a split second, then hides everything. Can someone shed some light on this? This is a new live site we just launched minutes ago and these browsers account for 12% of our overall traffic. UGH I hate you microsoft!!! Thanks all 🙂
Web Design | | TheDude0 -
W3C My site has 157 Errors, 146 warning(s) Is it an issue?
Is having this number of W3C errors & warnings an issue and will be impacting my site's performance? When the site was built 6 months ago my developers told me that it "was nothing to worry about", but I have read that any errors aren't good, let alone the huge number my site has? Your advice please Thanks Ash
Web Design | | AshShep10 -
New site or fix the old one
I have a delima. Basically the main business product I used to offer is not going to be offered anymore. The types of sales events we conducted for auto dealerships are not able to be insured any longer forcing the change. So I am pivoting to just offering direct mail and I plan on going into digital probably social, landing pages, content marketing and not sure what else. I was able to register http:www.roiautos.com and www.roidirectmail.com both variations of www.roiautosolutions.com withc was the original site. Also that is the closest to the actual name of the business. My question is whether to build a site focusing on direct mail using the direct mail dot com, or just to redo the current site. The current site doesn't have much rank if any because the old product was not something that was searched for. As a mater of fact 99% of my business came from referrals and word or mouth so I just never really bothered. My thoughts are that ROI Direct Mail will work better for search and I am even going to use that as a DBA and TM. But I am unsure of what to do for search. One thing that has to happen is that all references to offering staffed sales events have to be removed from any site per my insurance company. Any advice?
Web Design | | roiautos0 -
Finding a good wordpress web developer for SEO fixes
I have gotten a good audit done of my site now it is time to get the fixes done. I have a developer I worked with for a long time that is good at coding and fixes on my site. However I am not confident in his abilities to execute some of the seo changes that need to be done. Their are some common stuff he can tackle but when it comes to GWT and proper handling of 404's and 301's and other seo tasks I am not sure if he is the right choice.Maybe due to a lack of experience of dealing with the issues I have or is just not his specialty or web devs just don't know seo...lol.... Is obvious a lot of SEO's don't make the changes themselves but leave it up to the devs to handle it their suggestions and fixes. But from my experience devs are not so well versed in seo and you have a hard time knowing if they are doing it correct or can even do it.(of course they will say they can they want your cash and i understand that ). In particular a good amount of wordpress devs claim to know seo but i find that far from the truth.Even when guided to issues some of them will often leave you disappointed. Sorry for my rant! Now to my question , obviously not many SEO's make the actual code changes themselves (how i wish i knew a one that did) are their ones out their that do? If not how do i find a good wordpress dev that can make proper seo changes and knows his stuff....example i need someone who can trouble shoot and track down some serious GWT I have and deal with some hardcore 404 & 301 issues . A lot claim to know but when push comes to shove I have been left disappointed. Thanks in advance for your help and suggestions or recommendations.
Web Design | | chrisyak0 -
Duplicate content issue
I have recently built a site that has a main page intended to rank for national coverage. This site also has a number of pages targeted at local searches, these pages are slight variations of each other with town specific keywords. Does anyone know if google will see this as spam and quarantine my site from ranking? Thanks
Web Design | | stebutty0 -
Custom URL's with Bigcommerce Issue (Is it worth it?)
We're building out a store in Bigcommerce, who for all intensive purposes is perfect for SEO besides the fact that you can not change the URL's to be custom. My question is, does this kill the SEO value of bigcommerce, despite everything else being great? So for example the URL's for a category page would be something like this www.mysite.com/categories/keyword and the product URL's are pulled in by product name, so product URL's could be something like www.mysite.com/products/Product-Description-Long-223.html (notice the words will be capitalized and their is no way to remove the trailing .html) I could go with Interspire (the liscenced version of Bigcommerce) or Magento so I can custom edit this stuff. But then its a lot more work for my employee's on the buildout.
Web Design | | iAnalyst.com0