Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO Ranking: Can Child Theme Compete with Custom Theme?
Ranking for New York City commercial real estate is extremely competitive. We compete against: www.squarefoot.com, www.42floors.com, www.Loopnet.com, www.wework.com and a dozen other optimized sites. Our site was designed in 2012. We plan on upgrading it. From an SEO perspective, can we compete by purchasing a Wordpress real estate theme and customizing it into a child theme? Our better ranking competitors are using custom themes where the code has been very streamlined to make the sites quick and easy to index by spiders. Would we gain a significant edge by custom coding? This is somewhat technical for a business owner and I am trying to get my head around it. Our existing site is www.nyc-officespace-leader.com. Some of the themes we are considering are: -http://main.wpestatetheme.org/homepage -http://houzez01.favethemes.com/ -http://realhomes.inspirythemes.biz/property/ From an SEO perspective is creating a child theme from the above a good approach? Or will a custom theme give us an advantage. If there is an advantage is that edge so marginal that it is not significant? In terms of coding, is a custom site much more labor, 2x, 3x the time to code? Also is the maintenance of a custom site much more involved? Also, as a related question, my developer since 2012 has created many custom plugins for Wordpress. Is this a no, no? Will avoiding custom plugins add to the development cost? Even if we use a child theme from an existing real estate website, I would hope that the improved user interface will provide a boast in at least conversions if not SEO. Thanks, Alan
Web Design | | Kingalan10 -
Can Image File Names be Masked?
If we "mask" file names for our website but they are left their original name on the server, will Google notice this? Client wants to mask them in order to name them with keywords but not change on the actual server.
Web Design | | Atlanta-SMO0 -
How Can I Make My Site iPhone Friendly?
I have been looking into making my website for iphone friendly as my analytics are not great for the iphone and I know when I try to navigate around it on an iphone it can be tough. I was told that if I make changes to the layout that it would affect my layout across everything, which I did not want to do. So I have two questions: Is this correct regarding the layout? If so, if you did something like m.waikoloavacationrentals.com which would be the mobile version how would that possibly effect your rankings with regards to the traffic distribution? Any feedback would be appreciated. Also if anyone has any experience in doing this I would be interested in discussing further.
Web Design | | RobDalton0 -
Can someone please help with technical question!
I have noticed that our website tool to get a quote does not work with active scripting disabled is this bad? How many people have this disabled?
Web Design | | BobAnderson0 -
How can i embed my video into a table using SEO embed setting?
We use Wistia.com to embed our videos. They have different options for embed settings and we prefer to use the SEO embed setting, however, when we use that setting we aren't able to insert the video in a table side by side with another image or text. When we try, the video jumps out of the table and the table gets (for lack of a better work) out of wack. When we embed the video with the iframe embed setting, the video can be placed in a table with no issues, but then we don't get the SEO credit. We have our site in wordpress. I'm not sure if that has something to do with the tables getting messed up. Check out this link to see an example of how we want the video to show up. http://www.3000doorhangers.com/ Any suggestions as to how we can use the SEO embed setting within a table as shown in the above link?
Web Design | | JimDirectMailCoach0 -
My 404 page is showing a 4xx error. How can that be fixed?
My actual 404 page is giving a 4xx error.
Web Design | | sbetzen
The page address is http://www.ecowindchimes.com/v/404.asp It loads fine... it is the page all 404's are directed to. Why is it showing a 404 error. The page works. How can this be fixed? Stephen0 -
Can someone suggest a good wordpress plugin for compressing images?
I am building a new website and it has 500+ photo images. I've got less than 100 in there at the moment and the page speed is already very slow. I'm trying to reduce the size of the images before uploading but without a photoshop application it is very difficult. Are there any great plugins for wordpress that can help?
Web Design | | skehoe0