Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can forwarding users from one domain to a different domain damage rank and authority of first domain?
Preliminary Explanation: We launched a new website a couple months back but haven't had much luck in Google taking notice. One of the main attractions to our site is an old flash app that was made nearly a decade ago. As the original developer has long ago moved on and we are unable to figure out how to integrate it with our new site, we've been stuck hosting the flash app on a different domain. As such, users who come to our site and want to use the app must immediately navigate away from our site to this other domain. This has caused our primary domain's bounce rate and average site time to plummet while raising it for the other domain. My question: is this damaging our search rank and page authority with Google for this primary domain/site and counter-acting any other positive SEO changes we can make? How much weight does Google give towards bounce rate/average site time spent by users in its overall calculations for search rank and page authority? Our average site time for this primary domain is resting currently at 50-60 seconds, while for the secondary domain that hosts the old flash app it is 4-5 minutes.
Web Design | | Closetstogo0 -
Can someone help me understand Structured Data?
So I'm wondering if someone could explain Structured Data a little better to me and what the importance is. I also am wondering how to best add Scheme.org markup to certain pages. I tried a plugin for wordpress and I don't think it was working correctly. I'm specifically wanting to make sure my Google Profile is showing with my website in SERP. I have the ?rel=author tag in on the front page and when I checked it when the Google Structured Data checker it shows it to be correct but its not displaying in SERP. Thanks!
Web Design | | jonnyholt0 -
Can someone suggest a good wordpress plugin for compressing images?
I am building a new website and it has 500+ photo images. I've got less than 100 in there at the moment and the page speed is already very slow. I'm trying to reduce the size of the images before uploading but without a photoshop application it is very difficult. Are there any great plugins for wordpress that can help?
Web Design | | skehoe0 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
I am looking to improve my on page seo, can you provide any recommendations or suggestions for how?
I am relatively new to the world of SEO and recently built a new site. I have read as many books as I can to help increase my skill set rapidly, and have attempted to implement the best of what I have learned but I know many of you have been in this arena for a while and I would be extremely appreciative of any suggestions you can offer with regard to on page. Thanks in advance. http://luxuryhomehunt.com - home page http://luxuryhomehunt.com/homes-for-sale/orlando.html - city level http://luxuryhomehunt.com/homes-for-sale/orlando/bay-hill.html - community level
Web Design | | Jdubin0 -
Can't figure out what's going on with these strange 403's
The last crawl found a good number of 404's and I can't figure out what's going on. I don't even recognize these links. What's really strange is a few of them high a fairly decent page authority. For instance this one has a PA of 55: http://noahsdad.com/?path=http%3A%2F%2Feurosystems.it%2Fconf_commerciale%2Fimages%2Fdd.gif%3F%3F There are several more like this one also, it seems most of these new ones have "Feurosystems" in the link...I have no idea what that is. Just curious what you guys think is going on, why these are 404'ing, and how to fix it. Thanks. Edit: I took out the "%" from the links and I get this: http://noahsdad.com/?path=http3A2F2Feurosystems.it2Fconf_commerciale2Fimages2Fdd.gif3F3F which takes me to a page on my site. I have no idea what's going on, or what that link is. Hoping someone can chime in because this is strange. Another edit: I just checked out the Google Webmaster's and it looks like these errors are 403's and all started around March 21st. I have no idea what happend on March 21st to start causing all of these errors, but I'd sure like to get it fixed. 🙂
Web Design | | NoahsDad0 -
Can google crawl text in jquery sliders?
We are redesigning our website and want to present a fair amount of text within jquery sliders. Will google crawl this text or is it treated the same way as actual script? Perhaps there is a way to just have the text as plain html but use jquery to display it?
Web Design | | Netboost0 -
Can i do this? Will Google penalize me?
I have a page for a Criminal Defense Attorney and i set up a list of the type of criminal charges he is certified to deal with. I wanted to use title tags and put the Keyword "Miami Criminal Defense Attorney" & "Miami Traffic Defense Lawyer"... My question is will Google penalize me for plugging the same Key words over and over on the title tag for each ?? CHECK THE IMAGE to see what I'm talking about... thanks guys. x97dl
Web Design | | marig0