Robots.txt - Allow and Disallow. Can they be the same?
-
Hi All,
I need some help on the following:
Are the following commands the same?
User-agent: *
Disallow:
or
User-agent: *
Allow: /
I'm a bit confused. I take it that the first one allows all the bots but the second one blocks all the bots.
Is that correct?
Many thanks,
Aidan
-
Hi Aidan
I'm getting a similar problem on a site I'm working on. The on page rank checker "can't reach the page". I've checked everything obvious (at least I think I have!)
May I ask how you eventually resolved it?
Thanks Aidan
-
Hi
you can use this tool for be sure that the crawler see your files
http://pro.seomoz.org/tools/crawl-test
but you must wait for receive the report to a email.
when you say:
"get the following msg when I try to run On Page Analysis:"
the tools is this?
http://pro.seomoz.org/tools/on-page-keyword-optimization/new
for check the website you can use this:
http://www.opensiteexplorer.org
Ciao
Maurizio
-
Hi,
Thanks for the clarification. So the Robots.txt isn't blocking anything.
Do you know why then i cannot use SEOMoz On Page Analysis and Xenu and Screaming Frog only return 3 URLs?
I get the following msg when I try to run On Page Analysis:
"Oops! We were unable to reach the papge you requested for your report. Please try again later."
Would there be something else blocking me? GWMT Parameters maybe?
-
E' un piacere.
but I don't understand the problem.
if the site have this robots.txt
**User-agent: ***
Allow: /
every crawler can index and see all files of the this website and Seo moz also.
Maybe the problem is different?
Ciao
-
Thanks Maurizio,
I need to do some analysis on this site. Is there a way to use my SEO tools (screaming frog, SEOMoz) to ignore the robots.txt to enable me to do a good site audit?
Thanks again for the answers. Much appreciated
Aidan
-
Hi Aidan
User-agent: *
Disallow:and
User-agent: *
Allow: /are the same
Ciao
Maurizio -
Hi Maurizio,
The reason I asked is because I am working on a site and it's robots.txt is :
User-agent: *
Allow: /
Why would they have this?
I can't use On-Page Analysis or Screaming Frog as it only results in 3 URLs.
Thanks again,
Aidan
-
Hi
1° example:
User-agent: *
Disallow:all User-agent can index your files
2° example
User-agent: *
Disallow: /never User-agent"can index you files
other example here:
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449
Ciao
Maurizio
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I safely asume that links between subsites on a subdirectories based multisite will be treated as internal links within a single site by Google?
I am building a multisite network based in subdirectories (of the mainsite.com/site1 kind) where the main site is like a company site, and subsites are focused on brands or projects of that company. There will be links back and forth from the main site and the subsites, as if subsites were just categories or pages within the main site (they are hosted in subfolders of the main domain, after all). Now, Google's John Mueller has said: <<as far="" as="" their="" url="" structure="" is concerned,="" subdirectories="" are="" no="" different="" from="" pages="" and="" subpages="" on="" your="" main="" site.="" google="" will="" do="" its="" best="" to="" identify="" where="" sites="" separate="" using="" but="" the="" is="" same="" for="" a="" single="" site,="" you="" should="" assume="" that="" seo="" purposes,="" network="" be="" treated="" one="">></as> This sounds fine to me, except for the part "Google will do its best to identify where sites are separate", because then, if Google establishes that my multisite structure is actually a collection of different sites, links between subsites and mainsite would be considered backlinks between my own sites, which could be therefore considered a link wheel, that is, a kind of linking structure Google doesn't like. How can I make sure that Google understand my multisite as a unique site? P.S. - The reason I chose this multisite structure, instead of hosting brands in categories of the main site, is that if I use the subdirectories based multisite feature I will be able to map a TLD domain to any of my brands (subsites) whenever I'd choose to give that brand a more distinct profile, as if it really was a different website.
Web Design | | PabloCulebras0 -
SEO Ranking: Can Child Theme Compete with Custom Theme?
Ranking for New York City commercial real estate is extremely competitive. We compete against: www.squarefoot.com, www.42floors.com, www.Loopnet.com, www.wework.com and a dozen other optimized sites. Our site was designed in 2012. We plan on upgrading it. From an SEO perspective, can we compete by purchasing a Wordpress real estate theme and customizing it into a child theme? Our better ranking competitors are using custom themes where the code has been very streamlined to make the sites quick and easy to index by spiders. Would we gain a significant edge by custom coding? This is somewhat technical for a business owner and I am trying to get my head around it. Our existing site is www.nyc-officespace-leader.com. Some of the themes we are considering are: -http://main.wpestatetheme.org/homepage -http://houzez01.favethemes.com/ -http://realhomes.inspirythemes.biz/property/ From an SEO perspective is creating a child theme from the above a good approach? Or will a custom theme give us an advantage. If there is an advantage is that edge so marginal that it is not significant? In terms of coding, is a custom site much more labor, 2x, 3x the time to code? Also is the maintenance of a custom site much more involved? Also, as a related question, my developer since 2012 has created many custom plugins for Wordpress. Is this a no, no? Will avoiding custom plugins add to the development cost? Even if we use a child theme from an existing real estate website, I would hope that the improved user interface will provide a boast in at least conversions if not SEO. Thanks, Alan
Web Design | | Kingalan10 -
Can a cloud based firewall affect my search ranking?
Hi, I recently implemented a firewall on my website to prevent hacking attacks. We were getting a crazy amount of people per day trying to brute force our website. I used the sucuri cloud proxy firewall service which they claim because of the super fast caching actually helps SEO. I was just wondering is this true? Because we're slowly falling further and further down the SERPS and i really don't know why. If not, is there any major google update recently I don't know about? Thanks, Robert
Web Design | | BearPaw880 -
Block parent folder in robot.txt, but not children
Example: I want to block this URL (which shows up in Webmaster Tools as an error): http://www.siteurl.com/news/events-calendar/usa But not this: http://www.siteurl.com/news/events-calendar/usa/event-name
Web Design | | Zuken0 -
Can anybody recommend a good UK based SEO
Looking for a company or person willing to work on a small company website in other words not expensive 🙂 I looked on the recommended list on the MOZ website but they mostly in America and mostly only too willing to work on large company websites. There wasn't anything catering for a one-man band business like mine which is letting holiday cottages www.endeavourcottage.co.uk. I have just started talks with a responsive web designer, but if I got the right SEO company maybe they could do both but not necessary.
Web Design | | WhitbyHolidayCottages1 -
Can Image File Names be Masked?
If we "mask" file names for our website but they are left their original name on the server, will Google notice this? Client wants to mask them in order to name them with keywords but not change on the actual server.
Web Design | | Atlanta-SMO0 -
Can wordpress actually be bad for sites if it static?
Hello, So last year I did rank for my website. Yet I switched from Adobe Muse to wordpress. I thought it would be great for updating and blobbing if I ever do it. So I got a theme, and went for it I have Yoast and that's it for plugins. But if I take say another couple of years to blog, am I hurting myself with wordpress? Like Google knows I am using wordpress so it expects me to be creating content? I know its an odd question, just had to ask
Web Design | | Berner0 -
Can someone please help with technical question!
I have noticed that our website tool to get a quote does not work with active scripting disabled is this bad? How many people have this disabled?
Web Design | | BobAnderson0