Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question About creating seperate websites or using a subdomain or just subfolders...Thanks!
I am working with a company who has created a reservation software...We have been advertising this software aggressively and have been doing well with companies coming on board...Let's call the company ABC and the url was ABC.com...Now they are launching a customer facing side of things that will have all the clients listed for consumers to book at their location...So an online marketplace...Like VRBO or AirBnB...They are taking the name of the software company and making it the url for the marketplace site...So now the marketplace is ABC.com and we need to figure out if we go with a new site for the software side or if we use subdomains or a subfolder.... Do we go with ABCSoftware.com - New root domain ABC.software.com - Subdomain ABC.com/software - Subfolder There are a few other intricacies with this...Not all the companies that use the software will be on the marketplace side...And people who book through the company but using the software have a url embedded on their site that books there directly...This booking costs the company less than a booking that comes through the marketplace so this needs to be kept separate... Thanks for the help everyone!
Moz Pro | | TheMarketingInitiative1 -
Why doesn't OSE show results from sites like Wikipedia, YouTube, Twitter, etc.?
I know OSE used to provide link data from these domains. But I have been doing link profile lookups on sites that I know have links from these domains - and they don't show up in my results. Just to make sure, they don't even show up when I sort the sites by domain authority.
Moz Pro | | ProspectMX0 -
Some thoughts on MozTrust based on OSE Findings X ref'd with SERPS
I've been doing a bit of competitor analysis for a client using OSE. There are a group of about 4 websites (our clients website included) that all dominate the sector with none of the 4 clearly out in front (call this GROUP A). Then there are another group of about 5 websites, which come lower in the SE's consistently than the top 4 (Call this GROUP B) **I've been doing some analysis in OSE: ** ALL GROUP B Websites outrank all of GROUP A websites in the OSE Metrics (Including Trust Rank). I did some analysis on the backlinks in Group A VS Group B Group A - Generally a mixture of ok links from blog posts, sponsorship, and ok directories. Group B - As A, but with fewer numbers of links from quality blogs PLUS A high level of spammy links ( .edu and .gov spam filled pages), very low quality, almost non legible blog posts on MFA sites (think Digital Point sellers). From the above it is clear that the OSE metrics are out of whack with the real SE results. Clearly OSE has a few problems with working out what are spammy links and what are decent. Obviously google also has issues with working this out, so I am not surprised that OSE also does - but that doesn't solve the issue. This is a general discussion - so I would just throw in a few thoughts on how OSE may possibly try are overcome some of these issues : 1/. % Trust Links vs Non trust Links:
Moz Pro | | James77
Add in a metric to Trust Rank where the number of links close to trusted sites are also compared to the number of links not close to trusted sites. If you see a very high ratio of links from sites that are not close to trusted sites, it is a strong indicator of spammy links. 2/. Use seed "Non Trusted" sites to create a negative Trust Rank
Use something like a reverse of the "trusted sites" theory, but taking a load of very clearly spammy / link manipulative sites and work out in terms of links connections how far the site is away from these sites. Thoughts???0 -
Linking C Blocks - SEOMoz says its a good thing?
In the competitve analysis, one competitor have more Linking C Blocks, Seomoz has a tick by it almost like its a better thing. Surely a site with the same administrative relationship is not going to help you as much from a linking point of view.
Moz Pro | | sanchez19600 -
OSE Latest Update?
As of yesterday's OSE update alot of sites that I'm tracking saw their PA & DA drop significantly. This was for multiple sites, whereas last month's index saw those stats increase significantly. Has anybody else seen this happen to them and perhaps drawn any conclusions?
Moz Pro | | MichaelWeisbaum1 -
In competitive domain analysis my website shows pages that I no longer have.
When I go to my campaign, link analysis, under competitive comparison/followed backlinks, the 5 pages under Anchor Text/Target URL, supposedly on my website, are not on my website anymore. I recently moved my domain name to another server (before signing up for seomoz) and deleted the forum associated with my website. Those 5 pages are all from that now extinct forum. What's going on?
Moz Pro | | ovistomih0 -
Best way to use OSE Advanced Reports (Repost after Q&A issues)
I can see that Open Site Explorer's Advanced Reports feature is very powerful in terms of sifting through large amounts of link data, but does anyone have any useful tips on how to get the most out it? I want to use my 5 reports a month, but I want to make to maker sure I'm making the most of that allowance.
Moz Pro | | seanmccauley1 -
OSE Linking Domains CSV Export Blank?
For some client domains, I've tried to do a CSV export from the OSE Linking Domains page, I get a CSV that contains the typical OSE header text, but no output records. Before I click the CSV link I always choose "show domains with links to All pages on the Root Domain". It doesn't happen with every domain I've tried this with, and when I've had the time/patience, some of the time if I try hours later, I get data. The problem though is I rarely have the luxury of running reports a second time during the audit cycle. Has anyone else experienced this? Is it a bug, or just an export fail due to system resources or something?
Moz Pro | | AlanBleiweiss0