Search Engine blocked by robots.txt
-
I am getting this error whe I try to crawl http://photosales.belfasttelegraph.co.uk/ but my robots.txt file does not block any bots?
-
Hi Judith,
If you're still having problems, send an email to help@seomoz.org and they'll be able to help you figure out why Roger doesn't want to crawl your site.
-
I am useing the Bing SEO Toolkit, it worked fine for me,
try removeing robots for a test
-
the SEOmoz crawler
-
What are you iusing to crawl?
-
I'm sure why it let you scroll and not me?
-
I just scroled it ok.
but you dont need to put the user agent in twice, this will do below
It was bloking a lot of pages but it crawled the ones you allowedUser-agent: * Disallow: */wo/ Disallow: */ajax/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange - Search Console page indexing "../Detected" as 404
Anyone seen this lately? All of a sudden Google Search Console is insisting in Page indexing that there is a 404 for a page that has never existed on our client's site: https://........com.au/Detected We've noticed this across a number of sites, precisely in this way with a capitalised "/Detected" To me it looks like something spammy is being submitted to the SERPs (somehow) and Google is trying to index that and then getting a 404. Naturally MOZ isn't picking it up, cause the page simply never existed - it's just happening in Search Console 2afc7e35-71e4-4e25-80a3-690bf10776a7.png It comes and it goes in the 404 alerts in Console and is really annoying. I reckon it started happening late 2022.
Reporting & Analytics | | DanielDL0 -
Search Console Crawl Errors/Not Found - Strange URLs
Hello, In Google Search Console under Crawl > Crawl Errors > Not found I have strange URLs like the following: https://www.domain.com//UbaOZ/
Reporting & Analytics | | chuck-layton
https://www.domain.com//UPhXZ/
https://www.domain.com//KaUpZ/WYdhZ/SnQZZ/MOcUZ/ There is no info in Linked From tab. Have you seen this type of error??
Does anyone know whats causing it??
How should it be fixed?? Thanks for reading and the help!0 -
Google Search Console All Verfication Methods Failed
Hi Guys, We are trying to get google webmaster tools verified for this site: https://www.nookie.com.au/ We have implemented the recommended: HTML tag from Search Console which is: Which is installed see: http://prntscr.com/g66cle However, verification still failed. We then tried Google Analytics (failed), alternative methods (HTML file upload failed) Google Tag Manager. All failed. I have no idea why this has happened, any ideas? Cheers.
Reporting & Analytics | | jaynamarino0 -
Google Analytics and Bounce Rates Query - Should I block access from foreign countries ?
Hi , When I look at my google analytics for my UK Website, I can see alot of visits come from outside the UK , i.e Brazil and USA. Both of which give me almost 100% bounce rates from people visiting from there. I am wondering, if google looks at bounce rates with regards to ranking factors and should I therefore block access to my site from visitors outside the UK ?... Would this help increase my rankings ? Given that we only serve uk customers, I cant see any benefit of allowing non uk customers the ability to see the site . what does people think ? thanks pete
Reporting & Analytics | | PeteC121 -
Best way to measure local search keyword rank
I have several clients that have a regional or local presence and want to track their rankings for various keywords, but only in the areas they are located in. What are the best ways to track keyword rank in specific locations or regions of the US?
Reporting & Analytics | | TheURLdr0 -
Google Bot blocked
Google says that they are blocked from crawling my site. I have tried to get my web masters to look into this and they say the site apears fine. However, my SERP results have been falling and the errors are increasing. http://www.RealEstateMarketLeaders.com https://support.google.com/webmasters/bin/answer.py?hl=en&answer=2409441 I have sent an e mail into Google asking for help and I have been looking into every possible channel to find a solution for this problem. The site does have a 301 re direct pointing in from my old site http:///www.TriCitiesRealEstateAgent.com into my new site http://www.RealEstateMarketLeaders.com. However, I am not sure if that is the problem or if something else happend to make me see such a huge drop in SERP results.
Reporting & Analytics | | Brandon_Patton0 -
Search Traffic Drops Before It Improves?
I'm working on a site with tons of great, useful content....the owners of the site implemented a new site layout and design (complete overhaul) and they were lacking basics such as meta descriptions, 301 redirects, and, shockingly, they had the same Title tag for every single page on a site with thousands of unique how-to articles. Unsurprisingly their traffic dropped by about 300%. They generate most of their traffic from people learning how to build stairs, how to install crown molding, and other related matters. Beginning last Thursday I've been performing basic on-site SEO, things like having unique titles for each page and similar tasks. The week from Thursday when I began until yesterday (Wednesday), Google traffic dropped -29.73% - 17,715 vs 25,210 I believe this is a normal part of the "Google Shuffle" -- does anyone have a Matt Cutts link or similar proof that this is a normal part of the process?
Reporting & Analytics | | wattssw0 -
Search for signed in users
"As search becomes an increasingly customized experience, particularly for signed in users" What does this imply ? Does it mean that search is more customized for signed in users than those for non signed in users ?
Reporting & Analytics | | seoug_20052