Can I exclude pages from my Crawl Diagnostics?
-
Right now my crawl diagnostic information is being skewed because it's including the onsite search from my website. Is there a way to remove certain pages like search from the errors and warnings of the crawl diagnostic? My search pages are coming up as:
- Long URL
- Title Element Too Long
- Missing Meta Description
- Blocked by meta-robots (Which is how I want it)
- Rel Canonical
Here is what the crawl diagnostic thinks my page URL looks like:
website.com/search/gutter%25252525252525252525252525252525252525252525252525252525
252525252525252525252525252525252525252525252525252525252525252
525252525252525252525252525252525252525252525252525252525252525
252525252525252525252525252525252525252525252525252525252525252
52525252525252525252525252525252525252525252525252Bcleaning/
Thank you,
Jonathan
-
Thanks! That was easy.
-
You can do so in robots.txt
SeoMoz bot is called rogerbot
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Rank Report says #6 in Google but I can't find the page anywhere
So SEOMoz says that I've consistently ranked #6 for a certain keyword. But when I search I'm no where to be found. I've done regular searches, incognito and some non-seomoz reports and all come up with nothing in Google. I noticed it a week or two ago, but didn't think it would continue. This is no bueno. I wouldn't be surprised if I got penalized (luckily my homepage relatively well for similar keywords), an old seo consultant used very spammy tactics. I recently removed them, but not before I started to notice that I fell off the map. Why would SEOMoz not recognize this, and continue to say I'm ranking well? The keyword is bpi building analyst the page is http://www.cleanedison.com/courses/bpi-building-analyst
Moz Pro | | CleanEdisonInc0 -
How can I see the URL's affected in Seomoz Crawl when Notices increase
Hi, When Seomoz crawled my site, my notices increased by 255. How can I only these affected urls ? thanks Sarah
Moz Pro | | SarahCollins0 -
Initiate crawl
Anyway to start the crawl of a site immediately after changes have been made? Or must you wait for the next scheduled crawl? Thanks.
Moz Pro | | dave_whatsthebigidea.com0 -
Too many on-page links
one of my SEOmoz pro campaigns has given me the warning: Too many on-page links and the page in question is my html sitemap. How do i resolve this because I obviously need my sitemap. How do i get around this?
Moz Pro | | CompleteOffice1 -
Crawl Test produced only 1 page
Hi, I recently submitted a crawl for www.cirrato.com using SEOMoz Crawl Test Tool. I have a lot of pages, but the crawl result shows only 1 page, which is the front page and nothing else... Does anyone know what this could mean or what the problem is?
Moz Pro | | yusufcirrato0 -
Crawl still in progress ...
Hi guys, New crawl on one of my campaigns is still in progress since November 27th, i didn't get new data since November 19th 2011 ... What should i do ?
Moz Pro | | DavidEichholtzer0 -
Yellow Pages
We have just made a yellow pages site n in 3 weeks Google has just indexed 1700 pages out of 18000, so what can we do that Google index all the pages or how the process works? yellowpages.naitazi.com Regards
Moz Pro | | razasaeed0 -
Page Rank and offline sites
I have a domain with PR6 according to the Historical Pagerank Checker. But that last PR was calculated 2 years ago. I brought the site back online a few days ago and have checked that many/most of the backlinks are still valid. It is now in the Google index but the Historical Pagerank Checker shows PR0. Will it get back its previous rank or something close to it? How long will it take?
Moz Pro | | DomainOptions0