Can I exclude pages from my Crawl Diagnostics?
-
Right now my crawl diagnostic information is being skewed because it's including the onsite search from my website. Is there a way to remove certain pages like search from the errors and warnings of the crawl diagnostic? My search pages are coming up as:
- Long URL
- Title Element Too Long
- Missing Meta Description
- Blocked by meta-robots (Which is how I want it)
- Rel Canonical
Here is what the crawl diagnostic thinks my page URL looks like:
website.com/search/gutter%25252525252525252525252525252525252525252525252525252525
252525252525252525252525252525252525252525252525252525252525252
525252525252525252525252525252525252525252525252525252525252525
252525252525252525252525252525252525252525252525252525252525252
52525252525252525252525252525252525252525252525252Bcleaning/
Thank you,
Jonathan
-
Thanks! That was easy.
-
You can do so in robots.txt
SeoMoz bot is called rogerbot
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can anyone offer an example of a site or page that gets 100% or even close to that on Search Visibility?
I have a couple of sites that I manage that kill it in the SERPs and yet they get a low search visibility score from Moz Pro, I am talking 18%-19%, and another that ranks well has a search visibility score of 8.71%. I know there are factors that go into calculating the score, I am just curious if anyone is really up there.
Moz Pro | | -b.graves- 00 -
Crawl diagnostics incorrectly reporting duplicate page titles
Hi guys, I have a question in regards to the duplicate page titles being reported in my crawl diagnostics. It appears that the URL parameter "?ctm" is causing the crawler to think that duplicate pages exist. In GWT, we've specified to use the representative URL when that parameter is used. It appears to be working, since when I search site:http://www.causes.com/about?ctm=home, I am served a single search result for www.causes.com/about. That begs the question, why is the SEOMoz crawler saying there is duplicate page titles when Google isn't (doesn't appear under the HTML improvements for duplicate page titles)? A canonical URL is not used for this page so I'm assuming that may be one reason why. The only other thing I can think of is that Google's crawler is simply "smarter" than the Moz crawler (no offense, you guys put out an awesome product!). Any help is greatly appreciated and I'm looking forward to being an active participant in the Q&A community! Cheers, Brad
Moz Pro | | brad_dubs0 -
Too many on-page links
I received a warning in my most recent report for too many on-page links for the following page: http://www.fateyes.com/blog/. I can't figure out why this would be. I am counting between 60-70 including all pull downs, "read more's", archive, category and a few additional misc. links. Any ideas or suggestions on this? Or what I might do to rectify? Perhaps it's just an SEOmoz report blip... We currently don't have the post list rolling to additional pages so it's kind of passively set up to be endless, but it's in the works.
Moz Pro | | gfiedel0 -
Who wants to help go over my crawl diagnostics via skype?
I have run a crawl diagnostic on my site and have 194 errors and most of them are 404 errors in wordpress. Not sure why, but many of my pages had name changes (possibly a permalinks issue) but I have no idea how to fix it. I had 5 duplicate page titles, and 1 tile missing or empty. 72 crawl notices found (2 permanent redirect, 17 blocked by robots, 53 rel canonical) 19 Crawl warnings were found Who wants to have some fun?
Moz Pro | | starkSEO0 -
Where can I find the page strength tool?
I have an SEOmoz account and can't seem to find the page strength tool. Is it somewhere in research tools?
Moz Pro | | elenaroi0 -
Image Asset pages shown to have Page Authority
When looking at top pages for my site in www.opensiteexplorer.org I'm seeing a bunch of asset pages being listed to have page authority. How could this be? Is open site explorer mistaken? Here is a page with a PA: 24 http://www.minespress.com/catalogassets/thumbnails/0000437_atx_software_compatible_folders.jpg
Moz Pro | | smines0 -
Issue in number of pages crawled
i wanted to figure out how our friend Roger Bot works. On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site. Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again. I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had. Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue. Thanks!
Moz Pro | | cchhita0 -
Why would the SEOMoz Page analysis pick up exact keywords used in page title and text?
Hi, I am trying to optimise this URL : www.adaptiveconsultancy.com/ecommerce/features/advanced-ecommerce with the keyword being 'advanced ecommerce' With the 'On-Page Report Card' from SEOMoz that the exact keyword isn't featured in the page title or text, but it is in there. Why would this not be picked up? Thank you in advance,
Moz Pro | | adaptiveconsultancy
M0