How to set the crawler or reports to ignore
-
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off
I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
-
Thanks for the confirmation. I hadn't wanted to put that in the robots file because if inquisitive, non-robot traffic if you get my meaning, but in it goes; can't stand seeing errors.
-
Got it, thanks for jumping in so quick.
-
Hey Str8,
Donford is correct. You would need to specifically block our crawler from the mobile pages in your robots.txt file. Unfortunately, we don't currently have a way to disregard specific pages or errors in the web app, but we are looking to add that function sometime in the future.
I hope this helps. Please let me know if you have any other questions.
-
Hi Str8
**user-agent: rogerbot disallow: /*mobile=**
I think may do the trick... somebody may need to confirm this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Blog archive pages in Craw Error Report
Hi there, I'm new to MOZ Pro and have a question. My scan shows Archive pages as having crawl issues, but this is because Yoast is set up to block robots on these pages. Should I be allowing search engines to crawl these pages, or am I fine to leave them as I have it set up already? Any advice is greatly appreciated.
Moz Pro | | mhenshall
Marc0 -
Setting up MOZ to run on Staging
Hi Moz, We would like to setup Moz to run on our Staging Server. This would be extremely valuable as it alerts us to new SEO issues/risks in a controlled and secure environment that is not exposed to production. Our internal team has recommended potentially setting up a reverse proxy server that will validate either via Moz's/Rogerbot's http header or IP and allow Moz access to our Staging environment to crawl. Is this something that we can setup with Moz? Are there other ideas to enable Moz to crawl our Staging server?
Moz Pro | | kriskunisch0 -
Getting my top keywords separated out in SEOmoz reports
I am using the standard functionality to produce weekly Moz reports. There does not seem to be a setting to show rankings of my most important keywords. It would be nice to have those high-level keywords on the first page. For example, I have 200 keywords in an account. I want the report to show on a page my 10 most important keywords. Is there a way to set up a Label to my keywords in order to product a report page just for those keywords?
Moz Pro | | clicktoshop0 -
Plurals and the SEOmoz On Page Report Card
I have some pages on my website that receive an "A" grade with the SEOmoz On Page Report Card if I use a plural term in a phrase (say, "landscape designers in Dallas") but an "F" grade if I use the singular term (say, "landscape designer in Dallas"). Is this a true and accurate reflection of how search engines see the page? Am I going to rank well for plural terms but not for singular, given the specific on-page content I'm using in this example? Or is this an opportunity for SEOmoz to tweak its' algorithm to better reflect the search engines?
Moz Pro | | ahirai1 -
What Does the SEO Moz Crawler Take Into account?
I'm working on a page that has links from some decent pages pointing to it, but a lot of them are low-value blog comments. So I'm pretty sure that its Page Authority is higher than it should be, compared to where it's ranking. Does SEO moz take the type of link into account? i.e. if it's a footer link, blog comment, or forum signature link; this should carry less weight than a link in the content of the page itself, as it does with Google.
Moz Pro | | seanmccauley0 -
Possible to force the SEOmoz Ranking Report to update?
Hi guys, I have added about 100 keywords to my Rankings Report in SEOmoz and I'm just wondering if there is a way to force the application to get my rankings ASAP rather than waiting a while for the rankings to come through? I'm in a bit of a hurry! 🙂 Thanks. C.
Moz Pro | | CormacMoylan230 -
Are the Ranking Changes on SEOmoz ranking reports weekly or life of the keyword?
I am referring to the area where it says ranking changes and where it has a green and a red arrow for improved and declined.
Moz Pro | | JGar-2203710