How to set the crawler or reports to ignore
-
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off
I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
-
Thanks for the confirmation. I hadn't wanted to put that in the robots file because if inquisitive, non-robot traffic if you get my meaning, but in it goes; can't stand seeing errors.
-
Got it, thanks for jumping in so quick.
-
Hey Str8,
Donford is correct. You would need to specifically block our crawler from the mobile pages in your robots.txt file. Unfortunately, we don't currently have a way to disregard specific pages or errors in the web app, but we are looking to add that function sometime in the future.
I hope this helps. Please let me know if you have any other questions.
-
Hi Str8
**user-agent: rogerbot disallow: /*mobile=**
I think may do the trick... somebody may need to confirm this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to set up Goals/Conversions in Google Analytics with Moz
With my new Moz account, how would I go about setting up a useful Goal/Conversion on Google Analytics?
Moz Pro | | Brandon320 -
Are organic site links excluded in the rankings report?
If I'm pulling a rankings report for a specific subdomain and a different subdomain appears at the main result and the specific subdomain is a site link beneath it, does that show up in the results? It doesn't seem to in my report.
Moz Pro | | mattiasantin0 -
OSE Backlink results - reported link not actually there?
Not a complaint, but a question to understand how the research tool operates: When I run backlink checks on websites, often the reported link is not only not on the page, but it's not found anywhere on the site. I use several tools to search for the link url as well as for the keyword. Why does the tool report a link is there, but I cannot find the links in some cases? Is there a lag in the information the tool is using, making it not quite up to date, or is it something else? Thanks much!
Moz Pro | | AdamThompson0 -
SEOmoz crawler bug?
I just noticed that a few of my campaigns have number of pages crawled 1. Can someone tell me what this is.... from 5 campaigns 2 have only one pages crawled from which one is an online shop with over 2000 products 🙂
Moz Pro | | mosaicpro0 -
Our recently searched Open Site backlink reports are identical to last months across multiple clients. Why is this happening?
When processing the backlink data of our clients for this month (searches done on 28th December), we're finding that the information gathered from Open Site explorer is identical to the data that was found on searches done at the end of November. It appears to have happened for the reports we use for every single client so it must be more than a coincidence. From our perspective, it's almost as if no data has been tracked since the last time we gathered backlink reports. Could you help us get to the bottom of this?
Moz Pro | | queryclick0 -
Automated SEO Report for Agencies
I have a lot of SEO reports to send out, SEO Moz is awesome but it's not designed for this kind of work. I would like an automated report that does an on page optimization report on the potential client's site, finding errors etc it does not have to include fixes for each. This is just a basic report but it must look nice and be user friendly. I have looked at Raven and it looks ok however it does not have any decent On Page Optimization reports. I know I could crawl and then delete the campaign in SEO Moz but that is very time consuming and far from ideal. If anyone knows of a solution please let me know 🙂
Moz Pro | | SEOKeith0 -
MOZ Crawler only crawling one page per campaign
We set up some new campaigns, and now for the last two weekly crawls, the crawler is only accessing one page per campaign. Any ideas why this is happening? PS - two weeks back we did "upgrade" the account. Could this have been an issue?
Moz Pro | | AllaO0 -
Ranking reports don't seem to match reality
seomoz shows our site at #1 for a keyword that we're actuallly #2 we seem to rank #2 even when I disable personalization (setting URL token) or using incognito mode on chrome. Any insight on how seomoz verifies these ranks, and if I can place credibility in them?
Moz Pro | | brianjolney0