Truncate page URLs
-
We have some pages (for example a contact us form) for which the URL is modified by the CMS depending on the referring page (this helps to put the form submission in context for the sales reps who get the contact submission).
The SEOmoz crawler considers each URL a new page -- and so numbers like in diagnostics are all inflated as the same page is listed multiple times (e.g. for too many links)
Is there a setting to change what the crawler considers to be the same page?
Here are two URLs for the same page that the reports treat as separate pages:
http://www.spirent.com/About-Us/Contact_us.aspx?referurl=0F528F4D703D8BB3523738D6373AA8AD
http://www.spirent.com/About-Us/Contact_us.aspx?referurl=10ACDA6055244E369395223437FDCF30
The page is actually: http://www.spirent.com/About-Us/Contact_us.aspx
Thanks
Ken
-
As you can see here, this is an issue as Google are indexing many variations of the same page although this means that somewhere is linking to them unless your site is set up so that even a crawler passing through links to your contact page is creating the query parameter in the URL's.
To resolve this, you need to add the following to your robots.txt file:-
Disallow: ?referurl=
This will prevent any URL's passing that query parameter from getting crawled and indexed ensuring that only the originals of the pages will appear in search engines and not flag as duplicate content.
Hopefully, someone from SEOmoz can add as to whether there is an option for obeying robots.txt directives within their crawler so that these URL's are not listed as I'm not sure.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why am I ranking high for a keyword that is nowhere on my page?
There is a specific keyword phrase that I am ranking in the top 10 search results for. When I run page optimizer in Moz, my page is given a score of 73 and I'm predictably told I need to include this keyword somewhere on the page. This specific keyword phrase is not included anywhere on the page - not in image files, image alts, anchor text from inbound links (that I can find), text content, meta-titles or descriptions - it's nowhere. Meanwhile, a keyword phrase which Moz' page optimizer tells us we are scored 96 for we aren't even in the first three pages of search results for. What's going on here? On the surface, this seems to be telling me the Moz page optimizer is kinda...broken? Why am I ranking in the top 10 for a keyword phrase nowhere on my page? Thanks in advance for any help.
Moz Pro | | Closetstogo0 -
URL Parameters causing duplicate content - Login/Registration page
All, I just recently acquired a new client and right away I noticed an abundance of duplicate content being recorded after the moz crawl diagnostics was completed. After a quick digest of the issue, it seems that the majority (90%) of the outlined duplicated content is stemming from the client's Login/Registration page. Upon clicking (without being logged-in) any asset or forum discussion board link within the site, the user is automatically redirected to the Login/Registration page, which seems to create this massive redirect loop associated with dynamic url parameters. Ex. After clicking on a select internal link (asset or discussion board) the user is redirected to the Login/Register page which presents the page and a URL that looks a lot this this: Ex. 1 https://www.clientsite.com/register-login?ReturnUr...xxxx%xxxx%xxxx%...... Ex. 2 https://www.clientsite.com**/register-login?returnurl=/register-login?returnurl=/register-login?returnurl=/page-titl**e/ These URLs seem to becoming larger and larger... The client wants to ensure users have to Login/Register within their site before they're allowed to view the content. This process doesn't allow for any type of preview page to be viewed by a user prior to clicking on the internal link, which in turn doesn't allow any preview pages to be indexed. Right now, Moz is picking up all of the redirect and labeling them as duplicate page content/duplicate page titles based on the Login/Registration page. Questions/Comments: Would it be wise to create preview pages for the asset pages and discussion board pages to allow for proper indexing? - Could this be a CMS issue? Current being used on this is, Kentico. There are thousands of pages being recorded in the crawl as duplicate, however only 14 seem to be indexing with duplicate title tags. 301 or canonical redirect strategy? Moz crawl data issue? Again, this is my first look at this issue, so more information is bound to come out soon! Please let me know if anyone has run into this issue and if you have a possible solution to get rid of this redirect loop process. Thanks! -T
Moz Pro | | MattLacuesta0 -
Universal rankings has made my page not show in rankings any more
We've had a page with a video on it for ages; it always ranks well, generally around no. 5. But I've noticed that it's now ignored in SEMOZ as a page, and only shows as a Universal Result - so it looks as though our top page is no. 22, even thought the video shows as an icon still in the same position, no. 5. That's annoying as I now have to explain to my stakeholders how come our rankings appear to have dropped off a cliff - when i fact they haven't. The page appears in google SERPS exactly as it always has. Can anything be done to make it appear back as it did in SEMOZ? Thanks
Moz Pro | | challMOZ0 -
On Page URL's not updating?
I recently moved my site to Shopify which completely changed the URL structure. I went to my old pages and created permanent 301 redirects to my new pages on shopify. For some reason, the On Page reports are not picking up on one of the new pages. This is effecting my grading. Just wondering why this is happening & whether this may be an indication of a larger problem? Any help would be greatly appreciated! Thanks!
Moz Pro | | PedroAndJobu0 -
Broken links in the pdf of the On Page Report
Hi, I run an individual On Page report for a particular URL, then I export as pdf. The URL appears in the pdf and looks fine but when you click on it it goes to a 'page not found'. I know the URL is correct. When I hover over the URL in the pdf i notice that the word 'Good' is at the end of my URL but I did not put this in there. if I give the report to a client it doesn't look so good. http://www.narellanpools.com.au/local-contact/narellan-pools-alburywodongaGood Is this a bug? Cheers Virginia
Moz Pro | | VirginiaC0 -
Wrong duplicated page content
I found out that some errors on my website are considered as "duplicated page content" while they are not, the content is different on each page. I wonder why ? Is it an issue from Seomoz ?
Moz Pro | | Amadeus_eBC0 -
Too Many On-Page Links: Crawl Diag vs On-Page
I've got a site I'm optimizing that has thousands of 'too many links on-page' warnings from the SeoMoz crawl diagnostic. I've been in there and realized that there are indeed, the rent is too damned high, and it's due to a header/left/footer category menu that's repeating itself. So I changed these links to NoFollow, cutting my total links by about 50 per page. I was too impatient to wait for a new crawl, so I used the On Page Reports to see if anything would come up on the Internal Link Count/External Link Count factors, and nothing did. However, the crawl (eventually) came back with the same warning. I looked at the link Count in the crawl details, and realized that it's basically counting every single '<a href'="" on="" the="" page.="" because="" of="" this,="" i="" guess="" my="" questions="" are="" twofold:<="" p=""></a> <a href'="" on="" the="" page.="" because="" of="" this,="" i="" guess="" my="" questions="" are="" twofold:<="" p="">1. Is no-follow a valid strategy to reduce link count for a page? (Obviously not for SeoMoz crawler, but for Google)</a> <a href'="" on="" the="" page.="" because="" of="" this,="" i="" guess="" my="" questions="" are="" twofold:<="" p="">2. What metric does the On-Page Report use to determine if there are too many Internal/External links? Apologies if this has been asked, the search didn't seem to come up with anything specific to this.</a>
Moz Pro | | icecarats0 -
How Does On Page Analysis work
Hi guys, I just need to run something past you. when I look at my on page analysis I have 5 key terms I am focusing on. For instance one of them is "computer backup". According to the report the current grade is 'F' when looking at site page "/" which I assume is the home page.
Moz Pro | | cubetech
When I do a lookup on other pages of the site it gets a ranking of A. Which is good. But since the hompage ranking went from A to F my rankings have definitely been affect. So i guess my questions are: does "/" mean the hompage, or all pages overall. What should I really be looking at here. I am assuming that you select certain pages to target certain key words. Should i be focusing like this, or more to the "/". Thanks Guys hoping to clear this one up.0