/$1 URL Showing Up
-
Whenever I crawl my site with any kind of bot or a sitemap generator over my site. it comes up with /$1 version of my URLs. For example:
It gives me hdiconference.com & hdiconference.com/$1 and hdiconference.com/purchases & hdiconference.com/purchases/$1
Then I get warnings saying that it's duplicate content. Here's the problem: I can't find these /$1 URLs anywhere. Even when I type them in, I get a 404 error. I don't know what they are, where they came from, and I can't find them when I scour my code.
So, I'm trying to figure out where the crawlers are picking this up. Where are these things? If sitemap generators and other site crawlers are seeing them, I have to assume that Googlebot is seeing them as well.
Any help? My developers are at a loss as well.
-
Perfect. Thanks for the help, guys!
-
If you can't find them, you could put a disallow in your robots.txt files to keep them from being crawled.
-
I had a similar issue and found it was due to (in the case of a MozPro crawl at least) the bot crawling a JS command in the head. One of the commands included an anchor tag that was being read as a link rather than in context of the java script command. Check your JS files/scripts. It might be in there somewhere.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Broken URL Links
Hi everyone, I have a question regarding broken URL links on my website. Late last year I move my site from an old platform to Shopify, and now have broken URL links giving out 4xx errors. When I look at Moz Pro>Campaigns>Insights>links, I can see the top broken URL links, however there is a difference if copy & paste URL directly from Moz Pro and by Export CSV file. For example below, If I copy and paste links direct from Moz Pro, it has the “http://” in front as below: http://www.thehairhub.com.au/WebRoot/ecshared01/Shops/thehairhub/57F3/1D8F/D244/C675/E27D/AC10/003F/35AD/manic-panic-colours.jpg But when I export the list of links as an CSV file, the http:// is removed. www.thehairhub.com.au/WebRoot/ecshared01/Shops/thehairhub/57F3/1D8F/D244/C675/E27D/AC10/003F/35AD/manic-panic-colours.jpg Another Example below: By copy & paste URL direct from Moz Pro
Technical SEO | | johnwall
http://thehairhub.com.au/Shop-Brands/Vitafive-CPR/CPR-Rescue By export CSV file.
thehairhub.com.au/Shop-Brands/Vitafive-CPR/CPR-Rescue Which one do I use to enter into the “Redirect From” field in Shopify URL Redirects? Do I need to have the http:// in front of the URL? Or is it not required for redirects to work? Kind Regards, John Wall
The Hair Hub0 -
Why does my 301 show the old urls with new descriptions and titles?
Hi all, We've just rebranded. The 301 appears to have worked well and moved the results and rankings onto the new domain. However a site:olddomain.com search in Google brings up about a hundred pages that have the new titles and descriptions but show the old urls - does anyone have any idea how to make the old domain disappear from the SERPS? Many thanks, Richard
Technical SEO | | panini0 -
No Keyword in URL
SEOMoz (and other platforms) advise that I need to add my keyword to the page URL, however as far as I'm concerned it has been, so why don't these platforms see it. My home page URL is www.salesandinternetmarketing.com, but apparently I haven't added the keyword internet marketing to the URL, what advice can you give me please? Lindsay
Technical SEO | | lindsayjhopkins1 -
Correct Redirect method for switching pages from .html to /pretty urls/
I have a customer that has all his site files as .html extensions and i'm going to rebuild this site into a wordpress site for easier management, regarding the new permalink structure, should i just do a 301 redirect on this?
Technical SEO | | tgr0ss0 -
/index.php/ page
I was wondering if my system creates this page www my domain com/index.php/ is it better to block with robot.txt or just canonize?
Technical SEO | | ciznerguy0 -
Duplicate pages, overly dynamic URL’s and long URL’s in Magento
Hi there, I’ve just completed the first crawl of my Magento site and SEOMOZ has picked up 1,000’s of duplicate pages, overly dynamic URL’s and long URL’s due to the sort function which appends URL’s with variables when sorting products (e.g. www.example.com?dir=asc&order=duration). I’m not particularly concerned that this will affect our rankings as Google has stated that they are familiar with the structure of popular CMS’s and Magento is pretty popular. However it completely dominates my crawl diagnostics so I can’t see if there are any real underlying issues. Does anyone know a way of preventing this? Cheers,
Technical SEO | | WendyWuTours
Al.1 -
301 redirects and Dynamic URLs
I just ran my first diagnostic and one of my primary immediate problems are duplicate titles and duplicate content. My guess it that because the root URL http://sitename.com (which has not yet been redirected to www...) has generated an entire tree of content which is identical to the tree rooted at http://www.sitename.com. QUESTION: Do I need to do a redirect simply for the root url (sitename.com -> www.sitename.com) or do I now need to develop specific 301 redirects for each of the sub-nodes/pages? ie sitename.com/?q=about-us -> www.sitename.com/?q=about-us sitename.com/?q=our-team -> www.sitename.com/?q=our-team etc.
Technical SEO | | Barrycliff680 -
"/" at the end of a URL
I just noticed that I have the exact same page showing up separately in my Google Analytics reports. One has a "/" at the end and the other does not. Otherwise, these are the exact same URL's. Is this something I need to be aware of from a duplicate content perspective? If so, how do I go about fixing this? I thought the SE's would automatically see that a URL with a "/" at the end is the same as one without, but if that is the case, why is it showing up in my reports as two separate pages?
Technical SEO | | Blockinc0