Why do i get multiple variations of my url with ?order=asc and ?view=list at the end of it in my crawl report?
-
I just did a crawl for one my clients to validate any error in the structure. Next thing I know is that the website have multiple variation of the same url with query like ?order=asc and ?view=list at the end of it.
I am wondering why these url variations appears in the crawl I just did since bots aren't suppose to go further thant the ? normally.
Just to show you a couple of url's of my crawl test.
<colgroup><col width="484"></colgroup>
| https://test.com/exemple/?per_page=9 |
| https://test.com/exemple/?per_page=15 |
| https://test.com/exemple/?per_page=30 |
| https://test.com/exemple/?orderby=popularity |
| https://test.com/exemple/?orderby=date |
| https://test.com/exemple/?orderby=price |
| https://test.com/exemple/?orderby=price-desc |
| https://test.com/exemple/?order=asc |
| https://test.com/exemple/?order=desc |
| https://test.com/exemple/?view=list |Thank you Guys
-
Thank you Samantha your answer is very useful !
-
Hey there!
Sam from Moz's Help Team here! As far as I'm aware, Google and other crawlers do crawl past the '?', unless certain parameters are disallowed within the robots.txt. If the URL is: https://test.com/exemple/?per_page=9, a search engine will see something like test com exemple 'search' etc. Google recommends blocking all Internal Search Results in the Robots.txt file - for Rogerbot, it would look something like this
User-agent: Rogerbot
Disallow: ?utmHere is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt
I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site crawl only shows homepage
Hi everyone, A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage. Does somebody have an idea why the other pages of the website are not showing up? Thanks,
Moz Bar | | WeAreDigital_BE
Jens0 -
Custom Reporting cannot be set to daily???
Hi, I created a custom report and scheduled it to be sent daily on my email, the problem is even though I set it to daily the next reporting is always weekly? My other daily reports are working fine except this one. Can anyone shed a light on this and why it's happening. Thanks
Moz Bar | | en-gageinc0 -
SEOMOZ On page ranking reports update
I've recently made a load of changes to improve the on page ranking report grades for my site. I was givena C grade for this page: http://www.katalysticevents.com/luxury-wedding-teepee-hire-about/ based on two key phrases Wedding Tipi and Wedding Teepeee. I've made all the changes as advised (excluding H1 tag) by the report but when I rereun the report it's picking up old page data and showing the same problems. Is there some kind of cache thing I'm missing here? The confusing thing is I made the same series of changes for this page: www.katalysticevents.com/luxury-marquee-hire-services/ for Luxury marquee and luxury marquee hire and the results are fine...I've got an A grade. Examples are things like ALT image attirbutes which don't seem to show up on the first page but are fine on the second page.
Moz Bar | | EdoubleD
The site is a Word press site and I'm using the "All in one SEO pack" plugin. Can anyone advise?0 -
Can the Moz tool identify variations of a Chinese language branded keyword?
We've recently started a trial of the Moz Pro service and are tracking a selection of keywords. Our primary band / product name is 功夫英语(Kungfu English), so we've set a rule that any keywords containing those four Chinese characters (功夫英语) should be marked as a branded keyword. However, in the "Non-Paid Keywords Sending Search Visits" section of our traffic report, we see a few variations on our brand name that are not being marked as branded keywords. (See attached Images). Based on our rules, shouldn't these variations also be marked as branded keywords without our needing to manually add them as such? Or have I misunderstood the intent of this rule? For the English text, the brand rule about words containing "kungfuenglish" seems to have resulted in all of "www.kungfuenglish.com", "http://www.kungfuenglish.com", and kungfuenglish being labelled as branded keywords. However, I'm not seeing the same sort of result with variations on the Chinese keyword, 功夫英语. KedTi1D.png z5bcUIt.png
Moz Bar | | PaulCoffey0 -
Moz Keyword Ranking Report
Hi, Moz Keyword Ranking report shows different results then what i see on my desktop for some keywords. There wasn't much difference on most of them until now so ignored some of the differences but now i see huge differences. For instance, on my Google UK I see the keyword ranking 1 or 2 but Moz reports show 22 or higher than that . Search Engine is Google UK. Would you please advise if I am doing anything wrong here? Thank you,
Moz Bar | | Rubix0 -
Moz "Crawl Diagnostics" doesn't respect robots.txt
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like: Duplicate content Overly dynamic URLs Duplicate Page Titles The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Moz Bar | | Vitalized
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored): Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/ Many thanks for any info on this issue.0