Bingpreview/1.0b Useragent Using Adding Trailing Slash to all URLs
-
The Bingpreview crawler, which I think exists in order to take snapshots of mobile friendly pages, crawled my pages last night for the first time. However, it is adding a trailing slash to the end of each of my dynamic pages. The result is my program is giving the wrong page--my program is not expecting a trailing slash at the end of the urls. It was 160 pages, but I have thousands of pages it could do this to.
I could try doing a mod rewrite but that seems like it should be unnecessary. ALL the other crawlers are crawling the proper urls. None of my hyperlinks have the slash on the end. I have written to Bing to tell them of the problem.
Is anyone else having this issue? Any other suggestions for what to do?
The user agent is: Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
-
Will do. Forgot to mention Bing is checking into it. But for the reasons you mentioned I am still going to do the 301s. Thanks again.
-
Sounds like a plan. I'd also make every redirect a 301, just in case. Cheers.
-
Thanks for your reply Cyrus. Wow, so much to learn.
I will put in logic via a mod redirect to basically remove the trailing slash and go to the resulting url because otherwise all the trailing slash urls will be a different page of basically a 'no-product' business and the like.
These are all dynamically generated pages, so I think as long as I resolve to the 'proper' no-slash version then I won't need to worry about anything else, like a rel=canonical tag because there wont be any identical content.
Does that sound right to you?
-
On one hand I'd agree with you that you shouldn't have to rewrite those URLs on your end. On the other hand, it's usually best practice to make sure both versions of a URL (with slash and/or without) resolve to the same page. The reason for this is that:
- Search bots, including Google, will often "explore" variations of URLs for discoverability reasons - they want to make sure they are discovering all of your available content.
- People will link to you with and without trailing slashes. If they link to you with a trailing slash and your page breaks, you could be wasting link equity, to say nothing of the bad user experience of people visiting your site from the referral links
- For one reason or another it's common to append URLs with various parameters (for tracking reasons, campaings, etc) and often these URLs are generated by third party services when pointing at your site.
For all of these reasons, it's pretty common to either force redirect trailing slashes (via a 301) or make sure both versions resolve to the same content, and use a rel=canonical tag to indicate to search engines that these are indeed meant to be the same page.
On the other hand, if this is something not feasible and URLs ending in a slash are indeed different pages, you might want to carefully consider what those pages deliver to both humans and bots because it seems inevitable that both will eventually crawl and stumble upon them.
Perhaps not the answer you were looking for, but I hope it helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing the Trailing Slash in Magento
Hi guys, We have noticed trailing slash vs non-trailing slash duplication on one of our sites. Example:
Intermediate & Advanced SEO | | brandonegroup
Duplicate: https://www.example.com.au/living/
Preferred: https://www.example.com.au/living So, SEO-wise, we suggested placing a canonical tag on all trailing slash pointing to non-trailing slash. However, devs have advised against removing the trailing slash from some URLs with a blanket rule, as this may break functionality in Magento that depends on the trailing slash. The full site would need to be tested after implementing a blanket rewrite rule. Is any other way to address this trailing slash duplication issue without breaking anything in Magento? Keen to hear from you guys. Cheers,0 -
Product URL Optimisation
Hi guys, We are currently trying to add new products to our site but we are in a quandary on what type of URL structure to pursue. For example:
Intermediate & Advanced SEO | | michel_8
Product Name: Aspect Exfoliating Cleanser 240ml https://www.example.com.au/aspect-exfoliating-cleanser-240ml (including the size)
VS
https://www.example.com.au/aspect-exfoliating-cleanser 1.) Which is a better URL structure based on SEO 2018 and why?
2.) Is there any merit in removing the size from the URL key with the aim of attracting more traffic? Keen to hear from you guys! Cheers,0 -
SITEMAP - Does <changefreq>and <image:title>have any apreciable effect?</image:title></changefreq>
Hi everyone. It was hard to find some actual evidence that some of the atributes to be declared in a sitemap have some real impact.
Intermediate & Advanced SEO | | Gaston Riera
Particularly, im interested in these two: <changefreq></changefreq> and**image:title</image:title>** I've used them in a few cases just to check their effect and couldnt see any.
Do you have any experience with these? Or any other atribute that might be helpful, in order to create a more accurate and effective sitemap? Also, this could be a great topic to create a new Moz Blog post, the one about sitemap is 8years old.0 -
Confused: Url Restructure
Hello, We're giving our website a bit of a spring clean in terms of SEO. The site is doing ok, but after the time invested in SEO, content and last year's migration of multiple sites into one, we're not seeing the increase in traffic we had hoped. Our current urls look something like this: /a-cake-company/cup-cakes/strawberry We have the company name as the first level as we with the migration we migrated many companies into one site. What we're considering is testing some pages with a structure like this: /cup-cakes/cup-cake-company-strawberry So we'll lose a level and we'll focus more on the category of the product rather than the brand. What's your thoughts on this? We weren't going to do a mass change yet, just a test, but is this something we should be focusing on? In terms of organisation our current url structure is perfect, but what about from an SEO point of view? In terms of keywords customers are looking for both options. Thanks!
Intermediate & Advanced SEO | | HB170 -
Mixing static.htm urls and dynamic urls on a Windows IIS Server?
Hi all, We've had a website originally built using static html with .htm extensions ranking well in Google hence we want to keep those pages/urls. We are on a dedicated sever (Windows IIS). However our developer has custom made a new DYNAMIC section for the site which shows new added products dynamically and allows them to be booked online via shopping cart. We are having problems displaying them both on the same domain even if we put the dynamic section withing its own subfolder and keep the static htms in the root. Is it possible to have both function on IIS (even if they may have to function a little separately)? Does anyone have previous experience of this kind of issue or a way of making both work? What setup do we need to do on the dedicated server.
Intermediate & Advanced SEO | | emerald0 -
301 redirect with /? in URL
For a Wordpress site that has the ending / in the URL with a ? after it... how can you do a 301 redirect to strip off anything after the / For example how to take this URL domain.com/article-name/?utm_source=feedburner and 301 to this URL domain.com/article-name/ Thank you for the help
Intermediate & Advanced SEO | | COEDMediaGroup0 -
Google tagged URL an overly-dynamic URL?
I'm reviewing my campaign, and spotted the overly-dynamic URL box showing a few links. Reviewing it, they are my Google Tagged URLs (utm_source, utm_medium_utm_campaign etc) I've turned some internal links to Google Tagged URLs but should these cause concern?
Intermediate & Advanced SEO | | Bio-RadAbs0 -
URL rewrites
We have a problem whereby a number of our urls are adressable from different urls - I'm told because of a quirk of developing in .net. e.g. mysite/FundComparison mysite/Fund-comparison mysite/fund-comparison We asked our supplier who hosts this section of our site to do some url rewrites so that the duplicates would 301 to the correct url. They're on IIS 6.0 and are not ready to upgrade to IIS 7.0 (my recommendation, which makes it easier for them to do the rewrite using the rewrite module). They said it would take 6-8 weeks to implement a web controller to do this. "The bulk of the time for this implementation is in the build of the engine + the addition of all the possible permutations of the URL to redirect to the proper URL." This sounds absolutely insane to me. I would have thought it could be done in a matter of hours. What do people think?
Intermediate & Advanced SEO | | SearchPM0