How does a search engine bot navigate past a .PDF link?
-
We have a large number of product pages that contain links to a .pdf of the technical specs for that product. These are all set up to open in a new window when the end user clicks.
If these pages are being crawled, and a bot follows the link for the .pdf, is there any way for that bot to continue to crawl the site, or does it get stuck on that dangling page because it doesn't contain any links back to the site (it's a .pdf) and the "back" button doesn't work because the page opened in a new window?
If this situation effectively stops the bot in its tracks and it can't crawl any further, what's the best way to fix this?
1. Add a rel="nofollow" attribute
2. Don't open the link in a new window so the back button remains finctional
3. Both 1 and 2
or
4. Create specs on the page instead of relying on a .pdf
Here's an example page: http://www.ccisolutions.com/StoreFront/product/mackie-cfx12-mkii-compact-mixer - The technical spec .pdf is located under the "Downloads" tab [the content is all on one page in the source code - the tabs are just a design element]
Thoughts and suggestions would be greatly appreciated.
Dana
-
Thanks very much Christopher. This is an excellent explanation. What do you think of Charlie and EGOL's suggestions regarding making sure that there are links embedded in these PDFs pointing either back to the product page or even to the home page?
In your opinion, is this something worth doing? If so, why?
-
Hi Dana,
" ... you are right, one of the fundamental questions I still have is how does a bot behave when it finds an orphaned page like one of these? Does it just revert back to the sitemap and move one? Does it automatically go back to the last non-dead end page and move on from there? What does it do?"
Bots are not really like a single spider that has to crawl around the web that can get trapped when entering an orphaned page with no back-button. When a bot enters a site, it creates a list of all the internal pages that are linked from the home page. Then it visits each page on that list and keeps adding more linked pages to that list. Each time it adds more pages to the list, it only adds new unique pages and does not add duplicates. It also keeps track of which pages it has already visited. When all the pages have been visited once, and no new pages are discovered that are not already on the list, all of the pages have been crawled.
Best,
Christopher -
Hi Don,
Thanks so much for responding and while the answers I have received so far did give me some direction, you are right, one of the fundamental questions I still have is how does a bot behave when it finds an orphaned page like one of these? Does it just revert back to the sitemap and move one? Does it automatically go back to the last non-dead end page and move on from there? What does it do?
Thanks for chiming in. I'd love it if someone more familiar with how a bot actually crawls links like this on a page would jump in with an answer.
Dana
-
Thanks Charlie. I think this is a good suggestion. I work 9-6 too, and just so happen to be the in-house SEO strategist, so this stuff is what I'm there to do. I don't mind the mundane aspects of SEO because the payoff is usually pretty rewarding! Now I know what I'm doing on Monday (on top of a dozen other things!)
Thanks again!
-
I would spend the time needed to do an assessment of these pages.
** how many of them have external links
** how many of them pull traffic from search or other sites
** how many of them are currently useful (are people looking at them)
I would delete (and redirect the URL) of any page that answers "no" to the three items above. These are "dead weight" on your site.
Also, if these are .pdfs of print ads then they might simply be images in a pdf. (test this by searching for an exact phrase from one of them in quotes and include site:yourdomain.com in the query. Keep in mind that google can read the text in some images embedded in pdfs.
I had a lot of pdfs with images on one of my site and got hit with a panda problem. I think that Google thought that the .pdfs were thin content. So I used rel=canonical to assign them to the most relevant page using .htaccess. The panda problem was solved after a couple of months.
Also, keep in mind that .pdfs can be used for conversions. You can embed "add to cart" buttons and links into them and they will function just as on a web page.
If any of these pdfs are pulling in tons I traffic I would figure out how I can put the pdf to better use or create webpage (and redirect the pdf) to best monetize/convert or whatever you business goals dictate.
-
Can a bot navigate via a back button?
I don't think so. They can follow links but they can't "click".
-
Hi Dana
I think your question has been dodged a tad. I ways lead to understand that a .pdf or any page that opens in a new tab and does not link back to the original site, (dangling page) is not a problem. The reason being is that crawlers don't really care how a page is opened. Because the crawler will fork at every link and crawls each new page/link from each fork, when it finds a orphan or dangling page it just stops. This of course is not an issue since if the crawler has forked at each link.
So the question is how a SE treats .pdf's rather how does it treat orphan page. Maybe somebody who works with crawlers can confirm or educate us both on they work.
Don
-
Many thanks to both you and EGOL for excellent answers!
-
Thanks EGOL. Yes, many of these .pdfs could be and are referenced by other sites. Given that there's no link from the .pdf back to our site, we really are missing out on a huge opportunity. I thought this might be the case as I pondered the whole concept of "dangling links" that was discussed in a SEOMoz blog post this week.
I agree about the last point regarding opening in a new window being more of a usability issue than a problem for SEO. I agree with you completely that opening in the same window is way better for the end user.
Can a bot navigate via a back button?
Thanks very much to both you and Charlie for your excellent answers!
-
lol, thank heaven's they aren't spammy. However, they aren't particularly helfpul either. You see, about 3,000 of them are old .pdf versions of print advertising campaigns, going back as far as 2005. They contain obsolete pricing, products, etc. Unfortunately, instead of archiving them off the server, they've been continuously archived in a sub-directory of our main website.
Nearly all of it is indexed. It seems to me the best thing to do for these is to include a statement that the content is an old advertisement and include a llnk to our current "special offers" page.
What do you think of that as a strategy for at least giving engines and humans a means to navigate to someplace current on the site?
-
I see 6000 pdfs as an amazing opportunity. Get links on those pages and it will funnel a lot of power through your site.
If that was my site, we would be on that job immediately. Could be a huge gain for some easy work.
-
Go back and rework our .pdfs so they at least contain a link back to the homepage?
Yes! Absolutely! And, link them to other relevant pages. If these are reference documents they could be pulling in a lot of links and traffic from other sites.
In addition toAs well as configure the hyperlinks so they open in the same window instead of a new one?
In my opinion, this is not an SEO issue. This is a usability issue. I would have them open in the same window so the back button is available.
-
Thank you Charlie. In our case, our .pdfs contain no links in them at all. There is nothing for a bot (or a human) that will navigate them out of the .pdf....not even the back button.
Considering that, and EGOL's response below, would the best course of action be to include, at the very least, an active link back to our homepage from all of our .pdf files?
We have as many as 6,000 .pdfs.
Thanks,
Dana
-
Thanks EGOL,
Yes, I understand well that .pdf documents can be indexed. That's not my concern. My concern is that a bot that navigates to one of our many .pdf tech specs documents, which, incidentally, contains no outbound links to anything, would then become trapped and not be able to continue crawling the site. This is particularly true because we have them set up to open in a new window. In the example above, sure, there's a text reference back to the site "www.kingdom.com" - but it isn't a link in the .pdf. There are no links, in any of our .pdfs.
So, what is the best way to deal with this? Go back and rework our .pdfs so they at least contain a link back to the homepage? In addition toAs well as configure the hyperlinks so they open in the same window instead of a new one?
-
.pdf documents are crawled by bots and they accumulate pagerank just like .html pages.
You can include links in them to other documents on the web and bots will crawl those links and pagerank will flow through them.
.pdf documents can be given a "title tag" equivalent by opening their properties and giving the document a title. This title will display in the SERPs. .pfd documents can be hard to beat in the SERPs if they are optimized and have links from a competitive number of other web documents.
Lots of document formats behave this way. Excel, PowerPoint, Word for example.
In my opinion, .pdf documents can trigger a Panda problem for your site if you have a lot of them with trivial or duplicate content (as in print versions of web documents). They can be given rel=canonical through .htaccess to solve the Panda problem but Google often takes a long long time (sometimes months) to recognize the canonical and use that instruction.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
UGC - Product Reviews Search Engine accesibility
Hi everyone A question about using product reviews on a website - I completely understand genuine product reviews can be great for creating totally unique content and for seo in general. But my understanding is also that it is great only when it's accessible by google. So the question I have is that by using aggregate review services - i.e. reviews(dot)co(dot)uk widgets, are they always easily accessible by Google? Examples I've seen that use the widget are: https://www.oliverbonas.com/jewellery/silver-honeycomb-bee-necklace-29494 http://www.fires-cookers.co.uk/hoover-dyc169a-tumble-dryer-white.html#prod-reviews So when you view the page source for these pages, the actual content of the reviews are nowhere to be seen. I was wondering if anyone has experience of using any of these thrid party widgets, and if these come as issues, what you would do to make sure these reviews content are seen by Google? Thank you
Technical SEO | | MH-UK0 -
No PA and no DA and no back links.
Hello Friends, I am having some problem in my website. I hosted this before 4 months. I have done some seo work related to back links. But when I see in opensiteexplorer than it does not show any back links even It shows 1 DA and 1 PA but why?
Technical SEO | | docbeans0 -
Should I consider webmaster tools links and linked pages ratio to remove unnatural links?
I don't know this is a suitable place for post this question. Anyway I have done it. According to the Google webmaster tools, Links to your site page. My blog has considerable amount of links, from linked pages (from certain domain names). For an instance please refer following screenshot. When I am removing unnatural links, should I consider these, links from linked pages ratio? Almost all of these sites are social bookmarking sites. When I publish a new bookmark on those sites, they automatically add a homepage link. As a result of that, I got a huge number of home page links from linked pages. What is your recommendation? Thanks! webmaster.png web_master_tools.png
Technical SEO | | Godad0 -
Site is not displaying in Search Engines
My site is www.deoveritas.com it is in magento framework and it has a blog section in wordpress. When I enter Site:www.deoveroitas.com in google it shows all blog links in search result. The homepage and other innerpages are not getting displayed in search results at all. I even tried searching for www.deoveritas.com/about-us and it displays blogs in result. Checked Google webmaster fetch as google and it was index and successful. Can you please help me with this. Is my site de-indexed or banned by Google? the same issue is on Bing and Yahoo search engines too. Please help Thank you.
Technical SEO | | tpt.com0 -
Disallow: /search/ in robots but soft 404s are still showing in GWT and Google search?
Hi guys, I've already added the following syntax in robots.txt to prevent search engines in crawling dynamic pages produce by my website's search feature: Disallow: /search/. But soft 404s are still showing in Google Webmaster Tools. Do I need to wait(it's been almost a week since I've added the following syntax in my robots.txt)? Thanks, JC
Technical SEO | | esiow20130 -
Too many on page links
Hi All, As we all know, having to much links on a page is an obstacle for search engine crawlers in terms of the crawl allowance. My category pages are labeled as pages with to many "one page" links by the SEOmoz crawler. This probably comes from the fact that each product on the category page has multiple links (on the image and model number). Now my question is, would it help to setup a text-link with a clickable area as big as the product area? This means every product gets just one link. Would this help get the crawlers deeper in these pages and distribute the link-juice better? Or is Google smart enough already to figure out that two links to the same product page shouldn't be counted as two? Thanks for your replies guys. Rich
Technical SEO | | Horlogeboetiek0 -
Inter-link Optimization for Pages
Hi Guys, How much interlinking is okay for between pages? Can i start link my internal pages like Wikipedia does? 20,30 interlink per page? OR is it all depends on Page Authority? Can i go inter-link crazy on 1 year old legitimate site? Big Thanks?
Technical SEO | | DigitalJungle0 -
If you add a no follow to a time sensitive link, will it get picked up as broken link 404 in WMT report?
We have a client who publishes deals that are time sensitive. Links to the deals expire and so Google's crawlers are picking them up and finding a 404 If I no follow them, will the 404's still get picked up and reported in WMT? The same question applies to SEOMoz Pro.
Technical SEO | | Red_Mud_Rookie0