Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. How does a search engine bot navigate past a .PDF link?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    How does a search engine bot navigate past a .PDF link?

    Technical SEO
    5
    16
    5471
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • danatanseo
      danatanseo last edited by

      We have a large number of product pages that contain links to a .pdf of the technical specs for that product. These are all set up to open in a new window when the end user clicks.

      If these pages are being crawled, and a bot follows the link for the .pdf, is there any way for that bot to continue to crawl the site, or does it get stuck on that dangling page because it doesn't contain any links back to the site (it's a .pdf) and the "back" button doesn't work because the page opened in a new window?

      If this situation effectively stops the bot in its tracks and it can't crawl any further, what's the best way to fix this?

      1. Add a rel="nofollow" attribute

      2. Don't open the link in a new window so the back button remains finctional

      3. Both 1 and 2

      or

      4. Create specs on the page instead of relying on a .pdf

      Here's an example page: http://www.ccisolutions.com/StoreFront/product/mackie-cfx12-mkii-compact-mixer  - The technical spec .pdf is located under the "Downloads" tab [the content is all on one page in the source code - the tabs are just a design element]

      Thoughts and suggestions would be greatly appreciated.

      Dana

      1 Reply Last reply Reply Quote 0
      • danatanseo
        danatanseo @danatanseo last edited by

        Thanks very much Christopher. This is an excellent explanation. What do you think of Charlie and EGOL's suggestions regarding making sure that there are links embedded in these PDFs pointing either back to the product page or even to the home page?

        In your opinion, is this something worth doing? If so, why?

        1 Reply Last reply Reply Quote 0
        • ChristopherGlaeser
          ChristopherGlaeser @danatanseo last edited by

          Hi Dana,

          " ... you are right, one of the fundamental questions I still have is how does a bot behave when it finds an orphaned page like one of these? Does it just revert back to the sitemap and move one? Does it automatically go back to the last non-dead end page and move on from there? What does it do?"

          Bots are not really like a single spider that has to crawl around the web that can get trapped when entering an orphaned page with no back-button.  When a bot enters a site, it creates a list of all the internal pages that are linked from the home page.  Then it visits each page on that list and keeps adding more linked pages to that list.  Each time it adds more pages to the list, it only adds new unique pages and does not add duplicates.  It also keeps track of which pages it has already visited.  When all the pages have been visited once, and no new pages are discovered that are not already on the list, all of the pages have been crawled.

          Best,
          Christopher

          1 Reply Last reply Reply Quote 1
          • danatanseo
            danatanseo @donford last edited by

            Hi Don,

            Thanks so much for responding and while the answers I have received so far did give me some direction, you are right, one of the fundamental questions I still have is how does a bot behave when it finds an orphaned page like one of these? Does it just revert back to the sitemap and move one? Does it automatically go back to the last non-dead end page and move on from there? What does it do?

            Thanks for chiming in. I'd love it if someone more familiar with how a bot actually crawls links like this on a page would jump in with an answer.

            Dana

            ChristopherGlaeser danatanseo 2 Replies Last reply Reply Quote 0
            • danatanseo
              danatanseo @danatanseo last edited by

              Thanks Charlie. I think this is a good suggestion. I work 9-6 too, and just so happen to be the in-house SEO strategist, so this stuff is what I'm there to do. I don't mind the mundane aspects of SEO because the payoff is usually pretty rewarding! Now I know what I'm doing on Monday (on top of a dozen other things!)

              Thanks again!

              1 Reply Last reply Reply Quote 0
              • EGOL
                EGOL @danatanseo last edited by

                I would spend the time needed to do an assessment of these pages.

                ** how many of them have external links

                ** how many of them pull traffic from search or other sites

                ** how many of them are currently useful (are people looking at them)

                I would delete (and redirect the URL) of any page that answers "no" to the three items above.  These are "dead weight" on your site.

                Also, if these are .pdfs of print ads then they might simply be images in a pdf.  (test this by searching for an exact phrase from one of them in quotes and include site:yourdomain.com in the query.   Keep in mind that google can read the text in some images embedded in pdfs.

                I had a lot of pdfs with images on one of my site and got hit with a panda problem.  I think that Google thought that the .pdfs were thin content.  So I used rel=canonical to assign them to the most relevant page using .htaccess.  The panda problem was solved after a couple of months.

                Also, keep in mind that .pdfs can be used for conversions.  You can embed "add to cart" buttons and links into them and they will function just as on a web page.

                If any of these pdfs are pulling in tons I traffic I would figure out how I can put the pdf to better use or create webpage (and redirect the pdf) to best monetize/convert or whatever you business goals dictate.

                1 Reply Last reply Reply Quote 0
                • EGOL
                  EGOL @danatanseo last edited by

                  Can a bot navigate via a back button?

                  I don't think so.  They can follow links but they can't "click".

                  1 Reply Last reply Reply Quote 0
                  • donford
                    donford last edited by

                    Hi Dana

                    I think your question has been dodged a tad. I ways lead to understand that a .pdf or any page that opens in a new tab and does not link back to the original site, (dangling page) is not a problem. The reason being is that  crawlers don't really care how a page is opened. Because the crawler will fork at every link and crawls each new page/link from each fork, when it finds a orphan or dangling page it just stops. This of course is not an issue since if the crawler has forked at each link.

                    So the question is how a SE treats .pdf's rather how does it treat orphan page. Maybe somebody who works with crawlers can confirm or educate us both on they work.

                    Don

                    danatanseo 1 Reply Last reply Reply Quote 1
                    • danatanseo
                      danatanseo @danatanseo last edited by

                      Many thanks to both you and EGOL for excellent answers!

                      1 Reply Last reply Reply Quote 0
                      • danatanseo
                        danatanseo @danatanseo last edited by

                        Thanks EGOL. Yes, many of these .pdfs could be and are referenced by other sites. Given that there's no link from the .pdf back to our site, we really are missing out on a huge opportunity. I thought this might be the case as I pondered the whole concept of "dangling links" that was discussed in a SEOMoz blog post this week.

                        I agree about the last point regarding opening in a new window being more of a usability issue than a problem for SEO. I agree with you completely that opening in the same window is way better for the end user.

                        Can a bot navigate via a back button?

                        Thanks very much to both you and Charlie for your excellent answers!

                        1 Reply Last reply Reply Quote 0
                        • danatanseo
                          danatanseo @danatanseo last edited by

                          lol, thank heaven's they aren't spammy. However, they aren't particularly helfpul either. You see, about 3,000 of them are old .pdf versions of print advertising campaigns, going back as far as 2005. They contain obsolete pricing, products, etc. Unfortunately, instead of archiving them off the server, they've been continuously archived in a sub-directory of our main website.

                          Nearly all of it is indexed. It seems to me the best thing to do for these is to include a statement that the content is an old advertisement and include a llnk to our current "special offers" page.

                          What do you think of that as a strategy for at least giving engines and humans a means to navigate to someplace current on the site?

                          1 Reply Last reply Reply Quote 0
                          • EGOL
                            EGOL @danatanseo last edited by

                            I see 6000 pdfs as an amazing opportunity.  Get links on those pages and it will funnel a lot of power through your site.

                            If that was my site, we would be on that job immediately.  Could be a huge gain for some easy work.

                            1 Reply Last reply Reply Quote 2
                            • EGOL
                              EGOL @danatanseo last edited by

                              Go back and rework our .pdfs so they at least contain a link back to the homepage?

                              Yes!  Absolutely!  And, link them to other relevant pages.  If  these are reference documents they could be pulling in a lot of links and traffic from other sites.

                              In addition toAs well as configure the hyperlinks so they open in the same window instead of a new one?

                              In my opinion, this is not an SEO issue.  This is a usability issue.   I would have them open in the same window so the back button is available.

                              1 Reply Last reply Reply Quote 1
                              • danatanseo
                                danatanseo last edited by

                                Thank you Charlie. In our case, our .pdfs contain no links in them at all. There is nothing for a bot (or a human) that will navigate them out of the .pdf....not even the back button.

                                Considering that, and EGOL's response below, would the best course of action be to include, at the very least, an active link back to our homepage from all of our .pdf files?

                                We have as many as 6,000 .pdfs.

                                Thanks,

                                Dana

                                EGOL danatanseo 5 Replies Last reply Reply Quote 1
                                • danatanseo
                                  danatanseo @EGOL last edited by

                                  Thanks EGOL,

                                  Yes, I understand well that .pdf documents can be indexed. That's not my concern. My concern is that a bot that navigates to one of our many .pdf tech specs documents, which, incidentally, contains no outbound links to anything, would then become trapped and not be able to continue crawling the site. This is particularly true because we have them set up to open in a new window. In the example above, sure, there's a text reference back to the site "www.kingdom.com" - but it isn't a link in the .pdf. There are no links, in any of our .pdfs.

                                  So, what is the best way to deal with this? Go back and rework our .pdfs so they at least contain a link back to the homepage? In addition toAs well as configure the hyperlinks so they open in the same window instead of a new one?

                                  EGOL danatanseo 3 Replies Last reply Reply Quote 1
                                  • EGOL
                                    EGOL last edited by

                                    .pdf documents are crawled by bots and they accumulate pagerank just like .html pages.

                                    You can include links in them to other documents on the web and bots will crawl those links and pagerank will flow through them.

                                    .pdf documents can be given a "title tag" equivalent by opening their properties and giving the document a title.  This title will display in the SERPs.  .pfd documents can be hard to beat in the SERPs if they are optimized and have links from a competitive number of other web documents.

                                    Lots of document formats behave this way.  Excel, PowerPoint, Word for example.

                                    In my opinion, .pdf documents can trigger a Panda problem for your site if you have a lot of them with trivial or duplicate content (as in print versions of web documents).  They can be given rel=canonical through .htaccess to solve the Panda problem but Google often takes a long long time (sometimes months) to recognize the canonical and use that instruction.

                                    danatanseo 1 Reply Last reply Reply Quote 1
                                    • 1 / 1
                                    • First post
                                      Last post

                                    Got a burning SEO question?

                                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                    Start my free trial


                                    Browse Questions

                                    Explore more categories

                                    • Moz Tools

                                      Chat with the community about the Moz tools.

                                    • SEO Tactics

                                      Discuss the SEO process with fellow marketers

                                    • Community

                                      Discuss industry events, jobs, and news!

                                    • Digital Marketing

                                      Chat about tactics outside of SEO

                                    • Research & Trends

                                      Dive into research and trends in the search industry.

                                    • Support

                                      Connect on product support and feature requests.

                                    • See all categories

                                    Related Questions

                                    • JH_OffLimits

                                      Should we Nofollow Social Links?

                                      I've been asked the question of whether if we should nofollow all of our social links, would this be a wise thing to do? I'm not exactly getting a clear answer from search results and thought you guys would be best to ask 🙂 Thanks in advance.

                                      Technical SEO | | JH_OffLimits
                                      0
                                    • mundsack

                                      Is SEO effected of putting an external link in the primary navigation of a website?

                                      I have a customer, www.xxx.com.  This site has good traffic, low bounce rate (28%), 2:00 min avg time on site, and 45% return visitor rating.  No spam rankings, etc.  Good load time. Another site, www.yyy.com, has sent out a request for them to add them as a new link in www.xxx.com's primary navigation - using a title such as "abc" (not the name of the company or site of yyy.com).   This second site, www.yyy.com, has a bounce rate of 98%, avg time on site is :30, and 10.2% return visitor rate.  No spam flags noted in Open Site explorer.  Plus they are asking other sites similar to www.xxx.com to do the same thing. Questions/Concerns and Feedback appreciated: Will yyy.com's analytics and quality pass back to xxx.com and cause Google or algorithms to flag or penalize xxx.com?  (It ranks #1 for quite a few things.)  The relevancy between the sites is good -same industry, same business objectives. From a usability standpoint, isn't it more appropriate to place a link to another website in a different way? e.g. a promotional graphic wit a link or anchor text links? Isn't it more appropriate to ask another business for links - not using the primary nav of a site?  (It seems yyy.com is essentially asking other sites for 'free advertising/promotion.' Thanks!

                                      Technical SEO | | mundsack
                                      0
                                    • Koki.Mourao

                                      Image Search

                                      Hello Community, I have been reading and researching about image search and trying to find patterns within the results but unfortunately I could not get to a conclusion on 2 matters. Hopefully this community would have the answers I am searching for. 1) Watermarked Images (To remove or not to remove watermark from photos) I see a lot of confusion on this subject and am pretty much confused myself. Although it might be true that watermarked photos do not cause a punishment, it sure does not seem to help. At least in my industry and on a bunch of different random queries I have made, watermarked images are hard to come by on Google's images results. Usually the first results do not have any watermarks. I have read online that Google takes into account user behavior and most users prefer images with no watermark. But again, it is something "I have read online" so I don't have any proof.  I would love to have further clarification and, if possible, a definite guide on how to improve my image results. 2)  Multiple nested folders (Folder depth) Due to speed concerns our tech guys are using 1 image per folder and created a convoluted folder structure where the photos are actually 9 levels deep. Most of our competition and many small Wordpress blogs outrank us on Google images and on ALL INSTANCES I have checked, their photos are 3, 4 or 5 levels deep. Never inside 9 nested folders.  
                                      So... A) Should I consider removing the watermark - which is not that intrusive but is visible?
                                      B) Should I try to simplify the folder structure for my photos? Thank you

                                      Technical SEO | | Koki.Mourao
                                      0
                                    • Gavin.Atkinson

                                      Why has my search traffic suddenly tanked?

                                      On 6 June, Google search traffic to my Wordpress travel blog http://www.travelnasia.com tanked completely. There are no warnings or indicators in Webmaster Tools that suggest why this happened. Traffic from search has remained at zero since 6 June and shows no sign of recovering. Two things happened on or around 6 June. (1) I dropped my premium theme which was proving to be not mobile friendly and replaced it with the ColorMag theme which is responsive. (2) I relocated off my previous hosting service which was showing long server lag times to a faster host. Both of these should have improved my search performance, not tanked it. There were some problems with the relocation to the new web host which resulted in a lot of "out of memory" errors on the website for 3-4 days. The allowed memory was simply not enough for the complexity of the site and the volume of traffic. After a few days of trying to resolve these problems, I moved the site to another web host which allows more PHP memory and the site now appears reliably accessible for both desktop and mobile. But my search traffic has not recovered. I am wondering if in all of this I've done something that Google considers to be a cardinal sin and I can't see it. The clues I'm seeing include: Moz Pro was unable to crawl my site last Friday. It seems like every URL it tried to crawl was of the form http://www.travelnasia.com/wp-login.php?action=jetpack-sso&redirect_to=http://www.travelnasia.com/blog/bangkok-skytrain-bts-mrt-lines which resulted in a 500 status error. I don't know why this happened but I have disabled the Jetpack login function completely, just in case it's the problem. GWT tells me that some of my resource files are not accessible by GoogleBot due to my robots.txt file denying access to /wp-content/plugins/. I have removed this restriction after reading the latest advice from Yoast but I still can't get GWT to fetch and render my posts without some resource errors. On 6 June I see in Structured Data of GWT that "items" went from 319 to 1478 and "items with errors" went from 5 to 214. There seems to be a problem with both hatom and hcard microformats but when I look at the source code they seem to be OK. What I can see in GWT is that each hcard has a node called "n [n]" which is empty and Google is generating a warning about this. I see that this is because the author vcard URL class now says "url fn n" but I don't see why it says this or how to fix it. I also don't see that this would cause my search traffic to tank completely. I wonder if anyone can see something I'm missing on the site. Why would Google completely deny search traffic to my site all of a sudden without notifying any kind of penalty? Note that I have NOT changed the content of the site in any significant way. And even if I did, it's unlikely to result in a complete denial of traffic without some kind of warning.

                                      Technical SEO | | Gavin.Atkinson
                                      1
                                    • mshowells

                                      Fake Links indexing in google

                                      Hello everyone, I have an interesting situation occurring here, and hoping maybe someone here has seen something of this nature or be able to offer some sort of advice. So, we recently installed a wordpress to a subdomain for our business and have been blogging through it. We added the google webmaster tools meta tag and I've noticed an increase in 404 links. I brought this up to or server admin, and he verified that there were a lot of ip's pinging our server looking for these links that don't exist. We've combed through our server files and nothing seems to be compromised. Today, we noticed that when you do site:ourdomain.com into google the subdomain with wordpress shows hundreds of these fake links, that when you visit them, return a 404 page. Just curious if anyone has seen anything like this, what it may be, how we can stop it, could it negatively impact us in anyway? Should we even worry about it? Here's the link to the google results. https://www.google.com/search?q=site%3Amshowells.com&oq=site%3A&aqs=chrome.0.69i59j69i57j69i58.1905j0j1&sourceid=chrome&es_sm=91&ie=UTF-8 (odd links show up on pages 2-3+)

                                      Technical SEO | | mshowells
                                      0
                                    • Gordian

                                      Does Google Bot accept Cookies

                                      I am working with a per page results refinement that stores a cookie on the users computer and then keeps that same per page as the user goes around the site. I was just wondering if that was true for Google bot or Bing bot as well. Will they keep the cookie or would they not be able to accept it. I just want to know as I dont want different urls created if they can keep the cookie. Thanks!

                                      Technical SEO | | Gordian
                                      0
                                    • nicole.healthline

                                      How do search engines treat urls that end in hashtags?

                                      How do search engines treat urls that end in hashtags? For example, www.domain.com/abc#xyz.

                                      Technical SEO | | nicole.healthline
                                      0
                                    • Aspedia

                                      Add to Cart Link

                                      We have shopping cart links (<a href's,="" not="" input="" buttons)="" that="" link="" to="" a="" url="" along="" the="" lines="" of="" cart="" add="" 123&return="/product/123. </p"></a> <a href's,="" not="" input="" buttons)="" that="" link="" to="" a="" url="" along="" the="" lines="" of="" cart="" add="" 123&return="/product/123. </p">The SEOMoz site crawls are flagging these as a massive number of 302 redirects and I also wonder what sort of effect this is having on linkjuice flowing around the site. </a> <a href's,="" not="" input="" buttons)="" that="" link="" to="" a="" url="" along="" the="" lines="" of="" cart="" add="" 123&return="/product/123. </p">I can see several possible solutions: Make the links nofollow Make the links input buttons Block /cart/add with robots.txt Make the links 301 instead of 302 Make the links javascript (probably worst care) All of these would result in an identical outcome for the UX, but are very different solutions. What would you suggest?</a>

                                      Technical SEO | | Aspedia
                                      0

                                    Get started with Moz Pro!

                                    Unlock the power of advanced SEO tools and data-driven insights.

                                    Start my free trial
                                    Products
                                    • Moz Pro
                                    • Moz Local
                                    • Moz API
                                    • Moz Data
                                    • STAT
                                    • Product Updates
                                    Moz Solutions
                                    • SMB Solutions
                                    • Agency Solutions
                                    • Enterprise Solutions
                                    Free SEO Tools
                                    • Domain Authority Checker
                                    • Link Explorer
                                    • Keyword Explorer
                                    • Competitive Research
                                    • Brand Authority Checker
                                    • Local Citation Checker
                                    • MozBar Extension
                                    • MozCast
                                    Resources
                                    • Blog
                                    • SEO Learning Center
                                    • Help Hub
                                    • Beginner's Guide to SEO
                                    • How-to Guides
                                    • Moz Academy
                                    • API Docs
                                    About Moz
                                    • About
                                    • Team
                                    • Careers
                                    • Contact
                                    Why Moz
                                    • Case Studies
                                    • Testimonials
                                    Get Involved
                                    • Become an Affiliate
                                    • MozCon
                                    • Webinars
                                    • Practical Marketer Series
                                    • MozPod
                                    Connect with us

                                    Contact the Help team

                                    Join our newsletter
                                    Moz logo
                                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                    • Accessibility
                                    • Terms of Use
                                    • Privacy

                                    Looks like your connection to Moz was lost, please wait while we try to reconnect.