Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Let your business shine with Listings AI
      Moz Local

      Let your business shine with Listings AI

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Digital Marketing
    3. Web Design
    4. Best method to stop crawler access to extra Nav Menu

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Best method to stop crawler access to extra Nav Menu

    Web Design
    3
    4
    1571
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • DWJames
      DWJames last edited by

      Our shop site has a 3 tier drop down mega-menu so it's easy to find your way to anything from anywhere. It contains about 150 links and probably 300 words of text.

      We also have a more context-driven single layer of sub-category navigation as well as breadcrumbs on our category pages.

      You can get to every product and category page without using the drop down mega-menu.

      Although the mega-menu is a helpful tool for customers, it means that every single page in our shop has an extra 150 links on it that go to stuff that isn't necessarily related or relevant to the page content. This means that when viewed from the context of a crawler, rather than a nice tree like crawling structure, we've got more of an unstructured mesh where everything is linked to everything else.

      I'd like to hide the mega-menu links from being picked up by a crawler, but what's the best way to do this?

      I can add a nofollow to all mega-menu links, but are the links still registered as page content even if they're not followed? It's a lot of text if nothing else.

      Another possibility we're considering is to set the mega-menu to only populate with links when it's main button is hovered over. So it's not part of the initial page load content at all.

      Or we could use a crude yet effective system we have used for some other menus we have of base encoding the content inline so it's not readable by a spider.

      What would you do and why?

      Thanks,

      James

      1 Reply Last reply Reply Quote 0
      • blacey
        blacey @Click2Rank last edited by

        I agree Alan,

        Mega Menu's are a good way to dilate the link equity of your page and in most cases it isn't needed at all. Keep the top-level navigation simple and have a submenu on all pages that contain links relevant to that section.

        EG: Mega Menu could be:

        Home Mens (Mens Tops, Mens Jeans, Mens Coats), Women (Womens Tops, Womens Jeans etc) Contact us

        In this example it would be better to have one top level menu for:

        Home | Mens | Women | Contact us

        Then when your in the men or women section show links to "Tops", "Jeans" and "Coats". That way those links are relevant to the section you're in and reinforces the structure of that section to search engines.

        After giving it further thought I would suggest not having a mega menu at all, because it may harm your SEO on-page optimisation efforts in the long term.

        1 Reply Last reply Reply Quote 0
        • Click2Rank
          Click2Rank last edited by

          Ben's partially correct.  Unfortunately Google has been claiming they do process Javascript for a while, and they recently stated they've begun reading AJAX.  Of course they do a lousy job of it and don't always get it right, which just makes things even more muddy.

          So from an SEO best practices perspective, you shouldn't have the menu(s) in the first place, at all.

          You may also THINK their good for users but has any significant study been performed to confirm that?  You'd need to check click-through rates on all the links to know for sure.

          What I've found through years of auditing sites that have such menus is that it almost always turns out to be the case where most of the deeper links NEVER get clicked on from within these menus.  Instead, they're overwhelming to users.  This is why it's better to not have them from a UX perspective.

          If you abandon them and go with more traditional hierarchical index and sub-index pages, and if those are properly optimized, you'll not only eliminate the massive SEO problem but in fact get more of your category pages to have higher ranking strength and authority over time.

          IF you're going to keep them in any form because you don't want to go to the extreme I recommend, then yes - AJAX would likely be the only scenario that offers the least likelihood of search engines choking on the over-use of links.

          And for the record, the real current problem with all those links on every page is duplicate content confusion - all of those URLS at the source level dilutes the uniqueness of content on every page of the site.  And that also means you're harming the topical focus of every page as well.  So whatever you do, AJAX or doing away with them altogether is going to be of high value long term.

          • Alan Bleiweiss - Click2Rank's Search Team Director
          blacey 1 Reply Last reply Reply Quote 1
          • blacey
            blacey last edited by

            From my experience I don't think you can really 'hide' the megamenu links from a crawler if they are generated using a content management system (code server side). If the link is on the page in the HTML then it will be crawled by a bot etc.

            The general method of getting a mega menu to work is through the use of CSS and JavaScript, so you might want to have a look at using AJAX to get the relevant links from the database and then use JavaScript to put the links into the page.

            This isn't a great solution, but bots cannot load JavaScript, so what they will see is only the links that are served up from the content management system.

            1 Reply Last reply Reply Quote 0
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • GregLB

              Do things like using labels on an element that is not a form input affect how google sees us in regards to accessibility?

              Do things like using labels on an element that is not a form input affect how google sees us? It's an accessibility error that our devs have made - using a label element because it looks good, not because it's an actual label on a form field. Just wondering how that affects accessibility in Google's eyes.

              Web Design | | GregLB
              0
            • QBSEO

              What’s the best tool to visualize internal link structure and relationships between pages on a single site?

              I‘d like to review the internal linking structure on my site. Is there a tool that can visualize the relationships between all of the pages within my site?

              Web Design | | QBSEO
              0
            • cinchmedia

              Using Button Links vs Sidebar Menu

              I have a services page with a lot of rich text and a slideshow of images.  Currently, I am using a column of buttons to various services, and am wondering if a sidebar menu would be more effective for Google to crawl and rank?

              Web Design | | cinchmedia
              0
            • Kingalan1

              Lots of Listing Pages with Thin Content on Real Estate Web Site-Best to Set them to No-Index?

              Greetings Moz Community: As a commercial real estate broker in Manhattan I run a web site with over 600 pages. Basically the pages are organized in the following categories: 1. Neighborhoods (Example:http://www.nyc-officespace-leader.com/neighborhoods/midtown-manhattan)  25 PAGES Low bounce rate 2. Types of Space (Example:http://www.nyc-officespace-leader.com/commercial-space/loft-space) 
              15 PAGES Low bounce rate. 3. Blog (Example:http://www.nyc-officespace-leader.com/blog/how-long-does-leasing-process-take 
              30 PAGES Medium/high bounce rate 4. Services (Example:http://www.nyc-officespace-leader.com/brokerage-services/relocate-to-new-office-space)  High bounce rate
              3 PAGES 5. About Us (Example:http://www.nyc-officespace-leader.com/about-us/what-we-do
              4 PAGES High bounce rate 6. Listings (Example:http://www.nyc-officespace-leader.com/listings/305-fifth-avenue-office-suite-1340sf)
              300 PAGES High bounce rate (65%), thin content 7. Buildings (Example:http://www.nyc-officespace-leader.com/928-broadway
              300 PAGES  Very high bounce rate (exceeding 75%) Most of the listing pages do not have more than 100 words.  My SEO firm is advising me to set them "No-Index, Follow". They believe the thin content could be hurting me. Is this an acceptable strategy? I am concerned that when Google detects 300 pages set to "No-Follow" they could interpret this as the site seeking to hide something and penalize us. Also, the building pages have a low click thru rate. Would it make sense to set them to "No-Follow" as well? Basically, would it increase authority in Google's eyes if we set pages that have thin content and/or low click thru rates to "No-Follow"? Any harm in doing this for about half the pages on the site? I might add that while I don't suffer from any manual penalty volume has gone down substantially in the last month. We upgraded the site in early June and somehow 175 pages were submitted to Google  that should not have been indexed. A removal request has been made for those pages. Prior to that we were hit by Panda in April 2012 with search volume dropping from about 7,000 per month to 3,000 per month. Volume had increased back to 4,500 by April this year only to start tanking again. It was down to 3,600 in June. About 30 toxic links were removed in late April and a disavow file was submitted with Google in late April for removal of links from 80 toxic domains. Thanks in advance for your responses!! Alan

              Web Design | | Kingalan1
              0
            • mmewdell

              Is it cloaking/hiding text if textual content is no longer accessible for mobile visitors on responsive webpages?

              My company is implementing a responsive design for our website to better serve our mobile customers. However, when I reviewed the wireframes of the work our development company is doing, it became clear to me that, for many of our pages, large parts of the textual content on the page, and most of our sidebar links, would no longer be accessible to a visitor using a mobile device. The content will still be indexable, but hidden from users using media queries. There would be no access point for a user to view much of the content on the page that's making it rank. This is not my understanding of best practices around responsive design. My interpretation of Google's guidelines on responsive design is that all of the content is served to both users and search engines, but displayed in a more accessible way to a user depending on their mobile device. For example, Wikipedia pages have introductory content, but hide most of the detailed info in tabs. All of the information is still there and accessible to a user...but you don't have to scroll through as much to get to what you want. To me, what our development company is proposing fits the definition of cloaking and/or hiding text and links - we'd be making available different content to search engines than users, and it seems to me that there's considerable risk to their interpretation of responsive design. I'm wondering what other people in the Moz community think about this - and whether anyone out there has any experience to share about inaccessable content on responsive webpages, and the SEO impact of this. Thank you!

              Web Design | | mmewdell
              0
            • EcommerceSite

              What is the best tool to view your page as Googlebot?

              Our site was done with asp.net and a lot of scripting. I want to see what Google can see and what it can't. What is the best tool that duplicates Googlebot? I have found several but they seem old or inaccurate.

              Web Design | | EcommerceSite
              0
            • netbuilder

              Html 5 main and secondary navigation for SEO best performances

              I am building a website which will have a main navigation related to the site and each link of the main navigation will have a secondary navigation. We do not want to use a megamenu style navigation. I will try to explain it with a example: Let's start with an example for a computer store "My PC Store", the Main Navigation would be: Desktop PC's Notebook & Tablets
              Multimedia
              When clicking on the "Notebook & Tablets" the user is directed to the page domain.com/notebook-tablet.html and on this page the secondary navigation appears: Laptop Netbook Tablets / iPad I am confused on how I should organize the semantic navigation for best SEO performances and I need advice / suggestions. I thought about 2 different ways to do it but which one is more appropriate in terms of SEO? PROPOSITION A Home Page: <header> My PC Store <nav> Desktop PC's Notebook & Tablets Multimedia </nav> </header> Sub-Page (Notebook & Tablets): <nav>(or <aside>?) Desktop PC's Notebook & Tablets Multimedia </aside> </nav> <header> Notebook & Tablets <nav> Laptop Netbook Tablets / iPad </nav> </header> As you notice on the home page the Main Site Navigation is included in the <header>while it is not in the sub-pages. PROPOSITION B Home Page: <header> My PC Store <nav> Desktop PC's Notebook & Tablets Multimedia </nav> </header> Sub-Page (Notebook & Tablets): <header> Notebook & Tablets <nav> Desktop PC's Notebook & Tablets Multimedia </nav> # Notebook & Tablets * Laptop Netbook Tablets / iPad </header> The main navgation remains always in the <header>(home page / sub-pages) of all page. I need suggestions... How would you guys organize the nav ? </header> </header>

              Web Design | | netbuilder
              0
            • ChristineCadena

              Best Wordpress Hosting

              I've had a horrible experience with the security on wordpress hosting with GoDaddy.  Someone recommended Blue Host as my next option.   Does anyone have any experience with BlueHost and what other hosting companies would you recommend for wordpress hosting?

              Web Design | | ChristineCadena
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.