undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • MozCon
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Digital Marketers
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    Track AI Overviews in Keyword Research
    Moz Pro

    Track AI Overviews in Keyword Research

    Try it free!
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Digital Marketers

      Simplify SEO tasks to save time and grow your traffic.

    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Digital Marketing
  3. Web Design
  4. Bing Indexation and handling of X-ROBOTS tag or AngularJS

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Bing Indexation and handling of X-ROBOTS tag or AngularJS

Web Design
2
7
1.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • AU-SEO
    AU-SEO last edited by Dec 16, 2015, 3:24 AM

    Hi MozCommunity,

    I have been tearing my hair out trying to figure out why BING wont index a test site we're running.

    We're in the midst of upgrading one of our sites from archaic technology and infrastructure to a fully responsive version.
    This new site is a fully AngularJS driven site. There's currently over 2 million pages and as we're developing the new site in the backend, we would like to test out the tech with Google and Bing.

    We're looking at a pre-render option to be able to create static HTML snapshots of the pages that we care about the most and will be available on the sitemap.xml.gz

    However, with 3 completely static HTML control pages established, where we had a page with no robots metatag on the page, one with the robots NOINDEX metatag in the head section and one with a dynamic header (X-ROBOTS meta) on a third page with the NOINDEX directive as well. We expected the one without the meta tag to at least get indexed along with the homepage of the test site.

    In addition to those 3 control pages, we had 3 pages where we had an internal search results page with the dynamic NOINDEX header. A listing page with no such header and the homepage with no such header.

    With Google, the correct indexation occured with only 3 pages being indexed, being the homepage, the listing page and the control page without the metatag.

    However, with BING, there's nothing. No page indexed at all. Not even the flat static HTML page without any robots directive.

    I have a valid sitemap.xml file and a robots.txt directive open to all engines across all pages yet, nothing.

    I used the fetch as Bingbot tool, the SEO analyzer Tool and the Preview Page Tool within Bing Webmaster Tools, and they all show a preview of the requested pages. Including the ones with the dynamic header asking it not to index those pages.

    I'm stumped. I don't know what to do next to understand if BING can accurately process dynamic headers or AngularJS content.

    Upon checking BWT, there's definitely been crawl activity since it marked against the XML sitemap as successful and put a 4 next to the number of crawled pages.

    Still no result when running a site: command though. Google responded perfectly and understood exactly which pages to index and crawl.

    Anyone else used dynamic headers or AngularJS that might be able to chime in perhaps with running similar tests?

    Thanks in advance for your assistance....

    1 Reply Last reply Reply Quote 0
    • Everett
      Everett @AU-SEO last edited by Feb 19, 2016, 2:10 PM Feb 19, 2016, 2:10 PM

      Thank you for the update Kavit.

      1 Reply Last reply Reply Quote 0
      • AU-SEO
        AU-SEO @Everett last edited by Feb 18, 2016, 11:05 PM Feb 18, 2016, 11:05 PM

        Hi Everett and Fellow Mozzers,

        I have been away overseas so wasn't able to put up an update.

        Eventually, managed to get a hold of someone at BING within the tech team who told me that the reason that they didn't index the pages was simply because of popularity.

        It isn't enough to have unique content, design and structure on your site, it is also vital to have traffic, links and mentions as external signals.

        We also got word that dynamic sites and pre-render content will be acceptable for BING so we're resting easier at night these days.

        Development on the site continues as per schedule and we will be launching the proper site this year on a highly authoritative domain which should yield very different results to the test we put together.

        Hopefully, this will help someone else who is on a similar pathway.

        Everett, I would like to thank you again for taking the time to read, reply and help us with our analysis.

        Thanks!

        Everett 1 Reply Last reply Feb 19, 2016, 2:10 PM Reply Quote 0
        • topic:timeago_earlier,about a month
        • AU-SEO
          AU-SEO @Everett last edited by Jan 18, 2016, 4:00 AM Jan 18, 2016, 4:00 AM

          Hi Everett,

          Thank you for the analysis and deeper insights.

          I did make the changes to the test pages bar the design template.

          We added the unique titles, meta descriptions and meta keywords.

          We added completely unique content to all three pages with no other instances of this content appearing on the web at all.

          The pages are now also interlinked and also linked from the top of the homepage so none of them are orphan pages.

          sitemaps have been updated and resubmitted.

          The latest version has been out a week so far, but no response from BING as yet.

          Thanks,

          Kavit.

          1 Reply Last reply Reply Quote 0
          • topic:timeago_earlier,26 days
          • Everett
            Everett last edited by Jan 7, 2016, 2:02 AM Dec 23, 2015, 1:55 PM

            Hello Kavit,

            I would suggest putting unique Title tags, meta descriptions and content on those pages. They are very thin as it is, and all of the content is boilerplate.

            There are 57,100,000 results on Bing for: "Search for an Australian Business, Government Department or Person" which is the content on the home page you shared.

            There are 60,600 results on Bing for: ""There was a table set out under a tree in front of the house, and the March Hare and the Hatter were having tea at it" which is the content on this page: http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html .

            And so on. I can see why Bing wouldn't want to add yet another thin, duplicate, orphan page to their index. My advice would be to build out those test pages with a design template and to put original content, title tags and meta descriptions on all of them. Then repeat your test.

            AU-SEO 2 Replies Last reply Feb 18, 2016, 11:05 PM Reply Quote 1
            • AU-SEO
              AU-SEO last edited by Dec 23, 2015, 1:13 AM Dec 23, 2015, 1:13 AM

              Hi Everett,

              Thank you for taking the time out to read and respond.

              The URL we have setup for testing is: wp-seospike-weblbl.naws-sensis.com.au

              We have 3 control pages (all flat HTML pages) that we setup and put online for bing to crawl:

              http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/no-metatag.html - no robots metatag and allowed to crawl and index.

              http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag.html - page with a noindex metatag not to be crawled and indexed

              http://wp-seospike-weblbl.naws-sensis.com.au/bing-seo-control/metatag-header.html - X-Robots meta tag NOINDEX

              http://wp-seospike-weblbl.naws-sensis.com.au - homepage with no robots exclusion

              Ideally, I expected the homepage and the no-metatag page to be indexed at least.

              I am familiar with the builtvisible documentation that they've put out.

              My main pain point is that even the flat HTML pages are getting ignored, so I can't even test the deeper AngularJS developed pages since my control group is not delivering results as it should.

              a site command on the above domain on bing shows no results.

              Thanks again!

              1 Reply Last reply Reply Quote 0
              • Everett
                Everett last edited by Dec 22, 2015, 5:05 PM Dec 22, 2015, 5:05 PM

                Is there any chance of getting a URL for the domain in question?

                Have you read this yet?
                https://builtvisible.com/javascript-framework-seo/

                What are the URLs like that you're asking Bing to index? Which is closest?

                Hashbang
                http://www.IWishJSFramworkWebsitesWouldGoAway/#!

                Escaped Fragment
                http://www.IWishJSFramworkWebsitesWouldGoAway/?escaped_fragment=

                Base URL using Angular's $location service to construct URLs without the #! via the HTML5 History API http://www.IWishJSFramworkWebsitesWouldGoAway/

                I know this doesn't answer your question, but hopefully it will get the discussion started.

                1 Reply Last reply Reply Quote 0
                • 1 / 1
                1 out of 7
                • First post
                  1/7
                  Last post

                Got a burning SEO question?

                Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                Start my free trial


                Browse Questions

                Explore more categories

                • Moz Tools

                  Chat with the community about the Moz tools.

                • SEO Tactics

                  Discuss the SEO process with fellow marketers

                • Community

                  Discuss industry events, jobs, and news!

                • Digital Marketing

                  Chat about tactics outside of SEO

                • Research & Trends

                  Dive into research and trends in the search industry.

                • Support

                  Connect on product support and feature requests.

                • See all categories

                Related Questions

                • Joe_Stoffel

                  Have Your Thoughts Changed Regarding Canonical Tag Best Practice for Pagination? - Google Ignoring rel= Next/Prev Tagging

                  Hi there, We have a good-sized eCommerce client that is gearing up for a relaunch. At this point, the staging site follows the previous best practice for pagination (self-referencing canonical tags on each page; rel=next & prev tags referencing the last and next page within the category). Knowing that Google does not support rel=next/prev tags, does that change your thoughts for how to set up canonical tags within a paginated product category? We have some categories that have 500-600 products so creating and canonicalizing to a 'view all' page is not ideal for us. That leaves us with the following options (feel it is worth noting that we are leaving rel=next / prev tags in place): Leave canonical tags as-is, page 2 of the product category will have a canonical tag referencing ?page=2 URL Reference Page 1 of product category on all pages within the category series, page 2 of product category would have canonical tag referencing page 1 (/category/) - this is admittedly what I am leaning toward. Any and all thoughts are appreciated! If this were in relation to an existing website that is not experiencing indexing issues, I wouldn't worry about these. Given we are launching a new site, now is the time to make such a change. Thank you! Joe

                  Web Design | Feb 7, 2023, 8:46 PM | Joe_Stoffel
                  1
                • Brando16

                  What is the best way to handle annual events on a website?

                  Every year our company has a user conference with between 300 - 400 attendees. I've just begun giving the event more of a presence on our website. I'm wondering, what is the best way to handle highlights from previous years? Would it be to create an archive (e.g. www.companyname.com/eventname/2015) while constantly updating the main landing page to promote the current event? We also use an event website (cvent) to handle our registrations. So once we have an agenda for the current years event I do a temporary redirect from the main landing page to the registration website. I don't really like this practice and I feel like it might be better to keep all of the info on the main domain. Wondering if anybody has any opinions or feedback on that process as well. Just looking for best practices or what others have done and have had success with.

                  Web Design | Jul 19, 2016, 10:21 AM | Brando16
                  1
                • SEOguy1

                  Https pages indexed but all web pages are http - please can you offer some help?

                  Dear Moz Community, Please could you see what you think and offer some definite steps or advice.. I contacted the host provider and his initial thought was that WordPress was causing the https problem ?: eg when an https version of a page is called, things like videos and media don't always show up. A SSL certificate that is attached to a website, can allow pages to load over https. The host said that there is no active configured SSL it's just waiting as part of the hosting package just in case, but I found that the SSL certificate is still showing up during a crawl.It's important to eliminate the https problem before external backlinks link to any of the unwanted https pages that are currently indexed. Luckily I haven't started any intense backlinking work yet, and any links I have posted in search land have all been http version.I checked a few more url's to see if it’s necessary to create a permanent redirect from https to http. For example, I tried requesting domain.co.uk using the https:// and the https:// page loaded instead of redirecting automatically to http prefix version.  I know that if I am automatically redirected to the http:// version of the page, then that is the way it should be. Search engines and visitors will stay on the http version of the site and not get lost anywhere in https. This also helps to eliminate duplicate content and to preserve link juice. What are your thoughts regarding that?As I understand it, most server configurations should redirect by default when https isn’t configured, and from my experience I’ve seen cases where pages requested via https return the default server page, a 404 error, or duplicate content. So I'm confused as to where to take this.One suggestion would be to disable all https since there is no need to have any traces to SSL when the site is even crawled ?. I don't want to enable https in the htaccess only to then create a https to http rewrite rule; https shouldn't even be a crawlable function of the site at all.RewriteEngine OnRewriteCond %{HTTPS} offor to disable the SSL completely for now until it becomes a necessity for the website.I would really welcome your thoughts as I'm really stuck as to what to do for the best, short term and long term.Kind Regards

                  Web Design | Feb 25, 2016, 1:31 PM | SEOguy1
                  0
                • Kingalan1

                  Lots of Listing Pages with Thin Content on Real Estate Web Site-Best to Set them to No-Index?

                  Greetings Moz Community: As a commercial real estate broker in Manhattan I run a web site with over 600 pages. Basically the pages are organized in the following categories: 1. Neighborhoods (Example:http://www.nyc-officespace-leader.com/neighborhoods/midtown-manhattan)  25 PAGES Low bounce rate 2. Types of Space (Example:http://www.nyc-officespace-leader.com/commercial-space/loft-space) 
                  15 PAGES Low bounce rate. 3. Blog (Example:http://www.nyc-officespace-leader.com/blog/how-long-does-leasing-process-take 
                  30 PAGES Medium/high bounce rate 4. Services (Example:http://www.nyc-officespace-leader.com/brokerage-services/relocate-to-new-office-space)  High bounce rate
                  3 PAGES 5. About Us (Example:http://www.nyc-officespace-leader.com/about-us/what-we-do
                  4 PAGES High bounce rate 6. Listings (Example:http://www.nyc-officespace-leader.com/listings/305-fifth-avenue-office-suite-1340sf)
                  300 PAGES High bounce rate (65%), thin content 7. Buildings (Example:http://www.nyc-officespace-leader.com/928-broadway
                  300 PAGES  Very high bounce rate (exceeding 75%) Most of the listing pages do not have more than 100 words.  My SEO firm is advising me to set them "No-Index, Follow". They believe the thin content could be hurting me. Is this an acceptable strategy? I am concerned that when Google detects 300 pages set to "No-Follow" they could interpret this as the site seeking to hide something and penalize us. Also, the building pages have a low click thru rate. Would it make sense to set them to "No-Follow" as well? Basically, would it increase authority in Google's eyes if we set pages that have thin content and/or low click thru rates to "No-Follow"? Any harm in doing this for about half the pages on the site? I might add that while I don't suffer from any manual penalty volume has gone down substantially in the last month. We upgraded the site in early June and somehow 175 pages were submitted to Google  that should not have been indexed. A removal request has been made for those pages. Prior to that we were hit by Panda in April 2012 with search volume dropping from about 7,000 per month to 3,000 per month. Volume had increased back to 4,500 by April this year only to start tanking again. It was down to 3,600 in June. About 30 toxic links were removed in late April and a disavow file was submitted with Google in late April for removal of links from 80 toxic domains. Thanks in advance for your responses!! Alan

                  Web Design | Jul 3, 2014, 5:09 PM | Kingalan1
                  0
                • NoisyLittleMonkey

                  Is it bad to have /index.php at the end of a uri?

                  Is it bad for SEO if traffic is directed to "http://www.example.com/someuri/index.php" instead of "http://www.example.com/someuri/" and would it be works setting up a redirect rule at htaccess level?

                  Web Design | Oct 4, 2013, 10:54 AM | NoisyLittleMonkey
                  1
                • EscaladeSports

                  Other tags inside an H1 tag

                  So I have a situation with the website I'm currently redesigning where the H1 titles are supposed to mix colors per the current brand strategy. The branding crew is adamant that this has to be done so there is no use in saying "just don't do it". To accomplish this I'm wrapping the words that need to be the other color in a . Additionally, some pages have a "sub text" as part of the title, floated to the right and in a smaller font but with the same multi color treatment. I'm wondering if the sub text should be in an H2 and positioned to the right or if it would be beneficial to have the text in the H1 as well. An example of what I'm talking about would be something like this: "Big Shoes for Big Guys - Nike Shoes" In that, the "Big Shoes" and "Nike" would be one color and the "for Big Guys" and "Shoes" would be another. I can imagine having the "Nike Shoes" as part of the H1 would be a good idea in some respect but I'm not certain of that. In order to make that happen I can only think of one way to do it: -H1-
                  Big Shoes
                  -span- for Big Guys -/span-
                  -div- Nike
                  -span- Shoes -/span-
                  -/div-
                  -/H1- So that brings me back to the original concern, do search engines care about tags inside the H1? The only other way to accomplish the color changes that I can think of would be to have a fairly large chunk of javascript setup to go through H1's to colorize them using the span tags. That is unless GoogleBot has started to execute javascript while crawling the sites now...

                  Web Design | Jun 1, 2013, 8:48 AM | EscaladeSports
                  1
                • Jvalops

                  Google also indexed trailing slash version - PLEASE HELP

                  Hi Guys, We redesigned the website and somehow our canonical extension decided to add a trailing slash to all URLs. Previously our canonical URLs didn't have a trailing slash. During the redesign we haven't changed the URLs. They remained same but we have now two versions indexed. One with trailing slash one without. I've now fixed the issue and removed the the trailing slash from canonical URLs. Is this the correct way of fixing it? Will our rankings be effected in a negative way? Is there anything else I need to do. The website went live last Tuesday. Thanks

                  Web Design | Jul 21, 2012, 9:28 AM | Jvalops
                  0
                • superTallDan

                  Duplicate Content for index.html

                  In the Crawl Diagnostics Summary, it says that I have two pages with duplicate content which are: www.mywebsite.com/ www.mywebsite.com/index.html I read in a Dream Weaver tutorial that you should name your home page "index.html" and then you can let www.mywebsite.com automatically direct the user to index.html.  Is this a bug in SEOMoz's crawler or is it a real problem with my site? Thank you, Dan

                  Web Design | Jul 1, 2011, 7:41 PM | superTallDan
                  0

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                • Digital Marketers
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy

                Looks like your connection to Moz was lost, please wait while we try to reconnect.