undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Pro
  4. Link Explorer
  5. Rogerbot and Dotbot blocked by WP Engine hosting

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Rogerbot and Dotbot blocked by WP Engine hosting

Link Explorer
3
4
2.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • wearehappymedia
    wearehappymedia last edited by Sep 30, 2014, 4:46 AM

    We recently discovered a problem with our MOZ account which led to checking our host's settings. Turns out that WP Engine (who we love to death) automatically BLOCK Rogerbot and Dotbot from crawling sites they host. What? Yeah exactly expect I was cursing a little too. So we went through the steps of getting that sorted one bot at a time but a week on and we're still getting 403 returns via OSE. This is more of a discussion than a question expect what I would like to see is a force crawl button so that we can test that we're no longer serving a 403 error from our host MOZ. I thought WP Engine were huge and I'm surprised this has not come up before but there's no info on it anywhere. Is there a way to force a mozbots crawl??

    1 Reply Last reply Reply Quote 0
    • wearehappymedia
      wearehappymedia last edited by Oct 5, 2014, 4:52 AM Oct 5, 2014, 4:52 AM

      Thanks guys I'll wait until the month cycles through and see if anything has changed.

      1 Reply Last reply Reply Quote 1
      • DavidLee
        DavidLee last edited by Oct 1, 2014, 1:15 PM Oct 1, 2014, 1:15 PM

        Hi Radi

        Just wanted to add that dotbot only crawls once a month while rogerbot crawls weekly. So you will not see any new metrics on OSE until we update our next Index.

        🙂

        1 Reply Last reply Reply Quote 2
        • John-Henry
          John-Henry last edited by Sep 30, 2014, 9:07 PM Sep 30, 2014, 9:07 PM

          Hi Radi,

          Thanks for posting your question, I'm John-Henry Scherck, Web Strategist for WP Engine.

          After some testing on our end, we are seeing Rogerbot getting status 200s when it/he crawls sites on WP Engine, I even tested it on my personal site that also runs on WP Engine and was able to crawl it successfully. However, that doesn't mean everything is resolved - we want you to get the most out of Moz and have a great experience using WP Engine 🙂

          So, the best thing to do is to submit a ticket through our user portal and we can look into the 403 issue for you.

          However, I can confirm that Dotbot has been blocked by WP Engine for some time. At least as far back as 2012, probably earlier. Normally bots get blocked for being resource intensive, but we can take an internal look at allowing Dotbot for users who need to use it. If you want to use Dotbot on your site, please fill out a ticket in the user portal and our support and tech ops teams will look into for you.

          Thanks for being a customer and please feel free to reach out to me in the future if you have any issues,

          John-Henry

          1 Reply Last reply Reply Quote 2
          • 1 / 1
          1 out of 4
          • First post
            1/4
            Last post

          Got a burning SEO question?

          Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


          Start my free trial


          Browse Questions

          Explore more categories

          • Moz Tools

            Chat with the community about the Moz tools.

          • SEO Tactics

            Discuss the SEO process with fellow marketers

          • Community

            Discuss industry events, jobs, and news!

          • Digital Marketing

            Chat about tactics outside of SEO

          • Research & Trends

            Dive into research and trends in the search industry.

          • Support

            Connect on product support and feature requests.

          • See all categories

          Related Questions

          • g3mmab

            Unsolved Huge Increase In Internal Links + Domain Authority Tanked

            internal linking domain authority

            Hey, I am very concerned about the link metrics I am seeing, really hoping someone can help. In October last year there were a total of 34.7K 'total links' attributed to my E-commerce website . This compared to 41.2k and 213k links attributed to my two main competitors. In November, my website had a forced update due to incompatibility issues my shopping cart version was having with the server on which it is hosted. (I was told by my web developers / hosting company that their server was upgraded.) This update created many problems such as checkout issues, page layout issues, sitemaps not automatically regenerating and the big one - it reverted back to http instead of https. Along with my developers, I spent 2 weeks fixing all of the issues until everything worked correctly and https was reinstated. After initial warnings from Moz, I have not received any critical crawler issues or warnings, big ranking drops etc since November. I was satisfied that all was ok. However, in November my 'total links' jumped to 81.5k and have since continued to balloon. December - 309.1k, January -335k, February - 665k, March - 716k, April - 719k, May - 742k, June - 751k. In the same period (November - June), my external links went from 5.3k - 8.7k. So it appears the huge increase in links are internal. In my latest report it states that 'internal follow links' are up by 8k and 'internal no follow links' are up by 1k. This is insane. I cannot understand why I have so many internal links and why they keep increasing. I maintain the website with updates, blog articles etc but no more than I have in the past several years. Worryingly, once this increase in links started being reported, my Domain Authority has also tanked. In October my DA was 37 (higher than my competitors) and has since dropped as low as 19. It has gone up to 21 this month (still significantly lower than my competitors). I have spent hours researching why this huge increase in internal links could be happening but have got nowhere. If anyone could advise what the issue could be and how I can fix it, I would be eternally grateful. Issues I have discovered along the way that may help with analysis:
            My site has many 404s (hundreds) - all relate to urls of old images that have been removed from the website. The 'Top Hits' on my website are from blogs (some spammy) to these deleted image urls that return a 404. I don't use any plugins. I do get 'duplicate content' notices as my blog posts archive themselves and generate new urls - as an example - 'blog post name' archived December 2023. It may be the case that the old http urls have been redirected to https but I have no warnings for 'redirect issues' in Moz. Thanks for taking the time to read this and thanks in advance for any help.

            Link Explorer | Aug 31, 2024, 2:17 AM | g3mmab
            0
          • Faizali.786

            Unsolved How to decrease spam score

            spam score link explorr

            Why spam score of my website is 78 in moz. I had disaow bad backlinks in 2023 and it's still showing my spam score too high. Help me to decrease it. alt text

            Link Explorer | Oct 22, 2024, 12:36 PM | Faizali.786
            0
          • Be7sjsj38sj

            Unsolved I have question related to Website DA.

            da

            Website DA is 4 but Ahref is showing DR 5. Why its different. Top Follow APK
            Top Follow
            Top follow for PC
            Top Follow Mod APK

            Link Explorer | Sep 8, 2024, 3:19 PM | Be7sjsj38sj
            0
          • pakistansites2

            How To Track Backlinks?

            My question is about competitor analysis.
            How can we track the outbound links of any website that is our competitors.

            Link Explorer | Aug 26, 2024, 6:20 PM | pakistansites2
            0
          • stay.hungary

            Website showing High Spam Score

            I am having a website https://www.hemporganic.net/ Suddenly I am noticing the website showing around 42% spam score. I simply cannot understand the reason behind it. The website contains only one backlink and that too it is guest posting backlink. Please explain to me why this is occurring with my site.

            Link Explorer | Jun 11, 2024, 6:46 PM | stay.hungary
            10
          • waqazolqa

            Why my domain authority is low

            hey moz community i want to ask something i have blog on https://bestbeginnerguitars.net/ guitars.this blog is 7 months old and i created backlink on it.but today i am checking DA of my website its 7 how can i increase it

            Link Explorer | Oct 5, 2024, 5:55 PM | waqazolqa
            1
          • seoprojecter

            Does using Sucuri block Moz?

            I am using Opensite explorer on my site which uses Sucuri and it shows 0 for everything. Does Sucuri stop MOZ from reading the link? I also suspect that using Sucuri has made my SEO suffer because the first page is always saying "redirecting" . Anyone with experience to this? Thanks

            Link Explorer | Mar 17, 2015, 2:31 AM | seoprojecter
            0
          • vivekrathore

            Why Open Site Explorer is Showing 0 Total Internal Links?

            Open Site Explorer is reporting "0" value under "Total Internal Links" as well as "Internal Followed Links" metrics for my website. I have a good interlinking of pages throughout the site.
            Is there an issue with OSE (Open Site Explorer) crawler ? OR There are some crawling problems with our Website's structure ?
            Can anybody please give an idea or recommendations on this ?

            Link Explorer | Apr 4, 2024, 11:24 AM | vivekrathore
            1

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy

          Looks like your connection to Moz was lost, please wait while we try to reconnect.