On Page Grader can't access my URLs
-
HI-
I am trying to grade some specific pages for keywords with the on page grader but it keeps telling me "Sorry, but that URL is inaccessible. "
I can reach them via the browser and they are not https. Any thoughts?
Here is a sample:
www.bulkcandystore.com/kosher-candy
Any help is appreciated.
Ken
-
Hey Ken,
Thanks for following up. You can add a crawl delay for rogerbot in your robots.txt file to limit how quickly we make requests to the site. I would just recommend that you don't add a crawl delay higher than 10 because that can cause us to be unable to crawl the site in a reasonable amount of time to complete a full crawl.
I hope that helps!
Chiaryn
-
Thanks for the info. I checked with my ISP and it turns out that they automatically blocked Rogerbot because of excessive requests (this happened Jan 15). Is there a way to crawl dely between requests longer or was that maybe a one time fluke?
Thanks
Ken
-
Hey Ken,
Thanks for writing in and sorry for the confusion. I'm afraid the server for www.bulkcandystore.com/kosher-candy is returning a 403 forbidden error for our crawler, so we are not allowed to access that page with our tools. The server would respond differently to the browser vs. our tools so I would recommend contacting the webmaster for the site and white-labelling the user-agent rogerbot so that we can access the site in the future.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Https address has different result that http in Page Optimization Score toll in Moz PRO
The following url
Moz Bar | | TrueluxGroup
https://www.whichledlight.com/t/gu10-led-bulbs has (100 score for keyword 'GU10 LED')
has different on page opmisation score results to
http://www.whichledlight.com/t/gu10-led-bulbs (73 score for keyword 'GU10 LED') Anyone know if we've set something up wrong?
Also, is this even something to worry about, does google treat them differently? We're using the Page Optimisation Tool in Moz Pro ** UPDATE ** It's worth mentioning we are using emberjs, so the website is a single page application.
We use prerender to render the pages for google.0 -
Moz Bar doesn't Show Anything
Hi, I use the MozBar chrome extension, and it has worked fine in the past. But lately it doesn't show any metrics. Am I missing an update or something? Thanks.
Moz Bar | | TMI.com0 -
How does a non-traditional TLD impact Moz's crawl test?
I have a client who moved from a .com to .academy domain 6 months ago, and their current crawl tests are coming back with a universal page authority of 1, along with 0 indexed backlinks. The previous version of the site had an average page authority of 35-40, the site architecture and content are nearly identical, and there are no other errors or red flags in the crawl report that would hold back their organic rankings. In fact, looking at the site's analytics account, I can see dozens of sites that provide current and properly functioning backlinks, non of which are listed on the crawl test. So the question is - is Moz currently unable to properly crawl a .academy (or any other non-traditional TLD) site, or is there some deeper issue with the site's SEO that I'm not seeing? Thanks!
Moz Bar | | ThinkAOR1 -
Does anyone else have issues with Moz's keyword search volume tool for Google's search engine?
It will show the search volume for Bing even when Google is selected. Then, if you select Bing, you'll get the same data as it shows for when you selected "google". So basically, this tool does not work for Google's search engine. Or it is most likely not a reliable way to perform keyword research. Anyone else notice this? Does Moz even offer a way to submit a support ticket to get this fixed?
Moz Bar | | ShokIdeaGroup1 -
Keyword Grader & OmniExplorer_Bot Question
I found this in several of my clients Robots.txt "User-Agent: OmniExplorer_Bot Disallow: /" Right now I cannot grade any of them that are on a particular platform that has that written in. Would this block the keyword grader? Thanks!
Moz Bar | | OOMDODigital0 -
Optimise your Pages in Moz Crawl - where do the keywords come from?
I am just going through my first Crawl stats from the MOZ analytics Under the pages to optimise section I have pages that I have optimised for my best keyword with an A grade that are showing as an F grade and suggesting a different keyword? Where is this keyword coming from? I am assuming that my page has been analysed and a better keyword has been recommended? Can anyone advise? Thanks Roger
Moz Bar | | rnperki0 -
Moz Rank Tracker doesn't work with "PHRASE" Keywords!?
Hello, If at http://ranktracker.moz.com trying to track phrase KW - the system wont accept it. But adding [EXACT] match with [] - works well. Have I missed something? Cheers.
Moz Bar | | SEOisSEO0