Crawl diagnostic issue?
-
I'am sorry if my English isn't very good, but this is my problem at the moment:
On two of my campagnes I get a weird error on Moz Analytics:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
Moz Analytics points to an url that starts with: http:/**/None/**www.????.com. We don't understand how Moz indexed this non-existing page that starts with None? And how can we solve this error?
I hope that someone can help me.
-
Hi MOZ,
I'am sorry that I have not previously responded. The problem has been solved. Thanks!
Also thanks to Pixel for the response!
Greetz,
Sam
-
Hi Nettt!
I apologize for any confusion and can confirm there is no issue on your side. One of our crawlers failed causing some campaigns crawled on Aug 29th attempt to follow the strange /None/ URL you are seeing in your diagnostics. I've submitted a re-crawl for all of your campaigns affected so you should see updated data by this Friday.
Hope this helps!
-
"I have checked the URL, and it is not our own website that has the error."
is this the problem?
Could you take a screen grab of the problem it might help better.
-
Thanks for the respons, Pixelbypixel!
I have checked the URL, and it is not our own website that has the error.
We have checked the robots.txt and it should not cause any problem. We have n't recently changed it.
I Think that Moz is causing it, but I am not sure..
-
Is the URL correct on Moz pro? It also seems like your robots.txt is blocking Moz which you may want to look into.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will google be able to crawl all of the pages given that the pages displayed or the info on a page varies according to the city of a user?
So the website I am working for asks for a location before displaying the product pages. There are two cities with multiple warehouses. Based on the users' location, the product pages available in the warehouse serving only in that area are shown. If the user skips location, default warehouse-related product pages are shown. The APIs are all location-based.
Intermediate & Advanced SEO | | Airlift0 -
Main Nav Redirects Issue: Unnecessary Evil or Better Solution
Hi, I'm somewhat stumped on the best course of action for our navigation menu. Our site is "divided" into two areas; informational and transactional. Because of compliance, transnational users have to be geo targeted; therefore, we serve them a specific page (we have 6 different regions: uk, aus, eu, etc). If users visit informational side, it's not geo specific. Example: https://site/charts https://site/uk/money Within our main nav, we don't specify the geo transaction page and use a generic https://site/money/ (page doesn't exist) then when a user clicks that link, we'll detect their location and serve up a 301 redirect to the correct geo page. This has obviously caused a ton load of unnecessary redirects and a waste of powerful link equity from the header of the site. It's been recommended to dynamically change the linked URL in this header based on the location of the user. That sounds good but what about Google? Since we can't detect Google crawler IP, we would have to pick a default geo URL like /uk/money. If we do that, the other regional URLs suffer link equity. How do we minimize redirects and make Google happy for all our geo pages. Hope this makes sense and thanks for your time!
Intermediate & Advanced SEO | | Bragg0 -
Search Console Change of Address Tool Issue
We're currently migrating few "event" mini sites to a main site that will have a subfolder for each event.. Example:
Intermediate & Advanced SEO | | RichardUK
newsite.com/event1 The issue is that Search console is not able to verify this kind of redirect:
example.com --> 301 --> newsite.com/event Do you know any work around for this? I was thinking of using a subdomain instead which will in turn redirect to the /event subfolder.. but with each hop it will diminish the link's strength. I prefer not to leave it as subdomain as data gets mashed up in Google Analytics with subdomains and we have seen worse ranking results with subdomains. Any help is greatly appreciated.0 -
University website outbound links issue
Hi - I'm working on a university website and have found a load of (1) outbound links to companies that have commercial tie ups to the university and, beyond that, loads of (2) outbound links to companies set up by alumni and (3) outbound links to commercial clients of the university. Your opinions on whether I should nofollow these, or not, would be welcome. At the moment I'm tempted to nofollow (1) yet leave (2) and (3) - quite simply because the (1) backlinks may have been negotiated as part of a package (nobody can actually remember at the university!), yet (2) and (3) were freely given by the university. Your thoughts would be welcome!
Intermediate & Advanced SEO | | McTaggart0 -
Google can't access/crawl my site!
Hi I'm dealing with this problem for a few days. In fact i didn't realize it was this serious until today when i saw most of my site "de-indexed" and losing most of the rankings. [URL Errors: 1st photo] 8/21/14 there were only 42 errors but in 8/22/14 this number went to 272 and it just keeps going up. The site i'm talking about is gazetaexpress.com (media news, custom cms) with lot's of pages. After i did some research i came to the conclusion that the problem is to the firewall, who might have blocked google bots from accessing the site. But the server administrator is saying that this isn't true and no google bots have been blocked. Also when i go to WMT, and try to Fetch as Google the site, this is what i get: [Fetch as Google: 2nd photo] From more than 60 tries, 2-3 times it showed Complete (and this only to homepage, never to articles). What can be the problem? Can i get Google to crawl properly my site and is there a chance that i will lose my previous rankings? Thanks a lot
Intermediate & Advanced SEO | | granitgash
Granit FvhvDVR.png dKx3m1O.png0 -
Redirect issue launching duplicate product categories on another TLD
Dear Mozzerz We run this e-commerce website (superstar.dk) where we are selling all different kinds of wristwatches from different brand names (Casio, Garmin, Suunto etc). We just bought another website selling watches (xxx.com) and therefore we would like to move some of the content from superstar.dk to the new website xxx.com, making superstar.dk into a more niche website. So we are basically taking a brand with all the products in it and shutting it down on superstar.dk and instead launching it on xxx.com. Superstar.dk will still be running, just with a more niche product- and brand selection. So my question is, should we redirect all the old product categories that we are shutting down to the new website on another TLD where we are opening them again and the same for the products (e.g. superstar.dk/garmin -> xxx.com/garmin)? Or would it be better to keep the redirects within the same website/TLD (e.g. superstar.dk/garmin -> superstar.dk)? A few examples:
Intermediate & Advanced SEO | | superstardenmark
superstar.dk/garmin -> xxx.com/garmin
superstar.dk/suunto -> xxx.com/suunto
etc..
superstar.dk/product1 -> xxx.com/product1
superstar.dk/product2 -> xxx.com/product2
etc.0 -
Dropped ranking - Penguin penalty or duplicate content issue?
Just this weekend a page that had been ranking well for a competitive term fell completely out of the rankings. There are two possible causes and I'm trying to figure out which it is, so I can take action. I found out that I had accidentally put a canonical on another page that was for the same page as the one that dropped out of the rankings. If there are two pages with the same canonical tag with different content, will google drop both of them from the index? The other possibility is that this is a result of the recent Penguin update. The page that dropped has a high amount of exact anchor text. As far as I can tell, there were no other pages with any penalties from the Penguin update. One last question: The page completely dropped from the search index. If this were a Penguin issue, would it have dropped out completely,or just been penalized with a drop in position? If this is a result of the conflicting canonical tags, should I just wait for it to reindex, or should I request a reconsideration of the page?
Intermediate & Advanced SEO | | gametv0 -
Are links to on-page content crawled / have any effect on page rank?
Lets say I have a really long article that begins with links to <a name="something">anchors on the same page.</a> <a name="something"></a> <a name="something">E.g.,</a> Chapter 1, Chapter 2, etc, allowing the user to scroll down to different content. There are also other links on this page that link to other pages. A few questions: Googlebot arrives on the page. Does it crawl links that point to anchors on the same page? When link juice is divided among all the links on the page, do these links count and page rank is then lost? Thanks!
Intermediate & Advanced SEO | | anthematic0