Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler is not able to crawl my website
-
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe -
@jasontorney Hi, I had the same problem after moving several of my websites to a Virtual Private Server that had enhanced security features.
One of these features was specifically to stop the Moz bot from crawling websites, and the hosting engineers advised they had done this because it was particularly aggressive in nature.
With my VPS control panel I have found the switch that allows me to disable bot blocking, and I occasionally do this if I'm grading a page with Moz, but advice from hosting support was to otherwise leave it active to protect the websites from attached (which means I don't get feedback from Moz crawls).
If you check with your hosting company you may find that they have a similar bot blocker configured for security purposes.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website is constantly decreasing
For few weeks ago my website is constantly decreasing in search position. I lost keywords and is gooooing down.
Technical SEO | | Dan_Tala
Although it is well rated on several on page and off page seo verification software that I have tried.
I checked Google search console and Analytics and found no major problems. However… from one day to another it keeps going down.
I also checked what the main competitors are doing and they are not doing well, at all.
The main competitor actually has a creepy website. Totally devoid of onpage or offpage SEO but with an enormous number of backlinks. And of a very bad quality, which should disqualify it, still…
Few weeks ago I changed something.
In the pages I had H1, 4xH2, no H3 and an H4 without content.
An unnatural H tag structure.
Now I have H1, H2, H3, 3xH4, with the coherent information.
Theoretically, Google should have been “happy” or I’m missing something. I use a SAAS platform.
I just found out that they made changes to the keywords (tags).
I am selling toner cartridges for printers.
So…
The tags are printer models and generate a url in which they have the products.
Ex. https://www.sertit.ro/cartus-imprimanta-cilindru-color-hp-laserjet-pro-m-177fw goes to the products for that printer model.
The question is… should I make tag canonical?
Is it possible for products to loose so much in Google search?0 -
My WP website got attack by malware & now my website site:www.example.ca shows about 43000 indexed page in google.
Hi All My wordpress website got attack by malware last week. It affected my index page in google badly. my typical site:example.ca shows about 130 indexed pages on google. Now it shows about 43000 indexed pages. I had my server company tech support scan my site and clean the malware yesterday. But it still shows the same number of indexed page on google.
Technical SEO | | ChophelDoes anybody had ever experience such situation and how did you fixed it. Looking for help. Thanks FILE HIT LIST:
{YARA}Spam_PHP_WPVCD_ContentInjection : /home/example/public_html/wp-includes/wp-tmp.php
{YARA}Backdoor_PHP_WPVCD_Deployer : /home/example/public_html/wp-includes/wp-vcd.php
{YARA}Backdoor_PHP_WPVCD_Deployer : /home/example/public_html/wp-content/themes/oceanwp.zip
{YARA}webshell_webshell_cnseay02_1 : /home/example2/public_html/content.php
{YARA}eval_post : /home/example2/public_html/wp-includes/63292236.php
{YARA}webshell_webshell_cnseay02_1 : /home/example3/public_html/content.php
{YARA}eval_post : /home/example4/public_html/wp-admin/28855846.php
{HEX}php.generic.malware.442 : /home/example5/public_html/wp-22.php
{HEX}php.generic.cav7.421 : /home/example5/public_html/SEUN.php
{HEX}php.generic.malware.442 : /home/example5/public_html/Webhook.php0 -
How to allow bots to crawl all but WP-content
Hello, I would like my website to remain crawlable to bots, but to block my wp content and media. Does the following robots.txt work? I worry that the * user agent may conflict with the others. User-agent: *
Technical SEO | | Tom3_15
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/ User-agent: GoogleBot
Allow: / User-agent: GoogleBot-Mobile
Allow: / User-agent: GoogleBot-Image
Allow: / User-agent: Bingbot
Allow: / User-agent: Slurp
Allow: /0 -
Migrating micro site into existing website
My company is planning to migrate an existing (ecommerce) micro site - which sits on its own domain - into their main ecommerce site. This means that the content will be moved from www.microdomain.co.uk to www.maindomain.com/category. Some products already exist on the main domain. The micro site is fairly small with just over 400 pages - I am planning to map each URL to the new URL (exact corresponding page) and create 301 redirects for each. Where any additional content does not exist yet on the existing main domain, we will create it and 301 redirect to it. The micro site currently ranks fairly well for some keywords - being such a specialised micro site, (some of) the keywords also form part of the domain name, however, they won't on the main page although they may form part of the URL (category). As an example (using a made up URL), our micro site www.bread-sticks.co.uk ranks on page 1 for the keyword bread sticks - we don't just sell bread sticks on www.bread-sticks.co.uk but also rolls and bread though, bread sticks is one category of very closely related categories. Say our main domain is www.supermarket.co.uk (selling a wide range of food / drink products. The micro site will be moving to www.supermarket.co.uk/baked-products/ - which is a category. Within that category, there are sub categories, i.e. bread sticks, rolls and bread which will sit under www.supermarket.co.uk/bread-sticks/ etc. What would be the best way for ensuring that our main domain would take over the rankings from our micro site, given that it will be sitting on our main domain as a category (one of many)? Can we expect www.supermarket.co.uk/baked-products/ or www.supermarket.co.uk/bread-sticks/ to replace www.bread-sticks.co.uk in the rankings simply by 301 redirecting? Thanks for your help!
Technical SEO | | ViviCa10 -
How to SEO a Website Built off Godaddy?
I have a client whose website is built off Godaddy services. I know Godaddy is not the right choice for building a website, but what's done is done. The client has already bought the Godaddy services and there's no way I can tell him to go rebuild his website before we could optimize it for SEO. I'm already facing a lot of challenges while optimizing on-page elements. When I wanted to verify the ownership for Google Analytics and Webmaster Tool via his Godaddy account. the process failed many times. it looks like Godaddy is using some kind of caching not allowing us to modify the codes. For example, I'd applied the site verification codes for Webmasters Tool 48 hours ago, and the metatag for google site verification is not yet updated in the frontend. It's quite frustrating. What would you suggest?
Technical SEO | | suskanchan1 -
Why is my website banned?
IMy website is Costume Machine at www.costumemachine.com . My site has been banned for 1 year now. I have requested that google reconsider my site 3 times without luck. The site is dynamic and basically pulls in feeds from affiliate sites. We have added over 1,500 pages of original content. The site has been running great since 2008 without any penalties. I don't think I got hit with any linking penalty. I cleaned up all questionable links last November when the penalty hit. Am I being hit with a "thin" site penalty? If that is the issue what is the best way to fix the problem?
Technical SEO | | tadden0 -
What is the best website structure for SEO?
I've been on SEOmoz for about 1 month now and everyone says that depending on the type of business you should build up your website structure for SEO as 1st step. I have a new client click here ( www version doesn't work)... some bugs we are fixing it now. We are almost finished with the design & layout. 2nd question have been running though my head. 1. What would the best url category for the shop be /products/ - current url cat ex: /products/door-handles.html 2. What would you use for the main menu as section for getting the most out of SEO. Personally i am thinking of making 2-3 main categories on the left a section where i can add content to it (3-4 paragraphs... images maybe a video).So the main page focuses on the domain name more and the rest of the sections would focus on specific keywords, this why I avoid cannibalization. Main keyword target is "door handles" Any suggestions would be appreciated.
Technical SEO | | mosaicpro0
