Robots.txt file issues on Shopify server
-
We have repeated issues with one of our ecommerce sites not being crawled. We receive the following message:
Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
Are you aware of an issue with robots.txt on the Shopify servers? It is happening at least twice a month so it is quite an issue.
-
I leave you 2 url, but it seems that the robots do not let you modify it.
They propose you to put pages as non-index....
https://community.shopify.com/c/Shopify-Discussion/How-can-I-edit-robots-txt/td-p/432485/page/4
https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines
greetings
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
Meta Robots query
Hi guys, I was ranking really well on my home page for certain keywords which has all dropped pretty dramatically over the last 3/4 weeks - I think the issue is since since the configuration of Yoast SEO Wordpress plugin. In March (when my rankings were strong) my crawl test showed the top data in the attached image, and in May (now the rankings have dropped severly) they show the bottom data. I don't fully understand canonical and Meta Robots so I am hoping someone can shed some light on the following points. 1. Will the change result in my loss of rankings.
Moz Pro | | RocketStats
2. How can I put it back to how it was in March? PS. I haven't had any Google penalties. Thanks,
Joshua RfTar0 -
Issue: Title Element Too Long !
hello , i found this issue in seomoz campaigns. i see many links blocked by google for this reason . but something wrong here for example i see this link like that in seomoz dashboard Wagdy Hassan, Author at Seo Seek : Seo Tutorials , Seo Tools , Make Money Online http://seo-seek.com/author/wagdys/ and its already not in google. but when i open the link http://seo-seek.com/author/wagdys/ the title is " Wagdy Hassan " i fixed it 6 days ago. what is wrong with the site? also i still waiting google to put the new results .. wait for your answer, Thanks 🙂
Moz Pro | | Wagdys0 -
Meta-Robots noFollow and Blocked by Meta-Robots
On my most recent campaign report, I have 2 Notices that we can't find any cause for: Meta-Robots nofollow-
Moz Pro | | gfiedel
http://www.fateyes.com/the-effect-of-social-media-on-the-serps-social-signals-seo/?replytocom=92
"noindex nofollow" for the page: http://www.fateyes.com/the-effect-of-social-media-on-the-serps-social-signals-seo/ Blocked by Meta-Robots -Meta-Robots nofollow-
http://www.fateyes.com/the-effect-of-social-media-on-the-serps-social-signals-seo/?replytocom=92
"noindex nofollow" for the page: http://www.fateyes.com/the-effect-of-social-media-on-the-serps-social-signals-seo/ We are unable to locate any code whatsoever that may explain this. Any ideas anyone?0 -
Keyword rankings are not updating from past two weeks? Is there any issue in SEOmoz tool ?
I have added 215 keywords to my account. From past two weeks i am waiting to see the rankings, however it's not updating.
Moz Pro | | Sasank0 -
My account recently got deactivated due to billing issues. I reactivated my account, but all of my campaigns are gone. Is there a way I can get back my campaigns without having to redo all of them??
My account recently got deactivated due to billing issues. I reactivated my account, but all of my campaigns are gone. Is there a way I can get back my campaigns without having to redo all of them??
Moz Pro | | DorianDDR10 -
Are header directives such as X-Robots and Link supported by OSE/Linkscape/SEOMoz tools?
SEOMoz tool reports show lots of duplicate content where there are http header directives in place on pages to eliminate dupes. Googlebot obeys but Roger the robot doesn't. Are header directives such as X-Robots and Link (rel=canonical) supported by OSE/Linkscape? I'd like to put my mind and clients at ease. Thanks
Moz Pro | | Mediatorr0 -
Where do these URL's come from?! (Indexation issues)
We have an international webshop with languages in the URLs. Our URLs are now set up as follows: http://thermalunderwear.eu/eng/category/product Now, we know that there's some kind of strange redirect problem causing problems with our indexation, this is a technical issue that should be fixed soon. But whether this is the cause of some other strange problems, I do not know. I'd be happy with any help/advice/tips. 1. The SEOmoz site crawler starts at http://thermalunderwear.eu. This currently does not yet redirect to http://thermalunderwear.eu/eng like we want it to, but all the links on the page do include the default language code. So all links on the page are http://thermalunderwear.eu/eng/category etc. However, apart from those URLs, the site crawler finds many URLs in the form http://thermalunderwear.eu/category/product etc., so not including the language variable. Where it gets these I do not know, and since these URLs dont exist and the webshop simply shows the homepage, these URLs all have 50+ duplicate titles/content. Why oh why? 2. If I do a Google search for indexed URL's with English as language, I get many results formatted like this: Coldpruf Enthusiast mens thermal shirt - Thermal wear for men ...
Moz Pro | | DocdataCommerce
thermalunderwear.eu/eng/men/coldpruf-enthusiast-mens-thermal-shirt 170+ items – Fine-ribbed longsleeve thermal shirt men from Enthusiast ... {$SCRIPT_NAME} eng/men/coldpruf-enthusiast-mens-the {$ajax_url} http://thermalunderwear.eu/ajax What are those variables doing there? It looks like it's taking something from our Smarty debug console, which is hidden but still active in the source code, but also the ajax URL which is in a completely different location. What is Google trying to show here?0