We are using Hotlink Protection on our server for jpg mostly. What is moz.com address to allow crawl access?
-
We are using Hotlink Protection on our server for jpg mostly. What is moz.com crawl url address so we can allow it in the list of allowed domains? The reason is that the crawl statistics gives our a ton of 403 Forbidden errors.
Thanks.
-
Hi there!
Thanks for reaching out to us! I can certainly understand your need to have our crawler be accepted into your link protection program. Unfortunately our crawler doesn't operate using a url to crawl your site, we use a collections of IP addresses behind the scenes to mimic a search engine crawler to provide the best diagnoses around. With that said, a lot of our customer has had some success allowing our crawler on their server level through either their HTTP access file or other methods. Unfortunately I am not a Web developer or a server admin, so I couldn't exactly let you know how to implement it. I would recommend you to perhaps change up your question and post a new question regarding some work around for your software.
Thanks for your time, I hope that helps.
Peter
Moz Help Team.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Scraping a Password Protected Page
Hey Everyone, I'm looking for some help regarding my website. I have multiple pages on my site that are password protected (we only give the password to potential clients).... Somehow, when doing a google search, i found a directory that has successfully scraped and pulled all of the information from the password protected pages and is displaying this info on their site... This is very bad for us as obviously this information is critical and not intended to be shared with the rest of the world Does anyone have any advice on how to prevent my password protected pages from being scraped/crawled? Thanks!
Technical SEO | | Prime850 -
Use of iFrames to embed video content?
So we decided to add interesting video content we have shot ourselves to improve the user experience on our website. We have read though all the SEO material and advice we could find and decided to use Vimeo to host our videos instead of self-hosting. The videos themselves will not show up on Vimeo or Google as we set our account to 'private', but we notice that to embed the videos on our website - the embed code is an iFrame. My question's are, aren't iFrames supposed to be bad for SEO? Should we look to another video hosting solution instead? If other websites embed our videos on their site using the iFrame, do we get any SEO credit or the backlink? Thx.
Technical SEO | | Liam-Web0 -
Links from Instructables.com?
This is a silly newbie question. But will posting on www.instructables.com with some valuable content and url link back to my site help with "linking"? Or do they put a no-follow on all links on their site? Thanks for answering! Ron
Technical SEO | | yatesandcojewelers0 -
Getting Errors On Server Connectivity-??
Hi Guys I am getting a massive crawl errors on googlewebmaster ,stating there is over 2162 errors connect time out - anyone know where I can see exactly where the time out is from? I have browsed through my site and I do not see any connect timeout occured. Thanks Cary
Technical SEO | | ilovebodykits1 -
301 redirects on Windows server
Hi, We are soon moving www.ourumbrellaorganisationwebsite.co.uk/oldsubsidiaryname/index.aspx AND www.differentolddomainname.co.uk to just www.ourumbrellaorganisationwebsite.co.uk (an existing site which will no longer have the old subsidiary name sub section). How do we do the 301 redirects on a Windows server? Helicon has been suggested but I don't know it. I know we need to 301 redirect 'old' pages to the equivalent new ones, but is it a problem to do all of the old pages (there are lots) or should we just just do a few? is there ever a downside to doing individual redirects for an entire old site? Also, once the 301 redirects are in place from the old domain, is it possible to let the old domain expire and if so, at what point? Thanks
Technical SEO | | Houses0 -
Ranking on google.com.au but not google.com
Hi there, we (www.refundfx.com.au) rank on google.com.au for some keywords that we target, but we do not rank at all on google.com, is that because we only use a .com.au domain and not a .com domain? We are an Australian company but our customers come from all over the world so we don't want to miss out on the google.com searches. Any help in this regard is appreciated. Thanks.
Technical SEO | | RefundFX0 -
Magento - Google Webmaster Crawl Errors
Hi guys, Started my free trial - very impressed - just thought I'd ask a question or two while I can. I've set up the website for http://www.worldofbooks.com (large bookseller in the UK), using Magento. I'm getting a huge amount of not found crawl errors (27,808), I think this is due to URL rewrites, all the errors are in this format (non search friendly): http://www.worldofbooks.com/search_inventory.php?search_text=&category=&tag=Ure&gift_code=&dd_sort_by=price_desc&dd_records_per_page=40&dd_page_number=1 As oppose to this format: http://www.worldofbooks.com/arts-books/history-of-art-design-styles/the-art-book-by-phaidon.html (the re-written URL). This doesn't seem to really be affecting our rankings, we targeted 'cheap books' and 'bargain books' heavily - we're up to 2nd for Cheap Books and 3rd for Bargain Books. So my question is - are these large amount of Crawl errors cause for concern or is it something that will work itself out? And secondly - if it is cause for concern will it be affecting our rankings negatively in any way and what could we do to resolve this issue? Any points in the right direction much appreciated. If you need any more clarification regarding any points I've raised just let me know. Benjamin Edwards
Technical SEO | | Benj250 -
Does anyone see benefit in .com/en vs .com/uk for a UK site?
The client is already on /en and in my opinion there is not much to be gained by switching to /uk
Technical SEO | | Red_Mud_Rookie0