Please help to identify the following bots and spiders
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need help related to SEO ?
Hi, Hope you guys are doing well! I need help from experts, I have a query that "Does google allows SEO for websites related to crypto currency ?" If yes, what is best way to do SEO for crypto currency website ?
Technical SEO | | JessicaThompson0 -
Any SEO-wizards out there who can tell me why Google isn't following the canonicals on some pages?
Hi, I am banging my head against the wall regarding the website of a costumer: In "duplicate title tags" in GSC I can see that Google is indexing a whole bunch parametres of many of the url's on the page. When I check the rel=canonical tag, everything seems correct. My costumer is the biggest sports retailer in Norway. Their webshop has approximately 20 000 products. Yet they have more than 400 000 pages indexed by Google. So why is Google indexing pages like this? What is missing in this canonical?https://www.gsport.no/herre/klaer/bukse-shorts?type-bukser-334=regnbukser&order=price&dir=descWhy isn't Google just cutting off the ?type-bukser-334=regnbukser&order=price&dir=desc part of the url?Can it be the canonical-tag itself, or could the problem be somewhere in the CMS? Looking forward to your answers Sigurd
Technical SEO | | Inevo0 -
What online tools are best to identify website duplicate content (plagiarism) issues?
I've discovered that one of the sites I am working on includes content which also appears on number of other sites. I need to understand exactly how much of the content is duplicated so I can replace it with unique copy. To do this I have tried using tools such as plagspotter.com and copyscape.com with mixed results, nothing so far is able to give me a reliable picture of exactly how much of my existing website content is duplicated on 3rd party sites. Any advice welcome!
Technical SEO | | HomeJames0 -
Roger bot taking a long time to crawl site
Hi all, I've noticed Roger bot is taking a long time to crawl my new site. It started on the 28th Feb 2013 and is still going. There aren't many pages at the moment. Any ideas please? thanks a lot, Mark.
Technical SEO | | caterfor1 -
Would adding an SSL certificate help my website?
SSL certificates can obviously be a used as a ranking factor by Google, but would a site with no need for an SSL certificate notice a gain by adding one? Is it possible to demonstrate you have an SSL certificate without having some https pages on your site?
Technical SEO | | sthompson0 -
Help with this metadata code!
My website developer used the code below for my metadata but SEomoz is showing an error for my metadata. How can I change with this code so it will work?
Technical SEO | | CapitolShine0 -
Please help me critique my friends site
Hi All, I've regularly helped others on this forum when they've asked for tips on how to improve a specific site's SEO. Now I'm asking for the favour in return 🙂 My friend has come to me with a site (http://www.swishplastic.com.au) and asked me to help with SEO and on-page optimisation. I've got a few of my own ideas but there may be a few key things I'm overlooking. If anyone could give a few pointers, it would be much appreciated! Thanks!
Technical SEO | | bradkrussell0 -
Help needed with robots.txt regarding wordpress!
Here is my robots.txt from google webmaster tools. These are the pages that are being blocked and I am not sure which of these to get rid of in order to unblock blog posts from being searched. http://ensoplastics.com/theblog/?cat=743 http://ensoplastics.com/theblog/?p=240 These category pages and blog posts are blocked so do I delete the /? ...I am new to SEO and web development so I am not sure why the developer of this robots.txt file would block pages and posts in wordpress. It seems to me like that is the reason why someone has a blog so it can be searched and get more exposure for SEO purposes. IS there a reason I should block any pages contained in wodrpress? Sitemap: http://www.ensobottles.com/blog/sitemap.xml User-agent: Googlebot Disallow: /*/trackback Disallow: /*/feed Disallow: /*/comments Disallow: /? Disallow: /*? Disallow: /page/
Technical SEO | | ENSO
User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/themes/ Disallow: /trackback Disallow: /commentsDisallow: /feed0