How do I complete a reverse DNS check when completing log file analysis?
-
I'm doing some log file analysis and need to run a reverse DNS check to ensure that I'm analysing logs from Google and not any imposters. Is there a command I can use in terminal to do this?
If not, whats the best way to verify Googlebot?
Thanks
-
That's awesome! Glad to know there's a bulk tool out there!
-
Hi Tyler,
Thanks for your reply. I managed to get down to 98 unique IPs and ran a bulk reverse DNS/IP Look-up using this tool:
https://www.infobyip.com/ipbulklookup.php
Thanks for your help though!
-
Hey Daniel,
If you want to verify that a user-agent is actually Googlebot, you'll want to use a log file analysis tool to aggregate all of the IP addresses associated with Googlebot. Once you have a list of IP addresses, you can perform a reverse DNS lookup to verify whether the IP addresses are actually associated with Googlebot or not.
If you're on windows/pc these steps should work:
https://www.serverintellect.com/support/dns/reverse-dns/If you're on mac try these steps:
1. open Terminal
2. type "host" + ip address
for example: "host 66.249.66.1"
3. hit enter
4. view results. For example: "1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com"If the results are from Google.com or Googlebot.com, you can be sure it's actually Google crawling your site. Unfortunately, I don't know of any faster ways to achieve these results. I'm sure there's a tool out there, I just haven't found it yet.
This might also be a good resource for you: https://support.google.com/webmasters/answer/80553?hl=en
Good luck!
-Tyler
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help to decide of domain move reversal after 5 months!
Hello, first sorry for my bad english,it isn't my first lanugage.
Technical SEO | | Milad25
I have a website with 13 years of history and activity. 5 Months ago we received an warning message from our domain provider which would seize our domain because of sanctions (i live in Iran), and they have seized many of iranian domains since then, therefore i have decided to quickly change my domain to another address so i could save my website as much as possible before they take out my domain...
I have moved my website successfully to a new domain address and have done everything necessary for a good domain move (301 all links, change template and...) I have also used the "Change of Address Tool" provided in google search console so google knows my new domain address and change all of my links...
Unfortunately 90% of my traffics comes from google, therefore we are hevaily depending on organic traffic.
Since i have changed my domain address my traffic has been declining, and now i have only 30% of the traffic input left from google compared to my old domain 5 months ago. (i had recently some SEO troubles too which could effect this decline even more)
Fortunately my old domain wasn't seized by the domain provider and i have successfully transfered it to another provide recently so there is no danger for my old domain anymore.
My question is, should i move my website back to my old domain (cancel the google "Change of Address Tool" and use it again to move the new domain back to the old domain)? My old domain has more than 13 Years of history,has many backlinks within this 13years, and till now, i cannot get good rankings with new posts on the new domain, sometimes google even does not index my new articles several days, but my old domain ranks still well (i have tested a new article on the old domain to see how it performs and it was not very good, but i think it ranks still better than my new domain).
My top pages and categories has been redirected successful and are still ranking well on google on the new domain address and hasn't been affected negativly, my main problem are new posts that are not ranking well o even does not get indexed for several days!
I don't know what to do now, are 5months not enough for google to completly transfer all domain scores from my old domain to the new one? Will all scores of my old domain even transfer to my old domain eventually? How about the many Backlinks i have pointed to the old domain? (which 90% i cannot change or ask to change to my new address) Will the backlinks scores pointing to the old domain transfer to the new domain?In other hand i fear to move my site back to the old domain because i don't know how google would behave, would all my seo score and rankings come back after i move back to the old domain? Also as far as i know, after 6months of using the google "Change of Address Tool" i cannot cancle the domain change anymore, therefore i have roughly 1 month to decide to cancle the move or not...
Please if anyone could help or guide me what to do it would be life saving for me, because my whole income and my family depends on my website... 😞0 -
Reverse IP Lookup
I have a client that has over 90,000 incoming links from a single IP address. I can't figure out who's linking to them. I've used several different reverse IP lookup tools and can tell that the server is in Europe and ISP is AT&T Global Network Services Nederland B.V.. (http://www.ip-adress.com/reverse_ip/194.196.0.36) Says there's 0 hosts on that IP. Any suggestions?
Technical SEO | | DonnaDuncan0 -
Remove html file extension and 301 redirects
Hi Recently I ask for some work done on my website from a company, but I am not sure what they've done is right.
Technical SEO | | ulefos
What I wanted was html file extensions to be removed like
/ash-logs.html to /ash-logs
also the index.html to www.timports.co.uk
I have done a crawl diagnostics and have duplicate page content and 32 page title duplicates. This is so doing my head in please help This is what is in the .htaccess file <ifmodule pagespeed_module="">ModPagespeed on
ModPagespeedEnableFilters extend_cache,combine_css, collapse_whitespace,move_css_to_head, remove_comments</ifmodule> <ifmodule mod_headers.c="">Header set Connection keep-alive</ifmodule> <ifmodule mod_rewrite.c="">Options +FollowSymLinks -MultiViews</ifmodule> DirectoryIndex index.html RewriteEngine On
# Rewrite valid requests on .html files RewriteCond %{REQUEST_FILENAME}.html -f RewriteRule ^ %{REQUEST_URI}.html?rw=1 [L,QSA]
# Return 404 on direct requests against .html files RewriteCond %{REQUEST_URI} .html$
RewriteCond %{QUERY_STRING} !rw=1 [NC]
RewriteRule ^ - [R=404] AddCharset UTF-8 .html # <filesmatch “.(js|css|html|htm|php|xml|swf|flv|ashx)$”="">#SetOutputFilter DEFLATE #</filesmatch> <ifmodule mod_expires.c="">ExpiresActive On
ExpiresByType image/gif "access plus 1 years"
ExpiresByType image/jpeg "access plus 1 years"
ExpiresByType image/png "access plus 1 years"
ExpiresByType image/x-icon "access plus 1 years"
ExpiresByType image/jpg "access plus 1 years"
ExpiresByType text/css "access 1 years"
ExpiresByType text/x-javascript "access 1 years"
ExpiresByType application/javascript "access 1 years"
ExpiresByType image/x-icon "access 1 years"</ifmodule> <files 403.shtml="">order allow,deny allow from all</files> redirect 301 /PRODUCTS http://www.timports.co.uk/kiln-dried-logs
redirect 301 /kindling_firewood.html http://www.timports.co.uk/kindling-firewood.html
redirect 301 /about_us.html http://www.timports.co.uk/about-us.html
redirect 301 /log_delivery.html http://www.timports.co.uk/log-delivery.html redirect 301 /oak_boards_delivery.html http://www.timports.co.uk/oak-boards-delivery.html
redirect 301 /un_edged_oak_boards.html http://www.timports.co.uk/un-edged-oak-boards.html
redirect 301 /wholesale_logs.html http://www.timports.co.uk/wholesale-logs.html redirect 301 /privacy_policy.html http://www.timports.co.uk/privacy-policy.html redirect 301 /payment_failed.html http://www.timports.co.uk/payment-failed.html redirect 301 /payment_info.html http://www.timports.co.uk/payment-info.html1 -
Missing files in Google and Bing Index
We uploaded our sitemap a while back and we are no longer see around 8 out of 33 pages. We try submitting the sitemap again about 1-2 weeks ago and there but no additional pages are seen when I do site: option in both search engines. I reviewed the sitemap and it includes all the pages. I am not seeing any errors in the seo moz for these pages. Any ideas what I should try?
Technical SEO | | EZSchoolApps0 -
How should I properly setup my .htaccess file?
I have searched google for 'how to setup .htaccess file' and it seems that every website has some variation. For example: RewriteCond %{HTTP_HOST} ^yoursite.com RewriteRule ^(.*)$ http://www.yoursite.com/$1 [R=permanent,L] On SEOMOZ someone posted this: RewriteCond %{HTTP_HOST} !^www.yoursite.com [NC] RewriteRule (.*) http://www.yoursite.com/$1 [L,R=301] On yet another website, I found this: RewriteEngine On RewriteCond %{HTTP_HOST} !^your-site.com$ [NC] RewriteRule ^(.*)$ http://your-site.com/$1 [L,R=301] As you can see there are slight differences. Which one do I use? I'm on Apache CentOS and I have HTML5 websites and several Joomla! wesites. Would the HTACCESS File be different for both?
Technical SEO | | maxduveen0 -
Broken links shall i put them in my htaccess file to generate juice
Hi, when i had to rebuild my website after my hosting company made an error, i lost over 10,000 pages and lost many thousands of links coming to my site. What i want to know is, instead of trying to recreate those pages again which would take me a long time, should i put them into my htacess file and have them point back into my site. so for example, if i have a link coming to my site to an article which could be, holidays in benidorm are not selling well, would it be a good idea to have that link pointed at the main benidorm section which is benidorm news. And if i had an article which was people are finding it hard to lose weight, instead of writing a new article could i have the link pointing to my health section? If this is the correct way of doing it to grab back some link juice, would it slow my site down and how many links could i put in my htacess file. So what i am trying to say is, if i put in say 1000 redirects into my htaccess file, would it slow my site down and is this a wise thing to do or should i just let the links go.
Technical SEO | | ClaireH-1848860 -
Forward DNS Please HEEEELLLLPPPPPPP!!!
Sorry if I'm asking a rookie question here, I'm a hobbyist not a pro. I purchased a domain from Dodaddy and it's hosted on Network Solutions. We are now adding a shopping cart and need to create the subdomain shop.mydomain.com and forward the DNS to Volusion. I assumed this is done at Godaddy, where I purchased the domain. I called and they told me that Network Solutions needs to do it. I called NS and they told me GD had to do it. I called GD again hoping to get a different answer, but NOPE... guess what??? "everyone knows that it's done by whoever is hosting the site." UUUUGGGG So PLEASE let me know WHO has this responsibility???? Hopefully everyone has a common answer. 🙂 I still believe it's done at GD. Thanks Everyone
Technical SEO | | dmac0 -
Indexing of flash files
When Google indexes a flash file, do they use a library for such a purpose ? What set me thinking was this blog post ( although old ) which states - "we expanded our SWF indexing capabilities thanks to our continued collaboration with Adobe and a new library that is more robust and compatible with features supported by Flash Player 10.1."
Technical SEO | | seoug_20050