Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What is the best tool to view your page as Googlebot?
-
Our site was done with asp.net and a lot of scripting. I want to see what Google can see and what it can't. What is the best tool that duplicates Googlebot? I have found several but they seem old or inaccurate.
-
I just used it today. The simple crawl is free. Advanced is paid.
- topic:timeago_earlier,about a year
-
I think it's free again? I use it today...
- topic:timeago_earlier,2 years
-
Used to be free, now it's 9,99$ per year
certainly not unreasonable, but some sort of demo would be nice so one knows what to expect. Do you have another recommendation? - topic:timeago_earlier,8 months
-
Hi Niners52,
Personally, I like to use webconfs search engine spider tools (http://www.webconfs.com/search-engine-spider-simulator.php) but there are loads out there that you can use and it is your own preference really.
Also, I use ScreamingFrog to do similar tasks.
Matt.
-
I'm a fan of http://www.seo-browser.com/ - use their simple search and it's totally free.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have Your Thoughts Changed Regarding Canonical Tag Best Practice for Pagination? - Google Ignoring rel= Next/Prev Tagging
Hi there, We have a good-sized eCommerce client that is gearing up for a relaunch. At this point, the staging site follows the previous best practice for pagination (self-referencing canonical tags on each page; rel=next & prev tags referencing the last and next page within the category). Knowing that Google does not support rel=next/prev tags, does that change your thoughts for how to set up canonical tags within a paginated product category? We have some categories that have 500-600 products so creating and canonicalizing to a 'view all' page is not ideal for us. That leaves us with the following options (feel it is worth noting that we are leaving rel=next / prev tags in place): Leave canonical tags as-is, page 2 of the product category will have a canonical tag referencing ?page=2 URL Reference Page 1 of product category on all pages within the category series, page 2 of product category would have canonical tag referencing page 1 (/category/) - this is admittedly what I am leaning toward. Any and all thoughts are appreciated! If this were in relation to an existing website that is not experiencing indexing issues, I wouldn't worry about these. Given we are launching a new site, now is the time to make such a change. Thank you! Joe
Web Design | Feb 7, 2023, 8:46 PM | Joe_Stoffel1 -
Reason for robots.txt file blocking products on category pages?
Hi I have a website with thosands of products. On the category pages, all the products are linked to with the code “?cgid” in the URL. But “?cgid” is also blocked in the robots.txt file for some reason. So I'm thinking it's stopping all my products getting crawled by Google. Am I right here? Is there any reason why a website would want to limit so many URL's? I'm only here a week and the sites getting great traffic, so don't want to go breaking it!!! Thanks
Web Design | Apr 23, 2019, 6:34 AM | Frankie-BTDublin0 -
Is having a site map page necessary?
Hello all! So I know having a sitemap XML file is important to include in your robots.txt file. I also know it is important to submit your XML sitemap to Google and Bing. However, I am wondering if it is beneficial for your site's SEO value to have a sitemap page displayed on your website? Or is this just a redundant action if you have already done the above two actions with your XML sitemap? Thanks in advance!
Web Design | Oct 5, 2019, 8:23 AM | Myles920 -
Should i be using shortcodes for my my page content.
Hello, I have a question. Sorry if this is been answered before. Recently I decided to do a little face lift to my main website pages. I wanted to make my testimonials more pretty. Found this great plugin for testimonials which creates shortcodes. I love how it looks like, but just realised that when I use images in shortcodes, these are not picked up by search engines 😞 only text is. Image search ability is pretty important for me and I'm not sure if I should stick with my plain design and upload images manually with all alt tags and title tags or there is a way to adjust shortcode so it shows images to search engines. You can see example here. https://a-fotografy.co.uk/maternity-photographer-edinburgh/ Let me know your thoughts guys. Regards, Armands
Web Design | Apr 19, 2016, 2:35 PM | A_Fotografy1 -
Have an eBook. What is best practice for SEO?
Hello We have a free eBook - its a great resource and great piece of content. It is available to download on our website here - http://re-timer.com/the-product/how-to-sleep-better/ The book is available as a whole or as individual chapters (i.e. http://re-timer.com/app/uploads/2015/07/Chapter8.pdf?b0df38). The PDF chapters appear to be doing well in Google search for certain keywords. I can't measure this in GA though. I would like the eBook to assist the SEO of my website overall. If I create a web page and 'embedded' the PDF into it will Google still crawl this page? At the moment we are also using this to collect email addresses, this is a nice to have and it is OK if people get the eBook without doing this (if they find a chapter in Google they currently don't have to enter their email address). I'm sure lots of people have eBooks now. What is best practice and the best way to use this as a tool to maximise SEO for the whole website (http://re-timer.com)? Thank you! Laura
Web Design | Dec 10, 2015, 9:57 PM | LauraFalls1 -
Internal Linking: What is the best practice for pages not included in Nav bar?
I never quite understood why internal linking was such a big deal for SEO, but now I'm having second thoughts and perhaps understanding it more. I always thought since most websites have a navigation feature--usually the menu bar located at the top and often another one in the footer--that internal navigation was usually already built in to most websites and therefore, a silly topic to make a fuss over; however, I may be the silly one after all. I am now creating pages that are not included in the navigation so.... What is the best practice for this? If I am creating say, pages for certain locations and those location pages begin to number in the hundreds, it makes my navigation bar a little too cumbersome to have all those pages in a drop down menu. So I made a Locations page and just link to all those pages from that page (and from nowhere else). But now I'm wondering if this could be a bad internal linking practice and perhaps hurt my online visibility as an SEO ranking factor. Is this a crawl problem? And if so, is there a better option that provides a good visitor experience while appeasing the search engines.
Web Design | Sep 18, 2015, 4:31 PM | Dino640 -
Duplicate Content Issue: Mobile vs. Desktop View
Setting aside my personal issue with Google's favoritism for Responsive websites, which I believe doesn't always provide the best user experience, I have a question regarding duplicate content... I created a section of a Wordpress web page (using Visual Composer) that shows differently on mobile than it does on desktop view. This section has the same content for both views, but is formatted differently to give a better user experience on mobile devices. I did this by creating two different text elements, formatted differently, but containing the same content. The problem is that both sections appear in the source code of the page. According to Google, does that mean I have duplicate content on this page?
Web Design | Sep 8, 2015, 11:49 PM | Dino640 -
Link colour on page?
I always thought that the link colour has to be different from text colour? I have come across a site http://www.printandpackaging.co.uk/ and it has made me question this belief, they seem to only have bolded the link which would be very nice if this is fine.
Web Design | Apr 8, 2013, 9:15 PM | BobAnderson0