Do you know any tool(s) to check if Google can crawl a URL?
-
Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.
In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.
Thanks in advance!
-
I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
-
You could consider this online tool: http://web-sniffer.net
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Is Google Showing My Images Upside Down in the Index?
Hi, My client has PDFs of their catalog on the site which google is indexing. However, it seems that google is taking an image from the catalog and then showing it upside in the index for images/search results. The images are not upside down on the site. Has anyone heard of this happening before or does anyone know a way to fix it? Thanks
Web Design | | AliMac260 -
We're considering making notable changes to our website's navigation. Other than 301 redirects from old pages to new, what do I need to consider with this type of move or update?
We would like to make some navigation changes to our website: www.NetGainIT.com, specifically to the services section. I know that I will need a list of 301 redirects if I do not plan on keeping certain pages, but what else do I need to consider?
Web Design | | NetGainTech0 -
Can someone please help with technical question!
I have noticed that our website tool to get a quote does not work with active scripting disabled is this bad? How many people have this disabled?
Web Design | | BobAnderson0 -
Will google penalize a website for using a table layout?
I just got a new client today and his entire website layout and structure is using tables instead of divs. This client is on a tight budget and wants to avoid unnecessary hours for re-coding the website, but at the same time he wants me to improve his SEO organically. This is the first time I've been asked to do work on an existing website that uses pure tables for the entire layout and I'm wondering if this effects the SEO in any way. So my question is, will tables effect rankings and SEO in any way?
Web Design | | ScottMcPherson0 -
My 404 page is showing a 4xx error. How can that be fixed?
My actual 404 page is giving a 4xx error.
Web Design | | sbetzen
The page address is http://www.ecowindchimes.com/v/404.asp It loads fine... it is the page all 404's are directed to. Why is it showing a 404 error. The page works. How can this be fixed? Stephen0 -
How to create Google Section or Jump To Links
Hello all! i need create a jump to links on my site and when seach a keyword on google it will display jump to links http://techwyse.com/blog/wp-content/uploads/2009/12/google-jump-to-links.jpg i same with that images please help me how to do it, and have any plugin on wordpress can do that google-jump-to-links.jpg
Web Design | | ITVSEO.COM0 -
How important is URL length?
Is URL length really that important? I have many articles that would lose meaning if the URL was shortened and for most, they would have to be under the root domain instead of under the category in order to fit. Has anyone tested if they were negatively impacted by URL's that are too long?
Web Design | | HMCOE0