Do you know any tool(s) to check if Google can crawl a URL?
-
Our site is currently blocking search bots that's why I can't use Google Webmaster Tools' URL fetch tool.
In Screamingfrog, there are dynamic pages that can't be found if I crawl the homepage.
Thanks in advance!
-
I'm using a tool called GSiteCrawler at the moment, I'm new to it, however it will list all crawlable pages and create a sitemap.xml for you too!
-
You could consider this online tool: http://web-sniffer.net
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google analytic's API information
I have multiple websites and instead of having to log in to each google analytics I want to create a dashboard inside my MIS that has the audience overview graph, is there any way to use API to do this? Is there a way to
Web Design | | BobAnderson0 -
We added hundreds of pages to our website & restructured the layout to include 3 additional locations within the sub-pages, same brand/domain name. How long could Google take to crawl/index the new pages and rank the keywords used within those pages?
We added hundreds of pages to our website & restructured the layout to include 3 additional locations within the sub-pages, same brand/domain name. The 3 locations old domains were redirected to their sites within our main brand domain. How long could Google take to crawl/index the new pages and rank the keywords used within those pages? And possibly increase our domain authority hopefully? We didn't want our brand spread out over multiple websites/domains on the internet. This also allowed for more content to be written on pages, per each of our locations service's, as well.
Web Design | | BurgSimpson0 -
Multiple sites using same text - how to avoid Google duplicate content penalty?
Hi Mozers, my client located in Colorado is opening a similar (but not identical) clinic in California. Will Google penalize the new California site if we use text from our website that features his Colorado office? He runs the clinic in CO and will be a partner of the clinic in CA, so the CA clinic has his "permission" to use his original text. Eventually he hopes to go national, with multiple sites utilizing essentially the same text. Will Google penalize the new CA site for plagiarism and/or duplicate content? Or is there a way to tell Google, "hey Google, this new clinic is not ripping off my text"?
Web Design | | CalamityJane770 -
Site health - webmaster tools
A bit of an odd one. In Webmaster Tools, there's the option to order sites by site health. When we do this our site - http://www.neooptic.com/ - is near the bottom, despite there being little or no crawl errors. Any ideas why this could be happening?
Web Design | | neooptic0 -
Will changing our URL's to MVC friendly URL's have a positive or negative affect on our rankings and link juice?
We've recently changed our site over to a new hosting system, we've got similar pages and are now looking at changing the URL's to ensure we do not loose our link juice from our previous site. My question is regarding the URL's, is it worth us changing our URL's to MVC friendly URL have a good or bad affect on our rankings and or link juice? Thanks
Web Design | | SimonDixon0 -
I know frames aren't good, but are they bad?
About 3/4 of my website includes frames from the Amazon aStore, but the pages also have at least 500 words of content on them each. I understand that spiders aren't too good with frames but will search engines punish my site for having them or just disregard them? Thanks in advance.
Web Design | | Max_powers0 -
Looking for an open source or wordpress designer that knows seo best practices
I have almost lost my patience in trying to find a web developer for our project. I have searched high and low from freelancers to us based firms. All I can find; freelancers that can't get the job done, but promise they can and us based firms that are currently getting away with murder charging through the nose on work that is not acceptable to say the least. US based Firms 1. Seem to give you as little work as possible to increase their margin. I get it we all need to make money. 2. Everyone knows how to do everything until you start telling them that you have a little education in the industry and will be testing their work. All of the sudden they no longer talk to you. 3. Got a few recommendations and they are all subpar performers. After asking them why their builds load so slow or have so many errors they have excuses that point to the customer Freelancers over seas. 1. I am not sure where to start with this. I have searched high and low in freelancer for someone that I can trust to build a site. Of course there is a ton of junk to look through. After countless hours of narrowing down the individuals I am thinking of giving a shot I find that they are not capable of the job. All I want is a new website from a firm that is honest and knows what they are doing. That is educated in seo best practices. That can build a quality website and actually has references of sites they built that are still up and running and test out alright. It is pretty bad when web development companies miss simple items like h tags. Really? Does anyone know of someone that knows what they are doing? That can work with someone that knows how to run a dvd player. Just disappointing to see all these web companies and freelancers that get away with murder. Who earns their keep in this industry?!?!?!?
Web Design | | forecastedinvestments0