Is there a Modern GSiteCrawler
-
We have a site with 1000+ URLs. We'd like to index all of the pages capturing URL's, Page Titles, etc with 100% accuracy. Which SEOmoz tool would be best at this? PS - in the old days we used the likes of GSiteCrawler. Thanks for your direction.
-
Yes, actually the 'Crawl Diagnostics' tool that comes with SEOmoz Pro does let you export to CSV. There are just two limitations:
-
It isn't in real-time (and I've typically leaned towards instant gratification ).
-
It may not find every page (to give you a real life example, one of my many experimental sites has approximately 1,000,000 pages .... in checking the Crawl Diagnostics export just now, SEOmoz found 10,319). But for smaller sites that our agency works with, it's pretty much spot-on.
-
-
Thanks Corey. We'll investigate all of the helpful recommendations you made.
Also just confirming, in your experience doesn't SEOMoz's "SEO Web Crawler" do what we are looking for?
-
Sadly SEOmoz (or any SEO tool) never seems to solve more than 5% of the puzzle. But I think Screaming Frog will probably do what you're asking (plus, it's free). If not that, I like SEO PowerSuite's Website Auditor a lot, and always seems to pull back denser data than most alternatives (but isn't free).
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
cross canonicalization with redirect
I'm working with a website that has turned one of its pages into its own website within the main website - mostly for the ease of customers, making it simpler to access that page using www.page.com rather than www.mainsite.com/about/page.
Technical SEO | | Shrine.SEO.Gal
As a result, there are two urls for that page (the ones just mentioned), both pointing to the exact same page, but with different urls. Now, they have made it so www.mainsite.com/about/page permanently redirects to www.page.com. which I thought was a good call. However, what do I do about canonicalization? Is it good to point the canonicalization of www.page.com to www.mainsite.com/about/page so that the rankings and link equity are maintained in the main website? Or would the fact that the www.mainsite.com/about/page redirects to www.page.com mess that up? I hope this makes sense!0 -
Need help in diagnosing what I may be doing wrong
I have a site that has been having problems ranking. Initially, spam rate was at 18%. I have since changed the URL and forwarded to the original so now the spam rate is under 5%. Phone calls started picking back up very slowly but then by August 2024 things came to a screeching halt. Phone has been dead and very little business has been written. I did notice on the robots.txt file it had this: User-agent: *
Technical SEO | | SOM24
Disallow: /
User-agent: Googlebot
Disallow:
User-agent: bingbot
Disallow: /no-bing-crawl/
Disallow: wp-admin and now I have since changed it to this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php Sitemap: https://urlishere.com/sitemap_index.xml My question is what else do I need to do to get this site to start ranking again? We are blogging once a month, writing press releases once a month, updating the social media a few times a week. I feel like maybe there is something on the backend that needs to be done to get this site back to ranking. I am using SEO by Yoast and I have filled in the title and meta description fields for all pages. There is a spot in Yoast where I can validate the site with Google, Bing, etc. I'm trying to figure out how to do that. I do see in the site's Google Webmaster Tools there are several pages not indexing. Any ideas on what else I can do to get this site to start ranking again? Thank you.0 -
Recently rankings for "Tree Service" dropped
Hi, we've had a page which was ranking top 1 on the Google rankings but has complete dropped out to 5. We don't have any duplicate errors from that page on here. Do you have any suggestions?
Technical SEO | | FIT0 -
My posts are ignored by google not sure why
I have been investigating this problem for some time now and there must be a technical problem. My posts seem to be ignored by google. For example, if i put this title of my article in google, other sites come up instead of my website Lincolnshire Caravan Owners Struggle To Compete With Haven https://www.in2town.co.uk/skegness-news/lincolnshire-caravan-owners-struggle-to-compete-with-haven/ I am trying to find out what technical problem I have that is stopping google displaying my post. Can anyone advise me on what tools to use and how to find out what is going wrong
Technical SEO | | headlinesplus0 -
Google Not Picking Up Posts
I am trying to work out why from March 4th Google is not seeing my posts. Our google impressions have dropped from 8,000 to 40. If you put in the full article name with speach marks it does not find it, and instead shows the home page in google. We have not had any warnings. We did have work done on our site but nothing else i could think of to cause this. Can anyone let me know what may have caused this. All articles are original
Technical SEO | | headlinesplus0 -
Product photo links appear in search results
Since I'm new, I don't have any information. Product photo links appear in the search results. Is this a problem? because when you click on the link, it does not take you to a page, but to a blank page with only a photo. What should I do for this situation?
Technical SEO | | UgurKarabulut0 -
My backlinks are not showing in webmaster tools? Why
Hi Experts, I have follow backlinks from a domain for 6 months, but its not apear in Links to Your Site tools (search console) that domain has 302k indexed pages in google! Could you please explain me why google not showing this type of backlinks?
Technical SEO | | denakalami7 -
Google will index us, but Bing won't. Why?
Bing is crawling our site, but not indexing it, and we cannot figure out why -- plus it's being indexed fine in Google. Any ideas on what the issue with Bing might be? Here's are some details to let you know what we've already checked/established: We have 4 301’s and the rest of our site checks out We’ve already established our Robots is ok, and that we are fixing our site map/it's in fine shape We do not see anything blocking bingbot access to the site There is no varnish or any load balancers, so nothing on that end that would be blocking the access We also don't see any rules in the apache or the .htaccess config that would be blocking the access
Technical SEO | | Alex_RevelInteractive0