Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How much is the site architecture impacting my site?
Hi there, I'm interested to learn how much the site archecture of griffith.ie (higher education) maybe impacting our rankings. In recent months there was some changes to the the faculty landing pages but not to the site archecture. The rankings in the last 6 months have dipped a little. There are two main path ways to get to the course. 1. Course finder - https://www.griffith.ie**/find-a-course** => 2. Faculties - https://www.griffith.ie**/faculties** Most of the SEO authority is coming through the Faculties pages as this is where all the courses are found in term of the url structure. For example; https://www.griffith.ie**/faculties/**business/courses/ba-hons-accounting-finance The UX on the site tells a different story and directs people to the course finder. /find-a-course Ideally, I feel the site would benefit much more if all the traffic was directed through the course finder however this would require (I think) a big redevelopment of the search tool and I feel we are diluting our efforts as when somebody arrives to the site through the homepage they go through the course finder and if they come through specific searches they get taken to the specific course page under the faculty section. the site has has this archecture for the best part of 4 years and I'm considering recommend a change if it would greatly improve SEO and UX. Any feedback on this would be great. Many Thanks Rob
Web Design | | robhough9091 -
Why is google still crawling my old website pages?
Why is google still looking at my old indexed pages and not my new index. ? Why are they crawling my old website links when none of them are available? How do I overcome these problems?
Web Design | | optimalspaces0 -
Changing top level navigation between site sections
We've got an internal proposal to change our top level nav depending on the section of the site. For example, on our homepage it might read: Products, Library, About with relevant links dropping down below. As we have varied products, the drop down underneath it would include the various families. When arriving on the product family page the top-level nav would change to represent more specific offerings. For example: xxx.com 1. Products; 2. Library; 3. About xxx.com/xxx 1. Product family 1; 2. Product family 2; 3. Product family 3; 4. Library; 5. About What are the SEO/UX implications of this? It seems confusing but allows more specific navigation via the main nav depending on the section of the site. Also it seems that an alternating TLN might not be too Google-friendly.
Web Design | | gwelch0 -
Incorporating Spanish Page/Site
We bought an exact match domain (in Spanish) to incorporate with regular website for a particular keyword. This is our first attempt at this, and while we do have Spanish speaking staff that will translate/create a nice, quality page, we're not going to redo everything in Spanish page. Any advice on how to implement this? Do I need to create a whole other website in Spanish? Will that be duplicate content if I do? Can I just set it up to show the first page in Spanish, but if they click on anything else it redirects to our site? I'm pretty clueless on this, so if anything I've suggested is off-the-wall or a violation, I'm really just spit-balling, trying to figure out how to implement this. Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
Having a second homepage for a site would affect my SEO?
Hello guys, One of our clients is planning to have a new landing page for any users hitting the site for the first time. (returning users will still see the current homepage based on cookies ... in other words, the site would technically have 2 home pages). According to this client, they are planning to do something like this: https://www.websitename.com/ (for returning visitors) https://www.websitename.com/newuser (for first time visitors) Our instinct is that is not great to have 2 home pages (that would affect the SEO campaign we are managing for this company) and we are not sure how to handle this. That's why we would appreciate your opinion regarding this topic: From an SEO perspective, do you think this is a good idea? If not, what would you guys do differentiate first-time visitors vs returning visitors without affecting SEO? Maybe just a pop-up? Thanks in advance for your help !
Web Design | | Robertnweil10 -
How can i embed my video into a table using SEO embed setting?
We use Wistia.com to embed our videos. They have different options for embed settings and we prefer to use the SEO embed setting, however, when we use that setting we aren't able to insert the video in a table side by side with another image or text. When we try, the video jumps out of the table and the table gets (for lack of a better work) out of wack. When we embed the video with the iframe embed setting, the video can be placed in a table with no issues, but then we don't get the SEO credit. We have our site in wordpress. I'm not sure if that has something to do with the tables getting messed up. Check out this link to see an example of how we want the video to show up. http://www.3000doorhangers.com/ Any suggestions as to how we can use the SEO embed setting within a table as shown in the above link?
Web Design | | JimDirectMailCoach0 -
.me.uk or .com domain with hyphen?
Hi Mozzers Just after some opinions really. Basically a client wants a restaurant website developing and had already chosen a name for it as (example) "englishroserestaurant.co.uk". They wanted to register it themselves but unfortunately before they had time to someone else did. Now they have come back to me and said about using the name (example) "englishrose.me.uk". I know these .me.uk names are supposedly for personal websites and not businesses but I assume that, by itself, shouldn't be an issue as there are many commercial sites that do use them. However as all the combinations have been taken I have suggested that they use a "domain-domain.com as an alternative so just wanted to know which option you guys would go for out of: 1. englishrose.me.uk 2. english-rose.com Although I avoid hyphens whenever I can in this instance 100% I would go for number 2 as it's only one hyphen and it's a business and looks more professional......having said that I would welcome any comments or opinions. Thanks in advance.
Web Design | | TrevorJones0 -
Changing the domain - To do or not to do - that is the question
Hello, I am in the process of updating my website (hopefully to improve SEO). It is a major update as we are going from 20,000 product line to 200,000 product lines and hiring two marketing people to work on more content. Unfortunately, I think my domain isn't the best i.e. www.direct-tractorparts.com Some of the parts have long part numbers and I am worried that my URLs will be too long i.e. I will lose link juice from the search engines. typical url www.direct-tractorparts.com/6XY41210TTTAZCAz2 I am thinking of launching the new site with a shorter domain name rather than updating the old site. My site doesn't have that many backlinks but generates a decent level of business for my company. Not sure if it would be worth setting up another brand or not Anyone got any advice?
Web Design | | DavidLenehan0