Way to spider Wordpress site
-
I have an old Wordpress site and I want to move it to a new server and take it off Wordpress (too many hacks). I am trying to spider the site so as to get static, non-Wordpress, pages.
I am having trouble doing this. When I spider the site, it changes the URLs. For instance, if the URL is www.domain.com/page/ the URL I get out of the spider is /page/index.html And those are not the URLs in the search engine indices. There are about 2000 pages on this site, so it is not feasible to set up 301 redirects.
I tried using these spidering programs: WinHTTack Website Copier and PageNest
Does anyone know of another method of turning a Wordpress site into a non Wordpress site?
-
Hi Dan
Hmm that's a little strange. Two things;
- is WordPress updated? Do you get the normal URLs when viewing in your browser?
- have you tried Screaming Frog SEO Spider? It's free to crawl up to 500 pages Although it won't get the actual HTML on the pages, it could solve the URL issue perhaps.
This blackhat world thread has a few options too.
-Dan
-
Hi Dan, I'm not so experienced in migrating a WP to non -wp but I understand that the issue you're having is that the spider is returning index.htmlfiles for urls like domain/page/.
IT's normal, any spider you will use you'll always have and index.html file. Every directory has it's index.html which is the default file to show if you're not establishing something different with rewrite rules.
If you write /page/ the browser will read the index.html file. What you have to be sure is that you'll set up a 301 redirect to avoid any index.html url to show and have it redirected to the main / page (with wildcards is a one line rule) and that your internal links are pointing all to / pages and not to index.html version of it. You can jsut find and replace the /index.html" string into the html code with the /" text (dreamweaver or any html editor will do that in bulk.
Only one commentary on you idea is that you may consider useful to build a php driven site, using includes for header, footer and nav/sidebar, jsut because thinking ahead if you're willing to make changes to a portion of the page repeating throughout the site you'll have to make changes in all pages and uplaod them all which is quite huge to do and also let space for many human/machine errors.
Hope that helped you out!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wordpress and Redirects?
I want to update my permalinks - actually I want to change the URL's to fit the content and keywords better. I can choose "edit" the URL, but don't I need a redirect? I don't see any htaccess Plugin installed.......is that what I need to be able to change my URL's in Wordpress?
Technical SEO | | cschwartzel0 -
Wordpress Archive pages
In the SEOMOZ site report a number of errors were found. One of which was no or duplicate meta desctions on certain blog pages. When I drilled down to find these i noticed thosepages are the wordpress autocreated archive pages. When I searched for these through the wordpress control panel through both pages and blogs they were nowhere to be found. Does anyone know how to find these pages or are they not something I need to worry about?
Technical SEO | | laserclinics0 -
Ranking Multi-Language Site
Recently we updated our website to a new version. Our website has a structure in which the English page is our main page with about 50 subpages. All these pages are translated in 5 different languages. The different languages are divided into folders. For example www.ourdomain.com/de containts all german pages. The pages with products would be for example: www.ourdomain.com/products for english and www.ourdomain.com/de/produkte for the german page. On our previous website this used to be simililar. After the website update the SEOMoz crawls are showning duplicated page content/title errors for the pages saying that the pages in other languages have the same content/title as the basis English webpage. Any idea how I can solve these errors?
Technical SEO | | Exp0 -
Converting Old Web Site to Wordpress
I have a website I would like to update and convert to WordPress. My site rates very highly for the keywords I care about and I have a lot of domain authority and page authority that I don't want to lose. I'm concerned about the switch as I don't want to hurt my Google positioning, but would like the benefits of a WordPress site. Any assistance and advice is appreciated.
Technical SEO | | greg.baumgartner0 -
Site not indexing correctly
I am trying to figure out what is going on with my site listings. Google is only displaying my title and url - no description. You can see it when you search for Franchises for Sale. The site is www.franchisesolutions.com. Why could this happen? Also I saw a big drop off in a handful of keyword rankings today. Could this be related?
Technical SEO | | franchisesolutions0 -
Wordpress for e-commerce
What plugin should I use to make a webshop taht is good for seo as well? Should I use wordpress indeed or should i use some other open source CMS?
Technical SEO | | sesertin0 -
Are article sites a good way to generate traffic and links
I read a lot that article sites are a great way to generate links and traffic but i would like to hear from people on here to see about their experience with article sites. I have tried article sites and i have to be honest that most of them are no follow links and have also not generate any traffic. I have written over 30 articles for free article sites with no traffic coming from them and this has been done over a eight month period can anyone recommend any good article sites that can produce traffic and also ones that produce do follow If you feel i am wasting my time putting articles on free sites then please do tell me
Technical SEO | | ClaireH-1848860 -
The course of action to move my macro site to some mini sites- justin if you can help
We have a site that we want to break up into mini sites but keep the old site for the major brands. Empirecovers.com is the major and we want to break it off into Empire Truck Covers and Empire Boat covers. What I am thinking of doing is linking from the home to Empiretruckcovers.com instead of a mini page on the site and 301 redirect the mini page to empiretruckcovers.com. Than (there wont be duplicate content) making a small page for truck covers on empire just so people do not get confused. Is this the best way to go or what do you suggest? We are doing this because I feel there is seo value in having mini sites and also the user experience will be cleaner and people will trust it a lot more than inside a big site. The other problem is I have some great rankings on the pages so I want to do it so there is as little damage as possible. I guess once I start I will do all the free directories, yahoo directory and try to get links as fast as I can. Any suggestions would be great. I am going to do a/b testing to see if my adwords convert better on mini site or on the big site for certain keywords too
Technical SEO | | goldjake17880