Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Way to spider Wordpress site
-
I have an old Wordpress site and I want to move it to a new server and take it off Wordpress (too many hacks). I am trying to spider the site so as to get static, non-Wordpress, pages.
I am having trouble doing this. When I spider the site, it changes the URLs. For instance, if the URL is www.domain.com/page/ the URL I get out of the spider is /page/index.html And those are not the URLs in the search engine indices. There are about 2000 pages on this site, so it is not feasible to set up 301 redirects.
I tried using these spidering programs: WinHTTack Website Copier and PageNest
Does anyone know of another method of turning a Wordpress site into a non Wordpress site?
-
Hi Dan
Hmm that's a little strange. Two things;
- is WordPress updated? Do you get the normal URLs when viewing in your browser?
- have you tried Screaming Frog SEO Spider? It's free to crawl up to 500 pages
Although it won't get the actual HTML on the pages, it could solve the URL issue perhaps.
This blackhat world thread has a few options too.
-Dan
-
Hi Dan, I'm not so experienced in migrating a WP to non -wp but I understand that the issue you're having is that the spider is returning index.htmlfiles for urls like domain/page/.
IT's normal, any spider you will use you'll always have and index.html file. Every directory has it's index.html which is the default file to show if you're not establishing something different with rewrite rules.
If you write /page/ the browser will read the index.html file. What you have to be sure is that you'll set up a 301 redirect to avoid any index.html url to show and have it redirected to the main / page (with wildcards is a one line rule) and that your internal links are pointing all to / pages and not to index.html version of it. You can jsut find and replace the /index.html" string into the html code with the /" text (dreamweaver or any html editor will do that in bulk.
Only one commentary on you idea is that you may consider useful to build a php driven site, using includes for header, footer and nav/sidebar, jsut because thinking ahead if you're willing to make changes to a portion of the page repeating throughout the site you'll have to make changes in all pages and uplaod them all which is quite huge to do and also let space for many human/machine errors.
Hope that helped you out!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Not Indexing Pages (Wordpress)
Hello, recently I started noticing that google is not indexing our new pages or our new blog posts. We are simply getting a "Discovered - Currently Not Indexed" message on all new pages. When I click "Request Indexing" is takes a few days, but eventually it does get indexed and is on Google. This is very strange, as our website has been around since the late 90's and the quality of the new content is neither duplicate nor "low quality". We started noticing this happening around February. We also do not have many pages - maybe 500 maximum? I have looked at all the obvious answers (allowing for indexing, etc.), but just can't seem to pinpoint a reason why. Has anyone had this happen recently? It is getting very annoying having to manually go in and request indexing for every page and makes me think there may be some underlying issues with the website that should be fixed.
Technical SEO | | Hasanovic1 -
Move a Wordpress Site to HTTPS with Bluehost
HI Guys, do you think that the following guide is enoght to move a bluehost wordpress site to https in a seo best practive way? https://www.shoutmeloud.com/free-ssl-certificate-bluehost-hosting.html Basically their steps are: Install SSL on Bluehost panel Install Really Simple SSL Wp Plugin Edit Your .htacess File & Add The Code For HTTP To HTTPS Redirection Update All HTTP URLs In Database To HTTPS Using Search and Replace Plugin Use Broken Link Checker plugin & use its redirection module to find links to 3rd party sites with HTTP that should now be HTTPS. Last thing to do Submit your new HTTPS site to Google Search Console & submit your sitemap. Update your profile link on Google Analytics. Update your website links on social media profiles & anywhere else they exist. This step you can do in pieces in the coming days. Read this guide to learn more about HTTP to HTTPS migration & fixing mixed content. If you disabled Who.Is guard for your domain name, you can enable it now. Do you know a better practical guide for wordrpess? in term of usefull plugins to handle the migration? Tx to everyone!
Technical SEO | | Dreamrealemedia0 -
Why add .html to WordPress pages?
A site I may take over has a plugin that adds .html to the pages. I searched online but I’ve only found how to add it rather than why to add it. Is it needed? If I remove it, I’ll have to be careful with SEO / indexed pages and redirects. The site is running 3.x.x and 90% of the plugins have not been updated in over 5 years including this one. Before I update to 4.7.x, I am trying to understand the landscape (pros / cons) on why something could be used and if I need to find a suitable replacement for it.
Technical SEO | | acktivate2 -
Yoast and wordpress duplicate meta
I'm using the Yoast plugin with wordpress and have noticed in my HTML I have duplicate meta data. For example my header starts with
Technical SEO | | simonatkinsphoto
<title>(title) </title<span><<br /><meta </span><span class="html-attribute-name">property</span><span>="</span><span class="html-attribute-value">og:site_name</span><span>" </span><span class="html-attribute-name">content</span><span>=<br /><span><meta </span><span class="html-attribute-name">property</span><span>="</span><span class="html-attribute-value">og:description</span><span>" </span><span class="html-attribute-name">content</span><span>=<br /><br /></span></span>Then I have the 'This site is optimised by Yoast" tagline followed by the same meta -<br /> <span><meta </span><span class="html-attribute-name">name</span><span>="</span><span class="html-attribute-value">description</span><span>" </span><span class="html-attribute-name">content=<br /><span> <meta </span><span class="html-attribute-name">property</span><span>="</span><span class="html-attribute-value">og:title</span><span>" content=<br /><span> <meta </span><span class="html-attribute-name">property</span><span>="</span><span class="html-attribute-value">og:description</span><span>" </span><span class="html-attribute-name">content=<br /><span> <meta </span><span class="html-attribute-name">property</span><span>="</span><span class="html-attribute-value">og:site_name</span><span>" </span><span class="html-attribute-name">content</span><span>=<br /><br /></span></span></span></span>Is this likely to cause problems with Google and is there a way to stop both wordpress and Yoast adding meta to the header. </p></title>0 -
Is there a pinging tool to ping all sites at once
hi, i am just wondering if there is a tool that you can put on your toolbar that allows you to ping all the sites at once. The last thing i want to keep doing is to go through every single one and ping my article. I would like to find a tool that does it all for me, can anyone let me know if there is one out there. many thanks
Technical SEO | | ClaireH-1848860 -
Should I import external reviews to my site?
Hi everybody! I manage the website for a financial services company. We have more than 5000 reviews on a user review website. We have the possibility to import and display all these reviews on our site. Is this good for SEO? Will Google find it suspicious that our site suddenly displays a lot of new keyword-rich content? What about duplicate content? Please, share your thoughts. Thanks!
Technical SEO | | Georgios0 -
Sitefinity vs Wordpress
We're looking for a new CMS and out development company suggested Sitefinity. I've had great success with Wordpress. Is either system better. I love worpdress but have had no experience with Sitefinity. Thanks!
Technical SEO | | StandUpCubicles0 -
Index forum sites
Hi Moz Team, somehow the last question i raised a few days ago not only wasnt answered up until now, it was also completely deleted and the credit was not "refunded" - obviously there was some data loss involved with your restructuring. Can you check whether you still find the last question and answer it quickly? I need the answer 🙂 Here is one more question: I bought a website that has a huge forum, loads of pages with user generated content. Overall around 500.000 Threads with 9 Million comments. The complete forum is noindex/nofollow when i bought the site, now i am thinking about what is the best way to unleash the potential. The current system is vBulletin 3.6.10. a) Shall i first do an update of vbulletin to version 4 and use the vSEO tool to make the URLs clean, more user and search engine friendly before i switch to index/follow? b) would you recommend to have the forum in the folder structure or on a subdomain? As far as i know subdomain does take lesser strenght from the TLD, however, it is safer because the subdomain is seen as a separate entity from the regular TLD. Having it in he folder makes it easiert to pass strenght from the TLD to the forum, however, it puts my TLD at risk c) Would you release all forum sites at once or section by section? I think section by section looks rather unnatural not only to search engines but also to users, however, i am afraid of blasting more than a millionpages into the index at once. d) Would you index the first page of a threat or all pages of a threat? I fear duplicate content as the different pages of the threat contain different body content but the same Title and possibly the same h1. Looking forward to hear from you soon! Best Fabian
Technical SEO | | fabiank0