How deep should I let my forum be spidered
-
I run quote a niche website that's been running since late 1999 and over that time I've built up something like 4000 resources which consist of either text articles or image galleries and reviews along side another few thousand news stories relating to the niche interest. On top of the main site I also have a forum which isn't especially optimised for SEO and I was wondering, whilst was cleaning it up, whether anyone has any tips / suggestion / best practices for forum SEO.
Because it is all UGC the quality of the posts can be quite weak so I was wondering whether I should block robots completely from the forum, which seems a little harsh, whether I should let the whole forum be spidered (which seems a little excessive and potentially a bad thing) or whether I should restrict things to that only the main index and perhaps one page of topics and their posts be accessible to robots and then nofollow the rest?
Any thoughts?
-
Hey mate.
UGC can actually be very high quality content, and subsequently I would let Google run wild through all the content.
Forums can be tricky and you would want to make sure Google doesnt index any duplicate content.
I would use this tool, http://home.snafu.de/tilman/xenulink.html to crawl the forum. Export to Excel and then assess all the URL structures and see if it's picking up any strange parameters or duplicate content.
I would block the weird paramters in Google webmaster tools, and if technically possible 301 all the duplicate content issues. The problem with restricting things in robots.txt is that you might actually block link juice accidentally.
Hope that helps - my background is more in ecommerce but im sure we'll have similar site structure issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
There is a copy of our website that is ranking. How can I let Google know our website is the authentic site?
I just found another copy of my old website and have no way to take it down. Unfortunately, it's ranking so he didn't place it as a nofollow. (My boss hired someone to redevelop our website before I came on board and never finished the project). So, could this be hurting us? I tried to look to see if we were being penalized and couldn't find that we were. Also, ever since we migrated to a new domain name, our ranking is tumbling. I've redirected properly and tested to make sure they're resolving correctly and they are. I have no idea what is going on. We've virtually lost all ranking. Any help would be much appreciated.
On-Page Optimization | | npuffer790 -
Duplicate content because of member only restrictions on a forum.
Our website's Community Forum links to the membership profile pages, which by default are blocked for non-members. https://www.foodbloggerpro.com/community/ https://www.foodbloggerpro.com/community/member/1301/ We're getting warnings in Moz for duplicate content (and errors) on these member profile pages. Any ideas for how we can creatively solve this problem? Should we redirect those pages or just beef them up with more content? Just ignore it and assume that search spiders will be smart enough to figure it out? See attached video for further explanation. Community_Area.mp4
On-Page Optimization | | Bjork0 -
Duplicate Content Issues with Forum
Hi Everyone, I just signed up last night and received the crawl stats for my site (ShapeFit.com). Since April of 2011, my site has been severely impacted by Google's Panda and Penguin algorithm updates and we have lost about 80% of our traffic during that time. I have been trying to follow the guidelines provided by Google to fix the issues and help recover but nothing seems to be working. The majority of my time has been invested in trying to add content to "thin" pages on the site and filing DMCA notices for copyright infringement issues. Since this work has not produced any noticeable recovery, I decided to focus my attention on removing bad backlinks and this is how I found SEOmoz. My question is about duplicate content. The crawl diagnostics showed 6,000 errors for duplicate page content and the same for duplicate page title. After reviewing the details, it looks like almost every page is from the forum (shapefit.com/forum). What's the best way to resolve these issues? Should I completely block the "forum" folder from being indexed by Google or is there something I can do within the forum software to fix this (I use phpBB)? I really appreciate any feedback that would help fix these issues so the site can hopefully start recovering from Panda/Penguin. Thank you, Kris
On-Page Optimization | | shapefit0 -
Too many forum posts - may I delete the oldest ones?
Hello, We've got product pages with comments. We have collected so many comments for the last two years on these pages so they made problems to our website. We plan to cut every posts older than one year to solve this problem. Could it be bad for SEO? (Yes, these are really unique and relevant posts...) How could I predict the consequences? Thanks!
On-Page Optimization | | DDL0 -
SeoMoz Pro - Keyword stuffing but its a forum
hiya guys Just going through the Seomoz Pro on page thing One of the flagged items is keyword stuffing, I have 22 keywords (stuffed) but they want it down to 15, the thing is, my website is a forum. and the keyword stuffing is from the title name of each categories eg its a nightlife forum so Birmingham nightlife forum Brighton nightlife forum Leeds nightlife forum Manchester nightlife forum u get the picture To avoid this keyword stuffing, should I change some of the forum names to eg leeds nightlife forum > leeds nightlife forums? just by adding the 's that will half the keywords, but its not solving the problem in the long run, if i was to add more cities and areas with the XXXX nightlife foru. What you reckon guys? Cheers guys Luke
On-Page Optimization | | Lukescotty0 -
Yahoo small business host has two web sites and confuses the spiders
Both another consultant and my first seomoz test stated that I have two websites and that confuses the spiders. . One is www.mystore.com ( example) and the other is mystore.com. Yahoo states that's how they role and I can not delete one. I can do a 301 redirect and redirect one site to the other but they do not have a recommendaation as to which one to redicrect. Which one should I redirrect?
On-Page Optimization | | Wales0 -
Should I let Google index tags?
Should I let Google index tags? Positive? Negative Right now Google index every page, including tags... looks like I am risking to get duplicate content errors? If thats true should I just block /tag in robots.txt Also is it better to have as many pages indexed by google or it's should be as lees as possible and specific to the content as much as possible. Cheers
On-Page Optimization | | DiamondJewelryEmpire0 -
How deep should I go with a directory site?
I am creating a new site which has a directory component. Based on what I have ready I am inclined to keep the site architecture as flat as possible. However, the natural layout that I have come up with in my head has the directory listings 5 or 6 pages deep in the site structure. I saw in another post that someone in a similar situation was suggesting that going deep like this is fine so long as there are many internal links to the deeper pages to indicate that they are important. Should I make a conscious effort to make the site architecture as flat as possible? Are there any specific guides/resources that address this particular issue that I should be aware of? Thanks!
On-Page Optimization | | fastestmanalive0