Google uses indicators to establish what the page is about, Numerous times Matt Cutts from Google's Ranking and Web Spam team has said there are hundreds of things that Googlebot's looks for when establishing page contents and how to rank it.
Without going into detail on every factor because this has been asked many times before in several other questions you should ensure that your pages have the absolutely basic:
- Titles
- Meta Descriptions
- Headers (h1, h2, h3)
- Content (The actual text that fills the page).
- Pictures ALT tags.
- URL Keywords (Optional, but again a indicator).
The list goes on and on, again forums are no different than any other page. But forums are classed as 'User Generated Content' which will be more on the radar for 'User Generated Spam' but this only becomes an issue when sites are not moderated.
Most open source forum content management system has SEO built in and in no way should you require a in house built to rank, millions of sites use open source with no problem what's so ever, so if you didn't rank using say for example 'PHPBB' then its because it was either setup wrong or your site is being punished for one reason or another, maybe it got slapped by Penguin or Panda, and Google's even new new Hummingbird algorithm which was released just recently.
Once again CMS engines have little to non role in actual rankings assuming they have the very basics SEO indicators in place, I'm willing to bet that your ranking issues assuming that you used a popular open source forum platform is nothing to do with indicators by the CMS but rather penalties from poorly made content or bad off site links.