How does a custom homepage (not a CMS like vbadvanced) affect the traffic and feel of a forum? I was going to add a custom homepage to add a guide and helpful links for the specific industry. Any experience with a custom homepage? Good or bad idea?

The problem would be that dynamic content is needed IMO to keep traffic funneling through to the forums. IMO unless you're going to institute at least a 1-page CMS "homepage", better to redirect the root directory straight to your forums vs. a static homepage and/or find a way to get more "homepage" content into your forum index.

I think having an homepage can increase earnings for first time visitors and offers the members a place to view all the latest happenings.

Unless you are going to have a homepage that has a lot of contents, I would simply put the forum as the index page.
If your website is simply a forum, I don't think that there is much to gain by having an index page that is not the forum home.

depends on how things are balanced, if everything is off one server a small page and forum together can equal a big load once the site gets popular

I want to know..how about..if somebody got a forum in their blog? How much it will effect our page rank and traffic?

since, right now most of my friend provide a forum in another page of their blog.

I am not a fan of a vbadvanced type home page. I think you need to go custom and I think it is important for a serious minded forum owner to do this. The home page will be perfect to point people to content / tools NOT in the forum.