To restrict a forum only to members of a custom usergroup, select Forum Permissions from the Usergroup menu on the left menu bar.
Scroll down to the forum you want to restrict and click Deny All
NOTE:Subforums inherit permissions from parent forums. Be aware that your changes may be affecting more than you think they are. The Forum Permissions display color-codes nondefault permissions, so this change should be easy to see. See the manual for more information on permissions inheritance.
Return to your forum and click on the Edit link next to the usergroup to which you want to grant access.
At the very top of the screen, just click the radio button for "Use usergroup default permissions".
This will grant the usergroup its normal default permissions within this forum. If you followed these instructions, that means that they'll be able to act like normal registered users in this group, while everyone else is denied access.
Scroll to the bottom and click Save.