We've installed a Server Core box into our network here so we can run Hyper-V as well as VMware ESX. We have also set it up as a DC and DNS server so that of our SBS box is offline things might continue to work a bit better than they have up until now :-)

Instructions on how to do this are a bit sparse and my Technician got this information from the SBSC Newsgroups:

From your description, I know that you want to add Windows Server 2008 Server Core to SBS domain. If that's not correct, please don't hesitate to let me know.

Unfortunately there is no article talking about how to add Windows 2008 Core to SBS domain. You can add Windows 2008 Server Core to SBS domain with the steps below on the SBS server:

1. Open a Command Prompt window. To do this, Click Start, click Run, type cmd, and then click OK.

2. Perform the following commands in the sequence noted:

Note 

In these instructions, "drive" represents the letter of your DVD drive on SBS server with Windows 2008 DVD inserted.

2. At the command prompt, type drive:\sources\adprep\adprep.exe /forestprep

3. At the ADPREP warning, type c and then press ENTER.

4. At the command prompt, type drive:\sources\adprep\adprep.exe /domainprep

If you are installing the additional server as a read-only domain controller, at the command prompt, type drive:\sources\adprep\adprep.exe /rodcprep (on the SBS server)

5. On how to join the Windows Server Core to SBS 2003, please refer to "To join a domain" of the article below:

Server Core Installation Option of Windows Server 2008 Step-By-Step Guide

http://technet.microsoft.com/en-us/library/cc753802.aspx

Related info:

Adding a Server Running Windows Server 2008 to a Windows Small Business

Server 2003 Network

http://technet.microsoft.com/en-us/library/cc708131.aspx

I hope that helps someone else doing this.

As an aside if you are an SBSC and are not using the SBSC Newsgroups for support you are doing yourself and your clients a great disservice.