I decided I needed to re-install my lab environment. I wanted to keep all of the firewalls on during the install process and only open the ports that are actually needed. I installed SQL using a named instance as many customers use a SQL 2008 cluster.
After I installed the SCOM database on the SQL 2008 server with all firewalls on. I created a firewall rule to let port 1433 allow connections. As specified in the Supported Configurations doc
Root management server 1433 —> OperationsManager database
I also setup a firewall rule to allow port 1434 back to the RMS server from the SQL Instance Server. (Also in the guide)
Root management server 1434 UDP < — OperationsManager database
I start the install of SCOM to the RMS server. I unchecked Database as my database is already install the on the SQL instance.
I typed in my SC Database Instance Name and clicked Next
But I got this error “Setup cannot location the SC database”
In the SCOM setup I clicked back and then next.
I checked the firewall logs in %systemroot%\system32\Logfiles\Firewall\pfirewall.log and it looks like UDP port 1434 is being dropped
In the SCOM setup I click back and next again.
Once again same failure. “Setup cannot location the SC database”
Back to the firewall logs. It now needs TCP port 62756 (Not in the guide)
I create another rule on the SQL server to enable TCP port 62756
After that rule is enabled I am able to continue on a install SCOM successfully with all of the windows firewalls still on.