Wait a sec, so do you use both Standard switch and Distributed switch?
Standard switch for ESXi Management and dvSwitch for vCSHB?
Also, could Primary server ping Secondary management IP? Is it only Channel Network they can't communicate?
Number of physical network adapters won't matter as long as they are configured correctly.