get the widest possible element/core you can get. If you can't get wide cores, get it made with multiple rows. You basically want as much or more area in the tubes than the inlet/outlet size ... more is better 'cos you'll dissipate more heat 'cos the liquid will flow through slower.
http://www.crankshaftcoalition.com/wiki/Bu..._cooling_system
The custom unit you have isn't working well? Maybe there's not enough core area for the coolant to flow through. As far as I know, there' no baffles or anything within, just two tanks and the core in the middle (that is unless you purposely made a serpentine crossflow system)