The slowness we see with big models with lots of tools and parameters is caused by the need for ModelBuilder to validate all the parameters for all the tools. This validation is very helpful when putting together a workflow but when many tools and parameters (including those in nested models) are there, it requires a LOT of validation, and, well, that takes time.
I disagree that it is all or nothing. A middle way I have found when I am running into parameter validation problems or very slow validation is to take some of the workflow and convert it to a Python function inside the Calculate Value tool. This hides that workflow from ModelBuilder's slow (and sometimes buggy) parameter validation.
If the tool has a extensive, complex input like the ones Tim is working with, I'll use model builder to test to get the parameters I want, then export model to script to get the python code written for me. Then I can use that code, either in Calculate Value or in a script tool.
However, from where this thread went it is obvious Tim does need a .pyt for this job so he can do the value table thing. But he figured out a lot of his workflow using ModelBuilder!