Try a standard 32 bit RGB as displacement map ( heighmap ).
Also, when you have the terrain selected, check the Subdivision U and V settings in the Inspector.
Btw. What's the polycount of the original MAX terrain and what's the final rendered displacement map ?
Lemme know ..
Please read this ..
--> CityEngine does not handle textures larger than 4000x4000px ( not 4K ). That was decided quite some time ago, since usually people use many ( sometimes hundreds ) of textures, and they all must fit in the GPU memory. Exchanging lower res textures with the final resolution can easily be done, so I'd not recommend to use anything bigger than 2K in general.
Now, the terrain is obviously a special case, since you want to have a certain resolution which drives the resulting geometry. But also here, I'd recommend to rather use an approximated terrain and replace this before render time with the full resolution.
--> Sadly, CityEngine is a bit limited still 'on the terrain side of things'.
Does that help to understand the situation ?
Maybe if you describe a bit what you guys and girls are planning to do, I can give some more pointers.
Let me know ..