I've loaded 600+ million row point layers and very large polygon layers that qualified as
"extremely big," but I think your problem is that the geometries are "extremely complex."
So the first question would have to be: Is there any way to can generalize the layer, and
limit times you need to use it at full detail?
I've found that point-in-polygon performance with a 1:100k country boundaries layer is
improved by an order of manitude or three when a 5 degree grid is intersected with it,
chunking complex polygons (Russia, Canada,..) resulting in ~2000 polygons (vice 167).
There are two types of polygon parts, exterior rings and interior rings. You'll often get
better performance from drawing hundreds of simple polygons as multi-part (exterior)
polygons, but complex interiors often are processed faster with single exterior parts.
While there may be limitations to ST_GEOMETRY storage, they'll mostly be in regard to
row-chaining of the blocks containing the pages storing each geometry.
- V