I have pretty much the same setup. 3 layers and about the same number of objects in one.
I load the big one using an async task every time the view changes at all (including every pan operation). The dataset I am using contains 110,000 items so I only display when zoomed in past a threshold. So I may dump and reload 1,000 or more graphics features on every pan. I just dont think it was made for that kind of duty.
Right now I just deal with the crashes, but at some point i will revisit the underlying issue. It would be awesome if an ESRI person would run it and show the crash point in their debugger, but that is asking a bit much for them to do that. Everyone would ask then.
It is crashing in native code, so there isnt much you can do with it for troubleshooting or pinpointing. I tried to run it in the emulator to get the actual process dump when it crashes, but my computer cant run the emulator fast enough to make it happen.