I have a small OGC building dataset (~135k buildings) that I've extruded in a local scene. The navigation and display shows no difference to Pro's usual behavior with smaller data sets, yet it keeps popping up its annoying "GPU resources exceeded" message (on top of everything - at least stick to your own program window).
I've monitored the GPU usage with a couple of programs (GPU-Z for example, see screenshot) and none shows loads reaching even half of the card's capabilities. The card's drivers are up to date and it can handle other visually far more complex applications - is Pro's graphics engine simply not able to utilize the hardware properly?
Make sure drivers are updated from the manufacturer and...
Switch between video drivers DirectX and OpenGL to see if one behaves better than the other on your machine
http://pro.arcgis.com/en/pro-app/help/mapping/map-authoring/display-options.htm
Looking at the screenshot, it shows 3GB (video)memory usage. I wonder if there is an issue with Pro not seeing (or not getting properly reported by the video driver) what the actual available memory is, which should be 8GB for the GTX 1080. At least doing some searches on the internet seems to show this as quite a common problem with other software as well, not properly detecting available video memory...
What happens if you stay just below the 3GB mark with a smaller dataset?
Nah, the VRAM can go over 3 easily. After some more resource monitoring I suspected the cache being the real culprit here and let the project run overnight with a macro slowly sweeping across the scene. The cache is now 8+ GB and so far the error hasn't popped up again, so I guess this is the actual issue.