Update:
As suggested above, for me removing the gp history from db worked wonders.
We also have this issue. We're currently running 10.7.1. The initial connection to the MSSQLSERVER DB (LGIM schema) takes 12 seconds. After that, opening a feature dataset takes an additional 7 seconds. It isn't a show stopper but the problem didn't exist in the 10.3 era. I've used UNC and DNS in my connections and there was no difference. I also tried compressing and rebuilding indices. That was interesting because the initial connection actually took 2 seconds longer but opening a feature dataset took 2 seconds less, so it was a wash. QGIS connects relatively quickly and is opening each dataset from the get go in about the same amount of time arc takes to make the initial connection. SSMS opens the database in under a second. I suspect it isn't a database issue. It is something that was introduced in later Arc versions (maybe at 10.5). Maybe a look at release notes between 10.3 and 10.5 might provide some insight. IDK.
Further testing showed that if you recursively list all feature classes in the database (regardless of whether or not they reside in a feature dataset) using arcpy it takes 41 seconds. So apples to apples, QGIS is connecting to these twice as quickly and SSMS is exponentially faster. I'd be kind of interested in seeing a comparison between 10.3 and 10.7. I might be curious enough to try it. I'll post results here if I do. The takeaway for me is that it's looking more and more like Arc is responsible for the delay.
Here's a script to test for yourself. The first time you run it, it will take awhile. Run it again right after that. It will run twice as fast because the initial handshake has already been made. At the very least, you'll be able to quantify the initial connection lag for your specific DB so you'll have a baseline.
import os, arcpy, time
gdb = r'%AppData%\ESRI\Desktop10.7\ArcCatalog\yourconnection.sde'
start = time.time()
arcpy.env.workspace = gdb
datasets = arcpy.ListDatasets(feature_type='feature')
datasets = [''] + datasets if datasets is not None else []
for ds in datasets:
for fc in arcpy.ListFeatureClasses(feature_dataset=ds):
path = os.path.join(arcpy.env.workspace, ds, fc)
print(path)
end = time.time()
elapsed = (end - start)
print(elapsed)