Do you only have building points or do you have access to building footprints as polygons?. What I see from you screen shot of you DTM is that it includes higher values where the buildings have been filtered out.
You have to keep in mind that with Drone2Map the DSM is based on "measurements" and the DTM is a derivative (hence the lower resolution with a factor 5). Most likely you would like to use the elevation at the streets as the DTM value, but currently inside the buildings values increment (lighter areas).
In case you have access to building footprints you could:
- convert them to raster (using the ID)
- use expand to make the areas a bit larger
- use zonal statistics to define the minimum height inside each expanded footprint
- combine that result to replace the DTM values
- obtain the object height by calculating DSM - corrected DTM