RetinaNet läuft nur auf einem CPU-Socket und einer GPU statt auf 2 CPUs

Question

Ich teste derzeit mein Setup, um zu sehen, wie es für Deep Learning performt.<\/P> Mein aktuelles Setup umfasst:<\/P> * HP Proliant ML350 GEN9<\/LI> * CPUs: 2x E5-2698 v3<\/LI> * GPU: Tesla P40<\/LI> * 64 GB RAM<\/LI> * 2x NVMe 1TB<\/LI><\/UL>Ich versuche, diesem Kurs zu folgen: "Detection of electric utility features and vegetation encroachments from satellite images using deep learning | ArcGIS API for Python."<\/P> Detection of electric utility features and vegetation encroachments from satellite images using deep learning | ArcGIS API for Python<\/A><\/P>Während des Trainingsprozesses ist mir aufgefallen, dass es scheinbar nur auf 1 CPU läuft statt auf 2 und sich somit auf 50 % der CPU-Kapazität beschränkt.<\/SPAN> <\/span><\/P> Change Detection of Buildings from Satellite Imagery - Preview (arcgis.com)<\/A>,<\/P>Einige Wochen zuvor habe ich den Kurs "Change Detection of Buildings from Satellite Imagery - Preview" auf arcgis.com gemacht. Zu diesem Zeitpunkt hatte ich nur eine CPU verfügbar und bemerkte ebenfalls nur eine 50%ige CPU-Auslastung.<\/P> <\/DIV><\/span><\/P>Gibt es eine Einstellung im Jupyter Notebook, die ich anpassen muss? Oder sollte ich die Befehle anders ausführen, um die meisten Systemressourcen zu nutzen?<\/P>

MarcoBoeringa · Answer

Welcome to the wonderful world of "processor groups", "CPU sets", "processor affinity", NUMA (Non Uniform Memory Access - a technique to access local or remote RAM in a non-uniform manner, with local faster RAM being preferred) and other processor scheduling details you likely never heard of, but that do start to creep in on your work once you hit large numbers of logical processors on multiple socket server systems.

As I also learned the hard way, software doesn't just magically use all logical processors on your system. In fact, most software out there has never been designed to take full advantage of >=64 logical processor systems, simply because there were none out there up until very recently.

One minor introduction might be this read of Bitsum:

https://bitsum.com/general/the-64-core-threshold-processor-groups-and-windows/

There is a lot more to read out there about these subjects, but I can tell you it doesn't necessarily provide answers or working solutions.

However, as to practical advice, you might attempt to:

- Disable hyperthreading

- Disable NUMA

in your system's BIOS. This will create in your case a system that is more or less logically seen to be as a "single socket" 32 core system without hyperthreading, with both CPUs having equal and predictable access times to local and remote RAM memory sticks. In this configuration, it is more likely all cores will be used, but they will have (slightly) higher RAM latency / access times.

While both of these options are designed to potentially run your system faster, they do not necessarily do so, and may in fact reduce the overall performance of specific workflows if those workflows need access to all physical cores. Note that disabling NUMA does not seem to be recommended on older multi-socket AMD systems from what I have read so far, as the data links between different CPU's memory seem to have far to big latency, actually harming overall performance, but this seems to be less of an issue with INTEL systems. I don't know if that is still an issue with modern AMD systems (likely not).

If you do disable these options in the BIOS of your system, I strongly recommend you to run a controlled performance test of your workflow before and after the change, to see if one or the other configuration is faster, it may turn out that disabling the options is actually harmful (but in my case it was beneficial to disable them).