topic Memory leak makeNetCDFRasterLayer in Python Questions

Memory leak makeNetCDFRasterLayer

RobertGaugl — Mon, 12 Dec 2016 07:43:57 GMT

Hi!

I have to go through 30 years of Solar Incoming Surface Radiation Data and for each hour I have one NetCDF v3 (.nc) file, so I have 262800 files to go through.

I made a little Python script (see attachment) which loops through all these files and makes a Raster Layer from those files with "MakeNetCDFRasterLayer_md" and then I use "ZonalStatisticsAsTable_sa" with "mean" as the statistics type and a shape file with the NUTS statistical regions of Germany to get the mean Solar Radiation of each NUTS region.

My problem is, that I can see in the Task Manager that the RAM usage keeps increasing with each loop and when it reaches about 990.000K ArcMaps can't process any more NetCDF files. Then I have to restart ArcMaps and restart my script. Because it reaches this state after it went through about 680 files, I would have to restart ArcMaps 387 times to go through all my files!

As far as I tested it, the memory leaks with the "MakeNetCDFRasterLayer_md", altough I use "Delete_management" to delete the Raster Layer from the memory.

I also tried to write the Raster Layer to disk, but the memory just keeps leaking (and makes the whole process a lot slower).

I also tried to use the ModelBuilder with the "Iterate Files" function and MakeNetCDFRasterLayer, but I have the same problem there as well.

I'm using ArcMaps 10.4.1.

Maybe someone can help me.

Thanks,

Robert

Re: Memory leak makeNetCDFRasterLayer

DanPatterson_Retired — Tue, 13 Dec 2016 03:10:20 GMT

Are the results of the process being added to ArcMap as you are going? have you run the process outside of an arcmap session with the same results?

Re: Memory leak makeNetCDFRasterLayer

RobertGaugl — Tue, 13 Dec 2016 12:44:43 GMT

I made a workaround:

I have a main Python script, which opens my second Python script as a subprocess. The second Python script goes through 200 files and then gets closed (so I never hit a RAM limit or some other limit). The main Python script reopens the second Python script until there are no files left anymore.

Re: Memory leak makeNetCDFRasterLayer

DanPatterson_Retired — Tue, 13 Dec 2016 12:49:44 GMT

Good. That probably means there are *loc files being created and a del statement is of no use. Next time you run your script see if that is the case by monitoring your folder... unless of course you are working in a geodatabase where this won't be possible and python has little or no control over what goes on there