Introduction
Python 2.6, which includes the multiprocessing module, was first introduced in ArcGIS products with the release of ArcGIS 10.0 (Domestic release date: October 22, 2010). Since then, even on Esri's official blog in the US,Python Multiprocessing – Approaches and Considerations (August 2011),Multiprocessing with ArcGIS – Approaches and Considerations (Part 1): September 2012articles about multiprocessing have been published, and in recent years, many developers from around the world have gathered at the Esri Developer Summit whereParallel Python: Multiprocessing with ArcPy: March 2017 video,Vector and Raster Multiprocessing with ArcPy: March 2018 videohave been presented as technical sessions.
Multiprocessing
ArcGIS Pro Operating EnvironmentWith the spread of high-spec hardware that can be seen in ArcGIS Pro (64-bit OS standard, recommended CPU quad-core, optimal memory over 16GB), the amount of data used for analysis and processing has also dramatically increased, and efficient analysis and processing are still required today.
In this context, this time I would like to introduce multiprocessing processing with ArcPy and Python, which is established as an extension method for the ArcGIS platform, along with key points of implementation and sample coding patterns.
For actual coding in various processes, please refer to the related information provided and try writing code yourself. By referring to this article and grasping the points of parallel processing implementation, I hope it will help you successfully incorporate it into your current workflow.
Best Practices
In various materials shown in related information, as best practices for multiprocessing processing using ArcPy and Python, the following points are mentioned.
- To maintain the results of temporary layers and operate at high speed,using in_memory workspace is recommended.
- File Geodatabase (Reference:Overview of Geodatabase), avoid writing to GRID raster.
These data formats can cause issues such as file locks or synchronization problems. This is because File Geodatabase and GRID raster do not support simultaneous writing; in other words, only one process can write at a time. When trying to modify feature classes developed in ArcGIS with ArcPy, this problem occurs. Furthermore, if you hold a File Geodatabase and try to write to multiple feature classes simultaneously, this problem expands. Even if all feature classes exist independently, writing to feature classes within a File Geodatabase can only be done one feature class at a time.
- Use a 64-bit Python execution environment for script execution.
From ArcGIS Pro 1.4 or later, or ArcGIS Server 10.5 or later, or if Background Geoprocessing (64-bit) is installed on ArcGIS Desktop 10.5 or later, use a 64-bit Python execution environment. The 64-bit execution environment can solve problems related to memory limits (4GB) of 32-bit environments when handling large data. Especially for implementation, caution is needed regarding the use of File Geodatabases mentioned in point 2. At least when writing processes for each process are described in worker functions, you should avoid using the same File Geodatabase but instead use separate File Geodatabases for each process. However, it is possible to use master functions that merge multiple shape files or individual File Geodatabases into a single source by using those worker functions.
* One approach is to use an enterprise geodatabase as a write destination from multiple processes or to use worker functions as write destinations.
Sample of Multiprocessing Processing
This sample contains multiple shapefiles under folders for each of the 47 prefectures in Japan's 47 prefectures folder. After creating a File Geodatabase for each prefecture under these folders respectively, it is a sample code that converts them into respective feature classes. The implementation point is that as written in the above best practices, writing to the File Geodatabase is locked during writing so that "Write code so that one prefecture folder's shapefile corresponds to one prefecture folder's File Geodatabase."
Let's look at the actual coding pattern.
1) Worker function: batch_convert
As coding pattern, first write the worker function similarly to normal processing with ArcPy. What should be noted here is that as written in best practices, processing by this worker function corresponds to processing by process; if access from multiple processes (multiple worker functions) requires writing destinations, either use enterprise geodatabases or avoid using the same File Geodatabase.
def batch_convert(inws, outws):
'''
Processing executed by one process (worker function):
Because writing to a File Geodatabase cannot be done by multiple processes simultaneously,
Multiple shapefiles under folders for one prefecture folder
Are converted into multiple feature classes under one prefecture folder's File Geodatabase respectively.
Description
'''
#print("Conversion: {0} => {1}\n".format(inws,outws))
if not arcpy.Exists(outws):
outfolder=os.path.dirname(outws)
foldername=os.path.basename(outws)
arcpy.CreateFileGDB_management(outfolder,foldername,"CURRENT")
arcpy.env.workspace = inws
fcs = arcpy.ListFeatureClasses()
for fc in fcs:
l=len(fc)
infc=os.path.splitext(fc)[0]
# Change feature class name by padding zeros to four digits starting from zero index
# Example) P101_CITY.shp → P0101_CITY
newfc = fc[:1]+fc[1:l-4].zfill(4)
# If feature class already exists then Append
# If it does not exist then FeatureClassToFeatureClass
<SPAN class="keyword token">if</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>Exists<SPAN class="punctuation token">(</SPAN>os<SPAN class="punctuation token">.</SPAN>path<SPAN class="punctuation token">.</SPAN>join<SPAN class="punctuation token">(</SPAN>outws<SPAN class="punctuation token">,</SPAN>newfc<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
outfc<SPAN class="operator token">=</SPAN>os<SPAN class="punctuation token">.</SPAN>path<SPAN class="punctuation token">.</SPAN>join<SPAN class="punctuation token">(</SPAN>outws<SPAN class="punctuation token">,</SPAN>newfc<SPAN class="punctuation token">)</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>Append_management<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">,</SPAN>outfc<SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">else</SPAN><SPAN class="punctuation token">:</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>FeatureClassToFeatureClass_conversion<SPAN class="punctuation token">(</SPAN>fc<SPAN class="punctuation token">,</SPAN>outws<SPAN class="punctuation token">,</SPAN>newfc<SPAN class="punctuation token">)</SPAN>
<SPAN class="keyword token">del</SPAN> fcs
<SPAN class="keyword token">return</SPAN> <SPAN class="string token">" Conversion complete: {0}"</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>outws<SPAN class="punctuation token">)</SPAN>
<SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span></Span><Span> </span ><span >< < EM O J I _ 3 2 > ></span ></span >
2) Worker function wrapper: multi_run_batch_convert
When specifying a worker function from a multiprocessing process, if the worker function takes a single argument it is unnecessary, but if multiple arguments need to be passed, using a wrapper function is convenient.
<span class="keyword token">def</span> <span class="token function">multi_run_batch_convert</span>(args)<span class="punctuation token">:</span>
<span class="string token">'''
Wrapper for batch_convert:
A wrapper to pass multiple arguments to the execution process
'''</span>
<span class="keyword token">return</span> batch_convert<span class="punctuation token">(*</span>args<span class="punctuation token">)</span><<EMOJI_33><span class="line-numbers-rows"><span><<EMOJI_34></span><span><<EMOJI_35></span><span><<EMOJI_36></span><span><<EMOJI_37></span><span><<EMOJI_38></span><span><<EMOJI_39></span></span>
3) Multiprocessing process handling: exec_batch_convert
Two arguments (inws, outws) passed to the worker function are converted into a list called params.
Then,
pool = multiprocessing.Pool(cpu_cnt) is used to create a pool with the specified number of CPUs,
and results = pool.map(multi_run_batch_convert , params) is specified.
First argument: The name of the worker function to process (in this case, the wrapper function) multi_run_batch_convert
Second argument: The list of parameters params
<span class="keyword token">def</span> <span class="token function">exec_batch_convert</span>(infolder<span class="punctuation token">,</span> outfolder)<span class="punctuation token">:</span>
<span class="string token">'''
Processing with multiprocessing (master function):
'''</span>
<span class="keyword token">try</span>:
start = datetime.datetime.now()
<span class="keyword token">print</span>("-- Start: Multiprocess_ShapefileToFeatureClassWithRename --:", start)
cpu_cnt = multiprocessing.cpu_count()
arcpy.env.workspace = infolder
inwss = arcpy.ListWorkspaces("*","Folder")
<span class="comment token"># Convert parameters to each process as a list</span>
params = []
<span class="keyword token">for</span> inws <span class="keyword token">in</span> inwss:
param1 = inws <span class="comment token"># Prefecture folder (contains shapefiles)</span>
gdbname = "{0}.gdb".<span class="punctuation token">format(</span>os.path.basename(inws)<span class="punctuation token">)</span>)
param2 = os.path.join(outfolder, gdbname) <span class="comment token"># Prefecture file geodatabase path</span>
params.append((param1,param2))
<span class="keyword token">if</span> len(inwss) < cpu_cnt: <span class="comment token"># If the number of processing folders is less than CPU cores, no need to start extra processes</span>
cpu_cnt = len(inwss)
pool = multiprocessing.Pool(cpu_cnt) <span class="comment token"># Create python processes equal to CPU count</span>
results = pool.map(multi_run_batch_convert, params) <span class="comment token"># Processes are executed sequentially by dividing tasks among them</span>
pool.close()
pool.join()
<span class="comment token"># Output processing results from each process</span>
<span class="keyword token">for</span> r <span class="keyword token">in</span> results:
<span class="keyword token">print</span>(r)
<span class="comment token"># Merge feature classes within file geodatabases by prefecture folder</span>
<span class="comment token"># Write here if creating a nationwide version</span>
<span class="comment token"># ~ Omitted ~</span>
fin = datetime.datetime.now()
print "-- Finish: Multiprocess_ShapefileToFeatureClassWithRename --:", fin
print " Elapsed time:", fin - start
except:
print traceback.format_exc(sys.exc_info()[2])<SPAN class="line-numbers-rows"><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN><SPAN></SPAN></span>
Sample code of the completed version
Implemented according to the coding pattern up to above, change infolder and outfolder in the Python script (Multiprocess_Sample.py) to your own environment and run from the Python execution environment, then data conversion will be performed in multiprocessing.
#coding:cp932
import arcpy,os
import sys
import multiprocessing
import datetime
import traceback
reload(sys)
sys.setdefaultencoding('cp932')
def multi_run_batch_convert(args):
'''
Wrapper for batch_convert:
A wrapper to pass multiple arguments to multiprocessing execution
'''
return batch_convert(*args)
def batch_convert(inws, outws):
'''
Processing executed in one process (worker function):
Because writing to file geodatabase cannot be done with multiple processes,
multiple shapefiles under one prefecture folder,
convert multiple feature classes under one prefecture file geodatabase folder,
as described here.
'''
#print("Convert: {0} => {1}\n".format(inws,outws))
if not arcpy.Exists(outws):
outfolder = os.path.dirname(outws)
foldername = os.path.basename(outws)
arcpy.CreateFileGDB_management(outfolder, foldername, "CURRENT")
arcpy.env.workspace = inws
fcs = arcpy.ListFeatureClasses()
for fc in fcs:
l = len(fc)
infc = os.path.splitext(fc)[0]
# Change feature class name by zero-padding 4 digits at position 0
# Example) P101_CITY.shp → P0101_CITY
newfc = fc[:1] + fc[1:l-4].zfill(4)
# If the feature class already exists, Append it.
# If it does not exist, use FeatureClassToFeatureClass.
if arcpy.Exists(os.path.join(outws, newfc)) :
outfc = os.path.join(outws, newfc)
arcpy.Append_management(fc, outfc)
else:
arcpy.FeatureClassToFeatureClass_conversion(fc, outws, newfc)
del fcs
return " Conversion complete: {0}".format(outws)
def exec_batch_convert(infolder, outfolder):
'''
Processing with multiprocessing (master function):
'''
<SPAN class="keyword token">try</SPAN><SPAN class="punctuation token">:</SPAN>
start<SPAN class="operator token">=</SPAN>datetime<SPAN class="punctuation token">.</SPAN>datetime<SPAN class="punctuation token">.</SPAN>now<SPAN class="punctuation token">()</SPAN>
<SPAN class="keyword token">print</SPAN> <SPAN class="string token">"-- Start: Multiprocess_ShapefileToFeatureClassWithRename --:"</SPAN><SPAN class="punctuation token">,</SPAN>start
cpu_cnt<SPAN class="operator token">=</SPAN>multiprocessing<SPAN class="punctuation token">.</SPAN>cpu_count<SPAN class="punctuation token">()</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>workspace <SPAN class="operator token">=</SPAN> infolder
inwss <SPAN class="operator token">=</SPAN> arcpy<SPAN class="punctuation token">.</SPAN>ListWorkspaces<SPAN class="punctuation token">("*","Folder")</SPAN>
<SPAN class="comment token"># Convert parameters to a list for each process</SPAN>
params<SPAN class="operator token">=</SPAN><SPAN class="punctuation token">[]</SPAN>
<SPAN class="keyword token">for</SPAN> inws <SPAN class="keyword token">in</span> inwss<SPAN class="punctuation token">:</span>
param1<SPAN class="operator token">=</span>inws <span class="comment token"># Prefecture folder (contains shapefiles)</span>
gdbname<SPAN class="operator token">=</span><span class="string token">"{0}.gdb"</span><span class="punctuation token">.format(os.path.basename(inws))</span>
param2<SPAN class="operator token">=</span>os.path.join(outfolder,gdbname) <span class="comment token"># Prefecture file geodatabase base path</span>
params.append((param1,param2))
<span class="keyword token">if</span> len(inwss) < span class="operator token"><</span> cpu_cnt<span class="punctuation token">:</span> <span class="comment token"># If the number of processing folders is less than CPU cores, no need to start extra processes</span>
cpu_cnt = len(inwss)
pool <span class="operator token">=</span> multiprocessing.Pool(cpu_cnt) <span class="comment token"># Create python processes equal to CPU count</span>
results = pool.map(multi_run_batch_convert,params) <span class="comment token"># Processes run sequentially assigned by division</span>
pool.close()
pool.join()
<span class="comment token"># Output processing results from each process</span>
<span class="keyword token">for</span> r <span class="keyword token">in</span> results<span class="punctuation token">:</span>
<span class="keyword token">print</span>(r)
<span class="comment token"># Merge feature classes inside the file geodatabase for each prefecture folder</span>
<span class="comment token"># Write here if creating a nationwide version</span>
<span class="comment token"># ~Omitted~</span>
fin = datetime.datetime.now()
<span class="keyword token">print</span> <span class="string token">"-- Finish: Multiprocess_ShapefileToFeatureClassWithRename --:"</span>,fin
<span class="keyword token">print</span> <span class="string token">" Elapsed time:"</span>, fin-start
<span class="keyword token">except</span><span class="punctuation token">:</span>
<span class="keyword token">print</span> traceback.format_exc(sys.exc_info()[2])
<span class="keyword token">def</span> setup_batch_convert():
<span class="string token">'''
Set execution parameters from command prompt:
Source folder containing prefecture-wise shapefiles: infolder
Example)
|-Shapefiles
|- 01_Hokkaido
|-P101_CITY.shp
|-L102_RIVER.shp
・ ・ ・ ・ ・
|- 02_Aomori_Prefecture
|- 03_Iwate_Prefecture
・ ・ ・ ・ ・
Destination folder for prefecture-wise file geodatabases: outfolder
Example)
|-Filegdbs
|- 01_Hokkaido.gdb
|-P0101_CITY
|-L0102_RIVER
・ ・ ・ ・ ・
|- 02_Aomori_Prefecture.gdb
|- 03_Iwate_Prefecture.gdb
・ ・ ・ ・ ・
'''</span>
infolder = r"E:\DATA\Shapefiles" <span class="comment token"># Source folder containing prefecture-wise shapefiles</span>
outfolder = r"E:\DATA\Filegdbs" <span class="comment token"># Destination folder for prefecture-wise file geodatabases</span>
exec_batch_convert(infolder,outfolder)
<Span class='keyword'>if </Span> __name__ == '__main__':
setup_batch_convert<SPAN class="punctuation token">(<\/SPAN><SPAN class="punctuation token">)<\/SPAN><SPAN class="line-numbers-rows"><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN><\/SPAN><SPAN >< < EM O J I _ 3 2 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 3 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 4 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 5 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 6 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 7 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 8 > > < \/ S P A N > < S P A N > < < E M O J I _ 3 9 > > < \/ S P A N > < S P A N > < < E M O J I _ 4 0 > > < \/ S P A N > < S P A N > < < E M O J I _ 4 1 > > < \/ S P A N > < S P A N > < < E M O J I _ 4 2 > > < \/ S P A N > < S P A N > < < E M O J I _ 4 3 > > < \/ S P A N > < S P A N > < < E M O J I _ 4 4 > > < \/ S P A N > < S P A N > < < E M O J I _ 4 5 > > < \/ S P A N ><S PAN ><<< EMOJ I _46 >> </ SPAN >< SPAN ><<< EMOJ I _47 >> </ SPAN >< SPAN ><<< EMOJ I _48 >> </ SPAN >< SPAN ><<< EMOJ I _49 >> </ SPAN >< SPAN ><<< EMOJ I _50 >> </ SPAN >< SPAN ><<< EMOJ I _51 >> </ SPAN >< SPAN ><<< EMOJ I _52 >> </ SPAN >< SPAN ><<< EMOJ I _53 >> </ SPAN >< SPAN ><<< EMOJ I _54 >> </ SPAN >< SPAN ><<< EMOJ I _55 >> </ SPAN >< SPAN ><<< EMOJ I _56 >> </ SPAN >< SPAN ><<< EMOJ I _57 >> </ SPAN >< SPAN ><<< EMOJ I _58 >> </ SPAN >< SPAN ><<< EMOJ I _59 >> </ SPAN >< SPAN ><<< EMOJ I _60 >> </ SPAN >< SPAN ><<< EMOJ I _61 >> </ SPAN >< SPAN ><<< EMOJ I _62 >> </ SPAN >< SPAN ><<< EMOJ I _63 >> </ SPAN >< SPAN ><<< EMOJ I _64 >> </ SPAN >< SPAN ><<< EMOJ I _65 >> </ SPAN >< SPAN ><<< EMOJ I _66 >> </ SPAN >< SPAN ><<< EMOJ I _67 >> </ SPAN >< SPAN ><<< EMOJ I _68 >> </ SPAN >< SPAN ><<< EMOJ I _69 >> </ SPAN >< SPAN ><<< EMOJ I _70 >> </ SPAN >< SPAN ><<< EMOJ I _71 >> </ SPAN >< SPAN ><<< EMOJ I _72 >> </ SPAN >< SPAN ><<< EMOJ I _73 >> </ SPAN >< SPAN ><<< EMOJ I _74 >> </ SP AN ></ SPA N ></ SPA N >
Python script execution in a 64-bit environment
Note: As written in the best practices, please run from a 64-bit Python environment. Examples of Python script execution include ArcGIS Desktop and ArcGIS Pro.
Example) ArcGIS Desktop 10.6.1
>C:\Python27\ArcGISx6410.6\python.exe Multiprocess_Sample.py
Example) ArcGIS Pro 2.2
>"c:\Program Files\ArcGIS\Pro\bin\Python\scripts\propy.bat" Multiprocess_Sample.py
Reference blogs and websites
・Using Arcpy with multiprocessing – Part 1・Using Arcpy with multiprocessing – Part 2・Using Arcpy with multiprocessing – Part 3・Python Multiprocessing – Approaches and Considerations (August 2011: ArcGIS 10.0 environment)・Multiprocessing with ArcGIS – Approaches and Considerations (Part 1) (September 2012: ArcGIS 10.1 environment)・Material from the John A. Dutton e-Education Institute at Pennsylvania State University GEOG 489 - Advanced Python Programming for GIS - 1.6.6 ArcPy multiprocessing examples
Reference videos
・Parallel Python: Multiprocessing with ArcPy (Esri Developer Summit March 2017)・Vector and Raster Multiprocessing with ArcPy (Esri Developer Summit March 2018)
