|
POST
|
Hi all, I wrote a Python script where I use multiprocessing.Pool.map to run a function on different parts of a large dataset in parallel (read only, results are stored in a separate directory for each process). The python sub-processes produce the expected results but they never close down so I end up with many python.exe processes running forever (even after the "master" script has finished). Each of these processes consume around 100 MB of RAM and I think I will run out of memory when I run this script on a full set of input parameters. I've been running it with 6 CPU cores but I end up with many more python processes running. How come they are not shutting down? Have you seen something like that before? I'm using ArcGIS 10.3.1 with ArcGIS Pro 1.1 and Python 3.4 (64 bit) installed for stand alone scripts. While my real script is much more complicated, this is essentially what I am doing: import arcpy
import multiprocessing
def worker_function(pars)
"""Function to be run in parallel"""
wd = pars.get('wd')
# ... do some work with spatial analyst ...
return {"result": ["spam", "eggs"]}
# parameter sets is a list of dictionaries of primite Python strings
def main():
parameter_sets = [
# ...
{
"flow_direction_raster": '...',
"weight_raster": '...',
"output_raster": '...',
"relative_to": '...',
"extent": '...',
"wd": '...'
}
# ...
]
pool = multiprocessing.Pool(6)
results = pool.map(worker_function, parameter_sets)
return results
if __name__ == "__main__":
main()
... View more
03-30-2016
08:32 AM
|
0
|
4
|
28745
|
|
POST
|
Good point, Curtis. String substitution aka string formatting can make the whole code much more readable. Especially considering all the options the format method offers. F.
... View more
03-29-2016
03:54 PM
|
0
|
1
|
1874
|
|
POST
|
Haha, Thank you very much, mister Xander Bakker, that is the kind of "slide-of-hand-arcgis-trick" I was looking for. Nicely done, I'll go with that. By the way, dropping the 11 digits after the decimal place of each coordinate reduced the size of the output text file from about 150 MB to about 50 MB! To clarify the coordinate system, this was in British National Grid (EPSG:27700), i.e. projected in metres. Thanks everyone, Filip.
... View more
01-19-2016
01:28 AM
|
0
|
0
|
2600
|
|
POST
|
Hi, I have a polygon feature class and I want to produce a text file where each feature is a row with an ID and the polygon geometry formatted as Well Known Text (WKT). To start with I am looping though the features and I use the .WKT property to produce the result. My problem is that the WKT is often formatted with coordinates reported to 12 decimal places. I need to round the coordinates to the nearest meter. Ideally I want to be able to do this with other types of geometries so I'd prefer something else then looping over the rings and vertices. Is there a way to do this? I tried to set arcpy.env.XYTolerance and arcpy.env.XYResolution to 1.0 but that had no effect. Any hints? Filip.
... View more
01-18-2016
07:29 AM
|
0
|
4
|
5208
|
|
POST
|
My tile cache is finally fully functional again.This is what worked for me: First, I could not delete any of the cache files in Windows Explorer or using the Delete Map Server Cache tool. So I restarted the machine. Then I used Delete Cache Tiles to get rid of all the tiles. Luckily, the tool worked this time, but I had to define the caching in service properties again and I selected "update cache manually". Then I used Manage Map Server Cache Tiles tool to generate the cache from scratch. Fortunately my tile cache is quite small so it didn't take long to re-generate it. Then I installed ArcGIS 10.2.2 for Server Map Cache Consumption Patch which I believe is the one Michael Volz had in mind. I haven't had a chance to try it again just yet, but hopefully I won't run into the same issues next time. Many thanks for all your help. Filip.
... View more
01-11-2016
06:44 AM
|
1
|
0
|
1748
|
|
POST
|
The issue I am dealing with at the moment appears to be similar to this: Manage Map Cache Tile: Errors at 10.2.2 When I left "Update cache automatically" ticked while overwriting the service, I can then see my CPUs work harder. Once the CPU load drops again (caching has finished) and I preview the service, some tiles were obviously missing (HTML Error 404). I then tried to re-create the tiles using Manage Map Cache Tile tool but it errors out saying: Output failure, error string = Error moving bundle Failed to cache extent:... and then it repeats for each process (i.e. each CPU that worked on it). Filip.
... View more
01-08-2016
07:06 AM
|
1
|
3
|
3969
|
|
POST
|
Jennifer's answer is right. I found it to be the easiest way to update data sources or other properties of a service. See also: Overwriting a service in ArcGIS for Desktop—Documentation (10.3 and 10.3.1) | ArcGIS for Server However, things usually get complicated with tiled services. In theory it should work as the help says - if you keep "Update cache automatically", the server will be busy for some time bit it should deal with it. In practice though, I always run into difficulties with updating cached services and then spend long time fixing it. If you have any tips, please let me know. I use 10.2.2
... View more
01-08-2016
05:47 AM
|
1
|
6
|
3969
|
|
POST
|
Hello, Since many of the tools for exporting and importing metadata are not available in ArcGIS Pro, what is the recommended way of working with metadata in geoprocessing work flows? In Python 2.7 and ArcGIS for Desktop it is possible to import/export metadata using geoprocessing tools. I have been using this in my scripts that I need to migrate to Python 3.4 Specific example is the function 'meta' here: https://github.com/NERC-CEH/arcapi/blob/master/arcapi.py#L913 Kind regards, Filip.
... View more
12-07-2015
02:02 AM
|
0
|
1
|
3493
|
|
POST
|
Hi, Thank you for the suggestions. I ended up using the ExtracValuesToPoints repeatedly on in_memory feature classes and built the resulting feature class that way, renaming the RASTERVALU column each time (i.e. copying the RASTERVALUE column to a new column and deleting RASTERVALU column). Seems to work OK. F.
... View more
11-22-2015
02:05 PM
|
0
|
0
|
709
|
|
POST
|
Hello all, I need to extract values from about 30 raster datasets (each about 500 MB in size, signed 32 bit) for about 1500 points. When I use the Extract Multivalues to Points tool, it quickly consumes all my memory (16 GB) and it never finishes. I must admit I only left it running for about an hour, but that is more than the whole process took in ArcMap. It seems the tool is extremely memory intensive and I wonder if somebody has any suggestions how to get around it. An obvious option is to use the Extract Values to Points tool repeatedly. Any other ideas? Filip
... View more
11-19-2015
03:18 AM
|
0
|
2
|
2658
|
|
POST
|
Hi, What do you mean by "attribute"? The attr(name, value) method adds an attribute to the DOM node of the graphic, not to the graphic feature. So setting feature.attr('something', 123) will result in the graphic dom node to be: <path something="123" ...></path> (although I am not sure what types of elements graphics are). In order to add an attribute to the graphic feature, in the sense of adding a new field, use the setAttributes method. Filip.
... View more
11-18-2015
03:34 PM
|
1
|
3
|
2773
|
|
POST
|
Today I learned a new fact relevant to dissolving or intersecting large datasets. ArcGIS Pro has Parallel Overlay toolset with Parallel Dissolve and Parallel Intersect tools that I hope address this issue. I haven't tried these tools yet but the help looks promising. Filip.
... View more
11-18-2015
03:15 PM
|
0
|
0
|
4407
|
|
POST
|
Hi, You've made quite some progress here. In order to see the tool parameters in the tool dialog, you have to configure them in the script tool properties as it is described in Adding a script tool and in Setting script tool parameters. Make sure the order of parameters in the table matches the numbers you assigned to them in the script (0 is the first parameter). You will also likely need to use the SetParameter function by the end of your script to let ArcGIS know what values should the tool return. For example, say that the end point parameter is the third parameter of your tool and its direction is "output". In your script you will write arcpy.GetParameter(2) to access the value provided by the user and eventually arcpy.SetParameter(2, 'some value') to notify ArcGIS what the resulting value is. Hope this makes sense. Filip.
... View more
11-18-2015
02:47 PM
|
2
|
0
|
1007
|
|
POST
|
I tried this in ArcPro Python window I now am even more puzzled. When I pasted the code to the window and executed, all of the arcpy.Exists calls returned False. Even the first one so it suggested that the layer was never created. I then tried running just the MakeFeatureLayer tool and this is what I got: r = arcpy.management.MakeFeatureLayer(in_fc, lrname, w) r.getMessages() # 'Start Time: Fri Nov 13 14:30:25 2015\nSucceeded at Fri Nov 13 14:30:25 2015 (Elapsed Time: 0.03 seconds)' r.getOutput(0) # 'lr1' arcpy.Exists('lr1') # False arcpy.Exists(r.getOutput(0)) # False What is going on here?? By the way, no layer was added to the table of contents either, not even when I run MakeFeatureLayer tool manually, but that is not an issue. F.
... View more
11-13-2015
06:40 AM
|
0
|
1
|
2095
|
|
POST
|
Hi, On line 12 I delete the layer object (variable lr) and since that didn't work I added lines 13, 14 just to show all the things I tried and didn't help. I tried setting data_type parameter to 'FeatureLayer' and also 'Layer' but that didn't help either. The help suggests the data_type parameter is "read only". I am using the standalone python 3.4 that comes with ArcGIS Pro and I am executing this as a script file from windows command line. F.
... View more
11-13-2015
06:06 AM
|
0
|
1
|
2095
|
| Title | Kudos | Posted |
|---|---|---|
| 1 | 02-05-2014 04:40 AM | |
| 1 | 02-08-2015 12:49 PM | |
| 1 | 07-20-2014 12:41 PM | |
| 1 | 03-23-2017 01:48 PM | |
| 1 | 08-18-2014 04:14 PM |
| Online Status |
Offline
|
| Date Last Visited |
11-11-2020
02:23 AM
|