Select to view content in your preferred language

arcpy and pmf

5318
23
02-02-2010 10:29 AM
TedCronin
MVP Honored Contributor
Will the generation of pmf files be supported with arcpy?  I don't remember seeing anything in the help.
0 Kudos
23 Replies
MarkCederholm
Frequent Contributor
The comtypes package works just fine with 2.6.  There's an extra step if you're using ArcObjects rather than arcpy to initialize standalone licensing at 9.4(10) beta 2.  The only problem I've noticed with beta 2 is that ArcMap and ArcCatalog will bomb on closing if any pure python COM objects have been present.

I haven't tried psyco but I'll look into it. [UPDATE:  No significant performance boost - the real bottleneck in the benchmark is COM interop (as intended), and psyco apparently can't do anything for that.]

Speaking of arcpy, I spent some time working with the mapping module at beta 1, but I'd like to spend some more time on it again this week before the DevSummit.  It's a great start anyway, and I definitely plan to attend the map automation session.
0 Kudos
TedCronin
MVP Honored Contributor
Do you have any equivalent benchmark numbers for 9.4/10?
0 Kudos
MarkCederholm
Frequent Contributor
Well, I've discovered that my older numbers are no longer valid since I went from VS 2005 to 2008 (VB .NET is much worse now! :p).  Here are completely fresh scores for 9.3.1 vs. 9.4(10) beta 2:

Benchmark:  Find average area of polygon records in a file GDB
Count: 521751
Avg Area: 426300
Scores in seconds represent best of 3 runs

9.3.1 scores (seconds):
-----------------------
C++: 21 (1.0)
VBA/ArcCatalog: 22 (1.0)
VBA/Word2007: 30 (1.4)
VB .NET: 179 (8.5)
Python/comtypes: 47 (2.2)
Python/arcgisscripting: 51 (2.4)
IronPython: 62 (3.0)

Beta 2 scores (seconds):
------------------------
C++: 12 (1.0)
VBA/ArcCatalog: N/A
VBA/Word2007: 12 (1.0)
VB .NET: 74 (6.2)
Python/comtypes: 37 (3.1)
Python/arcpy: 110 (9.2)
IronPython: 48 (4.0)

What's a mystery to me is that VBA and C++ used to clock at 12-13 seconds on my 9.3.1 machine, and both are running slower now.  Python's about the same as what it was before.
0 Kudos
TedCronin
MVP Honored Contributor
Ironpython is faster than vb, and faster than arcpy, I bet (I hope) we will see an improved arcpy experience at Pre.
0 Kudos
MarkCederholm
Frequent Contributor
As unintuitive as it seems, I�??ve discovered that defragging a hard drive can adversely tweak the performance of a file geodatabase.  After deleting and recopying my test dataset, I�??m getting numbers that once more are in line with what I�??ve seen in the past.  Here are the revised stats, and I�??ve added a few extra languages for comparison:

9.3.1 scores (seconds):
C++/ATL: 13
VBA: 14
Python/comtypes: 40
Python/arcgisscripting: 43
VB .NET: 47
IronPython: 54
Java: 81
C#: 1235
F#: 1235
C++/CLI (unmanaged): 1561

Beta 2 scores (seconds):
C++/ATL: 7
VBA: 8
Python/comtypes: 33
VB .NET: 43
IronPython: 43
Java: 77
Python/arcpy: 111
C#: 1202
F#: 1204
C++/CLI (unmanaged): 1536

C# is the real shocker:  all along I�??ve assumed that C# has about the same performance as VB .NET.  Boy, was I wrong.

I�??ve also been looking at F#, which is supposed to be ideally suited for asynchronous parallel processing.  Sounds great for geoprocessing, doesn�??t it?  Not if you�??re using ArcObjects:  its performance is about the same as C#.

At the bottom of the dung heap is C++/CLI. 😛  I was enthusiastic about it for a brief time last year, because you could have both managed and unmanaged code in the same project.  Sounds like the best of both worlds, right?  Yeah, until you look at the performance.  Good bye.

The walk-away lesson here is:  avoid using fine-grained ArcObjects in anything but C++ (not CLI!!) or VBA.  Instead, create coarse-grained objects in C++ and then consume them in other languages.  See y'all at the Developer Summit!
😉
0 Kudos
FrankPerks
Emerging Contributor
Hi,

I am currently working on a modified version of comtypes specifically for arcgis, since there are certain little bugs with how comtypes generates its modules. Along with some other things that comtypes does that i am pretty unhappy with.... However overall comtypes does a lot of undesirable extra overhead, internally and at the module loading point. (if your interested i can explain). Anyways i've been stripping / rewriting some chunks of comtypes mainly in the area of how comtypes performs inter-opt (its ctypes based, however how it gets there isn't pretty).

The test i ran was, take a geodatabase of 25 fc's, each composed of 3 different columns where each column name is unique for the entire gdb. Combine all of the feature classes into a single FC, composed of all 25 columns. I wrote the base test in C++, and pretty much did line for line conversion to comtypes python.

These scores are in milliseconds, running ArcGIS 9.3.1, on a macbook running windows 7 via bootcamp (so performance is not good). I averaged the results of 3 runs:
C++ / ATL: 26194.0
Modified Comtypes 0.6.2: 84571
Comtypes 0.6.2: 96411
Comtypes 0.6.3dev: 10120

I don't know VB, so i can't really test it. Any chance you can post your benchmark snippets?

Also regarding C#:

http://resources.esri.com/help/9.3/ArcGISDesktop/dotnet/2c2d2655-a208-4902-bf4d-b37a1de120de.htm
0 Kudos
MarkCederholm
Frequent Contributor
The presentation I gave at the DevSummit, and all the sample code (including the benchmark snippets) are here:

http://www.pierssen.com/arcgis/misc.htm

I haven't really seen any bugs regarding comtypes and ArcObjects, except one:  it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array.  I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

Once I download the RC I'll run the benchmarks again but I don't expect to see any change.  I'm more concerned about checking on the pure Python COM objects bug.
0 Kudos
FrankPerks
Emerging Contributor
The presentation I gave at the DevSummit, and all the sample code (including the benchmark snippets) are here:

http://www.pierssen.com/arcgis/misc.htm

I haven't really seen any bugs regarding comtypes and ArcObjects, except one:  it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array.  I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

Once I download the RC I'll run the benchmarks again but I don't expect to see any change.  I'm more concerned about checking on the pure Python COM objects bug.


- comtypes has (even with latest svn 0.6.3~) problems properly generating modules from typelibs, (packing bugs with structs, some really really odd typeing that i still can't tell if its a bug or not, (sadily codegenerator module is cthulu, thats been really just hacked together.)
- Some IDL elements are not taken from typelib properly.
- built in methods (count->len, iter, etc) only really get generated when you have "Next Skip Reset Clone" methods, however i am pretty sure the minimum for ESRI is just next()
- Its currently mostly impossible to run sphinx against it to document the modules via docstrings.
- There is alot of very very useless things created, that should not be accessible by users.
- Alot of things still feel like i am using a C++ library

Sadly i didn't notice array, oops. I wonder how many methods actually just don't get generated. Hmm.

I've encountered the "Pure" python bug in the past, in 9.3.1, however the bug only appeared when running geoprocessor toolboxes and comtypes. What exactly did you attempt to run?
0 Kudos
MarkCederholm
Frequent Contributor
0 Kudos
MarkCederholm
Frequent Contributor
Adding the STAThread attribute dramatically changes C# and F# performance, and adding in the missing ReleaseComObject lines significantly improves VB .NET performance, bringing all three in line with each other as I had originally expected.  Also, setting the "CLR Thread Attribute" and "CLR Unmanaged Code Check" linker settings in the C++/CLI benchmark [the actual benchmark code is unmanaged] brings its performance to about the same as C++/ATL, which is what it should be.  :cool:

So here once again *sigh* are my benchmarks for 9.3.1:

Seconds:
Java: 81
IronPython: 54
Python: 40
VB/C#/F#: 27
VBA: 14
C++: 13
Python/arcgisscripting: 43

and now for the 10.0 RC:

Seconds:
Java: 78
IronPython: 44
Python: 33
VB/C#/F#: 19
VBA: 8
C++ : 8
Python/arcpy: 98

At the DevSummit, the ESRI geoprocessing team said for the 10.1 release they hope to improve the performance of arcpy and add Python support to the add-in framefork. :rolleyes:

I've also updated both the sample code and the presentation at my web site. 😛
0 Kudos