topic arcpy and pmf in Geoprocessing Questions

arcpy and pmf

TedCronin — Tue, 02 Feb 2010 18:29:02 GMT

Will the generation of pmf files be supported with arcpy? I don't remember seeing anything in the help.

arcpy and pmf

MarkCederholm — Fri, 12 Feb 2010 20:23:14 GMT

As far as I can tell, there is no Publisher support in arcpy, nor can you use it to check out the license. This is a textbook situation where you want to use ArcObjects instead (see my presentation at the upcoming DevSummit). I've attached an example script.

arcpy and pmf

TedCronin — Fri, 12 Feb 2010 20:43:30 GMT

yes, comtypes is pretty cool, thanks for the script, will give it a spin.

arcpy and pmf

BartHound — Thu, 25 Feb 2010 18:37:34 GMT

Mark -

Have you looked at IronPython with clrtypes, to do all the arcobjects, just inquiring?

arcpy and pmf

MarkCederholm — Fri, 26 Feb 2010 12:55:03 GMT

Not yet; I plan to eventually, though I hear IronPython's performance leaves something to be desired. Right now, I'm focusing on the Python shipped with ArcGIS.

arcpy and pmf

MarkCederholm — Mon, 15 Mar 2010 16:12:36 GMT

FYI, I've now added IronPython to my list of COM interop performance benchmarks. Here are the relative scores for calculating the average area of 500+K polygon features at 9.3.1:

VBA: 1.00
C++/Native: 1.00
Python/comtypes: 3.54
VB .NET: 3.54
Python/arcgisscripting: 3.62
IronPython: 4.78

arcpy and pmf

TedCronin — Mon, 15 Mar 2010 20:03:10 GMT

Which version of Ironpython does this refer to? What's interesting to me is the tie:

Python/comtypes: 3.54
VB .NET: 3.54

kinda makes you not want to use vb.

arcpy and pmf

MarkCederholm — Mon, 15 Mar 2010 20:26:11 GMT

IronPython is 2.6, CPython is 2.5.1, comtypes is 0.6.1

arcpy and pmf

TedCronin — Mon, 15 Mar 2010 21:22:09 GMT

Any testing with 2.6.2 with arcpy? Or is comtypes not ready for 2.6?

arcpy and pmf

JasonScheirer — Mon, 15 Mar 2010 22:26:02 GMT

Have you tried benchmarking your Python scripts against Python with psyco enabled? I wonder how much it would help, if the bottlenecks on the Python side can be significantly optimized away.

arcpy and pmf

MarkCederholm — Tue, 16 Mar 2010 13:09:15 GMT

The comtypes package works just fine with 2.6. There's an extra step if you're using ArcObjects rather than arcpy to initialize standalone licensing at 9.4(10) beta 2. The only problem I've noticed with beta 2 is that ArcMap and ArcCatalog will bomb on closing if any pure python COM objects have been present.

I haven't tried psyco but I'll look into it. [UPDATE: No significant performance boost - the real bottleneck in the benchmark is COM interop (as intended), and psyco apparently can't do anything for that.]

Speaking of arcpy, I spent some time working with the mapping module at beta 1, but I'd like to spend some more time on it again this week before the DevSummit. It's a great start anyway, and I definitely plan to attend the map automation session.

arcpy and pmf

TedCronin — Tue, 16 Mar 2010 14:31:30 GMT

Do you have any equivalent benchmark numbers for 9.4/10?

arcpy and pmf

MarkCederholm — Tue, 16 Mar 2010 18:26:17 GMT

Well, I've discovered that my older numbers are no longer valid since I went from VS 2005 to 2008 (VB .NET is much worse now! :p). Here are completely fresh scores for 9.3.1 vs. 9.4(10) beta 2:

Benchmark: Find average area of polygon records in a file GDB
Count: 521751
Avg Area: 426300
Scores in seconds represent best of 3 runs

9.3.1 scores (seconds):
-----------------------
C++: 21 (1.0)
VBA/ArcCatalog: 22 (1.0)
VBA/Word2007: 30 (1.4)
VB .NET: 179 (8.5)
Python/comtypes: 47 (2.2)
Python/arcgisscripting: 51 (2.4)
IronPython: 62 (3.0)

Beta 2 scores (seconds):
------------------------
C++: 12 (1.0)
VBA/ArcCatalog: N/A
VBA/Word2007: 12 (1.0)
VB .NET: 74 (6.2)
Python/comtypes: 37 (3.1)
Python/arcpy: 110 (9.2)
IronPython: 48 (4.0)

What's a mystery to me is that VBA and C++ used to clock at 12-13 seconds on my 9.3.1 machine, and both are running slower now. Python's about the same as what it was before.

arcpy and pmf

TedCronin — Tue, 16 Mar 2010 22:26:52 GMT

Ironpython is faster than vb, and faster than arcpy, I bet (I hope) we will see an improved arcpy experience at Pre.

arcpy and pmf

MarkCederholm — Sat, 20 Mar 2010 03:55:07 GMT

As unintuitive as it seems, I�??ve discovered that defragging a hard drive can adversely tweak the performance of a file geodatabase. After deleting and recopying my test dataset, I�??m getting numbers that once more are in line with what I�??ve seen in the past. Here are the revised stats, and I�??ve added a few extra languages for comparison:

9.3.1 scores (seconds):
C++/ATL: 13
VBA: 14
Python/comtypes: 40
Python/arcgisscripting: 43
VB .NET: 47
IronPython: 54
Java: 81
C#: 1235
F#: 1235
C++/CLI (unmanaged): 1561

Beta 2 scores (seconds):
C++/ATL: 7
VBA: 8
Python/comtypes: 33
VB .NET: 43
IronPython: 43
Java: 77
Python/arcpy: 111
C#: 1202
F#: 1204
C++/CLI (unmanaged): 1536

C# is the real shocker: all along I�??ve assumed that C# has about the same performance as VB .NET. Boy, was I wrong.

I�??ve also been looking at F#, which is supposed to be ideally suited for asynchronous parallel processing. Sounds great for geoprocessing, doesn�??t it? Not if you�??re using ArcObjects: its performance is about the same as C#.

At the bottom of the dung heap is C++/CLI. 😛 I was enthusiastic about it for a brief time last year, because you could have both managed and unmanaged code in the same project. Sounds like the best of both worlds, right? Yeah, until you look at the performance. Good bye.

The walk-away lesson here is: avoid using fine-grained ArcObjects in anything but C++ (not CLI!!) or VBA. Instead, create coarse-grained objects in C++ and then consume them in other languages. See y'all at the Developer Summit!
😉

arcpy and pmf

FrankPerks — Wed, 07 Apr 2010 02:53:39 GMT

Hi,

I am currently working on a modified version of comtypes specifically for arcgis, since there are certain little bugs with how comtypes generates its modules. Along with some other things that comtypes does that i am pretty unhappy with.... However overall comtypes does a lot of undesirable extra overhead, internally and at the module loading point. (if your interested i can explain). Anyways i've been stripping / rewriting some chunks of comtypes mainly in the area of how comtypes performs inter-opt (its ctypes based, however how it gets there isn't pretty).

The test i ran was, take a geodatabase of 25 fc's, each composed of 3 different columns where each column name is unique for the entire gdb. Combine all of the feature classes into a single FC, composed of all 25 columns. I wrote the base test in C++, and pretty much did line for line conversion to comtypes python.

These scores are in milliseconds, running ArcGIS 9.3.1, on a macbook running windows 7 via bootcamp (so performance is not good). I averaged the results of 3 runs:
C++ / ATL: 26194.0
Modified Comtypes 0.6.2: 84571
Comtypes 0.6.2: 96411
Comtypes 0.6.3dev: 10120

I don't know VB, so i can't really test it. Any chance you can post your benchmark snippets?

Also regarding C#:

http://resources.esri.com/help/9.3/ArcGISDesktop/dotnet/2c2d2655-a208-4902-bf4d-b37a1de120de.htm

arcpy and pmf

MarkCederholm — Wed, 07 Apr 2010 14:04:58 GMT

The presentation I gave at the DevSummit, and all the sample code (including the benchmark snippets) are here:

http://www.pierssen.com/arcgis/misc.htm

I haven't really seen any bugs regarding comtypes and ArcObjects, except one: it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array. I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

Once I download the RC I'll run the benchmarks again but I don't expect to see any change. I'm more concerned about checking on the pure Python COM objects bug.

arcpy and pmf

FrankPerks — Wed, 07 Apr 2010 15:47:59 GMT

The presentation I gave at the DevSummit, and all the sample code (including the benchmark snippets) are here:

http://www.pierssen.com/arcgis/misc.htm

I haven't really seen any bugs regarding comtypes and ArcObjects, except one: it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array. I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

Once I download the RC I'll run the benchmarks again but I don't expect to see any change. I'm more concerned about checking on the pure Python COM objects bug.

- comtypes has (even with latest svn 0.6.3~) problems properly generating modules from typelibs, (packing bugs with structs, some really really odd typeing that i still can't tell if its a bug or not, (sadily codegenerator module is cthulu, thats been really just hacked together.)
- Some IDL elements are not taken from typelib properly.
- built in methods (count->len, iter, etc) only really get generated when you have "Next Skip Reset Clone" methods, however i am pretty sure the minimum for ESRI is just next()
- Its currently mostly impossible to run sphinx against it to document the modules via docstrings.
- There is alot of very very useless things created, that should not be accessible by users.
- Alot of things still feel like i am using a C++ library

Sadly i didn't notice array, oops. I wonder how many methods actually just don't get generated. Hmm.

I've encountered the "Pure" python bug in the past, in 9.3.1, however the bug only appeared when running geoprocessor toolboxes and comtypes. What exactly did you attempt to run?

arcpy and pmf

MarkCederholm — Wed, 07 Apr 2010 16:06:31 GMT

Here's the thread:

http://forums.arcgis.com/threads/1311-Pure-Python-COM-objects-cause-Beta-2-crash

STAThread Attribute

MarkCederholm — Thu, 08 Apr 2010 20:54:22 GMT

Adding the STAThread attribute dramatically changes C# and F# performance, and adding in the missing ReleaseComObject lines significantly improves VB .NET performance, bringing all three in line with each other as I had originally expected. Also, setting the "CLR Thread Attribute" and "CLR Unmanaged Code Check" linker settings in the C++/CLI benchmark [the actual benchmark code is unmanaged] brings its performance to about the same as C++/ATL, which is what it should be. :cool:

So here once again *sigh* are my benchmarks for 9.3.1:

Seconds:
Java: 81
IronPython: 54
Python: 40
VB/C#/F#: 27
VBA: 14
C++: 13
Python/arcgisscripting: 43

and now for the 10.0 RC:

Seconds:
Java: 78
IronPython: 44
Python: 33
VB/C#/F#: 19
VBA: 8
C++ : 8
Python/arcpy: 98

At the DevSummit, the ESRI geoprocessing team said for the 10.1 release they hope to improve the performance of arcpy and add Python support to the add-in framefork. :rolleyes:

I've also updated both the sample code and the presentation at my web site. 😛