nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

arcpy and pmf

TedCronin

Will the generation of pmf files be supported with arcpy? I don't remember seeing anything in the help.

Encontre mais publicações marcadas com

Accepted answers

All comments

KimOllivier

fperks;16230 wrote:
Heres an easy way to make arcpy non iterable objects usable in a for each loop.

for each in iter(lambda: <your object>.next(), None):

This basically iterates through getting the next element until the termination condition of the element being None is met.

Thanks very much. Wonderfully Pythonic!

FrankPerks

I did notice that in arcpy the Array object is not iterable. I had to use the .next() function. Should I expect it to be iterable?

while row:
        feat = row.getValue(shapefield)
        qInterior = False
        for partNum in range(feat.partCount) :
            part = feat.getPart(partNum) 
            qInterior = False
            for ptNum in range(part.count):
                pt = part.next() 
                if pt != None:
                    arrayOuter.add(pt)
                else :
                    qInterior = True
                    break # ignore donut vertices
            arrayObj.add(arrayOuter)
            arrayOuter.RemoveAll()

Heres an easy way to make arcpy non iterable objects usable in a for each loop.

for each in iter(lambda: <your object>.next(), None):

This basically iterates through getting the next element until the termination condition of the element being None is met.

while row:
        feat = row.getValue(shapefield)
        qInterior = False
        for partNum in range(feat.partCount) :
            part = feat.getPart(partNum) 
            qInterior = False
            
            # to iterate you can use the iterable wrapper in
            # conjunction with a lambda.
            for pt in iter(lambda: part.next(), None):
                qInterior = True
                arrayOuter.add(pt)
            # old method
            for ptNum in range(part.count):
                pt = part.next() 
                if pt != None:
                    arrayOuter.add(pt)
                else :
                    qInterior = True
                    break # ignore donut vertices
            arrayObj.add(arrayOuter)
            arrayOuter.RemoveAll()

For the comtypes bug not generating esriSystem.Array its because of namespace pollution that comtypes generator does.

@Riverside

Can you tell me if your version of comtypes is actually generating any sort of builtin methods like len/iter/getitem/etc, for enumerable interfaces? Some Interfaces do, but the majority don't.

KimOllivier

I haven't really seen any bugs regarding comtypes and ArcObjects, except one: it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array. I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

I did notice that in arcpy the Array object is not iterable. I had to use the .next() function. Should I expect it to be iterable?

while row:
        feat = row.getValue(shapefield)
        qInterior = False
        for partNum in range(feat.partCount) :
            part = feat.getPart(partNum) 
            qInterior = False
            for ptNum in range(part.count):
                pt = part.next() 
                if pt != None:
                    arrayOuter.add(pt)
                else :
                    qInterior = True
                    break # ignore donut vertices
            arrayObj.add(arrayOuter)
            arrayOuter.RemoveAll()

TedCronin

Adding the STAThread attribute dramatically changes C# and F# performance, and adding in the missing ReleaseComObject lines significantly improves VB .NET performance, bringing all three in line with each other as I had originally expected. Also, setting the "CLR Thread Attribute" and "CLR Unmanaged Code Check" linker settings in the C++/CLI benchmark [the actual benchmark code is unmanaged] brings its performance to about the same as C++/ATL, which is what it should be. :cool:

So here once again *sigh* are my benchmarks for 9.3.1:

Seconds:
Java: 81
IronPython: 54
Python: 40
VB/C#/F#: 27
VBA: 14
C++: 13
Python/arcgisscripting: 43

and now for the 10.0 RC:

Seconds:
Java: 78
IronPython: 44
Python: 33
VB/C#/F#: 19
VBA: 8
C++ : 8
Python/arcpy: 98

At the DevSummit, the ESRI geoprocessing team said for the 10.1 release they hope to improve the performance of arcpy and add Python support to the add-in framefork. :rolleyes:

I've also updated both the sample code and the presentation at my web site. 😛

Well then I guess I will be looking forward to 10.1, then, just amazing on how slow arcpy is on these tests at 10.

MarkCederholm

Adding the STAThread attribute dramatically changes C# and F# performance, and adding in the missing ReleaseComObject lines significantly improves VB .NET performance, bringing all three in line with each other as I had originally expected. Also, setting the "CLR Thread Attribute" and "CLR Unmanaged Code Check" linker settings in the C++/CLI benchmark [the actual benchmark code is unmanaged] brings its performance to about the same as C++/ATL, which is what it should be. :cool:

So here once again *sigh* are my benchmarks for 9.3.1:

Seconds:
Java: 81
IronPython: 54
Python: 40
VB/C#/F#: 27
VBA: 14
C++: 13
Python/arcgisscripting: 43

and now for the 10.0 RC:

Seconds:
Java: 78
IronPython: 44
Python: 33
VB/C#/F#: 19
VBA: 8
C++ : 8
Python/arcpy: 98

At the DevSummit, the ESRI geoprocessing team said for the 10.1 release they hope to improve the performance of arcpy and add Python support to the add-in framefork. :rolleyes:

I've also updated both the sample code and the presentation at my web site. 😛

MarkCederholm

Here's the thread:

http://forums.arcgis.com/threads/1311-Pure-Python-COM-objects-cause-Beta-2-crash

FrankPerks

The presentation I gave at the DevSummit, and all the sample code (including the benchmark snippets) are here:

http://www.pierssen.com/arcgis/misc.htm

I haven't really seen any bugs regarding comtypes and ArcObjects, except one: it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array. I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

Once I download the RC I'll run the benchmarks again but I don't expect to see any change. I'm more concerned about checking on the pure Python COM objects bug.

- comtypes has (even with latest svn 0.6.3~) problems properly generating modules from typelibs, (packing bugs with structs, some really really odd typeing that i still can't tell if its a bug or not, (sadily codegenerator module is cthulu, thats been really just hacked together.)
- Some IDL elements are not taken from typelib properly.
- built in methods (count->len, iter, etc) only really get generated when you have "Next Skip Reset Clone" methods, however i am pretty sure the minimum for ESRI is just next()
- Its currently mostly impossible to run sphinx against it to document the modules via docstrings.
- There is alot of very very useless things created, that should not be accessible by users.
- Alot of things still feel like i am using a C++ library

Sadly i didn't notice array, oops. I wonder how many methods actually just don't get generated. Hmm.

I've encountered the "Pure" python bug in the past, in 9.3.1, however the bug only appeared when running geoprocessor toolboxes and comtypes. What exactly did you attempt to run?

MarkCederholm

The presentation I gave at the DevSummit, and all the sample code (including the benchmark snippets) are here:

http://www.pierssen.com/arcgis/misc.htm

I haven't really seen any bugs regarding comtypes and ArcObjects, except one: it doesn't wrap the esriSystem.Array object since there's a name clash with ctypes Array. I'd be curious as to what version of comtypes you've been using and what problems you've been seeing.

Once I download the RC I'll run the benchmarks again but I don't expect to see any change. I'm more concerned about checking on the pure Python COM objects bug.

FrankPerks

Hi,

I am currently working on a modified version of comtypes specifically for arcgis, since there are certain little bugs with how comtypes generates its modules. Along with some other things that comtypes does that i am pretty unhappy with.... However overall comtypes does a lot of undesirable extra overhead, internally and at the module loading point. (if your interested i can explain). Anyways i've been stripping / rewriting some chunks of comtypes mainly in the area of how comtypes performs inter-opt (its ctypes based, however how it gets there isn't pretty).

The test i ran was, take a geodatabase of 25 fc's, each composed of 3 different columns where each column name is unique for the entire gdb. Combine all of the feature classes into a single FC, composed of all 25 columns. I wrote the base test in C++, and pretty much did line for line conversion to comtypes python.

These scores are in milliseconds, running ArcGIS 9.3.1, on a macbook running windows 7 via bootcamp (so performance is not good). I averaged the results of 3 runs:
C++ / ATL: 26194.0
Modified Comtypes 0.6.2: 84571
Comtypes 0.6.2: 96411
Comtypes 0.6.3dev: 10120

I don't know VB, so i can't really test it. Any chance you can post your benchmark snippets?

Also regarding C#:

http://resources.esri.com/help/9.3/ArcGISDesktop/dotnet/2c2d2655-a208-4902-bf4d-b37a1de120de.htm

MarkCederholm

As unintuitive as it seems, I�??ve discovered that defragging a hard drive can adversely tweak the performance of a file geodatabase. After deleting and recopying my test dataset, I�??m getting numbers that once more are in line with what I�??ve seen in the past. Here are the revised stats, and I�??ve added a few extra languages for comparison:

9.3.1 scores (seconds):
C++/ATL: 13
VBA: 14
Python/comtypes: 40
Python/arcgisscripting: 43
VB .NET: 47
IronPython: 54
Java: 81
C#: 1235
F#: 1235
C++/CLI (unmanaged): 1561

Beta 2 scores (seconds):
C++/ATL: 7
VBA: 8
Python/comtypes: 33
VB .NET: 43
IronPython: 43
Java: 77
Python/arcpy: 111
C#: 1202
F#: 1204
C++/CLI (unmanaged): 1536

C# is the real shocker: all along I�??ve assumed that C# has about the same performance as VB .NET. Boy, was I wrong.

I�??ve also been looking at F#, which is supposed to be ideally suited for asynchronous parallel processing. Sounds great for geoprocessing, doesn�??t it? Not if you�??re using ArcObjects: its performance is about the same as C#.

At the bottom of the dung heap is C++/CLI. 😛 I was enthusiastic about it for a brief time last year, because you could have both managed and unmanaged code in the same project. Sounds like the best of both worlds, right? Yeah, until you look at the performance. Good bye.

The walk-away lesson here is: avoid using fine-grained ArcObjects in anything but C++ (not CLI!!) or VBA. Instead, create coarse-grained objects in C++ and then consume them in other languages. See y'all at the Developer Summit!
😉

TedCronin

Ironpython is faster than vb, and faster than arcpy, I bet (I hope) we will see an improved arcpy experience at Pre.

MarkCederholm

Well, I've discovered that my older numbers are no longer valid since I went from VS 2005 to 2008 (VB .NET is much worse now!

). Here are completely fresh scores for 9.3.1 vs. 9.4(10) beta 2:

Benchmark: Find average area of polygon records in a file GDB
Count: 521751
Avg Area: 426300
Scores in seconds represent best of 3 runs

9.3.1 scores (seconds):
-----------------------
C++: 21 (1.0)
VBA/ArcCatalog: 22 (1.0)
VBA/Word2007: 30 (1.4)
VB .NET: 179 (8.5)
Python/comtypes: 47 (2.2)
Python/arcgisscripting: 51 (2.4)
IronPython: 62 (3.0)

Beta 2 scores (seconds):
------------------------
C++: 12 (1.0)
VBA/ArcCatalog: N/A
VBA/Word2007: 12 (1.0)
VB .NET: 74 (6.2)
Python/comtypes: 37 (3.1)
Python/arcpy: 110 (9.2)
IronPython: 48 (4.0)

What's a mystery to me is that VBA and C++ used to clock at 12-13 seconds on my 9.3.1 machine, and both are running slower now. Python's about the same as what it was before.

TedCronin

Do you have any equivalent benchmark numbers for 9.4/10?

MarkCederholm

The comtypes package works just fine with 2.6. There's an extra step if you're using ArcObjects rather than arcpy to initialize standalone licensing at 9.4(10) beta 2. The only problem I've noticed with beta 2 is that ArcMap and ArcCatalog will bomb on closing if any pure python COM objects have been present.

I haven't tried psyco but I'll look into it. [UPDATE: No significant performance boost - the real bottleneck in the benchmark is COM interop (as intended), and psyco apparently can't do anything for that.]

Speaking of arcpy, I spent some time working with the mapping module at beta 1, but I'd like to spend some more time on it again this week before the DevSummit. It's a great start anyway, and I definitely plan to attend the map automation session.

JasonScheirer

Have you tried benchmarking your Python scripts against Python with psyco enabled? I wonder how much it would help, if the bottlenecks on the Python side can be significantly optimized away.

TedCronin

Any testing with 2.6.2 with arcpy? Or is comtypes not ready for 2.6?

MarkCederholm

IronPython is 2.6, CPython is 2.5.1, comtypes is 0.6.1

TedCronin

Which version of Ironpython does this refer to? What's interesting to me is the tie:

Python/comtypes: 3.54
VB .NET: 3.54

kinda makes you not want to use vb.

MarkCederholm

FYI, I've now added IronPython to my list of COM interop performance benchmarks. Here are the relative scores for calculating the average area of 500+K polygon features at 9.3.1:

VBA: 1.00
C++/Native: 1.00
Python/comtypes: 3.54
VB .NET: 3.54
Python/arcgisscripting: 3.62
IronPython: 4.78

MarkCederholm

Not yet; I plan to eventually, though I hear IronPython's performance leaves something to be desired. Right now, I'm focusing on the Python shipped with ArcGIS.

BartHound

Mark -

Have you looked at IronPython with clrtypes, to do all the arcobjects, just inquiring?

TedCronin

yes, comtypes is pretty cool, thanks for the script, will give it a spin.

MarkCederholm

As far as I can tell, there is no Publisher support in arcpy, nor can you use it to check out the license. This is a textbook situation where you want to use ArcObjects instead (see my presentation at the upcoming DevSummit). I've attached an example script.

PublishPMF.py.txt.zip

Participe da Conversa

Membros conectados podem postar, seguir atualizações e mais. Novo aqui? Registre uma conta gratuita.

Links Rápidos

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.