topic Re: Python script never terminates in Geoprocessing Questions

Python script never terminates

RobertStevens — Fri, 28 Aug 2015 23:40:06 GMT

Can someone divine why the attached python script never terminates?

I am running with ArcGIS10.3.1 basic.

When I run this using 32-bit python it gets to the last layer

and the terminates with a segmentation violation.

With a 64-bit python it never terminates.

The program runs fine on other maps I have, but

one in particular gives this problem. AFAIK

the map in question is not in any relevant

sense different from those on which the

script runs ok.

Re: Python script never terminates

DanPatterson_Retired — Sat, 29 Aug 2015 01:30:26 GMT

zips don't open easily on iThingys, if the script isn't too long, copy, paste and format it using the python syntax highligher accesses with the >> button using the advanced editor

Re: Python script never terminates

RebeccaStrauch__GISP — Sat, 11 Dec 2021 09:26:52 GMT

I've posted you script, as is (after the unzip), and it looks like your indentation is off onmany of the likes. 36, 37, and 55 have an extra space in front. And some of the if and elif: statements don't seem to be indented at all, so thinks wouldn't execute....which may be why the program doesn't end for some mxd's. If these are correct on you side, then it may be something happened in the zip file. Use Posting Code blocks in the new GeoNet to post the corrected copy of your code.

I've gone a completely different route for updating my broken links, since the .sde and .ags connections needed special handling. So I'll be interested if you code (once fixed) handles all those. My process reads from an excel/ (actually csv) file and can change paths, and feature class types, if needed. Seems to be working now, but running a test this weekend. Then I need to clean up the code a bit. You code is much shorted than mine, but not sure if I understand what it does complete (I'm not familiar with the OptionParser or glob for that matter).

import glob
import os
import sys
import arcpy
import arcpy.mapping as mapping
from optparse import OptionParser

# Previous invocations of this script were used to effect the following conversions
# from Program Files (x86)/ArcGIS/Desktop10.0/Business Analyst/US/Data/
# to   Business Analyst/2011Data/
# from RLIS_201402  to  RLIS
# from RLIS_201402  to  RLIS_WebMercator

parser = OptionParser()
parser.add_option("-f", "--fix", action="store_true", dest="f",
                  default=False,help="fix the source name")
parser.add_option("-o", "--old-src", action="store", dest="o",
                  default="RLIS-201402",
          help="the old source file name, or part of a name")
parser.add_option("-n", "--new-src", action="store", dest="n",
                  default="RLIS",
          help="the new source file name, or part of a name")

(options, args) = parser.parse_args()

old_src = options.o
new_src = options.n

def fixLayer(lyr):
    lyrModified = False
    if lyr.isFeatureLayer:
    print lyr.name
    if old_src in lyr.dataSource:
        print "\t\t" + lyr.dataSource
    if options.f:
         lyrModified = True
         lyr.findAndReplaceWorkspacePath(old_src,new_src,False)
    return lyrModified

def fixMap(mxd):
    mxdModified = False
    # loop thru all data frames in the map
    dataFrames = mapping.ListDataFrames(mxd, '')
    for frame in dataFrames:
        print "\tframe " + frame.name
        # get all layers in this data frame
        layers = mapping.ListLayers(mxd, '', frame)
        # loop thru all layers in the data frame
    print layers
    for lyr in layers:
        lyrModified = fixLayer(lyr)
        if lyrModified:
        mxdModified = True
    if mxdModified:
         mxd.save()

def fixit(fullPath,fileName):
    desc = arcpy.Describe(fullPath)
    if desc.datatype == "MapDocument":
    print fileName
    mxd = mapping.MapDocument(fullPath)
    fixMap(mxd)
    del mxd
    elif desc.datatype == "Layer":
    print fileName
    lyr = mapping.Layer(fullPath)
    if fixLayer(lyr):
        lyr.save()
    del lyr

for arg in args:
    if os.path.isdir(arg):
    for fileName in os.listdir(arg):
        fullPath = os.path.join(arg, fileName)
        fixit(fullPath,fileName)
    elif os.path.isfile(arg):
    fixit(arg,arg)
    else:
    print arg + " is not a map, nor a layer, nor a directory"

Re: Python script never terminates

RobertStevens — Sat, 29 Aug 2015 04:21:14 GMT

yes, you are right.

I don't know how that happened.

It was correct at one time.

I have no idea what Python would do with that

but I will correct it and retry.

Re: Python script never terminates

RebeccaStrauch__GISP — Sat, 29 Aug 2015 04:25:06 GMT

Are you using a good IDE that shows Python syntax errors? There are many....I use and like Wing. I use the Pro version but there is a free one too. But many others. It would bark at the extra space and would also give you an error when there is a "elif" with nothing to do, etc. Just a suggestion if you aren't using one. If it was in the .zip....no clue.

Re: Python script never terminates

RobertStevens — Sat, 29 Aug 2015 07:31:34 GMT

I fixed the indentation problems, and Wing seems to say all is OK.

But the behavior is unchanged.

Re: Python script never terminates

JoshuaBixby — Sat, 29 Aug 2015 14:29:46 GMT

If the code works fine on all other maps, and only crashes on one map, something is likely off with that one map. I have run into issues where MXDs are partially corrupt, not too corrupt for ArcMap to open but corrupt enough that ArcPy scripts will crash. Have you run DocDefragmenter or MXDDoctor on the problem map? If not, it might be worth seeing if running either of those makes a map that doesn't crash.

If the MXD is healthy and defragged, there might be a data source in the map that is exposing a bug with ArcPy.

Re: Python script never terminates

RebeccaStrauch__GISP — Sat, 29 Aug 2015 16:41:53 GMT

And to add what Joshua suggested....a SaveAs sometimes fixes slightly corrupted or bloated MXDs.

Do you have any .sde, groups, arcgis-server, etc connections in this mxd? And if so, does your code take those different items into consideration? im not sure if hitting one of those items would not let your script terminate, but it know they can/will cause issues when trying to auto fix broken links.

Re: Python script never terminates

RobertStevens — Sat, 29 Aug 2015 19:31:16 GMT

I have run the MXD Doctor and the Defrag utility all with no effect.

No the map in question does not have any .sde or server connections

-- all the data is local.

One other thing I should say is that I am presently running this script

to identify layers using a particular dataset. I am not changing anything

(so the -f option is false).

Just as an aside: this Python interface of arcpy is so slow. One notices

this also when invoking Python tools from within ArcGIS. Even to bring

up the dialogue screen is sluggish -- 10,15 even sometimes 20 seconds.

Given that Windows itself running under Parallels on my MAcbook Pro

takes a mere 5 seconds to startup, those numbers are astonishing.

Re: Python script never terminates

RobertStevens — Sat, 11 Dec 2021 09:26:55 GMT

The code that Rebecca kindly formatted did not properly show indentation

possibly because tabs and spaced were intermingled.

(Though there were the 3 errors she mentioned, since fixed).

I have re-posted the script here with the proper indentation.

import glob
import os
import sys
import arcpy
import arcpy.mapping as mapping
from optparse import OptionParser


# Previous invocations of this script were used to effect the following conversions
# from Program Files (x86)/ArcGIS/Desktop10.0/Business Analyst/US/Data/
# to  Business Analyst/2011Data/
# from RLIS_201402  to  RLIS
# from RLIS_201402  to  RLIS_WebMercator


parser = OptionParser()
parser.add_option("-f", "--fix", action="store_true", dest="f",
                  default=False,help="fix the source name")
parser.add_option("-o", "--old-src", action="store", dest="o",
                  default="RLIS-201402",
                  help="the old source file name, or part of a name")
parser.add_option("-n", "--new-src", action="store", dest="n",
                  default="RLIS",
                  help="the new source file name, or part of a name")


(options, args) = parser.parse_args()


old_src = options.o
new_src = options.n


def fixLayer(lyr):
    lyrModified = False
    if lyr.isFeatureLayer:
        #print lyr.name
        if old_src in lyr.dataSource:
            print "\t\t" + lyr.dataSource
        if options.f:
            lyrModified = True
            lyr.findAndReplaceWorkspacePath(old_src,new_src,False)
    return lyrModified


def fixMap(mxd):
    mxdModified = False
    # loop thru all data frames in the map
    dataFrames = mapping.ListDataFrames(mxd, '')
    for frame in dataFrames:
        print "\tframe " + frame.name
        # get all layers in this data frame
        layers = mapping.ListLayers(mxd, '', frame)
        # loop thru all layers in the data frame
        #print layers
        for lyr in layers:
            lyrModified = fixLayer(lyr)
            if lyrModified:
                mxdModified = True
    if mxdModified:
        mxd.save()


def fixit(fullPath,fileName):
    desc = arcpy.Describe(fullPath)
    if desc.datatype == "MapDocument":
        print fileName
        mxd = mapping.MapDocument(fullPath)
        fixMap(mxd)
        del mxd
    elif desc.datatype == "Layer":
        print fileName
        lyr = mapping.Layer(fullPath)
        if fixLayer(lyr):
            lyr.save()
        del lyr


for arg in args:
    if os.path.isdir(arg):
        for fileName in os.listdir(arg):
            fullPath = os.path.join(arg, fileName)
            fixit(fullPath,fileName)
    elif os.path.isfile(arg):
        fixit(arg,arg)
    else:
        print arg + " is not a map, nor a layer, nor a directory"


exit(0)

Re: Python script never terminates

JoshuaBixby — Mon, 31 Aug 2015 14:16:30 GMT

Try adding some print statements to see what layer causes the code to hang. If you change the order of the problem layer, does the code still hang on that layer?

Re: Python script never terminates

RobertStevens — Mon, 31 Aug 2015 17:49:02 GMT

I had already added some print, but I should have reported what I found.

The script does indeed go through all the layers (there is only

one frame in the map in question) and then hangs at the end

(To be more precise: the 64 bit version of python hangs at

the end; the 32 bit version experiences a segmentation violation).

I tried changing the order of layers. The script always reports

all the layers. Same behavior.

The map in question has many layers, but most of them

are basemap layers which display at different scales.

But the others maps I have are no different in that respect.

I can see nothing special about this map.

All the maps I have use the same basemap layers

but only this particular one (I believe) exhibits this

behavior.

To repeat: I have run Map Doctor and Defragger.

No change.

Re: Python script never terminates

RebeccaStrauch__GISP — Tue, 01 Sep 2015 01:04:55 GMT

Robert, not that it will fix you issue, but if you want to look at my solution to fixing broken links, I've posted my first version of Python addin for data inventory and “broken-link” repair. It's definitely a different approach than your code, but I'd be curious if you have a similar issue on you mxd. If so, than maybe it would point to the mxd and not your script.

Re: Python script never terminates

RobertStevens — Wed, 02 Sep 2015 01:26:50 GMT

Rebecca

I today ran your script, the one that identifies broken sources,

and it ran without any problem.

So there does appear to be something wrong with my Python Code.

I am hard pressed to figure out what.

I have many maps, in structure they are very much the same,

but only one causes this problem.

I will compare my code with yours.

Re: Python script never terminates

RebeccaStrauch__GISP — Wed, 02 Sep 2015 01:45:10 GMT

Robert, mine tests for all (or most anyway) of the anomalies. Seems like different data types, and TOC-groups, etc. need to be tested differently.....even more so when repairing the broken links. I wouldn't say mine is perfect yet, and I know it could be more efficient, but I'm at the point I need to move on, at least for a while. I'm hoping others give the broken link reporting a try and give feedback. I only have my datasets/mxds to test. So feel free to give scripting feedback....whether I merge it all in or not... .? I wrote it for our users of course, but hoping it can help others.