Has anyone else had problems with scripts that export PNGs not working after upgrading versions of ArcGIS?
I have a windows box that runs a script using arcpy each day to export > 900 PNGs. For the last year it's been running fine, with ArcGIS 10.0. Last week I decided it was finally time to upgrade... I uninstalled ArcGIS 10.0 and installed ArcGIS 10.3. My other scripts (that make 50 maps as opposed to 900) continued to work just fine.
However, when running the big script, python stops working after about 30 maps have been exported. This happens whether the script is running via Task Manager or if I'm running it manually using IDLE. The timing of the failure is not consistent, some times it makes more maps than others. But, it always fails sooner or later. I'm including a screenshot of the error dialog boxes I get when running the script with IDLE.
I tried exporting JPGs instead of PNGs and got the same error. I tried lowering the dpi of the images and it didn't help. I even uninstalled ArcGIS 10.3 and installed ArcGIS 10.2.2, hoping that it was just an issue with 10.3 - but that did not fix it either.
Any other ideas?
Solved! Go to Solution.
I have finally figured out how to get this script to run again.
One of the things the script does is calculate the title text (starting at line 267 in my code post). Some of the strings calculated for the text elements included <FNT> tags that would change the font size for the text enclosed in the tags. The maps whose title text included these tags were the ones that were failing. Once I removed the tags from the string, everything ran smoothly again.
I do not know why this fix worked. Let me know if you have any ideas / explanations.
Are you deleting stuff from memory as you go along? I see nothing in the log you showed, but it would be interesting to see the white screen behind in your screen grab
Well, I have garbage collector (gc) enabled, and if I ever use cursors I habitually del them when I'm done with them. But, I have the data creation section of the script commented out for my current test runs, so there isn't any geoprocessing going on before the export section runs right now. The only thing I have printing to the interactive window right now is when it starts - and is done with - each map PNG.
Are there other things I can do do free up memory while exporting?
I have noticed during my tests that the script will get significantly farther if I omit certain data sets from the list of maps to be exported. Python still eventually crashes, but some times I can get almost 500 maps (which is better than just 30 maps).
Maybe I should just split this one big script into two separate scripts...
This sounds exactly like the problem I have been trying to resolve for over a year (using both 10.2 and 10.3.1)
I'm attempting to loop through a list of MXDs and export the layouts as JPEGs. I can usually get about 10 to 20 maps into the process before it crashes Python completely: No traceback or error, just this popup in Windows...
In my research, I found another user who posted a description of the same problem in this thread:
Python Crashes when Running ExportToPDF(), ExportToPNG(), ExportToJPEG()
He was able to fix it, by using a Threaded approach, which would keep the entire application from crashing, but I was unable to replicate that in my application, and it seems like a workaround at best.
I thought that maybe the problem was somewhere in my code, which is moderately complex. I created a test script out of the 15 lines of code that loop through the files and export the JPEGs, and it crashes the same way. This is the code:
import arcpy import os.path rootdir = r'C:\Maps\Idaho' outpath = r'C:\test\output' for (dirName, subdirList, fileList) in os.walk(rootdir): for filename in fileList: if os.path.splitext(filename)[1].lower() == ".mxd": mxdfile = os.path.join(dirName, filename) mxd = arcpy.mapping.MapDocument(mxdfile) arcpy.mapping.ExportToJPEG(mxd, os.path.join(outpath, filename[:-3] + "jpg"))
I would be interested in seeing what your code looks like. I suspect the problem is in the ExportToJPEG(), ExportToPDF(), and ExportToPNG() methods.
I have a script that creates ~2200 Map Documents and exports the same number of PDFs and it runs without a hitch. It takes a day plus to run but I've never had it fail like that(Though I have had others that have had this sort of problem). I would always recommend cleaning up memory after each export. The only way mine varies is I put a del mxd in after I exporting the map to JPEG in the for-loop.
It seems like I attempted to explicitly delete the MXD after exporting each JPEG, but I didn't notice any difference. I'll try it again.
The only thing that ever did make a difference for me was the location of the MXD files. When they were hosted on my local hard drive, the script was able to run through all of the MXDs without crashing. When they're hosted on the network, they crash at about 10 to 15 files. I don't know why that should make any difference.
I also find that the ExportToPNG() line appears to be where my script fails. One difference that I see between our scripts is that I only use one mxd for exporting.
I have at least figured out that only certain map sets fail, so at this point I have gotten my script to run all the way through without crashing... but I now it only makes half of the maps that it used to. I wonder if this is related to Python 2.7, since everything was fine with ArcGIS 10.0 and Python 2.6.
I tried to shave my code down (I omitted the data creation / geoprocessing portion of the script since it works fine every time), but it is still kinda long for a forum post. Well, here it is:
import string, os, sys, datetime, shutil, arcpy, urllib, gc # Set Folder Folder = r'C:\Users\username\VegetationImpactProgram' print Folder #Other variables... FGDB = Folder + '\\freeze\\stations.gdb' FGDBras = Folder + '\\freeze\\rasters.gdb' FGDBgddras = Folder + '\\freeze\\gddrasters.gdb' mapsfolder = Folder + '\\maps\\freeze\\stationdata' themxd = Folder + '\\mxd_templates\\freeze_stationdata.mxd' dpi = 100 # Lists cwa010 = ('SEW', 'OTX', 'PQR', 'PDT', 'MFR', 'BOI') cwa020 = ('EKA', 'STO', 'REV', 'LKN') cwa030 = ('HNX', 'LOX', 'SGX', 'VEF', 'MTR') cwa040 = ('MSO', 'TFX', 'PIH', 'GGW', 'BYZ', 'RIW', 'CYS') cwa050 = ('SLC', 'GJT', 'BOU', 'PUB') cwa060 = ('PSR', 'FGZ', 'TWC', 'ABQ', 'EPZ') cwa070 = ('BIS', 'FGF', 'UNR', 'ABR', 'FSD') cwa080 = ('LBF', 'GID', 'GLD', 'DDC', 'TOP', 'ICT', 'OAX') cwa090 = ('AMA', 'LUB', 'OUN', 'TSA') cwa100 = ('MAF', 'SJT', 'FWD', 'EWX', 'HGX', 'CRP', 'BRO') cwa110 = ('DLH', 'MPX', 'ARX') cwa120 = ('DVN', 'LOT', 'ILX', 'DMX') cwa130 = ('EAX', 'SGF', 'LSX', 'PAH') cwa140 = ('LZK', 'SHV', 'LCH', 'MEG', 'JAN', 'LIX') cwa150 = ('MQT', 'GRB', 'MKX', 'APX', 'GRR', 'DTX') cwa160 = ('CLE', 'ILN', 'RLX', 'IWX', 'IND') cwa170 = ('LMK', 'OHX', 'JKL', 'MRX') cwa180 = ('FFC', 'HUN', 'BMX', 'MOB') cwa190 = ('CAR', 'GYX', 'BTV', 'ALY', 'BOX', 'OKX') cwa200 = ('BUF', 'BGM', 'CTP', 'PHI', 'PBZ') cwa210 = ('LWX', 'RNK', 'AKQ') cwa220 = ('RAH', 'MHX', 'ILM', 'GSP', 'CAE', 'CHS') cwa230 = ('TAE', 'JAX', 'MLB', 'TBW', 'MFL', 'KEY') cwaall = cwa010 + cwa020 + cwa030 + cwa040 + cwa050 + cwa060 + cwa070 + cwa080 + cwa090 + cwa100 + cwa110 + \ cwa120 + cwa130 + cwa140 + cwa150 + cwa160 + cwa170 + cwa180 + cwa190 + cwa200 + cwa210 + cwa220 + cwa230 areas = (['010',cwa010,'-117.0',0.1,10000,0],['020',cwa020,'-118.0',0.25,10000,25000],['030',cwa030,'-112.0',0.1,10000,0], \ ['040',cwa040,'-108.0',0.1,0,0],['050',cwa050,'-107.0',0.15,0,0],['060',cwa060,'-110.0',0.2,0,0], \ ['070',cwa070,'-100.0',0.1,0,0],['080',cwa080,'-100.0',0.1,0,0],['090',cwa090,'-99.0',0.2,0,0], \ ['100',cwa100,'-100.0',0.1,0,0],['110',cwa110,'-95.0',0.15,0,10000],['120',cwa120,'-91.0',0.15,0,0], \ ['130',cwa130,'-92.0',0.25,0,0],['140',cwa140,'-92.0',0.1,10000,0],['150',cwa150,'-88.0',0.1,0,0], \ ['160',cwa160,'-86.0',0.1,20000,-10000],['170',cwa170,'-88.0',0.2,-70000,0],['180',cwa180,'-88.0',0.1,10000,0], \ ['190',cwa190,'-78.0',0.05,-70000,0],['200',cwa200,'-80.0',0.05,-20000,0],['210',cwa210,'-81.0',0.2,-50000,0], \ ['220',cwa220,'-83.0',0.2,0,0],['230',cwa230,'-86.0',0.1,-10000,15000],['US',cwaall,'-100.0',-0.19,-150000,-50000]) types = ['pt', 'py'] temps = ['32', '28'] varis = ['dtf', 'date', 'days', 'dblw'] gdd_temps = ['mg', '42', '45', '50', '54'] mapsets = ['pt_sus','pt_lmin','py_lmin','pt_mmin','py_mmin'] # Start to Calculate the list of maps to make for vari in varis: for temp in temps: for type in types: mapsets.append(type + '_' + vari + '_' + temp) # Set today's date and time today = datetime.datetime.today() todaydate = today.strftime('%m%d') thisyear = today.strftime('%Y') thismonth = today.strftime('%m') if thismonth == '1' or thismonth == '2' or thismonth == '01' or thismonth == '02': marfirst = datetime.datetime((int(thisyear)-1), 3, 1) else: marfirst = datetime.datetime((int(thisyear)), 3, 1) date = today.strftime('%Y%m%d') # Figure out what month this is (to choose symbology) thismonthabv = today.strftime('%b') # Figure out which end dates to use for GDD (based on today's date) gdd_dates = [] for temp in temps: gdd_dates.append('lt' + temp) # Calculate the rest of the mapset list for temp in gdd_temps: for date in gdd_dates: for typ in types: mapsets.append(typ + '_gd_' + temp + '_' + date) # add all GDD to mapsets print 'mapsets:', len(mapsets), mapsets # # The next portions of the script include gathering and reading input data sources, # # creating point, polygon and raster data, and calculating these string variables: # == datadate, titledate, frzpoints_lyr, frzpoints_name, frzpoints_full_name, thiessens == # Make all the maps! gc.enable() try: # set mxd mxd = arcpy.mapping.MapDocument(themxd) # make up layer variables for the layers in the mxd dataframe = arcpy.mapping.ListDataFrames(mxd, 'Layers')[0] cwaareaslyr = arcpy.mapping.ListLayers(mxd, 'CWA_areas', dataframe)[0] cwaslyr = arcpy.mapping.ListLayers(mxd, 'CWA_bndy', dataframe)[0] stateslyr = arcpy.mapping.ListLayers(mxd, 'States', dataframe)[0] statesloreslyr = arcpy.mapping.ListLayers(mxd, 'States_lores', dataframe)[0] countieslyr = arcpy.mapping.ListLayers(mxd, 'US_Counties', dataframe)[0] countiesloreslyr = arcpy.mapping.ListLayers(mxd, 'US_Counties_lores', dataframe)[0] masklyr = arcpy.mapping.ListLayers(mxd, 'States_mask', dataframe)[0] # Set variables for legends/legend images legendpt = arcpy.mapping.ListLayoutElements(mxd, 'LEGEND_ELEMENT', 'PntLegend')[0] legendpy = arcpy.mapping.ListLayoutElements(mxd, 'LEGEND_ELEMENT', 'RasLegend')[0] legendptMar = arcpy.mapping.ListLayoutElements(mxd, 'PICTURE_ELEMENT', 'frz_legend_pt_date_Mar')[0] legendpyMar = arcpy.mapping.ListLayoutElements(mxd, 'PICTURE_ELEMENT', 'frz_legend_py_date_Mar')[0] legendptApr = arcpy.mapping.ListLayoutElements(mxd, 'PICTURE_ELEMENT', 'frz_legend_pt_date_Apr')[0] legendpyApr = arcpy.mapping.ListLayoutElements(mxd, 'PICTURE_ELEMENT', 'frz_legend_py_date_Apr')[0] legendptMay = arcpy.mapping.ListLayoutElements(mxd, 'PICTURE_ELEMENT', 'frz_legend_pt_date_May')[0] legendpyMay = arcpy.mapping.ListLayoutElements(mxd, 'PICTURE_ELEMENT', 'frz_legend_py_date_May')[0] legends = [legendpt, legendpy, legendptMar, legendpyMar, legendptApr, legendpyApr, legendptMay, legendpyMay] # clear out def queries off the bat cwaareaslyr.definitionQuery = '' cwaslyr.definitionQuery = '' stateslyr.definitionQuery = '' statesloreslyr.definitionQuery = '' countieslyr.definitionQuery = '' countiesloreslyr.definitionQuery = '' masklyr.definitionQuery = '' for mapset in mapsets: print 'mapset = ' + mapset temp = mapset[-2:] # figure out layer names for this mapset if mapset[3:5] <> 'gd': layernmpt = 'pt' + mapset[2:] if len(mapset) == 6: layernmras = 'sus_ras' if len(mapset) == 7: layernmras = mapset[3:] + '_ras' if len(mapset) == 9: layernmras = 'dtf_ras' if len(mapset) == 10: layernmras = mapset[3:-3] + '_ras' else: if mapset[6:8] <> 'mg': layernmpt = 'pt_gd' + mapset[6:8] + '_' + temp layernmras = 'gdd_ras' elif mapset[6:8] == 'mg': layernmpt = 'pt_mgdd_' + temp layernmras = 'mgdd_ras' if mapset[3:-3] == 'date': layernmpt = layernmpt + '_' + thismonthabv layernmras = layernmras + '_' + thismonthabv # make up layer variables for data layers try: ptthislyr = arcpy.mapping.ListLayers(mxd, layernmpt, dataframe)[0] ptthislyr.definitionQuery = "\"ST\" <> 'ON' AND \"ST\" IS NOT NULL" except: layernmpt = 'pt_blank' ptthislyr = arcpy.mapping.ListLayers(mxd, layernmpt, dataframe)[0] try: rasthislyr = arcpy.mapping.ListLayers(mxd, layernmras, dataframe)[0] except: layernmras = 'blank_ras' rasthislyr = arcpy.mapping.ListLayers(mxd, layernmras, dataframe)[0] # set data source for this mapset (use if statements if making points maps in addition to poly maps) if mapset[:2] == 'pt': ptthislyr.replaceDataSource (FGDB, 'FILEGDB_WORKSPACE', frzpoints_name, False) if mapset[:2] == 'py': if mapset[3:5] <> 'gd': if len(mapset) == 7: datanm = 'l' + mapset[4:] else: datanm = mapset[3:-3] + mapset[-2:] else: datanm = 'gd' + mapset[6:8] + mapset[-4:] if mapset[3:5] == 'gd': thisras = FGDBgddras + '\\' + datanm if arcpy.Exists(thisras): rasthislyr.replaceDataSource (FGDBgddras, 'FILEGDB_WORKSPACE', datanm, False) else: layernmras = 'blank_ras' rasthislyr = arcpy.mapping.ListLayers(mxd, layernmras, dataframe)[0] else: thisras = FGDBras + '\\' + datanm if arcpy.Exists(thisras): rasthislyr.replaceDataSource (FGDBras, 'FILEGDB_WORKSPACE', datanm, False) else: layernmras = 'blank_ras' rasthislyr = arcpy.mapping.ListLayers(mxd, layernmras, dataframe)[0] # set visibility for data layers and mask layer for lyr in arcpy.mapping.ListLayers(mxd): lyr.visible = False if mapset[:2] == 'pt': ptthislyr.visible = True rasthislyr.visible = False masklyr.visible = False elif mapset[:2] == 'py': ptthislyr.visible = True rasthislyr.visible = True masklyr.visible = True for reg in areas: area = reg[0] # area-specific variables cm = reg[2] scaleadj = reg[3] xadj = reg[4] yadj = reg[5] # set visibility of basemap layers if area <> 'US': cwaareaslyr.visible = False cwaslyr.visible = True stateslyr.visible = True countieslyr.visible = True statesloreslyr.visible = False countiesloreslyr.visible = False else: cwaareaslyr.visible = False cwaslyr.visible = False stateslyr.visible = False countieslyr.visible = False statesloreslyr.visible = True countiesloreslyr.visible = False if mapset[:2] == 'py': ptthislyr.visible = False # def query for cwas in area defq = '' for regst in reg[1]: defq = defq + " OR \"CWA\" = '" + regst + "'" defq = defq[4:] # Adjust spatial reference spatref = dataframe.spatialReference spatrefstring = 'PROJCS["US_National_Atlas_Equal_Area",GEOGCS["GCS_Sphere_Clarke_1866_Authalic",DATUM["D_Sphere_Clarke_1866_Authalic",SPHEROID["Sphere_Clarke_1866_Authalic",6370997.0,0.0]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Azimuthal_Equal_Area"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",' + cm + '],PARAMETER["Latitude_Of_Origin",45.0],UNIT["Meter",1.0],AUTHORITY["EPSG",2163]]' spatref.loadFromString(spatrefstring) dataframe.spatialReference = spatref # Select area's CWAs and zoom to them cwaslyr.definitionQuery = '' arcpy.SelectLayerByAttribute_management(cwaslyr, 'NEW_SELECTION', defq) dataframe.extent = cwaslyr.getSelectedExtent(False) arcpy.SelectLayerByAttribute_management(cwaslyr, 'CLEAR_SELECTION') # Zoom and/or pan if needed newscale = dataframe.scale newscale = newscale + (newscale * scaleadj) dataframe.scale = newscale ext = dataframe.extent ext.XMin = ext.XMin + xadj ext.XMax = ext.XMax + xadj ext.YMin = ext.YMin + yadj ext.YMax = ext.YMax + yadj dataframe.panToExtent(ext) # Set title and other text if mapset == 'pt_sus': titletext1 = 'Freeze Susceptibility' + '\n' + ' *Trial Product*' elif mapset[3:] == 'lmin': titletext1 = 'Lowest Minimum Temperature (' + u'\u00B0F)' + '\n' + 'since 8/1 <FNT size="12">(' + u'10\u00B0F to 50\u00B0F' + ')</FNT>' elif mapset[3:] == 'mmin': titletext1 = 'Lowest Minimum Temp (' + u'\u00B0F)' + '\n' + 'since 8/1 <FNT size="12">(' + u'-38\u00B0F to 10\u00B0F' + ')</FNT>' elif mapset[3:-3] == 'dtf': titletext1 = 'Date of First ' + temp + u'\u00B0F Freeze' + '\n' + 'since 8/1' elif mapset[3:-3] == 'date': titletext1 = 'Date of Most Recent ' + temp + u'\u00B0F Freeze' + '\n' + 'since 8/1' elif mapset[3:-3] == 'days': titletext1 = 'Days since Most Recent ' + temp + u'\u00B0F Freeze' + '\n' + 'since 8/1' elif mapset[3:-3] == 'dblw': titletext1 = 'Number of Days with Minimum Temperature <= ' + temp + u'\u00B0F' + '\n' + 'During the Past 14 Days' elif mapset[6:8] == 'mg' and mapset[9:11] <> 'lt': titletext1 = 'Accumulated Modified Growing Degree Days' + '\n' + 'Since ' + str(int(mapset[-4:-2])) + '/' + str(int(mapset[-2:])) elif mapset[6:8] == 'mg' and mapset[9:11] == 'lt': titletext1 = 'Accumulated Modified Growing Degree Days' + '\n' + 'Since Most Recent ' + temp + u'\u00B0F Freeze' elif mapset[3:5] == 'gd' and mapset[9:11] <> 'lt': titletext1 = 'Accumulated Growing Degree Days <FNT size="12">(base ' + mapset[6:8] + u'\u00B0F)</FNT>' + '\n' + 'Since ' + str(int(mapset[-4:-2])) + '/' + str(int(mapset[-2:])) elif mapset[3:5] == 'gd' and mapset[9:11] == 'lt': titletext1 = 'Accumulated Growing Degree Days <FNT size="12">(base ' + mapset[6:8] + u'\u00B0F)</FNT>' + '\n' + 'Since Most Recent ' + temp + u'\u00B0F Freeze' elm = arcpy.mapping.ListLayoutElements(mxd, 'TEXT_ELEMENT', 'Title1')[0] elm.text = titletext1 elm = arcpy.mapping.ListLayoutElements(mxd, 'TEXT_ELEMENT', 'Title2')[0] elm.text = 'As of ' + titledate # Move around legends if needed for legend in legends: legend.elementPositionY = 7.49 if mapset[3:-3] == 'date' and (thismonthabv == 'Mar' or thismonthabv == 'Apr' or thismonthabv == 'May'): if mapset[:2] == 'pt' and thismonthabv == 'Mar': legends[2].elementPositionY = 6.49 elif mapset[:2] == 'py' and thismonthabv == 'Mar': legends[3].elementPositionY = 6.49 elif mapset[:2] == 'pt' and thismonthabv == 'Apr': legends[4].elementPositionY = 6.49 elif mapset[:2] == 'py' and thismonthabv == 'Apr': legends[5].elementPositionY = 6.49 elif mapset[:2] == 'pt' and thismonthabv == 'May': legends[6].elementPositionY = 6.49 elif mapset[:2] == 'py' and thismonthabv == 'May': legends[7].elementPositionY = 6.49 else: if mapset[:2] == 'pt': legends[0].elementPositionY = 6.49 elif mapset[:2] == 'py': legends[1].elementPositionY = 6.49 # Use ArcPy to make PNG arcpy.RefreshActiveView() if mapset[-3:] == 'sus': pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[-3:] + '_' + datadate elif mapset[-4:] == 'lmin' or mapset[-4:] == 'mmin': pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[-4:] + '_' + datadate elif mapset[3:5] <> 'gd': pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[3:-3] + '_' + temp + '_' + datadate else: if mapset[9:11] == 'lt': if mapset[6:8] <> 'mg': pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[3:5] + mapset[6:8] + '_' + temp + '_' + datadate else: pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[6:8] + 'dd_' + temp + '_' + datadate else: if mapset[6:8] <> 'mg': pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[3:5] + mapset[6:8] + '_' + mapset[-4:] + '_' + datadate else: pngname = area + '_freeze_' + mapset[:2] + '_' + mapset[6:8] + 'dd_' + mapset[-4:] + '_' + datadate # Export and Copy maps out_png = mapsfolder + '\\' + pngname + '.png' print out_png arcpy.mapping.ExportToPNG (mxd, out_png, '', 0, 0, dpi) print pngname + ' exported' try: # Copy PNG to (M:) pngcopy = 'M:\\' + pngname[:-9] + '.png' shutil.copyfile (out_png, pngcopy) except: print '...FAILED to copy ' + pngname + ' to (M:)' try: # Copy PNG to (W:) pngcopy = 'W:\\ROOT\\VIP\\frz_maps\\images\\' + pngname[:-9] + '.png' shutil.copyfile (out_png, pngcopy) except: print '...FAILED to copy ' + pngname + ' to (W:)' except: print '...Map exports not complete', arcpy.GetMessages() # Save and close MXD mxd.save() del mxd
I have finally figured out how to get this script to run again.
One of the things the script does is calculate the title text (starting at line 267 in my code post). Some of the strings calculated for the text elements included <FNT> tags that would change the font size for the text enclosed in the tags. The maps whose title text included these tags were the ones that were failing. Once I removed the tags from the string, everything ran smoothly again.
I do not know why this fix worked. Let me know if you have any ideas / explanations.