I want to loop many mxds within a folder to retrieve layer,connection,etc. information using python.
I just need the syntax for looping through the mxds.
This is part of my Python addin for data inventory and “broken-link” repair. tool box, but I'll include the first script here. It inventories and lists the fgdb to out put csv, and it also coverts it to an .xsl (not .xlsx). Maybe you can pick out what you need from this.
EDIT: bTW replace all the myMsgs with print or arcpy add message .
''' --------------------------------------------------------------------------- Tool: FCInventoryReport Toolbox: CheckAndFixLinks.tbx Script: 2_InventoryFCs.py Purpose: This script will walk thru all the feature classes within a folder and create a text report, comma-delimted and an Excel .xls file. Include shapes, coverages, rasters, tables, connections, and FGDB Column names: FType FCname FullPath ------------------------------------------------------------------------------- Author: Rebecca Strauch - ADFG-DWC-GIS Created on: 4/10/2013 last modification: August 10, 2015 Description: To create list of the features classes within a folder, including coverages (pts, poly, arc, anno), shapes, grids and FGDB data. Outputs list to a text report file (not much formatting) in the folder being scanned named FCInventoryYYYYMMDD_HHMM.txt with the date and time as part of the name. This should always create a new file, unless you run it twice within same minute. Must have write permissions on the output folder in question. Known issue (with built-in workaround): for some reason some, but not all (ArcInfo) grids want to duplicate the folder name before the describe. I'm now checking to make sure it exists, if not, I am removing the duplicate portion, and letting it run. Seems to work. ------------------------------------------------------------------------------ Arguments: [0] theWorkspace: Folder/directory to search (walk thru, includes subfolders) [1] outFile: output base filename, default GDBList, script appends YYYYMMDD_HHMM Updates: --------------------------------------------------------------------------- ''' # Import modules import arcpy import os from _miscUtils import * from _gpdecorators import * # catch_errors decorator must preceed a function using the @ notation. @catch_errors def main(): """ Main function to create text file report of all Feature Classes in folder """ #setup environment arcpy.env.overwriteOutput = True # Script arguments... """ If running as standalone, hardcode theWorkspace and outFile """ theWorkspace = arcpy.GetParameterAsText(0) if not theWorkspace: theWorkspace = r"C:\__Data1\_TalkeetnaBU" # r"D:\_dataTest" outFile = arcpy.GetParameterAsText(1) if not outFile: outFile = "FCInventory" # Create new output name name tagged with YYYYMMDD_HHMM fileDateTime = curFileDateTime() currentDate = curDate() # Create new output name tagged with YYYYMMDD_HHMM outfileTXT = os.path.join(theWorkspace, outFile) + fileDateTime + ".txt" #theWorkspace + "\FCInventory" + fileDateTime + ".txt" outFileCSV = os.path.join(theWorkspace, outFile) + fileDateTime + ".csv" #theWorkspace + "\FCInventory" + fileDateTime + ".csv" outFileXLS = os.path.join(theWorkspace, outFile) + fileDateTime + ".xls" arcpy.AddMessage(theWorkspace + ", " + outfileTXT) reportFile = open(outfileTXT, 'w') csvFile = open(outFileCSV, 'w') arcpy.AddMessage( "File {0} is open? {1}".format(outfileTXT, str(not reportFile.closed))) arcpy.AddMessage( "File {0} is open? {1}".format(str(outFileCSV), str(not csvFile.closed))) #arcpy.AddMessage( "File " + str(csvFile) + " is closed? " + str(csvFile.closed)) arcpy.AddMessage("Writing the report to: " + outfileTXT + " and " + outFileCSV) outText = "List of all GIS data in " + theWorkspace + " on " + currentDate + '\n' outText += " Includes coverages (pts, poly, arc, anno), shapes, and FGDB data." + '\n' outText += "-----------------------------------------------------" + '\n' reportFile.write(outText) csvFile.write("FType, FCname, FullPath\n") def inventory_data(workspace, datatypes): for path, path_names, data_names in arcpy.da.Walk( workspace, datatype=datatypes): if "tic" in data_names: data_names.remove('tic') for data_name in data_names: fcName = os.path.join(path, data_name) #arcpy.AddMessage("Show for debug: " + fcName) if not arcpy.Exists(fcName): # workaround for raster folder name duplicating fcName = os.path.dirname(fcName) desc = arcpy.Describe(fcName) #arcpy.AddMessage("debug, desc it to me: " + desc.dataType) yield [path, data_name, desc.dataType] #, desc] i = 0 for feature_class in inventory_data(theWorkspace, "FeatureClass"): """ last modified data not working for gdb or FC in fgdb ...""" #lastMod = time.strftime('%m/%d/%Y %H:%M', time.localtime(os.path.getmtime(feature_class[0]))) if i == 0: #arcpy.AddMessage("{0} modified: {1}".format(feature_class[0], lastMod)) arcpy.AddMessage("{0}".format(feature_class[0])) outText = ' ' + feature_class[0] + '\n' reportFile.write(outText) path0 = feature_class[0] i =+ 1 elif not path0 == feature_class[0]: #arcpy.AddMessage("{0} modified: {1}".format(feature_class[0], lastMod)) arcpy.AddMessage("{0}".format(feature_class[0])) outText = ' ' + feature_class[0] + '\n' reportFile.write(outText) i = 0 if feature_class[2] == "ShapeFile": shpfile = arcpy.os.path.join(feature_class[0], feature_class[1]) lastMod = time.strftime('%m/%d/%Y %H:%M', time.localtime(os.path.getmtime(shpfile))) arcpy.AddMessage(" {0}: {1} modified: {2}".format(feature_class[2], feature_class[1], lastMod)) outText = (" {0}: {1} modified: {2}\n".format(feature_class[2], feature_class[1], lastMod)) else: arcpy.AddMessage(" {0}: {1}".format(feature_class[2], feature_class[1])) outText = (" {0}: {1}\n".format(feature_class[2], feature_class[1])) reportFile.write(outText) csvFile.write("{},{}, {}\n".format(feature_class[2], feature_class[1], feature_class[0])) reportFile.close() csvFile.close() arcpy.AddMessage( "File {0} is closed? {1}".format(outfileTXT, str(reportFile.closed))) arcpy.AddMessage( "File {0} is closed? {1}".format(outFileCSV, str(csvFile.closed))) # Creates Excel .xls file from the .csv ....easier to edit (ver 1) arcpy.TableToExcel_conversion(outFileCSV, outFileXLS) arcpy.AddMessage('!!! Success !!! ') # End main function if __name__ == '__main__': main()
I have looked at many regarding csv writer, yet I am not successful yet.
Devin,
As soon as I found something that worked I stopped searching.
For reading and writing I use mostley simple 'object=open(file,mode)' and for reading I loop over the file one line at the time 'for line in object:'. Very compact and simple code. Writing is 'object.write(reg) where 'reg' is a string.
Readline() is more complicated I think. See
https://docs.python.org/2/tutorial/inputoutput.html#methods-of-file-objects
The csv module is useful for reading parts of complicated csv-files, it produces a list of dictionaries. It is possible that it makes more elegant code for writing csv-files than what I used so far, if there is time I'l try, never to old to learn.
PS, my code uses the semicolon as delimiter, I'm from Holland. You want to change it to colon I think.
Met vriendelijke groet,
G.J. (Gerard) Havik
Dataspecialist
een gedachte voor het milieu: is printen van deze mail echt nodig?
Verstuurd vanaf mijn iPad
Op 23 mei 2016 om 22:37 heeft Devin Underwood <geonet@esri.com<mailto:geonet@esri.com>> het volgende geschreven:
GeoNet <https://community.esri.com/?et=watches.email.thread>
How do I loop a folder of mxds
reactie van Devin Underwood<https://community.esri.com/people/EscondidoAnalyst?et=watches.email.thread> in Python - Bekijk de volledige discussie<https://community.esri.com/message/610396?et=watches.email.thread#comment-610396>
the csv module is well documented 13.1. csv — CSV File Reading and Writing — Python 2.7.11 documentation and there are thousands of examples online
I am still working on trying to get the csv to write. You know about readlines and writelines? I just leaned of these, yet I haven't noticed anyone mentioning this option, ever.
I found a script.
import arcpy, os #Read input parameters from GP dialog folderPath = arcpy.GetParameterAsText(0) if folderPath=="": folderPath = r"D:\TESTFOLDER" #Loop through ech MXD file for filename in os.listdir(folderPath): fullpath = os.path.join(folderPath, filename) if os.path.isfile(fullpath): if filename.lower().endswith(".mxd"): #open rapportfile for MXD outFilename=fullpath[:fullpath.rfind(".")]+".csv" mes= '\nMXD: %s' % (fullpath) print mes arcpy.AddMessage(mes) rapportfile=open(outFilename,"w") header='MXD;WORKSPACE;FEATURECLASS' rapportfile.write(header+'\n') mxd = arcpy.mapping.MapDocument(fullpath) for df in arcpy.mapping.ListDataFrames(mxd): layerList = arcpy.mapping.ListLayers(mxd, "", df) mes='MXD %s bevat %s layers' % (filename, len(layerList)) arcpy.AddMessage(mes) print mes for lyr in layerList: if lyr.supports("dataSource"): workspace=lyr.workspacePath fc=lyr.datasetName print 'WorkspacePath: %s' % workspace print 'FeatureClass: %s' % fc reg='%s;%s;%s' % (filename,workspace,fc) #arcpy.AddMessage(reg) rapportfile.write(reg+'\n') mes='Papportbestand: %s\n' % (outFilename) print mes arcpy.AddMessage(mes) rapportfile.close() del mxd
I am going with csv instead, so I can share the script not worrying whether a person may or may not have openpyxl. Csv should suffice, but I am having a little trouble.
I have the following and can see the csv file in windows explorer processing/looping the files, but when I open the csv it has one file just repeated in several rows. You know what may be the cause?
with open('CSVLISTLAYERS.csv', 'wb') as outputcsv:
writer = csv.writer(outputcsv, dialect = 'excel')
for filename in (fullpath + lyr_source):
writer.writerow ([fullpath + lyr_source])
I have tried write to csv also, but it seems that openpyxl is more versatile, e.g. write to a native xlsx .
I will see what works out for me and let you know.
This is what worked for me. I used only what I needed and added openpyxl so I can write to excel. Yet I am still working on it successfully writing to excel
#Import Modulesimport arcpy,osfrom openpyxl import Workbook
#Set folder spacefolder = xyz
#Set variables# create excel worksheetswb = Workbook()ws1 = wb.create_sheet("yyy")
for filename in os.listdir(folder): fullpath = os.path.join(folder, filename) if os.path.isfile(fullpath): basename, extension = os.path.splitext(fullpath) if extension.lower() == ".mxd": arcpy.AddMessage("Processing: " + basename) mxd = arcpy.mapping.MapDocument(fullpath) dfs = arcpy.mapping.ListDataFrames(mxd) for df in dfs: arcpy.AddMessage("DataFrame: " + df.name) layers = arcpy.mapping.ListLayers(mxd, "", df) for layer in layers: if layer.isFeatureLayer: lyr_source = layer.dataSource lyr_name = layer.name.encode("utf8", "replace") arcpy.AddMessage("Copying: {}".format(lyr_name)) print fullpath + lyr_source
wb.save('aaaaaaa.xlsx')
Thank you for your help. My next step is the openpyxl writing to excel.
Sorry, was in a rush to leave at the end of the day and didn't properly read your post.
First you need to get a listing of all the data frames and go through each of them:
DataFrameList = arcpy.mapping.ListDataFrames(MxdFile)
for DataFrame in DataFrameList:
There is a bunch of info you can get on the data frame. Then in each data frame loop get a listing of the layers in each data frame and go through each of them:
LayerList = arcpy.mapping.ListLayers(MxdFile, "", DataFrame)
for Layer in LayerList:
You can also get a listing of tables:
TableList = arcpy.mapping.ListTableViews(MxdFile, "", DataFrame)
for Table in TableList:
I have attached the script I created.
Hi Gerard,
I am not having trouble going through each mxd in a folder. I am having trouble reporting the colour, linestyle type, thickness, point symbol, etc... that a symbol is using. There doesn't seem to be any way of doing that. I want to report all the information so that someone could, if needed, recreate the mxd from scratch and get the same result - or if I want to know how the mxd was at certain times and report on it and compare with other times in a table format, without having to back up the mxd and opening it up to see all the settings, symbology, etc... .
Hi Devin,
I don't know if this is the best way to do this (probably not), and it doesn't do subfolders (Gerard's code seems to do this) but this is what I do (it works 😞
##Set the mxd folder path.
MxdFolderPath = r"C:\GIS"
##Loop through each file in the folder.
for FileName in os.listdir(MxdFolderPath):
FileFullPath = os.path.join(MxdFolderPath, FileName)
##If the file exists then...
if os.path.isfile(FileFullPath):
##Initialize the file extensions to look for list.
FileExtensionsToReportList = [".mxd",".Mxd",".MXD"]
##For each file extension to report on...
for FileExtensionToReport in FileExtensionsToReportList:
##If the file ends with the file extension to report on then...
if FileName.endswith(FileExtensionToReport):
Hi Devin
I wrote the following python script to copy the datasets (feature classes & rasters) for each mxd within a folder into a new File Geodatabase. You could use the following as a starting point and instead of copy it out write the location of the layers (feature classes and rasters) into a summary table.
''' Created on Jan 20, 2016 Copy all feature classes and rasters from each mxd in a folder into a new File Geodatabase. @author: PeterW ''' import os import time import arcpy # set arguments folder = arcpy.GetParameterAsText(0) out_gdb = arcpy.GetParameterAsText(1) # folder = arcpy.GetParameterAsText(0) # out_gdb = arcpy.GetParameterAsText(1) # Processing time def hms_string(sec_elapsed): h = int(sec_elapsed / (60 * 60)) m = int(sec_elapsed % (60 * 60) / 60) s = sec_elapsed % 60 return "{}h:{:>02}m:{:>05.2f}s".format(h, m, s) start_time1 = time.time() # copy layers function def copy_features(): try: if arcpy.Exists(os.path.join(out_gdb, layer.datasetName)): arcpy.AddMessage("Feature class already exists, it will be skipped") else: arcpy.FeatureClassToGeodatabase_conversion(lyr_source, out_gdb) except: arcpy.AddMessage("Error copying: " + layer.name) arcpy.AddError(arcpy.GetMessages()) def copy_rasters(): try: if arcpy.Exists(os.path.join(out_gdb, layer.datasetName)): arcpy.AddMessage("Raster already exists, it will be skipped") else: out_raster = os.path.join(out_gdb, layer.datasetName) arcpy.CopyRaster_management(lyr_source, out_raster) except: arcpy.AddMessage("Error copying: " + layer.name) arcpy.AddError(arcpy.GetMessages()) # Loop through each data frame, layer and copy to new file geodatabase for filename in os.listdir(folder): fullpath = os.path.join(folder, filename) if os.path.isfile(fullpath): basename, extension = os.path.splitext(fullpath) if extension.lower() == ".mxd": arcpy.AddMessage("Processing: " + basename) mxd = arcpy.mapping.MapDocument(fullpath) dfs = arcpy.mapping.ListDataFrames(mxd) for df in dfs: arcpy.AddMessage("DataFrame: " + df.name) layers = arcpy.mapping.ListLayers(mxd, "", df) for layer in layers: if layer.isFeatureLayer: lyr_source = layer.dataSource lyr_name = layer.name.encode("utf8", "replace") arcpy.AddMessage("Copying: {}".format(lyr_name)) copy_features() if layer.isRasterLayer: lyr_source = layer.dataSource lyr_name = layer.name.encode("utf8", "replace") arcpy.AddMessage("Copying: {}".format(lyr_name)) copy_rasters() # Determine the time take to copy features end_time1 = time.time() print ("It took {} to copy all layers to file geodatabase".format(hms_string(end_time1 - start_time1)))
Let me know if you need help amending the following to meet you needs.
David,
Try the following:
import arcpy,os def printlayers(mxdfile): mxd = arcpy.mapping.MapDocument(mxdfile) layers = arcpy.mapping.ListLayers(mxd) for layer in layers: if layer.supports("dataSource"): print layer.dataSource del mxd folderPath = (xyz) for root,dirs,files in os.walk(folderpath): for file in files: if file.endswith('.mxd'): printlayers(os.path.join(root,file))
For using os.walk study: OS.walk in Python
part of the fun in coding is figuring out how to use something for your own project.
Gerard Havik
The following code prints out the data I want for a specified mxd. I want to iterate every mxd in a specified folderpath and return the layers. Any idea how to do this ?
import arcpy,os
folderPath = (xyz)
mxd = arcpy.mapping.MapDocument (xyz)
layers = arcpy.mapping.ListLayers (mxd)
for layer in layers:
if layer.supports("dataSource"):
print layer.dataSource
del mxd
Thank you, I will take a look and parse only what I need. Which is the most difficult part, there are examples and suggestions that I wish only had just what I needed.
go only one niveau ?
Just finished a project to extract as much relevant layer information as possible from MXD's in a folder with folders.
The choice to go only one niveau is intentional. This makes backup folders 'invisible'.
Hope this helps, suggestions for improvements are welcome.
I have created a Python script to give info on all mxds in a specified folder. The output is a separate text file for each mxd. I based it off of this page:https://community.esri.com/thread/19655. I found that not all available document/layer/table... info was being reported on so I used this page http://resources.arcgis.com/en/help/main/10.2/index.html#//00s300000008000000 to get as much info as I could (although I am having trouble getting data driven page info). There is still info I would like to get out (like feature symbology), but this is the best I could do/find.
Keep in mind, if I remember correctly, the inventory lists all whether broken link or not.
The broken link list will list only those, and may differ depending on relative relationship, and user permissions, if you copy the data.....which I still recommend for testing. I'm using it to find where users have local data for mxd's that are shared and in a network drive. helps me find the local data that needs to be centrally located and fixed. That might not be your need....but just a tidbit of possible usage.
This looks great, I just browsed it quickly. Good advice use on a non-working testing folder/files. I will test this out. Thank you.
Devin, it definitly is worth trying my data inventory addin, as Dan mentioned. since there are so many different types of data and data connections that can be within any mxd, there are many tests that need to be filtered thru to find the right pe. The add in does that fairly well, but still may not catch ALL oes yet. But it does spit out an excel and .csv output so you can see what you have. It will do recursive mxds in a folder, and runs fairly fast so, the inventory part you should try.
if it doesn't get what you need, and .addin is just a zip file....change the extension and unzip. You'll see how I looped thru everything, and you can grab snippets from thatand modify if needed.
Rebecca Strauch, GISP produced this Python addin for data inventory and “broken-link” repair.
which does project inventory, if it doesn't serve purposes, perhaps she might have some commentary on what you are trying to accomplish
You can use the Walk function in python that Michael had mentioned to loop throught the mxds. For each mxd, you can use what he later suggested with looping through the dataframes and layers.
A loop within a loop.
Can I input the workspace for listdataframes & listlayers since I want info on 19 mxds, not just one ?
You'll need to get the DataFrames in your mxd
arcpy.mapping.ListDataFrames("name of mxd")
then you'll need to get the layers in each data frame
arcpy.mapping.ListLayers("name of mxd, "", "name of data frame")
I have the following since I am not sure whether os.walk limitation apples to me since I want to get feature classes information within mxds.
import arcpy, os
workspace = ' xyz '
for root, dirs, files in os.walk(workspace):
for f in files: if f.endswith(".mxd"): mxd = root + '\\' + f print f
I get a list of mxds which is the first successful step but I need the feature classes.
FeatureClasess with info of their respective dataset.
What kind of database related features are you looking for?
Looks like I need the arcpy.da.Walk and not the os.walk which doesn't return database related features.
Still working on it though.
Thank you for the advice. I have been using X-ray, but it doesn't quite fit my purpose. I need the mxd info written to a excel sheet to be used as a ArcGIS table.
Also, the most import need is to loop which X-ray is used on the currently opened mxd.
If you are interested in the innards of the mxd, many recommend ... https://www.arcgis.com/home/item.html?id=f0ae73e90c1a4992a1059e7d370966d4 X-ray
Search through geonet threads for Walk function in python. You can also search outside geonet for python and walk.
Signed in members can post, follow updates, and more. New here? Register a free account.
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.