arcpy.da.Walk()-- read KMLs, Personal GDBs, etc.

[AlfredBaldenweck]

I'm giving arcpy's Walk a shot, and I've run into a disappointing setback: It doesn't notice several file types I'd like to see.

For example, I'd like to see KMLs (supported by Pro???), Personal GDBs, PDFs, Word Documents, and zip files.

I don't even need to read the contents of things like Personal GDBs right now; I just want to know they're there. (Sidenote: It will read coverages but not recognize if a Personal GDB or KML is there???)

I'm disappointed because arcpy.da.Walk takes about 15 seconds to evaluate on test data, and the functions I'd built with os.path.walk (including filtering out shapefile component files, etc), takes about 70 seconds on the same data, but return things like PDFs.

Does anyone know of a way to get arcpy's Walk to see other files?

[GhislainPrince]

Hi Alfred,

arcpy.da.Walk only returns object types recognized by the arcgis Pro application. Probably best analog is that walk will return data you see in Pro's catalog pane.

Pro doesn't support Personal GDBs, so we don't see/find them in the catalog pane, nor in arcpy.da.Walk.

For objects which happen to be a file on disk (eg: .zip, .pdf, ...) it's totally valid and good to use os.walk to find those.

[AlfredBaldenweck]

That's what I was afraid of. (Speaking of what we can see in the Catalog Pane, we still can't add file types for viewing there.)

So, then, when can we get some sort of function to let us do a file inventory that gives us useful files? 

I've been working on this for a couple months. My current workflow, using os.walk, is to do ListFiles, ListTables, ListRasters, ListFeatureClasses, and then filter out all the extraneous files that I really don't need to see, followed by any files that appeared in the other 3 lists: 

That's 96 different formats that I've had to enumerate, and I'm definitely still missing some. 

And this works! But also it took much longer than I'd've liked to compile this list and figure out how to filter it all.

Our options are to either get not enough or way too much. 

Heck, I would take a function that listed non-GIS files. I'd be fine pairing it with arcpy.da.Walk() for a useful inventory. I just really, really don't ever need to see an .ixs file.

And I get it; Pro isn't made to work with MDBs, so it doesn't see them. Same with coverages. But there's a ton of legacy data in those formats out there, and it'd be nice to at least know it's there so we can try to convert it before we lose access to ArcMap. I don't know, maybe having them appear in Catalog but greyed out.

Also, again, it's weird that it ignores KMLs (which can be used in Pro and appear in the Catalog Pane), but can tell where a coverage is, even though Pro can't see them. 

KMZ, as seen in the Catalog Pane	Coverage, as exposed in arcpy.da.Walk() in Pro 2.9.5

[GhislainPrince]

Hi Alfred, 

community ideas are good way to go for that catalog pane "add file type" feature, also getting attached to ENH requests with tech support.

We have arcpy.da.Walk to get data we know about, for the data formats Pro doesn't know about the approach of os.walk seems right to me.   Your approach of a white and/or black list for your org's other data seems right.

Finally we've worked on arcpy.da.Walk in recent releases, there's improvements to the online doc, as well as improved results for KML, .geodatabase, .atbx, parcel fabrics in the Pro 3.x releases.

Cheers

[Anonymous User]

Could you possibly multithread the Listing and filtering?  Run the ListFiles, ListTables, ListRasters, ListFeatureClasses in parallel and then combine the results to maybe speed it up until another solution is found?

I'd add for @GhislainPrince  that Walk is returning Rasterbands as Featureclasses. There is another post on here I helped with and when I tested the Walk for Featureclasses, it returned a Rasterband file. This was within the last couple weeks in 3.1.

[AlfredBaldenweck]

Posted an Idea about this here Functio to List Non-GIS files - Esri Community