listfeatureclasses bugged?

[RyanSchuermann]

I'm really stumped...
simple code here....not working....

import arcpy
arcpy.env.workspace = "C:/Users/Ryan/Routes.gdb"
shplist = arcpy.ListFeatureClasses()
print shplist

my Routes.gdb has ~9,000 shapefiles in it. Each shapefile is very small, maybe 3 to 50 line features (a route) The list returns empty...the output is []

If i run this exact code above, but point it to a Routes.gdb with only 25 shapefiles, it returns a correct list. This smaller gdb was generated from the exact same code that generated the large gdb, just run against a very small sample of input data.

I know the ~9,000 shapefiles are in Routes.gdb because I can see them in ArcCatalog 10.1 ..it just takes 30minutes for them to load in the Catalog Tree, or the "select input datasets to Merge window".

I even tried getting a subset of the files in Routes.gdb by using a mask, shplist = arcpy.ListFeatureClasses("Route0*")still no go...yet it works with the small gdb.

I thought file geodatabases were suppose to be able to handle large datasets? I could even understand if later the merge failed hitting the 2GB limit I read everywhere, but I can't even get there without a file list...the file list should not fail...at least I have yet to see anyone post that it has a limit.

Only thing I can think of is the list is too long, in which case, a subset, selecting, ~300 files should be no problem but that fails too as mentioned above.

I'm going to try and wade through the lag that's encountered when trying to use Merge from ArcCatalog, but that may be the only way to merge large numbers of small shapefiles? I would rather NOT have to leave python, just to do this and then go back to python, thus forcing me to run multiple programs instead of one nice and clean, fully functional program.

what am I doing wrong, or could improve on? thanks!

I'm running this on a server-grade system octocore 2.66GHz w/ 20GB ECC FB-DIMMS so it's not a wimpy system issue.


...and YES 🙂 in hindsight (and i've already changed the code to do so) I should have been merging them in small chunks as I generated the files..but it's too late now. This process took over 100 hours to run non-stop (running on a RAMDISK) and I don't have time to run it again.

[RyanSchuermann]

I am trying to help you solve the original problem, .......
Here is an example that is close to your problem. It is an OD matrix of 13,000 routes. ......I have access to all the routes with from and to IDs, a unique name for each route and the total length. What else do you need?

But....the reality is, you seem to be hijacking this thread with a non-problem. This is the python section, and I am trying to find some help concerning listfeatureclasses ...if there are any known limitations/bugs/etc, of if the problem lies in my application of said arcpy function.

If I wanted help on how to combine ArcGIS's OD Matrix and Find Route functionality, I would have posted it in the Network section 🙂

I appreciate your insight on my Network Analysis approach....but I never said what I did was the optimal way, and have since said once I realized what the client's needs were exactly, I modified it to run in half the time...

However...
You are still completely missing the point of my process. Solving an OD Matrix in ArcGIS does not tell me what streets are used in the OD Matrix, and at what frequency. You just said yourself, you only have the start,end points and cost of the route. To actually get the route, you have to solve the matrix in a nested loop, and even then it only gives you the route, not the streets involved in the solution. So, you then have to spatially select the streets based on the route solution, and then save the Street segment IDs to a lookup table (or as I did, a list) and then preform frequency analysis (which 2.7 does in a flash). Your whole spiel on, "no process takes longer than a cup of coffee", doesn't hold (water) here. Considering the Network contains MILLIONS of street segments, do you realize how long it takes to solve a route from one end of the network to another?...and then do a spatial select against those millions of street segments..and then repeat that 20,000 times? 4 days. They key here to reducing that time in half was not needing to save each route's streets as a feature class...which was, as the previous poster said, redundancy.

if you can show me a better way than going through the OD Matrix w/ nested solve loops, I'm all ears for future applications! 🙂