Latest Contributions by KimOllivier

I just define no parameters for the tool. Then if you add it to the menus using Customise.. you can add the tool as a button. Executing the button will not popup if there are no parameters. ... View more

People seem to be having some serious problems geocoding a small file of one million records! You should never have to split a table for geoprocessing since it is a serial process. Unless you are trying to use a server geocoder for batch. In this case the app posts all the records across and back again, not really likely to be fast enough before a timeout. As a benchmark, I expect to be able to geocode addresses at the rate of at least 1M/hour and sometimes up to 6M/hour on my desktop machine. My typical file contains half a million addresses. I have the reverse performance experience with single line versus batch processing, single line slows down to one tenth the speed because zone indexes cannot kick in. Therefore I hope you can keep looking for the non-obvious error that is causing the problem for you. My laptop has no trouble geocoding any size file, so it is not the size of the machine. It is likely that indexes are essential for normal use. Do you have parts of your address that can be indexed to allow the geocoder to break up the search? Such as state, zip or city? Any geoprocessing across a network drive is a mistake! This includes geocoding. Networks are just not fast enough with all the packing and unpacking to send each request in a reasonable time. Put your geocoder, source and output on the same local machine on a local drive. I understand this will break "corporate policies" nearly everwhere, but do a demo to show the problem to your administrator. You will have to copy-process-replace. This will also apply if you are using a remote database using SDE. I personally copy any source from a spreadsheet, shapefile or text file to a local filegeodatabase table. That enables indexing and unlimited sizes and makes it easy to put back in an enterprise database. You have complete control over the field types, null values and other unexpected data that will trip up the geocoder. Null values are sure to cause indexing to fail. It is very hard to set up a custom locator properly, I have yet to make them work properly myself after trying to tweak the defaults. The default timeout in my locator is 100 seconds, but who wants to wait for that for each failure? If it takes even a millisecond it is too long, so I set it for 1 second. If timing out is a problem, then indexing is not kicking in. If indexing is not running, why not? Either the reference does not have suitable indexes being built or you are throwing address candidates at it that do not have indexable components, eg no state when the locator expects a state. ... View more

The server is setup to only handle single line addresses designed for interactive matching. You can Post batches of multiline addresses for batch processing on the server, but that is not really a very good idea for large databases. You should expect to handle megabyte databases locally (ie on a LAN). There is only limited customisation available for batch format, change the field names etc. You have to have all the address components in four fields like a mailing label. I wanted to split out the components into flat, house, road similar to the format required to build the locator, but you cannot do that because the geocoding engine is hard coded to only accept a few formats. You have to have only flat_house_road suburb_town_locality state_region_city zip_postcode country Only the names can change, plus you can parse the contents a few different ways. You have control of the returned fields, but it is very intricate to change. ... View more

I would not be happy with colours unless they are accessible as a descriptive attribute for programming. My experience with users colour coding cells in Excel has been awful. It is impossible to convert the colours to codes for scripting analysis without very low level coding if at all. I had to go through and add more classification columns and then add a legend to show what they meant. ... View more

I would not be happy with colours unless they are accessible as a descriptive attribute for programming. My experience with users colour coding cells in Excel has been awful. It is impossible to convert the colours to codes for scripting analysis without very low level coding if at all. I had to go through and add more classification columns and then add a legend to show what they meant. ... View more

Why would you want 125 MXD files in the first place?? Surely the point of Python scripting is that you manipulate one MXD to avoid the proliferation of MXD files. Out of curiosity I have just done a count on my system and I only have 222 MXD files after 10 years of ArcGIS processing. The mapping module helps a lot with reducing the number of MXDs required, but you can also use templates, layer files, styles to reduce the number. ... View more

How can there be 5 different names for a street? I think you must have missed the key features of a locator that can match alternative abbreviations and spellings for prefixes and suffixes. It can handle a lot of variation for a street name. Are you also including alternative locations as a different street name? These can be added as alias list for the same street name. You can edit the XML configuration file to add more alternatives to handle the combinations. What about posting a few examples so we can see if they are really different? ... View more

Send a request to Google's geocoding service and you can get back a single address. There is a limit of 50,000 in a day. Since it has to go off across the internet it may not be faster than using in_memory featureclasses (that is not a real file). Did you miss the clever trick suggested by Chris Fox using in_memory as a virtual disk drive? I assume that if you want a fast response from one address you are trying to geocode a lot of them? If so, take out the loop and geocode them all in one featureclass that will be really fast. You should expect 1 million per hour if done locally on real files. Here is a way to use Google: # geopy example # http://code.google.com/p/geopy/wiki/GettingStarted from geopy import geocoders apikey = "your_api_key_adfvaklfmuyrdghbndfzcvazdfzcvsfgsfvsfgsdfbvsfgsghsdfgd" g = geocoders.Google(apikey)  place, (lat, lng) = g.geocode("380 New York Street,Redlands,California,USA")  print "%s: %.5f, %.5f" % (place, lat, lng) ... View more

Yes..yes. sorry to be so confusing. Again, can we please get past the issue of what I should or should not have done programatically to reduce the number of feature classes? I've already fixed that for future runs 🙂 But what's done is done and I am trying to access the geodatabase, just like someone else who may be dealing with a file geodatabase with 20k+ feature classes...which is the real issue here. The total size of the gdb is ~20GB I am trying to help you solve the original problem, not get you down from being stuck up the wrong tree. I think you should abandon the 100 hours of effort and reconsider your approach. You have clearly found some limits with your approach. It think it is because the software was not designed to handle the problem by splitting into individual featureclasses. Here is an example that is close to your problem. It is an OD matrix of 13,000 routes. It was done by loading all the points into one layer and solving for all routes at once. The processing time took less than one hour. The file geodatabase is 112MB. I didn't even have to write a script. I have access to all the routes with from and to IDs, a unique name for each route and the total length. What else do you need? ... View more

Ok, there must be a limit somewhere, and it won't be the length of a python list. Have you tried making a list of one? That is, instead of a default, put in a wild card that is an exact file name. arcpy.env.workspace = "path_to_gdb" # essential print arcpy.env.workspace # just to check listOfOne = arcpy.ListFeatureClasses("thefirstfile") print listOfOne If this works, then maybe you could make a wild card listing of a subset listOf2k = arcpy.ListFeatureClasses("image1*") print len(listOf2k) Second suggestion, if you already know what the names are, just use a python list to process them. No need to get a list at all eg suppose they are image00000 to image19999 (this really hurts because there should not be so many) for n in range(20000):     fc = "image"+str(n)     if arcpy.Exists(fc):         process(fc)     else :         print "Oops, guessed wrong",fc Maybe arcpyDescribe would be a workaround, this has a child_list property: desc = arcpy.Describe(gdb) lstChildren = desc.children print len(lstChildren),"featureclasses in ",gdb, "from",lstChildren[0]," to ",lstChildren[-1] Maybe you could split the data into multiple databases of 5000 featureclasses? It would help if your terminology was precise, computers like that. You cannot have shapefiles in a geodatabase, they are featureclasses. Do you really have a folder of shapefiles or a filegeodatabase of featureclasses? Is it really a personal geodatabase in an Access database? If they are really shapefiles, that might explain a lot. You could use the glob module to make a list. import glob listOfShapefiles = glob.glob(*.shp") Perhaps you could install PostGIS which being an Enterprise geodatabase will not have any limits. IMHO you haven't got a large geodatabase, just a disfunctional one. A large database is something with one featureclass greater than 2 GB.  Why would you persevere with something that takes 30 minutes to open in ArcCatalog? Surely this indicates a drastic rethink of you schema and workflow? Of course you can reconfigure you problem to run in the time to drink a cup of coffee. My suggestion is to abandon your 4 days of processing and work out a better way over a cup of coffee. You must be making a fundamental mistake. I have never run a process for 4 days and would not consider it a solution if it took that long. ... View more

Kimo, thanks for the reply. however I don't know of a better way to acquire the information that is required. Because arcgis does not do route analysis for an OD Matrix (it only provides a cost report), if I am wanting to show/analyze all of the streets that are involved in, say a 70x300 OD Matrix for (a very large city) You seem to be reinventing the available tools in a script. If you can think of a workflow that processes whole featuresets in once step you can avoid stepping through each feature. Think of the processes like an SQL query. In a relational database you do not have a loop to interate over each row. The same applies to featureclasses, they are supposed to be "geo-relational". It takes a while to get out of the "pilot mindset" - think of a solution for one feature, then wrap a loop around that. Instead think in a more global, generalised way on the relationship between features and find a tool to do it in one step. If any process takes longer than a cup of coffee, interrupt the process and find a better way! I have never had to run a model that takes longer than drinking the cup of coffee. After all you have to do many trials. ... View more

Short of waiting for a bugfix here are some strategies that people have used: Encapsulate the tool in a python function using local variables in the hope that they are removed on exit from the function Use a python system call to start a new thread so that the objects are destroyed on exit. You have to pass parameters, temporary featureclass references and other complications. You might also try installing the 10.1 SP 1 Python64x which can use Python 64 bit for geoprocessing tools running in background. However if it is a memory leak then it would just postpone the crash. Any process that opens a cursor to interate over features using another geoprocessing tool is a poor workflow design for tools that are designed to process the whole dataset as one entity. Could you not redesign the flow to process the featureclass in one pass? Perhaps post your workflow for help in doing this. ... View more

That is the catch with old MXD files. The graphic elements did not have element tags for a name, so you have to go through by hand and give each one a name to use the new mapping module to edit them. ... View more

They are DOS BAT file commands. They are in the windows help, or were at XP. Strange, does MSoft expect us all to stop using batch files and scheduler?? http://technet.microsoft.com/en-us/library/bb490890.aspx The idea is to cascade back status of each tool to the command script and log all messages to a file so you can see what happened. But now I have found that a lock is always going to happen if you try to update a filegeodatabase with two scripts simultaneously. I have now been reminded why it is not working while looking at the Esri Resources blog. A filegeodatabase is a single user database, and multiple scripts are effectively multiple users, unlike an Enterprise database (Oracle, SQL Server etc) where locks are per featureclass. http://blogs.esri.com/esri/arcgis/category/subject-analysis-and-geoprocessing/page/7/ [INDENT]Here are some important considerations before deciding to use multiprocessing: The scenario demonstrated in the first example, will not work with feature classes in a file geodatabase because each update must acquire a schema lock on the workspace. A schema lock effectively prevents any other process from simultaneously updating the FGDB. This example will work with shapefiles and ArcSDE geodatabase data. [/INDENT] ... View more

Maybe your scripts are not as independent as you think. If you are changing a schema of a featureset, then the filegeodatabase will get a lock, not just the featureclass. Locks happen all the time in ArcCatalog if you have it open while running a script from Pythonwin. It is often hard to get rid of the lock without exiting python completely, or ArcCatalog due to arcpy holding on to locks. I have found a tool (ExportToKML) that does not release the XML file. Not quite a featureclass, but same principle, so I expect bugs like this in other tools. You can test for schema locks in arcpy. Perhaps put in a test and wait a few minutes in a loop? Also try to ensure that locks are released. You should be able to track down the lock and then it will be easier to recode to avoid it. At 10.1 use the new capability with arcpy.da.SearchCursor() as cur:      for row in cur:         # do something Make sure that you try to release all potential locks. One way is to wrap tools in a python function, this does a garbage collection on exit. try using arcpy.RefreshCatalog(gdb) If you do use BAT files, add in the logging module to log results and return a value so that the batch file can stop on a failure. on exit set the return value to 0 or an error message In the batch file log any messages using: set log_file=logPath python myscript.py >> %log_file% 2>&1 if %errorlevel% neq 0 exit /b %errorlevel% Cursors and files are only closed when Python does a garbage collection. The "with" construct is better at closing files immediately. It is the closest we have to a file_handle.close() for geoprocessing tools. ... View more