Latest Contributions by JoshuaBixby

I was really thinking it might be something specific to that machine, but obviously that isn't the case.  I would say or suggest opening Support case, but best case scenario would be a bug that will be very low in terms of priority since it involves personal geodatabases.  This is where I would just find a workaround and say c'est la vie. Good luck ... View more

Seems like a pretty deep or pervasive problem, which should provide some sense of urgency one would think, but it could also be the excuse for not addressing the issue at all.  I have seen the latter more than a handful of times. Thanks for pursuing with Esri Support and getting a bug logged. ... View more

10k records isn't that many, really.  I assume the limit or problem is tied to the number of records being returned by the UpdateCursor and not the number of records in the underlying dataset, right?  Also, have you tried the older arcpy.UpdateCursor instead of the arcpy.da.UpdateCursor?  I know the former is slower, but if it works and doesn't generate duplicate rows. ... View more

Do you have another machine to try it on?  Does this happen with all machines you are working on or just one? ... View more

What's making the MXDs grow so much?  Good question, I wish we had clear answers.  Part of it is likely user behavior, although it is unclear what all behaviors cause the problem, but another part is how the MXDs are structured (design structure) and how ArcGIS interacts with them.  Geoprocessing history may play a part in some of the MXDs, but it isn't the dominant issue with the majority of the MXDs. MXD bloat isn't an issue amongst GIS staff as much as the casual GIS user in our organization.  There are numerous worst practices that casual users seem to develop over time, and some of them may contribute to bloat, but there are limits to what can be expected from educating users on best practices.  The bottom line for us is that we have an issue, prevention has its limits, and so we need to explore ways of cleaning it up on the back end. Opening up MXDs and trimming geoprocessing history will shrink some files to varying extents.  What also works is just opening up MXDs and saving a copy using the ArcMap UI.  The shortcoming of this approach is that it is manual.  We are looking for means to automate cleaning it up for thousands or tens of thousands of files.  Right now, either spawning docDefragmeter from Python or shell scripting it works, it is just clunkier and slower than we would like. ... View more

Heard back from Esri Development, "this is very much a known limit."  It seems to a handful or two of folks in Redlands, this is "very much" known; unfortunately up until now, no one has bothered to share it with ArcGIS users by actually documenting it.  At least now they have a bug to document it and to document they aren't going to do anything to address it. I "very much" get this response too often, can't help but think there is a hashtag reply in it: #findanotherworkflow #SOOL. ... View more

When doing Python scripting in ArcGIS, I tend to favor native ArcPy methods over geoprocessing tools.  The geoprocessing tools are convenient, but that convenience can come with a performance hit, and the hit can be significant with some tools.  Even within the ArcPy realm, I have found methods in the data access module (da) noticeably more efficient than corresponding methods in the base site package (arcpy).  (I think this has been one of the main selling points of the da module from the beginning) # import functions from modules that are available but not commonly imported from collections import defaultdict from numpy import fromiter, dtype # sum area by ID stats = defaultdict(int) with arcpy.da.SearchCursor(inputshp, ['ID','AREA']) as cur:     for k, v in cur:         stats += v # sum SUM_AREA - using sum over an iterable of dict's values fieldsum = sum(stats.itervalues()) # create iterable and populate numpy array stats_iterable = ((k, v, 100.0 * v / fieldsum) for (k, v) in stats.iteritems()) tmp1 = fromiter(stats_iterable,                 dtype([('ID', 'i4'), ('SUM_AREA', 'f8'), ('PROZENT', 'f8')])) # Dump numpy array to table arcpy.da.NumPyArrayToTable(tmp1, out_table) del tmp1 del stats .... The code above only uses 2 arcpy functions, both of which are in the data access module.  On a randomly generated million-record test data set, cutting out most of the arcpy functions reduced the runtime by more than 75%.  Profiling the original code shows the bulk of the extra runtime comes from a single function: analysis.py:(Statistics).  I have seen this numerous times, i.e., scripts that call Statistics_analysis on large data sets get really bogged down.  I am not sure how large your data sets are, but if they are large and you want to speed them up, rolling your own functions can usually get you performance gains. ... View more

Thanks for sharing.  Given the ArcSDE SDK is more niche than VBA was back in the day, I assume it won't be given multiple stays of execution, which makes your thoughts on moving forward that much more relevant. Spatial partitioning through the FGDB API, now that could get interesting.  Good to know, and I will have to spend some more time thinking about it. I spend most of my time with Python and SQL, don't dive into the APIs much.  When the day does come, I look forward to seeing the next generation of se_toolkit-like tools. ... View more

Interesting post with some food for thought and good reminders.  Seeing these tests were run with se_toolkit, and the ArcSDE SDK won't be supported beyond ArcGIS 10.2, it makes me wonder how easy or hard it would be to run similar tests in ArcGIS 10.3 or beyond. Esri's DEPRECATION PLAN FOR ARCGIS 10.1 AND ARCGIS 10.2. states: ArcGIS 10.2 will be the last major release to support the ArcSDE SDK with the ArcSDE C and Java APIs. Today many other options are available for developers, including SQL, which is available as a result of the widespread adoption of spatial types, the File Geodatabase API (introduced in 2011), and ArcGIS Runtime SDKs that offer modern environments for the creation of compelling custom applications. I don't think the File Geodatabase API is really applicable here, which leaves "SQL" and ArcGIS Runtime SDKs.  Maybe it is a long week catching up with me, but I am struggling to see where ArcGIS Runtime SDK fits in here.  Thinking of these specific tests, even going down the SQL path seems onerous, or at least not clear at first glance. What options do you see for recreating these types of test post se_toolkit? ... View more

Are you not able to use SQL Server's native spatial types (GEOMETRY, GEOGRAPHY)?  Either native spatial type works with ArcSDE and allows one to perform SQL directly against the data.  If the data is registered as version, you may need to query the versioned view instead of the base tables. ... View more

SDEBINARY?  I don't think it is possible, regardless of Oracle or SQL Server.  Are you using SDEBINARY with the code snippet above?  It looks like SDO_Geometry.  With SQL Server, you would need to use GEOMETRY or GEOGRAPHY. Look at STWithin or STContains depending whether you want to look at it from the point's or polygon's perspective. ... View more

Daniel, to code wrap you need to use the 'advanced editor' option in the upper right corner.  Once in the advanced editor, you can do syntax highlighting. Just to clarify, the code you just posted is generating the cursor object error you mention earlier?  Can you try running the command and pasting the exact error messages that are returned, all of them? One potential problem I see, although it wouldn't generate the cursor object error, is your string representing the feature class.  If you are going to use Windows-style paths (single backslashes for directories), you should put in 'r' in front of the first quote to signify a raw string.  For example, r"C:\Users\damrine" instead of "C:\Users\damrine".  In Python, a backslash is an escape character, which can create problems with Windows-style paths if people don't realize that. ... View more

A few things come to mind. I: drive, mapped network drive or local storage?  If mapped network drive, have you tried copying the MDB to local storage and testing it. Can you create a new MDB and copy the data into it?  If so, does that give same error messages? Can you make an OLE DB connection to the MDB in question? ... View more

Are you using my second code snippet?  The error message you are getting is likely caused by using a Python "with" statement and the older cursors.  The second code snippet I posted most recently does not use the with statement and should work. Your issue with working over the network has nothing to do with Python.  I would argue ArcGIS Desktop has historically handled network-based data, like UNC paths, very poorly. I believe the second code snippet I posted will work, and with any data set.  To defend scripting with Python, it is way more powerful than most of the GUI tools, which one would expect.  And, GUI tools don't always work as well, and I find troubleshooting them much more frustrating than Python. ... View more