Latest Contributions by JoshuaBixby

Simply running compress on a geodatabase doesn't ensure all of the edits get pushed into base tables, unless running compress fully compresses the geodatabase.  Is the VERSIONS table showing the state ID for the Default version as 0?  Also, does the associated versioned view show all the records? ... View more

When you say "in SQL Server," are you querying against the base tables or the associated multiversioned view or versioned view?  You may only have the Default version, but the base tables don't reflect what is in the Default version unless the geodatabase has been compressed.  Since you are using 10.2.2, versioned views should be created automatically when you register the layers as versioned.  Try querying against those views and see if your missing records are there. ... View more

In your original post, you state the SelectLayerByAttribute_management tool is "an expensive transaction."  Assuming you mean expensive in terms of time, what are you using to time the tool and what are you comparing it against? Generally, I stick with the arcpy functions and classes as much as possible and use the geoprocessing tools only when necessary or beneficial.  In this type of situation, i.e., getting the bounding extent of selected features or a subset of features, I find the geoprocessing tools are faster for most of the datasets I interact with regularly. I have had success using a mapping layer and the SelectLayerByAttribute_management tool: def GetExtent(fc, sql):     lyr = arcpy.mapping.Layer(fc)     arcpy.SelectLayerByAttribute_management(lyr, "NEW_SELECTION", sql)     return lyr.getSelectedExtent() Writing simple functions and using timeit on one of my datasets with ~250,000 records, I find using a map layer and SelectLayerByAttribute at least an order of magnitude faster than a search cursor.  I can't say for your datasets, but it might be worth trying the geoprocessing tools instead of relying on a search cursor. ... View more

The complexity, size, or both of the dataset is driving whether or how tiling is being used.  If you have access to ArcGIS 10.1 or higher, you could try using 64-bit Background Geoprocessing.  The 64-bit Background Geoprocessing tools allow for much more RAM to be utilized, which could change or eliminate tiling and get you the results you are after. ... View more

Tile processing of large datasets was introduced in ArcGIS 9.2 (well, it could be done in 9.1, but it was more manual and not integrated or automatic).  Even without looking at the Help, the progress window for the Union tool shows there is tiling occurring: Maybe there were not enough qualifiers on your statement about the Union tool and new vertices because the Union tool has to add new vertices in many cases: If your statement was meant for self unions, then I agree that new vertices seem spurious. What steps are you using to find the extra vertices? I am running ArcGIS 10.2.2 with all but the last couple patches, and I did find extra vertices.  I only closely looked at a handful of the extra vertices, but they were all duplicate vertices.  If it turns out that all of the extra vertices are duplicates, then there is no difference in the geometries of the polygons, well the perimeter and such. The reason I asked what steps you took to find extra vertices is that I used the Feature Vertices to Points tool.  Without doing more digging, I can't say whether the issue resides in the Union tool or Feature Vertices to Points tool. Beyond the tools, have you checked how clean the polygons are?  Looking at a few of the duplicate vertices, they seem to be near areas where the polygons seem to be self intersecting, or really close. ... View more

Do you have access to the SDE schema tables?  Have you looked to see if there are possibly entries for the view even though it doesn't exist?  Have you tried creating the view with a slightly different name?  I know in the past, if I had an mv_view that I didn't clean up completely, I would sometimes get this error message when trying to recreate the view. ... View more

Vince Angelo‌, I remember that blog post.  It was quite interesting and good material to be referenced and looked at in relation to this discussion.  I very much look forward to seeing some of the ideas discussed in that blog post come to fruition, understanding they are side projects of course. The reason I linked to that thread in the SQL Server Spatial Forum was that I thought Jakub K's recursive-quartering tessellation scheme was fairly elegant, and quite simple to implement through a couple table-valued functions.  From what I have seen, the geoprocessing/spatial functions in question do seem sensitive to vertex count, so reducing vertex count definitely helps. For most datasets, maybe tessellation is overkill and gridding will suffice.  For one of the datasets I was working with, that wasn't the case.  Looking at a gridding scheme and recursive-quartering tessellation scheme that both generate similar numbers of partitions for one of the complex polygons in my dataset, one can see how the tessellation balanced out the number of vertices better. Balancing out the vertices better allowed for the next part of the geoprocessing/spatial functions to execute quicker.  Using just the DBMS, no RAM caching, the recursive-quartering tessellation scheme got me in the range of 0.5 ms/feature while the gridding scheme was closer to 1.5 ms/feature.  I admit I didn't spend much time on optimizing a gridding scheme, so there is likely room for improvement there, but I am not sure if a more efficient gridding scheme alone will do much better than the current overall performance of using the tessellation scheme. I am pretty content with 0.5 ms/feature, and Jakub K's tessellation scheme got me there, so I thought I would share.  I know there are techniques that can get even better performance, but for now that extra investment isn't necessary for my situation. ... View more

I bit more information about the database would be helpful.  For example, what version of SDE and what DBMS and version?  Also, are we talking about two feature classes in the feature dataset?  A feature class and a table?  What command and syntax are you using to try to create it from the command line? ... View more

Not allowing nulls is one way of forcing users to populate a field, but it sets the bar pretty low in terms of data quality expectations.  If users are forced to enter something because nulls aren't allowed, and there are no restrictions on what can be entered, people will get creative on what they enter.  In these situations, not allowing nulls is worse than allowing them because at least when nulls are allowed and a field is populated, the odds are better than not someone didn't just make up a value. If you want to make users populate a field and you also want to increase the odds of having decent data, then it might make sense to look into Attribute Domains and Subtypes.  If you want to allow non-null, free-form text than some level of programming will likely be needed to achieve what you are after. ... View more

Are you wanting to change the default value for a field?  Does the Assign Default To Field (Data Management) tool not work for you? Regarding your last question about finding a way to check whether the valued entered is blank or not, are you talking about doing this in real-time as a user inputs text or going back through the records to see which ones are blank and not blank? ... View more

SQL Server Express is good for what it is, a free and limited DBMS, but I don't think it will take much to start hitting resource limits in SQL Server Express once you take the 3-connection limitation away.  Depending on your usage, you might want to consider upgrading to SQL Server Standard or Enterprise or even PostgreSQL if cost is an issue for upgrading SQL Server. ... View more

Thinking back to the statement about a point-in-polygon query taking 17 hours to run, my guess is that you have some complex or dense municipal boundaries.  SQL Server, even Enterprise Edition, doesn't seem to handle point-in-polygon queries well when there are complex or dense polygons.  Some say it is tied to MS's implementation of quad-tree spatial indexes or a limitation of quad-tree indexes for spatial data.  I am not deep enough into databases to really have an informed opinion, but I do have to tweak spatial indexes much more in SQL Server than in PostgreSQL/PostGIS. For the complex/dense point-in-polygon situation, I think there is something more with how SQL Server implements some of its spatial methods.  If you either don't or can't go down the geoprocessing route, I encourage you to read through this MSDN forum thread on SQL 2K8R2: Any performance hints for bulk loading data from spatial/GIS db into a data warehouse?  Although the title doesn't make it seem relevant to this post, it is.  The author of the thread has an approach and some code for using tessellation or on-the-fly partitioning to break up complex polygons and dramatically reduce run times.  I have successfully used the technique and seen reductions of 90%+. ... View more

This one was on Esri.  The beta licenses expired, but the pre-release licenses were never automagically loaded when they were supposed to, which wasn't a problem until the beta licenses did expire.  New licenses were loaded, and we are back in business. Take home message, don't hesitate to contact your Esri Customer Service Representative if what you see in AGOL for Pro licenses doesn't make sense or seems like it should work but doesn't. ... View more

I agree with James Crandall‌ about SELECT * . Are you using pyodbc?  If so, fetchall() returns an empty list, even if no rows are returned, so I don't get why rows wouldn't be defined.  If a successful query, even one with no results, returns a list that could instantiate rows; maybe the queries are generating an error and fetchall is returning nothing, so rows doesn't exist. ... View more