This post is the culmination of countless hours supporting Runtime Apps and Collector with ArcGIS Server using SQLServer. We have built and deployed over 30 runtime apps for field data collection over the past 3 years, but the issues we are currently experiencing at 10.3 are forcing us to reconsider our options.
Our problems primarily relate to 3 significant bugs in ArcGIS Server. All of these are known to Esri, and are documented as bugs in some form.
- ArcGIS Server fails to check-in large uploads. The specific size limit is unknown, but a large upload (say with many attachments) will fail. This seems to be a known bug. Workarounds at this point involve re-sizing images in Runtime apps... or using external photo software to re-size photos before adding to collector.
- Production environments with multiple web servers are prone to a service extent error. If you have a source SDE database with one projection (say WGS84) and a service with another (typically Web Mercator) the pooled web servers may enter a problem state where one server correctly reports the service extent in WGSS84 and the other reports it being in Web Mercator. You can see this by refreshing the rest endpoint for the feature service, the extent projection will change as the load balancer alternates requests between the two web servers. The primary manifestation of this is checkout geodatabases with invalid extents. You will end up with a geodatabase with a collectible area in Web Mercator, defined with lat/long values... creating an extremely small valid area somewhere near NULL island. With a 2 machine web pool an end user effectively has a 50/50 chance of downloading a valid database.
- SQLServer based geodatabases at 10.2.2 and 10.3 somehow enter a state where edits to existing features will fail. This can be observed in ArcMap by editing a feature and attempting to save. The reported error in this case is "Underlying DBMS error [[Microsoft][SQL Server Native Client 11.0][SQL Server]There are no rows in the current fetch buffer.] [GDBName.DBO.LayerName]". This error can be fixed by stopping services and deleting and recreating the archive for the feature classes in the geodatabase (which will most likely break existing replicas). This error seems to have become worse at 10.3 because check-ins now appear to fail silently, and in our most recent case the data being checked in was deleted from the iPad. This creates a situation where data could potentially be lost, unacceptable in a production environment.
FYI - the thing to do in this situation is to IMMEDIATELY look in this directory:
Find your service and look for the uploaded geodatabase. This geodatabase is a diff databse containing adds/edits/deletes and can be converted to a file geodatabase to manually apply edits. These files are cleared out by the system periodically so it's important you get a copy of the file right away.
At this point the only conclusion I can come to is that the platform (SQLServer, ArcGIS Server, Runtime/Collector) is not production ready. I would caution anyone preparing to deploy with this configuration to seriously consider other options. This may include using a different DBMS (Oracle, PostgreSQL) or hosting feature classes on AGOL, or abandoning ESRI tools all together for something like iForms.
I am eager to hear of other's experience with alternate DBMS platforms, has PostgreSQL proven to be a reliable DBMS for mobile app services?
Are our issues just a complete fluke? We are running ArcGIS Server 10.3 in an ESRI-recommended configuration. Based on the number of "failed sync" posts over in the Collector forums I'm guessing we are not. Take a glance over there at the number of complaints about data mysteriously not checking in, I'm guessing many of those issues are related to one or more of the bugs above.