Suddenly can't update feature layer - 'Database error has occurred.'

[RogerAsbury]

I have an application that pushes updates from a bunch of air sensors to a feature layer in ArcGIS Enterprise. It's been working happily for quite some time (a couple years?) but suddenly stopped working on Friday. Basically, it grabs a bunch of info in json and then uses that to update the layer. It uses the fairly standard edit_features() function -
added_sensors = item_layer.edit_features(adds=data)

where data is the json returned from all the sensors. This has worked up until Friday. Now, when I attempt to do this I get the following returned from the 'addResults':

[{'success': False, 'globalId': None, 'error': {'code': 10500, 'description': 'Database error has occurred.'}, 'objectId': None}]

Which is horribly non-descriptive other than knowing something database isn't happy. 

Other things I know happened on Friday that may or may not be relevant:
1. I created a view layer from this layer. I wouldn't expect that to affect this, and when I deleted that view layer, things remained broken.

2. A new employee was given access to this layer, but as far as I can tell, he's done nothing with it, and edit capabilities are still intact as they were before.

Any ideas? Searching up the error code brought up a few hits, but as far as I can tell, the answers weren't relevant. 😞 

PS: This layer has around 25,000,000 rows of data... Is there a limit

PPS: As it's open data, non-proprietary, etc... in case it helps, below is an example of the data. The ID, aqi, updated_date and is_fnsb_owned fields are irrelevant to this step (as in, the fields don't exist in this db for them and are (or at least... were) ignored:

[

  {
  "geometry": {
    "x": -147.82834,
    "y": 64.90278,
    "spatialReference": { "wkid": 4326 }
  },
  "attributes": {
    "ID": null,
    "name": "Fairbanks Ballaine & Lynx",
    "location_type": 0,
    "sensor_index": 7776,
    "humidity": 42.0,
    "temperature": 19,
    "latitude": 64.90278,
    "longitude": -147.82834,
    "date_created": 1518660699,
    "uptime": 29940,
    "pm25_cf_1_a": 0.29,
    "pm25_cf_1_b": 0.78,
    "pm25_atm_a": 0.29,
    "pm25_atm_b": 0.78,
    "pm25_alt_a": 0.5,
    "pm25_alt_b": 0.6,
    "pm25_a": 0.3,
    "pm25_b": 0.8,
    "time_stamp": 1702416227000,
    "aqi": null,
    "updated_date": null,
    "is_fnsb_owned": null
  }
},
{
"geometry": {
  "x": -147.72636,
  "y": 64.845314,
  "spatialReference": { "wkid": 4326 }
},
"attributes": {
    "ID": null,
    "name": "NCore 4",
    "location_type": 0,
    "sensor_index": 16315,
    "humidity": 53.0,
    "temperature": 16,
    "latitude": 64.845314,
    "longitude": -147.72636,
    "date_created": 1537903417,
    "uptime": 53618,
    "pm25_cf_1_a": 8.98,
    "pm25_cf_1_b": 0.0,
    "pm25_atm_a": 8.98,
    "pm25_atm_b": 0.0,
    "pm25_alt_a": 3.4,
    "pm25_alt_b": 0.0,
    "pm25_a": 9.0,
    "pm25_b": 0.0,
    "time_stamp": 1702416227000,
    "aqi": null,
    "updated_date": null,
    "is_fnsb_owned": null
  }
}

]

[MarcoBoeringa]

Well, there seem to be at least a couple of references to this particular error number and ArcGIS Enterprise / Server. Maybe there is something relevant to find in these links, e.g. did the format of the JSON change, or was there an upgrade to ArcGIS Enterprise using the security patch referenced in the last link, and you now need to patch it further to version 'B' of the patch to get it fixed?:

https://community.esri.com/t5/arcgis-pro-questions/paste-features-failed-error-code-10500/td-p/12575...

https://community.safe.com/s/question/0D54Q00009HHHyfSAH/arcsde-data-failing-to-insert-or-update-to-...

https://support.esri.com/en-us/bug/cannot-create-or-move-existing-features-in-a-hosted-fea-bug-00015...

[RogerAsbury]

For the first two links, I have double checked all the data types, sizes and field names. They all match up between the requested data and what the feature layer should be expecting. 

For the third, I asked one of our GIS admins and we are on version 10.9, which is a minor revision beyond when that issue was reported. I've asked them when we updated to that version, but have not heard back yet. (edit: 2 years ago, so not, like... Friday. 🙂 )

[RogerAsbury · Accepted solution]

As a follow up to this, it looks like a server space issue... Received the following from one of our GIS admins from the logs: