Hello. Our data store server filled up the hard drive with all the backups. We have deleted most of the backups to free up space but now we are having issues with our hosted feature layers. We can't create new records in anything. We can't edit any hosted feature layers. Workforce isn't working since we can't update any work orders or create new ones. I have rebooted everything. I re-ran the ArcGIS Server configuration wizard to no avail. Does anyone know how to fix this? We are running version 10.9.1 of enterprise in a federated environment.
I have ticket into ESRI but the tech keeps asking me about my ODBC 17 driver version. It's Portal, no driver is used. Losing faith in ESRI support so wanted to see if my fellow GIS peers had any ideas.
Thanks!
Solved! Go to Solution.
changedatastoremode readwrite --store relational --prompt no
describedatastore Available ArcGIS Data Stores ============================================================== Relational Data Store Tile Cache Data Store Spatiotemporal Big Data Store Graph Store Object Store General Information of ArcGIS Data Store on machine.domain.com ============================================================== ArcGIS Data Store release....11.2.0.1234 Staging location.............C:\arcgisdatastore\staging Log location.................C:\arcgisdatastore\logs Free disk space..............174.00GB Threshold for READONLY mode..2048MB Information for relational data store ds_sthiu0_5T ============================================================== Backup location...........\\nwshare\dsbackups Is backup folder shared...true Backup schedule...........{"schedule-starttime":"00:00:00","schedule-frequency":"Every 7 DAYS"} Days backup retained......31 Data store status.........Started SSL enabled...............true Last failover.............20150130190334005 Member machines...........MACHINE1.DOMAIN.COM, MACHINE4.DOMAIN.COM Maximum connections.......150 Owning system URL.........https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL.....https://portal_webadaptor.domain.com/portal Number of connections.....8 connection(s) to managed database Data Store mode.....................READWRITE Is Point-in-time recovery enabled...No Query optimizer enabled.............Yes Information for tile cache data store ds_wztxj7um ============================================================== Data location...................C:\arcgisdatastore\nosqldata Data store status...............Started Backup location.................\\mybackups\scenebackup Backup schedule.................{schedule-frequency":"Every 1 DAYS"} Is backup folder shared.........true Mode............................cluster Member tile cache machines......MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Registered tile cache machines..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Owning system URL...............https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL...........https://portal_webadaptor.domain.com/portal Information for spatiotemporal data store ds_qpko99Cl ============================================================== Backup location.....................\\nwshare2\sbds_bu Max rebalance off time..............60 minutes Automatic rebalance ................On Machines in spatiotemporal cluster..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Current coordinator in cluster......MACHINE1.DOMAIN.COM Registered spatiotemporal machines..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Owning system URL...................https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL...............https://portal_webadaptor.domain.com/portal Data Store mode.....................READWRITE Information for graph store x2b7s0n ============================================================== Deployment mode..................primaryStandby Access endpoint..................MACHINE1:9829 Backup location..................\\nwshare3\graphbu Registered graph store machines..MACHINE1.DOMAIN.COM Owning system URL................https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL............https://portal_webadaptor.domain.com/portal Information for object store w1tiknx ============================================================== Deployment mode...................singleInstance Access endpoint...................MACHINE1:9900 Backup location...................\\nwshare4\myfiles\ods_bu Registered object store machines..MACHINE1.DOMAIN.COM Owning system URL.................https://gisserver.domain.com:6443/server/admin Portal for ArcGIS URL.............https://portal_webadaptor.domain.com/portal
By default, the relational data store is placed in read-only mode when free disk space drops to 1,024 MB. In the following example, there are other apps writing to the machine, so the administrator configures the relational data store to be placed in read-only mode sooner: when free disk space on the machine drops to 4,000 MB:
changedbproperties --store relational --disk-threshold-readonly 4000 You are changing the following database properties: ...disk space threshold to change relational data store into READONLY mode to 4000MB) Changing database configurations could cause the database to restart. Please do not interrupt the process once it has started. Do you want to continue (Yes or No)?Yes
describedatastore command will show if the database is in read-only mode.
This utility is used with relational and spatiotemporal big data stores.
The changedatastoremode utility allows you to place a relational data store in read-only mode while you perform maintenance on the data store. For example, if you need to perform a maintenance task that will cause the data store to restart, such as changing the backup location from one drive to another drive or changing database properties, you can place the relational data store in read-only mode so no users are in the process of publishing or editing data when the data store restarts.
The changedatastoremode utility is also used to place the relational data store back in a read-write mode after you finish maintenance or after you add enough disk space to the primary data store machine to allow the data store to function properly in read-write mode.
You must run the changedatastoremode utility on the primary relational data store machine, not the standby machine.
If ArcGIS Data Store places your relational data store in read-only mode due to insufficient disk space, automatic backups are also disabled to avoid filling the disk further. Therefore, you must reset your automatic backup schedule using the updatebackupschedule utility after you place the relational data store back in read-write mode.
When your spatiotemporal big data store gets close to running out of disk space, ArcGIS Data Store places it in read-only mode to avoid data corruption. You can use the changedatastoremode utility to return a spatiotemporal big data store to read-write mode after you have added sufficient disk space to the spatiotemporal big data store machines.
changedatastoremode {readonly | readwrite} [--prompt {yes | no}] --store {relational | spatiotemporal}
In this example, the relational data store is placed back in read-write mode, meaning clients can resume such activities as publishing hosted feature layers to ArcGIS Enterprise, editing data through a hosted feature layer, or adding CSV files to a map viewer.
changedatastoremode readwrite --store relational --prompt no
To place a spatiotemporal big data store in read-write mode, specify spatiotemporal with the --store option as shown in the following example:
changedatastoremode readwrite --prompt no --store spatiotemporal
Thank you so much for your reply!!! So the datastore put itself into read only mode when the hard drive had no space left? How do I get it out of read only mode? Restarting the data store server didn't work.
changedatastoremode readwrite --store relational --prompt no
describedatastore Available ArcGIS Data Stores ============================================================== Relational Data Store Tile Cache Data Store Spatiotemporal Big Data Store Graph Store Object Store General Information of ArcGIS Data Store on machine.domain.com ============================================================== ArcGIS Data Store release....11.2.0.1234 Staging location.............C:\arcgisdatastore\staging Log location.................C:\arcgisdatastore\logs Free disk space..............174.00GB Threshold for READONLY mode..2048MB Information for relational data store ds_sthiu0_5T ============================================================== Backup location...........\\nwshare\dsbackups Is backup folder shared...true Backup schedule...........{"schedule-starttime":"00:00:00","schedule-frequency":"Every 7 DAYS"} Days backup retained......31 Data store status.........Started SSL enabled...............true Last failover.............20150130190334005 Member machines...........MACHINE1.DOMAIN.COM, MACHINE4.DOMAIN.COM Maximum connections.......150 Owning system URL.........https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL.....https://portal_webadaptor.domain.com/portal Number of connections.....8 connection(s) to managed database Data Store mode.....................READWRITE Is Point-in-time recovery enabled...No Query optimizer enabled.............Yes Information for tile cache data store ds_wztxj7um ============================================================== Data location...................C:\arcgisdatastore\nosqldata Data store status...............Started Backup location.................\\mybackups\scenebackup Backup schedule.................{schedule-frequency":"Every 1 DAYS"} Is backup folder shared.........true Mode............................cluster Member tile cache machines......MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Registered tile cache machines..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Owning system URL...............https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL...........https://portal_webadaptor.domain.com/portal Information for spatiotemporal data store ds_qpko99Cl ============================================================== Backup location.....................\\nwshare2\sbds_bu Max rebalance off time..............60 minutes Automatic rebalance ................On Machines in spatiotemporal cluster..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Current coordinator in cluster......MACHINE1.DOMAIN.COM Registered spatiotemporal machines..MACHINE1.DOMAIN.COM, MACHINE2.DOMAIN.COM, MACHINE3.DOMAIN.COM Owning system URL...................https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL...............https://portal_webadaptor.domain.com/portal Data Store mode.....................READWRITE Information for graph store x2b7s0n ============================================================== Deployment mode..................primaryStandby Access endpoint..................MACHINE1:9829 Backup location..................\\nwshare3\graphbu Registered graph store machines..MACHINE1.DOMAIN.COM Owning system URL................https://gisserver_webadaptor.domain.com/server Portal for ArcGIS URL............https://portal_webadaptor.domain.com/portal Information for object store w1tiknx ============================================================== Deployment mode...................singleInstance Access endpoint...................MACHINE1:9900 Backup location...................\\nwshare4\myfiles\ods_bu Registered object store machines..MACHINE1.DOMAIN.COM Owning system URL.................https://gisserver.domain.com:6443/server/admin Portal for ArcGIS URL.............https://portal_webadaptor.domain.com/portal
Thank you! That worked perfectly! You are a life saver! Or at least my database saver!
You are welcome!