Workflow to prevent unwanted or inintended mass data upload to ESRI service/account?

758
2
Jump to solution
06-10-2021 03:43 PM
Labels (3)
JansenLyons
Occasional Contributor III

Hi All,

I was curious if there were limits or safeguards in place from ESRI or documented ways to prevent mass data upload to servers, databases, or content pages; malicious or otherwise? For example, tens or hundreds of gigabytes, perhaps even terabytes if sizes are capable of such on certain platforms, or uploads in the millions of items being suddenly added to an ESRI feature service, database, or content page on AGOL. Besides bogging down or charging an unexpected amount of storage credits via AGOL/Enterprise, this scenario would theoretically break existing maps, apps, and dashboards and is a concern. My curiosity is about having a limit on amount or size (GB or # of features etc.) of data being uploaded, sent, or made into an ESRI databases, apps, or services, as well as what kind of actions can be taken if such an upload is successful besides backups and system services offsite. I look forward to the discussion!

Much appreciated, God bless,

JLyons

Jansen Lyons - Records and GIS Section - Public Works - City of Rio Rancho, NM
1 Solution

Accepted Solutions
EduardoFernandez1
Esri Contributor

Hello Jansen 

Good question and a very broad one. In general, within a service properties or configuration, there is no parameter to limit the number of records to append\upload or to cut the transaction when a file size is reached. 

When a member adds a hosted item in either ArcGIS Online or ArcGIS Enterprise (Portal for ArcGIS And ArcGIS Data Store),  there is a 500GB limit. (What can you add to ArcGIS Online?—ArcGIS Online Help | Documentation). What can you add to ArcGIS Enterprise?—Portal for ArcGIS | Documentation for ArcGIS Enterprise.

As for referenced services in Portal ( whereby the data source is a enterprise geodatabase), there is no file size limit. Any time based disruptions to the load might come from network traffic load or timeouts through IIS\tomcat or load balancers. Databases too have connection timeout type settings that can be fine tuned to disconnect a connection after a period of time.

I recommend limiting the sharing option of the feature layer in ArcGIS Online or Portal for ArcGIS to the owner only and then share it to a group with participating members who are approved to undertake feature level data management on a feature layer\feature service (being Creators user types with data editor or publisher role). The downside here is that if you need to overwrite the web layer, then the sharing options get overwritten and you have to manually reapply the sharing settings. It is inconvenient but in my opinion, keeping content secure and avoiding anonymous access to any end point trumps.

Cheers Ed

View solution in original post

2 Replies
EduardoFernandez1
Esri Contributor

Hello Jansen 

Good question and a very broad one. In general, within a service properties or configuration, there is no parameter to limit the number of records to append\upload or to cut the transaction when a file size is reached. 

When a member adds a hosted item in either ArcGIS Online or ArcGIS Enterprise (Portal for ArcGIS And ArcGIS Data Store),  there is a 500GB limit. (What can you add to ArcGIS Online?—ArcGIS Online Help | Documentation). What can you add to ArcGIS Enterprise?—Portal for ArcGIS | Documentation for ArcGIS Enterprise.

As for referenced services in Portal ( whereby the data source is a enterprise geodatabase), there is no file size limit. Any time based disruptions to the load might come from network traffic load or timeouts through IIS\tomcat or load balancers. Databases too have connection timeout type settings that can be fine tuned to disconnect a connection after a period of time.

I recommend limiting the sharing option of the feature layer in ArcGIS Online or Portal for ArcGIS to the owner only and then share it to a group with participating members who are approved to undertake feature level data management on a feature layer\feature service (being Creators user types with data editor or publisher role). The downside here is that if you need to overwrite the web layer, then the sharing options get overwritten and you have to manually reapply the sharing settings. It is inconvenient but in my opinion, keeping content secure and avoiding anonymous access to any end point trumps.

Cheers Ed

JansenLyons
Occasional Contributor III

Good morning Ed, ( @EduardoFernandez1  ),

Thank you for the detailed response, this is excellent and definitely covers what I am concerned about. Thankfully nothing has come up again to warrant the immediate implementation of any strict protocols beyond organizational sharing, but these details will come in handy should we need them. I will read into the links and get back to you in this thread if I have any further questions.

Thank you again and have a blessed week,

 

Jansen Lyons - Records and GIS Section - Public Works - City of Rio Rancho, NM