Storage and Data Governance Recommendations in Education

[GeriMiller]

As ArcGIS Online usage grows across educational institutions, we often get questions about managing the growing amount of content that comes with that. The concept of Data Governance is instrumental in effectively addressing such questions. Data governance is about setting institution-wide standards - data policies - that apply to how data is gathered, stored, accessed, processed, and disposed of.

 A data governance plan for managing one’s ArcGIS Online organization can help institutions plan for growth and avoid problems.  Many institutions are still in the early stages of implementing Modern GIS (ArcGIS Online and related apps); in such cases, “do not delete data” is a reasonable short-term policy. For institutions with thousands of users, however, now is a good time to develop data governance policies.

What should be addressed in a comprehensive data governance plan?

• Monitoring utilization of ArcGIS Online
• Handling of user content when a user leaves an organization (off-boarding)
• Plans for increasing capacity, prior to encountering limits.
• Regularly informing users of your data governance plan and how it will impact them

In developing a data governance plan for ArcGIS Online, a good starting point can be to review an institution's existing data governance practices for other campus systems, and mirror those. Answer questions such as “How often is data purged?”, “What happens to existing content when a user leaves?”, etc.

Frequently Asked Question about ArcGIS Online Storage

The purpose of this blog is not to address everything that goes into a data governance plan, but to focus specifically on addressing storage-related issues and options for increasing capacity prior to encountering storage limits. Below are Frequently Asked Questions which will help explain how to get information about your organization’s specific data storage, along with suggestions for next steps.

Q: What are the different types of storage in ArcGIS Online?

Broadly speaking, there are different types of storage in ArcGIS Online:

• Feature storage is specific to feature layers hosted in ArcGIS Online.   
• File storage includes other types of layers and files, including attachments in your hosted feature layers.
• Imagery storage is specific to image layers in ArcGIS Online.

Q: How do I gain insights as to where my organization stands in terms of storage? Is it possible to run a report?

We recommend creating an organization item report by going to Organization à Status page and creating an “Item” report. You can gain insights on items which consume large amounts of file or feature storage. To identify items in the organization that consume large amount of file or feature storage, open the report and sort the respective fields.  

For further information, please check the View and report status documentation. Please also check the Tracking your ArcGIS Online Feature Data Store Key Health Indicators blog specifically relating to feature storage.  

Q: What are my options and next steps if my storage is getting large?

Consult your IT colleagues/centralized IT support to evaluate whether existing data governance practices for other systems (e.g., Sharepoint, file servers) can be applied to ArcGIS Online. Some modifications may be required but you can leverage your colleagues’ expertise to build a strong foundation.

•  Remove data sets that consume large file or feature storage and have not been accessed or are redundant.

After running an Item report as described above, consider removing any large items that are not being accessed or are redundant. Students inevitably leave the institution and become inactive users with inactive content. In addition, redundant data can result from course activities in which dozens of students follow the same workflow.

In addition, this section of the ArcGIS Online documentation provides additional guidance for actions such as identifying old or unused items and finding large hosted feature layers.

Another consideration for managing content are 3rd Party tools, such as Geo Jobe’s Admin Tools, Back Up My Org, and Clean My Org solutions. Note that there is a cost associated with these tools.

• Implement additional storage solutions

Below are additional options to consider and both have an associated cost.

1. License Premium Feature Data Store: For organizations who are reaching the feature storage limits, and would like to keep maintaining similar workflows without removing existing content, a Premium Feature Data Store is available. In addition to providing additional storage, the Premium Feature Data Store provides additional dedicated database resources such as memory, CPU, and input/output (I/O) and could provide improved database performance.  
2. Leverage ArcGIS Enterprise to host large datasets: ArcGIS Enterprise runs in your own infrastructure, and while included as part of the Education Institution Agreement there is a cost associated with using an institution’s infrastructure/servers (on premises or cloud), as well as time cost associated with setting up and maintaining it. In addition, you may need to move content between portals, which could result in broken links to any applications leveraging the content if not managed carefully.

Either of the above approaches could be a viable option and it is up to your organization to consider which storage solution provides the best ROI in your specific context. In one of the options (Premium Data Store), Esri is providing the infrastructure and management at a cost, while in the other (ArcGIS Enterprise), the technology itself is available with your existing Esri license agreement but there is an associated cost with maintaining infrastructure/servers (on premises or cloud), as well as time cost for someone to stand up and maintain.

Any feedback and questions are welcome.