ArcGIS Governance in Education: Content and Storage Management

GeriMiller

INTRODUCTION

As ArcGIS Online usage grows across educational institutions, Education customers should start thinking about managing the amount of content and storage - depending on their institutional policies, as well as Esri license allocations. The concept of governance is instrumental - setting institution-wide standards and policies that apply to how data is gathered, stored, accessed, processed, and disposed of.

A governance plan for managing ArcGIS Online can help institutions plan for growth and avoid issues in the future.

The purpose of this blog is not to address everything that goes into a governance plan, but to focus specifically on addressing storage-related issues for content, and options for increasing capacity prior to encountering storage limits.

CONTENT/STORAGE – UNDERSTANDING

Understanding the three different types of storage in ArcGIS Online

Feature storage is specific to feature layers hosted in ArcGIS Online.
File storage includes other types of layers and files, including attachments in your hosted feature layers.
Imagery storage is specific to image layers in ArcGIS Online.

While all storage types can account for increasing the overall storage credit consumption in ArcGIS Online, the feature storage is most important to address, as it consumes most of the overall credits in Education ArcGIS Online organizations.

Understanding feature storage quota in ArcGIS Online

By default, an ArcGIS Online subscription comes with Standard Feature Data Store, which includes 500 GB of feature data storage and supports the full range of hosted feature layer operations. This quota includes feature storage only and does not include file or imagery storage.

As Education customers start getting near this quota, they need to start thinking of content stewardship and maintaining their feature data store threshold below this limit.

Administrators and members will receive quota (and other) notifications, as outlined in the When do I get email notifications related to ArcGIS Online documentation.

Understanding patterns of use in Education

On average, for Education ArcGIS Online organizations, 85% of credit use is consumed by feature storage. Check Credits by Capability section of the documentation. Additional information is available in the Credit Management blog.

Further, the costs associated with monthly or yearly feature storage - if one consumes the entire 500GB of storage - are as follow:

500 GB per month = 120,000 credits per month = 1,440,000 credits per year

Therefore, you will need to check your credit consumption, as outlined in the Credit Management blog, and work toward a threshold that is within your Education license allocation, or take actions of removing content or adding storage, as described later in the blog.

For some institutions, a few items consume most feature storage (typically associated with research projects or student/faculty testing of various workflows). For others, there is no small number of “big” items, rather, a lot of small items that add up, such as student content created through course work.

Removing old users/content is not often efficient in terms of freeing up storage – in the past, content/items were relatively small. Instead, focusing on the ~Top 50 feature storage items and addressing them, works better in terms of freeing up storage. However, this should be done with caution and verified with content owners.

In a future blog about managing users, we will address removing smaller items that have accumulated over time, that no one is using or accessing – as a way of reducing the overall storage.

Understanding how your institution handles other technologies

Consult your IT colleagues/centralized IT support within your institution to evaluate whether existing data

governance practices for other systems (e.g., Sharepoint, Google Drive, DropBox, LMS such as Canvas, Blackboard, etc.) can be applied to ArcGIS Online.

Some modifications may be required but you can leverage your colleagues’ expertise to build a strong foundation.

CONTENT/STORAGE – BEST PRACTICES

Our goal with the recommendations below is to help you understand when you need to act, and plan proactively for future years. Being a good steward of resources and operating within the bounds of existing allocations – credits, storage, users, performance - is important for future management of ArcGIS resources.

Create organization item report

Leverage the reporting capabilities of ArcGIS Online (Organization > Status page > Reports) to create an “Item” report. You can gain insights on items which consume large amounts of file or feature storage. To identify items in the organization that consume large amounts of file or feature storage, download the report, open it in your preferred spreadsheet software, and sort the respective fields.

For further information, please check the View and report status documentation and the Tracking your ArcGIS Online Feature Data Store Key Health Indicators blog. You can also schedule the report to run automatically every week or month – for future record keeping and predictions.

Once you have an idea of your items/content, below are options that you can take – you can choose one that works for you, or a combination of them.

Reduce content

Delete items that consume large file or feature storage

After running an Item report as described above, consider removing any large items that are not being accessed or are redundant. Students inevitably leave the institution and become inactive users with inactive content. In addition, redundant data can result from course activities in which dozens of students follow the same workflow.

This section of the ArcGIS Online documentation provides additional guidance for actions such as identifying old or unused items and finding large, hosted feature layers.

NOTE: It is important before removing any items to verify with item owners whether the items are still needed. If the items are still used, alternative options for hosting this content could be discussed. A suggested workflow, if applicable, could be a practice of un-sharing items for a grace period, and then deleting (if no inquiries are made about the missing data). Archiving content, discussed later in this blog, is an option as well, to ensure data is not lost, if needed.

Transfer of content between organizations

In certain instances, a strategy for reducing space in an ArcGIS Online organization could be exporting content to another format, for delivery in another organization, then deleting the content once it has been exported.

Below are additional options, some of which are unsupported, to transfer content between organizations. There is no one-size-fits-all solution.

The important point to take away is that one needs to verify whether certain content really belongs to the academic ArcGIS Online organization. We have seen instances of academic ArcGIS Online organizations hosting content that should be hosted in administrative/operational ArcGIS Online organizations. Or, instances where content should be hosted by outside entities, such as State Data repository items, that should be widely available to the public.

Guidance on transferring content could be found here, and options are below:

ArcGIS Assistant (note, while useful, this tool is not supported by Esri Technical Support)
3rd Party tools, such as Geo Jobe’s Admin Tools, Back Up My Org, and Clean My Org
ArcGIS API for Python clone_items method is best option (requires scripting)
Migrate Content for ArcGIS Enterprise/ArcGIS Online – additional resource

Implement backup and archival

It is always a good idea to have a workflow in place for disaster recovery and make your own backups/archival of ArcGIS Online content to an external destination.

Backup is a process of duplicating data that can be retrieved in the event of a data loss – an organization could consider layers of importance to be backed-up. Archival is the process of storing historical data for future reference, which is often a practice for other data management systems.

Backups do not reduce storage, though backup is a recommended practice for critical content (i.e. important research projects, publications, etc.). Archival could reduce storage, if the original content is removed from ArcGIS Online and archived in a separate system.

As an institution, you’d need to decide what should be backed up or archived, and what actual backup or archival option would work for you. For archival purposes, consider items that need to be immediately accessible (which would stay in the ArcGIS Online organization), versus items that only need to be accessible once in a while (that could be placed in “cold storage” and archived).

Some examples of options are below:

Backup hosted feature layers to cloud storage as file geodatabases
Backup ArcGIS StoryMaps as a JSON and file resources
Leverage ArcGIS Enterprise as an archival solution
ArcGIS Online Notebook: Demo Backup to Dropbox
- After opening link, Sign in to your ArcGIS Online organization
- Requires developer access to Dropbox, usually included with Education institution licensing
Check Introducing the ArcGIS Well-Architected Framework for additional considerations.

Increase Storage

Options for increasing feature storage capacity are listed below - both have an associated cost.

License Premium Feature Data Store

For organizations who are reaching the Standard Feature Data Store storage limits, and for which reducing feature storage usage is not a viable option, then a Premium Feature Data Store is available at a cost. In addition to providing additional storage, the Premium Feature Data Store provides additional dedicated database resources such as memory, CPU, and input/output (I/O) and could improve your organization's overall performance.

To make a clear distinction between the two feature data store types - with a Standard Feature Data Store, which is the default with the Education agreement - feature storage typically consumes a significant number of your agreement's credits, thus it is the most important type of storage to manage.

If you have upgraded to a Premium Feature Data Store, then feature storage consumes no credits. Rather, you need to manage your feature storage to remain within the quota defined for your Premium Feature Data Store level.

Note: Purchasing Premium Feature Data store is more cost effective than buying additional credits to cover storage costs.

Leverage ArcGIS Enterprise to host large datasets

ArcGIS Enterprise runs on infrastructure your institution provides, and while licensing for ArcGIS Enterprise is included as part of the Education Institution Agreement, there is the additional cost associated with using your own infrastructure/servers (on premises or cloud), as well as effort costs associated with setting up and maintaining it. In addition, moving content from ArcGIS Online to ArcGIS Enterprise, will require effort to update links and apps that depend on the migrated data.

ArcGIS Enterprise can be leveraged for hosting large datasets, but also, for storage/archival purposes, or for storing restricted information.

Below are considerations for possible configurations of purchasing Premium Feature Data Store or standing up ArcGIS Enterprise.

Upgrade ArcGIS Online to Premium Feature Data Store that covers an institution’s projected feature storage needs for the year - M4 Premium Feature Data Store being the ‘largest’, which provides total of 4TB of feature data storage.
Upgrade ArcGIS Online to a "smaller" Premium Feature Data Store (M2 or M3 Premium Feature Data Store) and move select "big" projects to an institutional supported ArcGIS Enterprise set up.
Keep ArcGIS Online with Standard Feature Data Store and move select "big" projects to an institutional supported ArcGIS Enterprise set up.

Options 2 and 3 are hybrid approaches, where institution would choose to host big feature storage items on ArcGIS Enterprise to help manage costs, and the rest of the feature storage needs could be handled by one of the Premium Feature Data Store options, or the Standard feature data store - depending on projected storage needs.

Either of the above approaches could be a viable option and it is up to your organization to consider which storage solution provides the best ROI in your specific context. In one of the options (Premium Feature Data Store), Esri is providing the infrastructure and management at a cost, while in the other (ArcGIS Enterprise), the technology itself is available with your existing Esri license agreement, but there is an associated cost with standing-up, operating, and maintaining your own infrastructure (on premises or cloud) for ArcGIS Enterprise.

Note: Cost comparisons of “Premium Feature Data Store” versus ArcGIS Enterprise are not easy to compute, as they are dependent on individual institution's policies and practices. Assessment could be done by Esri Professional Services to compare staffing, support, hardware, software to run ArcGIS Enterprise, to cost of purchasing Premium Feature Data Store. Esri Professional Services can provide recommendations on specifications for hardware depending on your requirements/systems, cost of doing installations, etc.

Experience has shown that often purchasing Premium Feature Data Store, could be cheaper as an option, compared to standing-up ArcGIS Enterprise (once the costs of servers, ongoing maintenance, people, etc., are accounted for).

Regardless of approach, it is crucial that enough lead time is provided to allow for conversations and proper scoping of requirements. A lot of pre-cursor conversations would need to occur with other business units/divisions in the university (IT, Budgeting, etc.) and since the solutions above take time to discuss, it is always good to start early. The resources below are helpful to share with IT colleagues, who are knowledgeable in supporting similar requirements for other systems:

CONCLUSION – NEXT STEPS AND WHERE TO GO FOR HELP

Invariably, there is complexity associated with the various options above. What is important is that good stewardship of resources is maintained. Some of the solutions above will continue to evolve, and we’d like you to be part of this journey – a contributor with ideas, processes and workflows.

Please share any comments and feedback here. If you have a workflow in place that has worked for you, we’d like to hear it.

For any additional questions, please contact your Account Manager or highered@esri.com.

AlexG69

Great !