CA_Census_FINAL_FINAL2: Using Item Reports to Help Manage Storage Space
“Ahhh! Our storage is at 110%... what do we do!??!” We don’t hear this every day, but it’s definitely a recurring conversation with ArcGIS Online administrators. As ArcGIS Online use grows across campuses, many of our users have started to see their ‘feature storage’ nearing 100%. While there are a range of ways to delete content and manage users, the focus of this blog is to simply show you how you can use the ArcGIS Online report functionality to take action on items (and users) that are eating up the majority of your organizations storage.
Note: There can be complex webs of item dependencies in ArcGIS Online. For example, 1 StoryMap might include dozens of web maps, that include hundreds of feature layers. By deleting one feature layer, it could create a cascading impact. All to say, ensure that you are aware of WHAT you are deleting, or create local backups of content that you plan on deleting.
Yet, I can almost guarantee that there are also hundreds of junk, 'test', and 'exercise' feature layers and data that are eating up a lot of your storage that can be removed.
How does storage work?
Every education organization comes equipped with a ‘Standard’ feature data store. The data store provides storage and compute resources for your organization. The standard tier comes equipped with 500GB of storage.
If your ‘% storage used’ crosses the 100% threshold, your ArcGIS Online organization will not ‘shutdown’. Yet, your users will start to see performance degradation with certain tasks. We highly recommend that users work to stay below the 100% threshold.
To assist you in managing your users and content, there are a range of tools and reports that can be used to answer that common question: ‘what/who is using up all of our storage?!?’.
There is also the option to upgrade to a Premium Feature Data Store, which come in different size tiers. Be aware that each of these incur a monthly cost that is in addition to the cost of your institution agreement.
So, let’s take a look at how you can use the tools in ArcGIS Online to better manage your organization’s storage.
ArcGIS Online Reports
From the Overview page of ArcGIS Online, users can start interrogating their storage and usage.
In the screenshot below, a user can quickly see how much storage is currently used.
By selecting ‘Feature Data Store’, users can also get a more detailed snapshot of resource usage by time of day, day of week.
Next – we are going to select to create an ‘item report’ – from the link in the details above.
Getting More Detail – building reports
This selection brings up to the report page, which can also be navigated to by going to Organization, Status, and Reports.
From within this page – users have the ability to build out an array of different reports that are then saved into the ArcGIS Online organization. For our purposes, we want to generate an Item Report. After the report is generated, it can then be downloaded as a CSV and should look something like this (I removed a few unnecessary fields and formatted the CSV as a table).
The fields that we care the most about are: Feature Storage Size (listed in MB), View Counts (is this item being used/viewed at all?), Share Level (Who is this shared with? Who is potentially using it?), Owner (username of the items owner), and Title
After sorting the data by the Feature Storage Size column, we already have some great information.
For this exercise - we don't care about the 'File Storage' column, so make sure you are looking at the right one. 'File Storage' plays no impact of your Feature Data Store, because it stores data in a different way/location (imagery, GDBs, etc). File storage will consume your organizations credits, but at a much slower rate than feature storage. To learn more about WHAT the differences are between the storage types, please read this documentation: Understand credits.
Right away, I can see that the 2 largest files in the organization (which are both nearly 1 GB in size) are duplicates of each other (same file name with the exception of ‘FINAL’). We can also see that the vast majority of the larger files in the organization are set to ‘private’, meaning that they aren’t being used by anyone but the owner. Also, looking at the view count for the files, I can see that some of them have views at ‘0’ and many are <10. Lastly, the owner information is listed along with the items. If you are unfamiliar with the username, you can use this to search across your members in ArcGIS Online and then drill down into the members settings to find their e-mail address.
In most instances, I have found that organizations have a few offending items that are consuming the majority of storage space. Running these reports on a quarterly basis, is a great way to track content and check to see what might be eating up all of your organization’s storage. All of this could be automated as well, using the ArcGIS Python API.
Close
If you are still struggling with storage, there are a few other options worth considering:
- ArcGIS Enterprise
- All institution agreements include ArcGIS Enterprise in the agreement. By standing up a dedicated environment for specific types of projects or features, ArcGIS Enterprise could serve as a dedicated environment for long standing data.
- Premium Feature Data Store
- As mentioned earlier in the blog, there is the option to upgrade your storage from Standard to Premium. This upgrade features more storage and processing power dedicated to your ArcGIS Online environment.
- Dedicated ArcGIS Online organization for research/projects
- Most tiers of the education agreements include the ability to stand up additional ArcGIS Online organizations. Requesting an organization for your specific users, research centers, etc. that are the using the majority of your ‘general’ university ArcGIS Online organization storage is a potential option to alleviate usage.
Lastly, there is currently no way to limit storage on a per user basis. So, even though you can assign credit budgets to users, these limits do not apply to the storage that users could be consuming.
As always, if you have any questions or need anything – please reach out below or directly to us!