on Oct 20, 2014All,.
I am looking at the credit usage in ArcGIS On-Line. I have roughly 2.5 gb of features services which are stored in a database. On average these services are consuming 17 credits a day. I currently cannot find out which features are consuming the credits as it aggregates all the usinge to one number. Is there a way I can find out
a) which are my largest consumers of database storage. i.e. largest feature classes
b) which feature services are consuming the most credits
Regards
Gordon
Gordon,
Check out this tool which will create a spreadsheet of all of the content in your organization. You can then sort based on file size. That would be the easiest way to go.
But you can also create some custom scripts using the Python modules (PortalPy, agoTools, ArcREST) found on ESRI's ArcGIS Online Admin GitHub site. I've found scripting using these tools to be very helpful and surprisingly easy to use.
Heath
Gordon
You get charged 2.4 credits per 10mb of data stored per month, so all of your feature services will be consuming credits to some degree, a credit estimator can be found here:
ArcGIS Online | Credits Estimator
Regards
Anthony
I've not tried the tool suggested by Heath, but I suspect it won't tell you what you want to know about specific feature layer storage. This is because all feature layer are stored in a single database, and the database currently doesn't allow us to calculate storage on a per layer basis.
Look for feature layers that have attachments. Picture from a camera can be several megabytes. Thus the storage of these features can add up quickly. If your feature layer was created before July 2014, it will store attachments as blobs in the database. After July, we made a change to AGOL to store attachments as files, so the credit usage is much less, as file storage is much less.
Hope this helps.
Mike
Mike, the ArcGIS Online Catalog tool that I first mentioned does collect the file size of each item. There is a checkbox that you must check to include the file size in the catalog. Once it runs, you can then download the file to a csv and sort as you like, particularly based on "size". If you want to see what sort of information can be obtained about each item, check the REST API Help.
Also, each user has a "storageusage" amount tied to that user which can be accessed through the REST API. If you are more familiar with Python than using REST calls, like I am, then the getUsers function in the agoTools that I referenced above (or the get_org_users function in the PortalPy tools) can be used to search all of your users for the biggest data hogs.
Heath
***UPDATE*** Oct 2016. The size of a hosted feature layer as listed on the item details page now reflects the current database storage size of the feature layer.
***Original Post***
OK, I've found out a little more about the size parameter on feature layers. The size parameter is a reflection of the size at publish time. Unfortunately, it doesn't get updated as the feature layer is edited. So, as long as you haven't edited your data, it would be a good way to compare different layer sizes...
Thanks,
Mike
Thanks Mike,
A lot of our feature services have attachments and pre-date July 2014. We are going to go through and republish these to take advantage of the file storage rather than the database storage.
Regards
Gordon
Yes, you can export to FGDB and republish.
Mike
I am having a similar problem with data consumption. AGOL Catalog worked great for identifying the hogs. However I am unclear as to the type of storage that applies to each item.
Is there a list of what content "type" in AGOL is charged at each rate: "Feature Service Storage" or "Tile and Data Storage".
Thanks,
Josh
Hi Joshua,
You will only be charged Feature Service Storage for a Hosted Feature Service in your organization. All other data types stored in your content will be charged credits based on the data and tile storage rate.
This documentation goes into further details regarding credit charges.
-Kelly
Kelly,
When I run AGOL Catalog it list 9 separate content types. All are hosted services on AGOL.
Do I have this breakdown correct?
Feature Service Storage (2.4 cr / 10 mb) =
Map Service
Map Package
Feature Service
Tile and Data Storage (1.2 cr / 1 GB) =
Tile Package
Web Map
Service Definition
Image
Code Attachment
Web App
Thanks,
Hi Joshua,
I have updated your list below:
Feature Service Storage (2.4 cr / 10 mb) =
Hosted Feature Service
Tile and Data Storage (1.2 cr / 1 GB) =
Map Service
Map Package
Tile Package
Web Map
Service Definition
Image
Code Attachment
Web App
Only Hosted Feature Services (Published to ArcGIS Online are charged feature storage credits. If you were to reference a feature service from ArcGIS Server, this wouldn't count towards your feature service storage.
Let me know if you have any more questions,
-Kelly
Kelly,
This is driving me nuts. AGOL Catalog results show 295 mb of total storage, 64 mb of which are Hosted Feature Services. We're burning almost 10 credits per day from 1.44 GB of Feature Service Storage according to the My Organization page. Where's the discrepancy?
Thanks,
Hi Joshua,
The sizes reported in catalog only reports to the size of the data of hosted feature services when they were published. Were any of the services published prior to July 2014 that contain attachments? This can be a cause for the discrepancy.
There is currently an enhancement regarding showing the current size of feature services available at this link
http://support.esri.com/en/knowledgebase/techarticles/detail/43869
Please Contact Technical Support for help in determining where the discrepancy is.
Thanks,
Kelly
Update:
The size reported on the item details page is the current size of data storage for feature services (not the size at the time of publishing).
So... to confirm. If I've extracted the sizes from the catalog using something like agolTools from the ESRI github, then I should be able to calculate my hourly credit usage on an item by item basis.
If I've got the data in excel, then the formula to calculate this hourly credit usage would look something like this;
=IF("Feature Type Column"="Feature Service", "SizeinMB"/10*2.4/720,"SizeinGB"*1.2/720)
720 = the hrs in the month
Credits are charged at 2.4 credits per 10 MB per month, calculated on an hourly basis. ArcGIS Online | Service Credits
The problem with this is that the size reported in the catalog for items created before July 2014 is not the actual size... (as per above)
So... If we delete all the items created before July 2014 and republish any that we need, we should be able to create a broken down list of items and hourly credit usage. This could be re-calculated every hour to get a total number of credits used by an item...
The reason I'm looking at this, is that we currently don't have any way for our IT dept. who is responsible for purchasing credits, to recoup storage costs from the various groups using our organizational account. I'm sure we're not the only ones who need to be able to do break down these costs.
So to update, I'm having some issues getting the above referenced formula to match with the reported credit usage in my organizational account status... So far, the closest I can get is if I double the number that the above referenced formula gives me.
I have a call in with Tech Support to investigate, and will update this post when I get a response. I'm also hoping to expand the script to add other credit usage...
Here are some facts I think will be useful
type:"feature service" typekeywords:hosted
Wallace, Cameron I'm not sure your expression is correct. We do calculate credit charges based on hourly database consumption. However, I think it would be difficult for your to do the same calculation. It would require that you have the total database size for each hour in the month to be able to calculate the fractional credit consumption for that hour, then sum all the hourly credit usage. Basically, the hour unit is the finest level of granularity we use to calculate cost, because database size is dynamic.The best you can easily do is determine the current credit consumption, based on the current database size.
We are working to improve our reporting abilities for item sizes so that people will be able to more easily find items that are consuming a lot of credits.This will eventually come. In the meantime, you can get an estimate of the current consumption by going to your My Organization page, signed in as an administrator
Thanks,
Mike
Without the ability to recreate those credit charges on an item/individual basis, we can't continue to use ArcGIS Online (tragedy of the commons), as we need to be able to assign charges on an individual or project basis.
So, we're actually considering re-running the script every hour to pull the data from the REST API to store the granularity that we can't get from ArcGIS Online on our system.
The aggregated view does not let us effectively pass the cost of usage on to the project or user, meaning that it becomes an overhead cost which can and will be abused by the various users/projects, and is therefore not sustainable for us to use as an organization.
So... assuming the size of the reported feature is accurate, and that we're pulling the data from the REST API on an hourly basis, we, at least theoretically, should be able to recreate the credit usage total, but also support an itemized list of storage credits.
If the feature service is not hosted, it doesn't have an associated size, and is therefore not counted... so calculating credit usage based on whether or not the item is a "Feature Service" should still produce accurate results.
Currently, if I drill down as far as I can into the credit usage I can see that we're using 0.43 credits an hour, primarily in Feature Storage.
If I pull the data from the REST API using the above script, and calculate the credits used, I get, 0.404856765... So I'm close... but not quite correct. Once I can confirm that I have the correct calculation method, then it's just a matter of setting up a machine to pull the data and store/aggregate it for a monthly billing cycle. We'll also have to include credits used in other ways (i.e. enrichment, etc.). I agree this is difficult and painful, but our other option is to stop using ArcGIS Online, which I'm trying to avoid.
I've not tried the tool suggested by Heath, but I suspect it won't tell you what you want to know about specific feature layer storage. This is because all feature layer are stored in a single database, and the database currently doesn't allow us to calculate storage on a per layer basis.
Look for feature layers that have attachments. Picture from a camera can be several megabytes. Thus the storage of these features can add up quickly. If your feature layer was created before July 2014, it will store attachments as blobs in the database. After July, we made a change to AGOL to store attachments as files, so the credit usage is much less, as file storage is much less.
Hope this helps.
Mike