topic Re: ArcGIS On-line Credit usage in ArcGIS Online Questions

ArcGIS On-line Credit usage

GordonSumerling — Tue, 21 Oct 2014 00:04:47 GMT

All,.

I am looking at the credit usage in ArcGIS On-Line. I have roughly 2.5 gb of features services which are stored in a database. On average these services are consuming 17 credits a day. I currently cannot find out which features are consuming the credits as it aggregates all the usinge to one number. Is there a way I can find out

a) which are my largest consumers of database storage. i.e. largest feature classes

b) which feature services are consuming the most credits

Regards

Gordon

Re: ArcGIS On-line Credit usage

HeathBrackett1 — Tue, 21 Oct 2014 21:32:25 GMT

Gordon,

Check out this tool which will create a spreadsheet of all of the content in your organization. You can then sort based on file size. That would be the easiest way to go.

But you can also create some custom scripts using the Python modules (PortalPy, agoTools, ArcREST) found on ESRI's ArcGIS Online Admin GitHub site. I've found scripting using these tools to be very helpful and surprisingly easy to use.

Heath

Re: ArcGIS On-line Credit usage

AnthonyGiles — Tue, 21 Oct 2014 22:10:15 GMT

Gordon

You get charged 2.4 credits per 10mb of data stored per month, so all of your feature services will be consuming credits to some degree, a credit estimator can be found here:

ArcGIS Online | Credits Estimator

Regards

Anthony

Re: ArcGIS On-line Credit usage

MikeMinami — Wed, 22 Oct 2014 16:06:18 GMT

I've not tried the tool suggested by Heath, but I suspect it won't tell you what you want to know about specific feature layer storage. This is because all feature layer are stored in a single database, and the database currently doesn't allow us to calculate storage on a per layer basis.

Look for feature layers that have attachments. Picture from a camera can be several megabytes. Thus the storage of these features can add up quickly. If your feature layer was created before July 2014, it will store attachments as blobs in the database. After July, we made a change to AGOL to store attachments as files, so the credit usage is much less, as file storage is much less.

Hope this helps.

Mike

Re: ArcGIS On-line Credit usage

HeathBrackett1 — Wed, 22 Oct 2014 21:43:33 GMT

Mike, the ArcGIS Online Catalog tool that I first mentioned does collect the file size of each item. There is a checkbox that you must check to include the file size in the catalog. Once it runs, you can then download the file to a csv and sort as you like, particularly based on "size". If you want to see what sort of information can be obtained about each item, check the REST API Help.

Also, each user has a "storageusage" amount tied to that user which can be accessed through the REST API. If you are more familiar with Python than using REST calls, like I am, then the getUsers function in the agoTools that I referenced above (or the get_org_users function in the PortalPy tools) can be used to search all of your users for the biggest data hogs.

Heath

Re: ArcGIS On-line Credit usage

MikeMinami — Thu, 23 Oct 2014 00:01:46 GMT

***UPDATE*** Oct 2016. The size of a hosted feature layer as listed on the item details page now reflects the current database storage size of the feature layer.

***Original Post***

OK, I've found out a little more about the size parameter on feature layers. The size parameter is a reflection of the size at publish time. Unfortunately, it doesn't get updated as the feature layer is edited. So, as long as you haven't edited your data, it would be a good way to compare different layer sizes...

Thanks,

Mike

Re: ArcGIS On-line Credit usage

GordonSumerling — Thu, 23 Oct 2014 01:43:24 GMT

Thanks Mike,

A lot of our feature services have attachments and pre-date July 2014. We are going to go through and republish these to take advantage of the file storage rather than the database storage.

Regards

Gordon

Re: ArcGIS On-line Credit usage

MikeMinami — Thu, 23 Oct 2014 15:37:36 GMT

Yes, you can export to FGDB and republish.

Mike

Re: ArcGIS On-line Credit usage

JoshuaFord — Tue, 03 Feb 2015 19:40:39 GMT

I am having a similar problem with data consumption. AGOL Catalog worked great for identifying the hogs. However I am unclear as to the type of storage that applies to each item.

Is there a list of what content "type" in AGOL is charged at each rate: "Feature Service Storage" or "Tile and Data Storage".

Thanks,

Josh

Re: ArcGIS On-line Credit usage

KellyGerrow — Tue, 03 Feb 2015 20:37:06 GMT

Hi Joshua,

You will only be charged Feature Service Storage for a Hosted Feature Service in your organization. All other data types stored in your content will be charged credits based on the data and tile storage rate.

This documentation goes into further details regarding credit charges.

-Kelly

Re: ArcGIS On-line Credit usage

JoshuaFord — Tue, 03 Feb 2015 21:26:49 GMT

Kelly,

When I run AGOL Catalog it list 9 separate content types. All are hosted services on AGOL.

Do I have this breakdown correct?

Feature Service Storage (2.4 cr / 10 mb) =

Map Service

Map Package

Feature Service

Tile and Data Storage (1.2 cr / 1 GB) =

Tile Package

Web Map

Service Definition

Image

Code Attachment

Web App

Thanks,

Re: ArcGIS On-line Credit usage

KellyGerrow — Tue, 03 Feb 2015 22:22:16 GMT

Hi Joshua,

I have updated your list below:

Feature Service Storage (2.4 cr / 10 mb) =

Hosted Feature Service

Tile and Data Storage (1.2 cr / 1 GB) =

Map Service

Map Package

Tile Package

Web Map

Service Definition

Image

Code Attachment

Web App

Only Hosted Feature Services (Published to ArcGIS Online are charged feature storage credits. If you were to reference a feature service from ArcGIS Server, this wouldn't count towards your feature service storage.

Let me know if you have any more questions,

-Kelly

Re: ArcGIS On-line Credit usage

JoshuaFord — Tue, 03 Feb 2015 22:59:54 GMT

Kelly,

This is driving me nuts. AGOL Catalog results show 295 mb of total storage, 64 mb of which are Hosted Feature Services. We're burning almost 10 credits per day from 1.44 GB of Feature Service Storage according to the My Organization page. Where's the discrepancy?

Thanks,

Re: ArcGIS On-line Credit usage

KellyGerrow — Tue, 03 Feb 2015 23:40:44 GMT

Hi Joshua,

The sizes reported in catalog only reports to the size of the data of hosted feature services when they were published. Were any of the services published prior to July 2014 that contain attachments? This can be a cause for the discrepancy.

There is currently an enhancement regarding showing the current size of feature services available at this link

http://support.esri.com/en/knowledgebase/techarticles/detail/43869

Please Contact Technical Support for help in determining where the discrepancy is.

Thanks,

Kelly

Re: ArcGIS On-line Credit usage

KellyGerrow — Tue, 05 Apr 2016 23:09:06 GMT

Update:

The size reported on the item details page is the current size of data storage for feature services (not the size at the time of publishing).

Re: ArcGIS On-line Credit usage

deleted-user-FfGbgR253rTy — Thu, 08 Sep 2016 18:32:44 GMT

So... to confirm. If I've extracted the sizes from the catalog using something like agolTools from the ESRI github, then I should be able to calculate my hourly credit usage on an item by item basis.

If I've got the data in excel, then the formula to calculate this hourly credit usage would look something like this;

=IF("Feature Type Column"="Feature Service", "SizeinMB"/10*2.4/720,"SizeinGB"*1.2/720)

720 = the hrs in the month

Credits are charged at 2.4 credits per 10 MB per month, calculated on an hourly basis. ArcGIS Online | Service Credits

The problem with this is that the size reported in the catalog for items created before July 2014 is not the actual size... (as per above)

So... If we delete all the items created before July 2014 and republish any that we need, we should be able to create a broken down list of items and hourly credit usage. This could be re-calculated every hour to get a total number of credits used by an item...

The reason I'm looking at this, is that we currently don't have any way for our IT dept. who is responsible for purchasing credits, to recoup storage costs from the various groups using our organizational account. I'm sure we're not the only ones who need to be able to do break down these costs.

Re: ArcGIS On-line Credit usage

deleted-user-FfGbgR253rTy — Wed, 05 Oct 2016 00:13:30 GMT

So to update, I'm having some issues getting the above referenced formula to match with the reported credit usage in my organizational account status... So far, the closest I can get is if I double the number that the above referenced formula gives me.

I have a call in with Tech Support to investigate, and will update this post when I get a response. I'm also hoping to expand the script to add other credit usage...

Re: ArcGIS On-line Credit usage

MikeMinami — Wed, 05 Oct 2016 17:55:17 GMT

Here are some facts I think will be useful

The size of a hosted feature layer as shown on the item details page now accurately reflects the actual database storage size for that layer.
This search expression (I believe) will find all hosted feature layers in your organization if you are an admin and can see all private items.
type:"feature service" typekeywords:hosted

Wallace, Cameron I'm not sure your expression is correct. We do calculate credit charges based on hourly database consumption. However, I think it would be difficult for your to do the same calculation. It would require that you have the total database size for each hour in the month to be able to calculate the fractional credit consumption for that hour, then sum all the hourly credit usage. Basically, the hour unit is the finest level of granularity we use to calculate cost, because database size is dynamic.The best you can easily do is determine the current credit consumption, based on the current database size.

We are working to improve our reporting abilities for item sizes so that people will be able to more easily find items that are consuming a lot of credits.This will eventually come. In the meantime, you can get an estimate of the current consumption by going to your My Organization page, signed in as an administrator

As an administrator, navigate to My Organization.
Click on View Status.
Scroll to the bottom of the page and you'll see a circular credit utilization chart. Click on the segment of the chart that reflects storage. The chart will update to show storage.
Click again on the chart on the segment that reflects "Feature storage". Below the chart, you'll see a table that reflects the total database storage and file storage associated with feature layers and the credit consumption during the specified time period, which you set at the top of the page.

Thanks,

Mike

Re: ArcGIS On-line Credit usage

deleted-user-FfGbgR253rTy — Wed, 05 Oct 2016 18:14:32 GMT

Without the ability to recreate those credit charges on an item/individual basis, we can't continue to use ArcGIS Online (tragedy of the commons), as we need to be able to assign charges on an individual or project basis.

So, we're actually considering re-running the script every hour to pull the data from the REST API to store the granularity that we can't get from ArcGIS Online on our system.

The aggregated view does not let us effectively pass the cost of usage on to the project or user, meaning that it becomes an overhead cost which can and will be abused by the various users/projects, and is therefore not sustainable for us to use as an organization.

So... assuming the size of the reported feature is accurate, and that we're pulling the data from the REST API on an hourly basis, we, at least theoretically, should be able to recreate the credit usage total, but also support an itemized list of storage credits.

If the feature service is not hosted, it doesn't have an associated size, and is therefore not counted... so calculating credit usage based on whether or not the item is a "Feature Service" should still produce accurate results.

Currently, if I drill down as far as I can into the credit usage I can see that we're using 0.43 credits an hour, primarily in Feature Storage.

If I pull the data from the REST API using the above script, and calculate the credits used, I get, 0.404856765... So I'm close... but not quite correct. Once I can confirm that I have the correct calculation method, then it's just a matter of setting up a machine to pull the data and store/aggregate it for a monthly billing cycle. We'll also have to include credits used in other ways (i.e. enrichment, etc.). I agree this is difficult and painful, but our other option is to stop using ArcGIS Online, which I'm trying to avoid.