Automate Feature Layer Limit Usage / Access with ArcGIS python API.

[JACooper]

After exploring options with an ESRI Support Services representative, it was determined that no current functionality exists with the ArcGIS Python API to limit access / usage for secure services feature layers with stored credentials. 

Manually in ArcGIS Online (AGOL) now, one can create a secure service of an existing hosted feature layer by storing their credentials (ArcGIS Online username and password) within the new service itself (see here). After this, one can then alter the settings of the new secure service to Limit Usage of the service by enabling a timed rate limit and specifying very specific IP or URL addresses where the service can appear. Following the documentation in the following links will lead you to this look on a browser:

• https://support.esri.com/en/technical-article/000017029
• https://doc.arcgis.com/en/arcgis-online/manage-data/limit-usage-of-secure-services.htm 

While it is possible to add a web layer with stored credentials to ArcGIS Online using ArcGIS API for python, there does not seem to be a way to set limited access to these secured hosted services using the ArcGIS python API. This would be a valuable feature for organizations that want to be able to use the AGOL platform to publicly display and share data with personally identifiable information while protecting that data from full public access. Particularly in combination with Hosted Feature Layer Views (see here and here), these secure services can allow curated snapshots of datasets that include data that the publishing organization or their data partners/stakeholders do not want to be accessible to the general public in ways that public feature layers are on AGOL.

Current workflows are still being explored on how to circumvent this python API limitation by using python in combination with other API services like JavaScript API and REST API. However, the ability to achieve these ends all from within the ArcGIS python API itself would be a valuable feature. 

Current arcgis library version: 1.8.4

Note: Cache control may be a parallel yet complimentary additional missing python API functionality for ArcGIS.

[HuubZwart]

I believe this is already possible. After accessing the item, check the serviceProxyParam property, there you will find the information required. I believe this is the correct syntax:

{   "referrers": ["https://foo.com", "https://bar.com"],
   "hitsPerInterval": 1000,
   "intervalSeconds": 60
}

[JACooper]

@HuubZwart My issue is that I cannot yet find the `serviceProxyParam` property in python. I can access the web service item in python and see a good number of its other properties. I can also manually edit these service proxy parameters in ArcGIS Online. Exactly how in python should I access an item's `serviceProxyParam` property?

I'm not finding a lot of other documentation on this in the arcgis python API reference documentation or elsewhere.

[JACooper]

Update: @HuubZwart was correct. This is a item of the properties dictionary for secure services that already have URLs or rate limiting data attached to them. The documentation for this can be found here. Furthermore, I wrote a wrapper script below for myself that others may find useful. This idea can be closed.

def change_url_parameters_secserv(secserv: arcgis.features.FeatureLayer, url, append_rather_than_replace: bool = True) -> arcgis.features.FeatureLayer:
    """ Add allowable URLs to a secure service! """
    # Suggested by a reply to my idea on AGOL API ideas, code developed by me.
    #  https://community.esri.com/t5/arcgis-online-ideas/automate-feature-layer-limit-usage-access-with/idc-p/1144162/highlight/true#M8692
    # Proxy parameter URLs in root feature service, not individual sub-layers.
    #  Safely get serviceProxyParams, avoiding issues if they don't exist
    existing_restrictions = getattr(secserv, 'serviceProxyParams', None) or {}
    new_restrictions = existing_restrictions
    """
{
  "hitsPerInterval":2,
  "intervalSeconds":60,
  "referrers":["https://server1","https://server2"],
  "[https://*arcgis.com] //*.arcgis.com allows any subdomain from arcgis.com to be added
}
    """
    referrers = 'referrers'
    if type(url) == str:
        # User wants to add a single URL
        # convert to list
        url = [url]
    if type(url) != list:
        raise TypeError("URL must be a single URL string or a list of such strings.")

    start_clean = False
    if append_rather_than_replace:
        # Add, don't replace
        if referrers in new_restrictions:
            # Add new URLs, avoiding duplicates
            new_restrictions[referrers] = list(set(new_restrictions[referrers] + url))
        else:
            # It's new! Start adding.
            start_clean = True
    else:
        start_clean = True
    if start_clean:
        # initialize the list of URLs the Sec Serv can show up in content.
        new_restrictions[referrers] = url

    # Now actually update the thing
    update_restrictions = {"serviceProxyParams": new_restrictions}
    secserv.update(update_restrictions)
    return secserv

[JACooper]

I wrote a wrapper function for myself that does this. Head start for ESRI if they ever want to implement this. 

_DO_NOTHING = object()

def limit_secserv_usage(
        secserv: arcgis.features.FeatureLayer,
        urls: list = None,
        hits_per_sec_interval: int = _DO_NOTHING,
        interval_seconds: int = _DO_NOTHING,
        append_rather_than_replace_urls: bool = True
) -> arcgis.features.FeatureLayer:
    """
    Limit the URLs and/or IP addresses in which a secure service hosted on ArcGIS Online can appear.
    Limit the rate at which secure services can be accessed.

    Parameters:
        secserv (arcgis.features.FeatureLayer):
            The feature layer to which the usage restrictions will be applied.
        urls (list, optional):
            A list of allowed referrer URLs for accessing the service. If `append_rather_than_replace_urls`
            is `True`, the URLs will be added to the existing list. Defaults to None.
        hits_per_sec_interval (int, optional):
            The maximum number of hits allowed per interval. If `_DO_NOTHING` is passed, the current limit
            is preserved. Defaults to `_DO_NOTHING`.
        interval_seconds (int, optional):
            The length of the rate-limiting interval in seconds. If `_DO_NOTHING` is passed, the current interval
            is preserved. Defaults to `_DO_NOTHING`.
        append_rather_than_replace_urls (bool, optional):
            If `True`, appends the provided `urls` to the existing list of allowed URLs. If `False`,
            replaces the existing list. Defaults to `True`.

    Returns:
        arcgis.features.FeatureLayer:
            The updated feature layer with the applied usage restrictions.
     """
    
    ssp = "serviceProxyParams"
    existing_restrictions = getattr(secserv, ssp, None) or {}
    new_restrictions = existing_restrictions.copy()
    """
    Example of a `restrictions` dict
    {'hitsPerInterval': 100,
    'intervalSeconds': 1,
    'referrers': [
'https://arcgis.com/apps/dashboards/a69a3699cbc14b4abab94c03eb22779d',
        'https://storymaps.arcgis.com/stories/736a3c899cb54ebeba6e725e9739851b/edit',
        'https://storymaps.arcgis.com/stories/736a3c899cb54ebeba6e725e9739851b',
        ]
    }
    """

    # STEP 1: Limit the referrer URLs and IPs.
    referrers = 'referrers'
    if isinstance(urls, str):
        # User wants to add a single URL
        # convert to list
        urls = [urls]
    elif not urls:
        urls = []
    if not isinstance(urls, list):
        raise TypeError("URL must be a single URL string or a list of such strings.")

    append_rather_than_replace = bool(append_rather_than_replace_urls)
    if not urls and append_rather_than_replace:
        # No URLs to add.
        # Leave `new_restrictions` alone.
        #  If it had URLs, don't replace them.
        #  If it didn't have URLs, don't add a referrers item to the new restriction properties
        pass

    else:
        # We need to update something,
        #  whether it is adding URLs to the existing list (append = True)
        #  or overwriting the whole list (append=False)
        #   either to refresh (urls != [])
        #   or remove (urls == [])
        #   the URL list.
        start_clean = False
        if append_rather_than_replace:
            # Add, don't replace
            if referrers in new_restrictions:
                # Add new URLs, avoiding duplicates
                new_restrictions[referrers] = list(set(new_restrictions[referrers] + urls))
            else:
                # It's new; Start adding.
                start_clean = True
        else:
            start_clean = True
        if start_clean:
            # initialize the list of URLs the Sec Serv can show up in content.
            # NOTE: If urls==[], that is fine to pass to update.
            #  It will result in no URLs being added,
            #  and no 'referrers' item will be added to downstream serviceProxyParams.
            new_restrictions[referrers] = urls

    # STEP 2: Enable rate limiting

    def handle_rate_limit(rate_arg,old_rate_var_name: str) -> int:
        """ Create a new rate limit based off of the pre-existing rate and the newly passed argument."""
        # Format arguments of rate limiting.
        #  Ensure valid values of rate arguments as integers or Nones"""
        if rate_arg in [_DO_NOTHING, True]:
            # We want to handle this downstream. This is intended to Not overwrite existing parameters by default
            new_rate_val = _DO_NOTHING
        elif rate_arg and not isinstance(rate_arg, bool):
            # Force this to become an integer. This should ensure arguments passed are numeric.
            new_rate_val = int(rate_arg)
        else:
            # Force `not` and bool arguments passed to become None
            #  (in case someone passed `False` or something to indicate no rate limiting on this SFL)
            new_rate_val = None

        # Check for pre-existing rate limits.
        #  Set the value of that pre-existing parameter if it exists, else None
        if old_rate_var_name in existing_restrictions:
            preexisting_rate_val = existing_restrictions[old_rate_var_name]
        else:
            preexisting_rate_val = None

        # Handle new versus pre-existing rate limits
        # def handle_new_versus_old_rates(old_rate_val, new_rate_val):
        if preexisting_rate_val:
            # There was a pre-existing value here. Handle
            if new_rate_val == _DO_NOTHING:
                # There was a pre-existing value, and the new command didn't want to edit that. So don't edit it.
                #  Trick by setting new val to old value so nothing functionally changes.
                new_rate_val = preexisting_rate_val
            else:
                # New value was intended to overwrite the old value. Do it.
                #  Even if it means that a None overwrites a former value.
                #  The user wanted to take away limits in this case.
                # new_rate_val = new_rate_val
                pass
        else:
            # No pre-existing rate limits. So adding even Nones won't matter.
            if new_rate_val == _DO_NOTHING:
                new_rate_val = None
            # return new_rate_val

        return new_rate_val

    intsec = 'intervalSeconds'
    hpi = 'hitsPerInterval'
    hits_per_sec_interval = handle_rate_limit(old_rate_var_name=hpi,rate_arg=hits_per_sec_interval)
    interval_seconds = handle_rate_limit(rate_arg=interval_seconds,old_rate_var_name=intsec)

    #  These will only take effect if they both are not none.
    #   But the API will sort this out on its own.
    #   No need for me to put fixes into this wrapper code for it as of arcgis2.4
    if not hits_per_sec_interval or not interval_seconds:
        print(f"No rate limiting will be applied to {getattr(secserv, 'title', None) or 'the secure feature layer'}.")

    # Add the rate limits to the dictionary (Finally!)
    new_restrictions[intsec] = interval_seconds
    new_restrictions[hpi] = hits_per_sec_interval

    # STEP 3: Now actually update the thing
    # NOTE: if
    #   `new_restrictions` is empty or
    #   has exclusively any or all elements of `{'hitsPerInterval': None, 'intervalSeconds': None,'referrers': []}`,
    #  there is no harm in updating the secure service, as long as user intended to not set these parameters.
    #  It will either not touch these elements or remove existing parametrs therein.
    update_restrictions = {ssp: new_restrictions}
    secserv.update(update_restrictions)

    return secserv