Updated best practices and process for using ArcGIS Python API in AWS Lambda layer

Jay_Gregory — Thu, 10 Nov 2022 15:25:03 GMT

I believe that the post from 2018 (https://community.esri.com/t5/arcgis-api-for-python-questions/arcgis-api-on-aws-lambda/m-p/785315#M1344) does not necessarily still apply.

In short, would like to access the ArcGIS Python API in AWS Lambda for basic scripting. So what I'd like to do is install the --no-deps version of the ArcGIS Python API (https://developers.arcgis.com/python/guide/install-and-set-up/#install-with-minimum-dependencies), along with required dependencies.

The documentation on the minimum dependencies is a little off - for example the documentation says one of the dependencies is requests_toolbelt. For conda, it needs to be specified as requests-toolbelt. Pip has a package requests_toolbelt. The same goes for ntlm_auth (Conda it is ntlm-auth and Pip it is ntlm_auth).

It also seems like there are some other dependencies that are not included in the documentation: cachetools, lxml, and reqeusts_oauthlib are two of them that are required when trying to leverage the --no-deps version of the ArcGIS Python API with Python3.9.

Lastly, unclear on exactly what process should be used for implementing this in an AWS lambda layer (if this is even possible). Conda to build environment (which packages the Python runtime as well so makes the layer even larger). Pip to build? What directory structure is needed for the .zip file? These are all things I'm struggling with that I feel like other folks might have solved.

I have heard AWS lambda has issues with certain libraries built with C, and also can't accept zip folders larger than a certain size (even if uploaded to S3 first).

Re: Updated best practices and process for using ArcGIS Python API in AWS Lambda layer

EarlMedina — Mon, 14 Nov 2022 14:08:03 GMT

I've used the Python API in a Lambda before and ran into the bloat issue with dependencies that you hint at. In my case, I was able to get it under the size limit but not without some work. My recommendation would be to take a moment to consider what functionality you actually need from the api and any other 3rd party libraries. If all you're using the api for is 2-3 things it might make sense just to code those parts yourself and keep things lean.

As far as setup goes, I'd recommend trying out a Serverless - it makes things much easier and leaves you with a template you can use for future Lambdas.

topic Updated best practices and process for using ArcGIS Python API in AWS Lambda layer in ArcGIS API for Python Questions

Updated best practices and process for using ArcGIS Python API in AWS Lambda layer

Re: Updated best practices and process for using ArcGIS Python API in AWS Lambda layer