topic Re: Arcgis API on AWS Lambda in ArcGIS API for Python Questions

Arcgis API on AWS Lambda

DanielHunter — Fri, 13 Oct 2017 17:58:50 GMT

I'm trying to upload a package to AWS Lambda and can not find/install the ArcGIS Python API for my package. Has anyone done this?

Thanks.

Re: Arcgis API on AWS Lambda

DanielHunter — Mon, 23 Oct 2017 15:41:50 GMT

Figured this out...

Use Amazon EC2 and spin up a free tier Linux instance.

Install Python 3.6

Install Anaconda and use Conda to install the ArcGIS Modules

Zip up the install directory along with your Lambda_function.py file and upload it to Lambda (or s3)

Dependencies required for Lambda to work include ArcGIS, NumPy, Pandas and PyTz

Putty Tools were a good resource to work between Windows and Linux environments

Re: Arcgis API on AWS Lambda

BillMajor — Thu, 16 Nov 2017 14:38:09 GMT

Nice work Daniel. I too recently went through this exercise. One thing you might consider for the future is to create a Python virtual environment, pip install numpy and pandas, and then copy the arcgis module into this folder. This way, the only thing in this folder than needs to be zipped up is /arcgis, /numpy, and /pandas; not the entire Anaconda install folder. This should significantly reduce your upload package.

Nice job!

Re: Arcgis API on AWS Lambda

DavidAskov1 — Sat, 18 Nov 2017 05:15:05 GMT

Bill Major‌, Daniel Hunter‌ -

My question is not just about Lambda, but also integrating this into a preexisting python environment.

Do you know where this is on the Windows Anaconda install? Anaconda is in C:\ProgramData\Anaconda3 by following the installer's default. There is a folder C:\ProgramData\Anaconda3\Lib\site-packages\arcgis with a bunch of .py scripts. Is that folder all I need to grab, or are there other entries written elsewhere (like a registry of modules)? Does the arcgis API imports other modules I need to get?

-- For Lambda, I assume I just drop the arcgis folder under my scripts and zip it up, does that sound right?

-- For the existing python, do I assume I just re-create whatever conda created. I see there is a tarball in the Esri Anaconda channel. Is that a better starting point? All I could find on GitHub (via Google) were training docs and samples.

thanks!

Re: Arcgis API on AWS Lambda

DanielHunter — Mon, 20 Nov 2017 13:34:14 GMT

These instructions from Esri should guide you.

install and set up | ArcGIS for Developers

For Lambda - I believe the package needs to be compiled in Linux environment for it to work properly. You'll need a linux machine to do so...you can create a new one on ec2.

For Existing Python - Virtual environments seem to be the way to go. Consider creating a new virtual environment and installing arcgis api through conda.

Re: Arcgis API on AWS Lambda

JonNordling1 — Sat, 08 Sep 2018 02:18:13 GMT

One Question I would have is about the licensing, for example some of the scripts I would want to put in Lambda checkout extension. CheckOutExtension—Help | ArcGIS for Desktop

Does anyone know how this might look for Lambda since it would not have the ArcGIS License Manager Integration? This would be huge I think if there was a way.

Re: Arcgis API on AWS Lambda

DavidAskov1 — Tue, 11 Sep 2018 00:54:39 GMT

Jon - I think you are looking for info on ArcPy. This thread is about Esri's other major Python API, called the "ArcGIS API for Python" (not a very creative name). ArcPy requires licensing, whereas the ArcGIS API for Python does not. That is more about working with server products, so you usually have to authenticate to the server, but you don't need any license to use the API itself.

Re: Arcgis API on AWS Lambda

SimonJackson — Fri, 01 Nov 2019 01:12:09 GMT

Is there an easier way of doing this without spinning up a Linux machine?

I am trying to run through the steps in this blog post, which suggests using PIP, but makes no mention of the machine needing to be linux.

However, we know that "AWS Lambda runs under a Linux environment"

Can we just download the linux package from Conda?

linux-64/arcgis-1.7.0-py37h39e3cac_863.tar.bz2 - is this the right one for Lambda?

And then upload a zipfile of this package into Lamdba?

Re: Arcgis API on AWS Lambda

DanielHunter — Fri, 01 Nov 2019 11:17:56 GMT

Its been a while since I've done this but don't let the linux instance deter you, its really very simple. If I remember correctly the linux machine was needed for compilation and running the scripts...I'm unfamiliar with the linux package for Conda. I've attached some notes I made a few years ago - They're probably pretty dated. Good luck.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Built EC2 instance using Amazon Linux AMI (AWS Linux AMI for Lambda)
http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

Use Putty to connect to Linux machine from windows
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Download WinSCP to transfer between Windows and Linux:
https://winscp.net/eng/download.php

Use this workflow steps 5 through 9
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html

5. Install Python 3.6 and virtualenv using the following steps:
sudo yum install -y gcc zlib zlib-devel openssl openssl-devel
wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
tar -xzvf Python-3.6.1.tgz
cd Python-3.6.1 && ./configure && make
sudo make install
sudo /usr/local/bin/pip3 install virtualenv
6. Choose the virtual environment that was installed via pip3
/usr/local/bin/virtualenv ~/shrink_venv
source ~/shrink_venv/bin/activate
7. Install libraries in the virtual environment
pip install Pillow
pip install boto3
8. Add the contents of lib and lib64 site-packages to your .zip file.

cd $VIRTUAL_ENV/lib/python3.6/site-packages
zip -r9 ~/CreateThumbnail.zip *
9. Add your python code to the .zip file
cd ~
zip -g CreateThumbnail.zip CreateThumbnail.py

I couldn’t get the py file added to the zip so I copied the zip back to the windows environment and simply rezipped the original contents along with the py file. Loaded that to AWS Lambda…

Re: Arcgis API on AWS Lambda

SimonJackson — Sun, 03 Nov 2019 23:35:14 GMT

Daniel Hunter‌ - big thanks for taking the time to share your workings. It would appear others on StackOverflow also take a similar approach.

Will take a look at working on this later today. Thanks again.

Re: Arcgis API on AWS Lambda

joonpark — Sun, 27 Sep 2020 02:52:55 GMT

Hi guys,

- Packaged ONLY in Linux env ? I was playing with Anaconda / Conda in window 10 ( all that virtual env. mimicking) and install ArcGIS package but no luck.

below is error from AWS Lambda with simple "from arcgis.gis import GIS" testing

- {
"errorMessage": "Unable to import module 'main': No module named 'requests'",
"errorType": "Runtime.ImportModuleError"
}

Thanks

Re: Arcgis API on AWS Lambda

AndresCastillo — Tue, 18 Oct 2022 15:18:26 GMT

This is gold.

Re: Arcgis API on AWS Lambda

AndresCastillo — Tue, 18 Oct 2022 15:23:49 GMT

@JonNordling1 ,

I believe if you use the checkOutextension function in the uploaded lambda script, it becomes a matter of access.

If the aws environment where the lambda is deployed has port access to the license manager, it should be able to grab the license for the script.

I have not verified this, but this are my thoughts on this.

Re: Arcgis API on AWS Lambda

AndresCastillo — Tue, 18 Oct 2022 15:36:15 GMT

Seems like ArcPy is too big to fit into lambda restrictions:
https://community.esri.com/t5/python-ideas/allow-arcpy-to-be-loaded-and-executed-through-aws/idi-p/1063441

There should be other alternatives to explore, such as AWS Fargate, or AWS Batch

Re: Arcgis API on AWS Lambda

MichaelKarikari1 — Tue, 13 Dec 2022 14:30:28 GMT

I was curious if anyone has recent success with this using the latest arcgis api module. The size of the site-package folder that gets filed up with ArcGIS API python dependencies seems to always be larger that was is allowed. Ive tried striping the package that would be uploaded to just the API module, but seems that lambda always complains of need additional modules in order to run

Re: Arcgis API on AWS Lambda

DrewDowling — Fri, 17 Oct 2025 19:20:08 GMT

Esri is releasing a docker container that has the ArcGIS API library and it's dependencies contiled in Linux. This container can be uploaded to an AWS ECR and run from there.

"Push to your private AWS ECR instance, and configure lambda to run from this container image. As of this writing, public AWS ECR instances are not supported for lambda."

https://github.com/Esri/arcgis-python-api/tree/master/docker