Arcgis API on AWS Lambda

[DanielHunter]

I'm trying to upload a package to AWS Lambda and can not find/install the ArcGIS Python API for my package.  Has anyone done this?

Thanks.

[DanielHunter · Accepted solution]

Figured this out...

Use Amazon EC2 and spin up a free tier Linux instance.

Install Python 3.6

Install Anaconda and use Conda to install the ArcGIS Modules

Zip up the install directory along with your Lambda_function.py file and upload it to Lambda (or s3)

Dependencies required for Lambda to work include ArcGIS, NumPy, Pandas and PyTz

Putty Tools were a good resource to work between Windows and Linux environments

[DavidAskov1]

Bill Major‌, Daniel Hunter‌ -

My question is not just about Lambda, but also integrating this into a preexisting python environment.

Do you know where this is on the Windows Anaconda install? Anaconda is in C:\ProgramData\Anaconda3 by following the installer's default. There is a folder C:\ProgramData\Anaconda3\Lib\site-packages\arcgis with a bunch of .py scripts. Is that folder all I need to grab, or are there other entries written elsewhere (like a registry of modules)? Does the arcgis API imports other modules I need to get?

-- For Lambda, I assume I just drop the arcgis folder under my scripts and zip it up, does that sound right?

-- For the existing python, do I assume I just re-create whatever conda created. I see there is a tarball in the Esri Anaconda channel. Is that a better starting point? All I could find on GitHub (via Google) were training docs and samples.

thanks!

[DanielHunter]

These instructions from Esri should guide you.

install and set up | ArcGIS for Developers 

For Lambda - I believe the package needs to be compiled in Linux environment for it to work properly.  You'll need a linux machine to do so...you can create a new one on ec2.

For Existing Python - Virtual environments seem to be the way to go.  Consider creating a new virtual environment and installing arcgis api through conda.

[SimonJackson]

Is there an easier way of doing this without spinning up a Linux machine?

I am trying to run through the steps in this blog post, which suggests using PIP, but makes no mention of the machine needing to be linux. 

However, we know that "AWS Lambda runs under a Linux environment"

Can we just download the linux package from Conda?

linux-64/arcgis-1.7.0-py37h39e3cac_863.tar.bz2  - is this the right one for Lambda?

And then upload a zipfile of this package into Lamdba? 

[DanielHunter]

Its been a while since I've done this but don't let the linux instance deter you, its really very simple.  If I remember correctly the linux machine was needed for compilation and running the scripts...I'm unfamiliar with the linux package for Conda.  I've attached some notes I made a few years ago - They're probably pretty dated.  Good luck.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Built EC2 instance using Amazon Linux AMI (AWS Linux AMI for Lambda)
http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

Use Putty to connect to Linux machine from windows
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Download WinSCP to transfer between Windows and Linux:
https://winscp.net/eng/download.php

Use this workflow steps 5 through 9
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html

5. Install Python 3.6 and virtualenv using the following steps:
sudo yum install -y gcc zlib zlib-devel openssl openssl-devel
wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
tar -xzvf Python-3.6.1.tgz
cd Python-3.6.1 && ./configure && make
sudo make install
sudo /usr/local/bin/pip3 install virtualenv
6. Choose the virtual environment that was installed via pip3
/usr/local/bin/virtualenv ~/shrink_venv
source ~/shrink_venv/bin/activate
7. Install libraries in the virtual environment
pip install Pillow
pip install boto3
8. Add the contents of lib and lib64 site-packages to your .zip file.

cd $VIRTUAL_ENV/lib/python3.6/site-packages
zip -r9 ~/CreateThumbnail.zip *
9. Add your python code to the .zip file
cd ~
zip -g CreateThumbnail.zip CreateThumbnail.py

I couldn’t get the py file added to the zip so I copied the zip back to the windows environment and simply rezipped the original contents along with the py file.  Loaded that to AWS Lambda…

[SimonJackson]

Daniel Hunter‌ - big thanks for taking the time to share your workings.  It would appear others on StackOverflow also take a similar approach. 

Will take a look at working on this later today.  Thanks again.

[AndresCastillo]

This is gold.

[BillMajor]

Nice work Daniel.  I too recently went through this exercise.  One thing you might consider for the future is to create a Python virtual environment, pip install numpy and pandas, and then copy the arcgis module into this folder.  This way, the only thing in this folder than needs to be zipped up is /arcgis, /numpy, and /pandas; not the entire Anaconda install folder.  This should significantly reduce your upload package.

Nice job!

[JonNordling1]

One Question I would have is about the licensing, for example some of the scripts I would want to put in Lambda checkout extension. CheckOutExtension—Help | ArcGIS for Desktop  

Does anyone know how this might look for Lambda since it would not have the ArcGIS License Manager Integration? This would be huge I think if there was a way.