Arcgis API on AWS Lambda

2047
10
Jump to solution
10-13-2017 10:58 AM
Highlighted
New Contributor III

I'm trying to upload a package to AWS Lambda and can not find/install the ArcGIS Python API for my package.  Has anyone done this?

Thanks.

1 Solution

Accepted Solutions
Highlighted
New Contributor III

Figured this out...

Use Amazon EC2 and spin up a free tier Linux instance.

Install Python 3.6

Install Anaconda and use Conda to install the ArcGIS Modules

Zip up the install directory along with your Lambda_function.py file and upload it to Lambda (or s3)

Dependencies required for Lambda to work include ArcGIS, NumPy, Pandas and PyTz

Putty Tools were a good resource to work between Windows and Linux environments

View solution in original post

10 Replies
Highlighted
New Contributor III

Figured this out...

Use Amazon EC2 and spin up a free tier Linux instance.

Install Python 3.6

Install Anaconda and use Conda to install the ArcGIS Modules

Zip up the install directory along with your Lambda_function.py file and upload it to Lambda (or s3)

Dependencies required for Lambda to work include ArcGIS, NumPy, Pandas and PyTz

Putty Tools were a good resource to work between Windows and Linux environments

View solution in original post

Highlighted
Occasional Contributor

Bill Major‌, Daniel Hunter‌ -

My question is not just about Lambda, but also integrating this into a preexisting python environment.

Do you know where this is on the Windows Anaconda install? Anaconda is in C:\ProgramData\Anaconda3 by following the installer's default. There is a folder C:\ProgramData\Anaconda3\Lib\site-packages\arcgis with a bunch of .py scripts. Is that folder all I need to grab, or are there other entries written elsewhere (like a registry of modules)? Does the arcgis API imports other modules I need to get?

-- For Lambda, I assume I just drop the arcgis folder under my scripts and zip it up, does that sound right?

-- For the existing python, do I assume I just re-create whatever conda created. I see there is a tarball in the Esri Anaconda channel. Is that a better starting point? All I could find on GitHub (via Google) were training docs and samples.

thanks!

Reply
0 Kudos
Highlighted
New Contributor III

These instructions from Esri should guide you.

install and set up | ArcGIS for Developers 

For Lambda - I believe the package needs to be compiled in Linux environment for it to work properly.  You'll need a linux machine to do so...you can create a new one on ec2.

For Existing Python - Virtual environments seem to be the way to go.  Consider creating a new virtual environment and installing arcgis api through conda.

Reply
0 Kudos
Highlighted
Occasional Contributor III

Is there an easier way of doing this without spinning up a Linux machine?

I am trying to run through the steps in this blog post, which suggests using PIP, but makes no mention of the machine needing to be linux. 

However, we know that "AWS Lambda runs under a Linux environment"

Can we just download the linux package from Conda?

linux-64/arcgis-1.7.0-py37h39e3cac_863.tar.bz2  - is this the right one for Lambda?

And then upload a zipfile of this package into Lamdba? 

Reply
0 Kudos
Highlighted
New Contributor III

Its been a while since I've done this but don't let the linux instance deter you, its really very simple.  If I remember correctly the linux machine was needed for compilation and running the scripts...I'm unfamiliar with the linux package for Conda.  I've attached some notes I made a few years ago - They're probably pretty dated.  Good luck.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Built EC2 instance using Amazon Linux AMI (AWS Linux AMI for Lambda)
http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

Use Putty to connect to Linux machine from windows
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

Download WinSCP to transfer between Windows and Linux:
https://winscp.net/eng/download.php

Use this workflow steps 5 through 9
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html

5. Install Python 3.6 and virtualenv using the following steps:
sudo yum install -y gcc zlib zlib-devel openssl openssl-devel
wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
tar -xzvf Python-3.6.1.tgz
cd Python-3.6.1 && ./configure && make
sudo make install
sudo /usr/local/bin/pip3 install virtualenv
6. Choose the virtual environment that was installed via pip3
/usr/local/bin/virtualenv ~/shrink_venv
source ~/shrink_venv/bin/activate
7. Install libraries in the virtual environment
pip install Pillow
pip install boto3
8. Add the contents of lib and lib64 site-packages to your .zip file.

cd $VIRTUAL_ENV/lib/python3.6/site-packages
zip -r9 ~/CreateThumbnail.zip *
9. Add your python code to the .zip file
cd ~
zip -g CreateThumbnail.zip CreateThumbnail.py

I couldn’t get the py file added to the zip so I copied the zip back to the windows environment and simply rezipped the original contents along with the py file.  Loaded that to AWS Lambda…

Highlighted
Occasional Contributor III

Daniel Hunter‌ - big thanks for taking the time to share your workings.  It would appear others on StackOverflow also take a similar approach. 

Will take a look at working on this later today.  Thanks again.

Reply
0 Kudos
Highlighted
New Contributor III

Nice work Daniel.  I too recently went through this exercise.  One thing you might consider for the future is to create a Python virtual environment, pip install numpy and pandas, and then copy the arcgis module into this folder.  This way, the only thing in this folder than needs to be zipped up is /arcgis, /numpy, and /pandas; not the entire Anaconda install folder.  This should significantly reduce your upload package.

Nice job!

Reply
0 Kudos
Highlighted
New Contributor II

One Question I would have is about the licensing, for example some of the scripts I would want to put in Lambda checkout extension. CheckOutExtension—Help | ArcGIS for Desktop  

Does anyone know how this might look for Lambda since it would not have the ArcGIS License Manager Integration? This would be huge I think if there was a way. 

Reply
0 Kudos
Highlighted
Occasional Contributor

Jon - I think you are looking for info on ArcPy. This thread is about Esri's other major Python API, called the "ArcGIS API for Python" (not a very creative name). ArcPy requires licensing, whereas the ArcGIS API for Python does not. That is more about working with server products, so you usually have to authenticate to the server, but you don't need any license to use the API itself. 

Reply
0 Kudos