Arcpy in Apache Air Flow

[SoratoSouza_e_Silva]

Hi,

It´s possible orchestrate pipelines in python with arcpy library in Apache Air Flow?

Thank´s

[HaydenWelch]

As far as I know, the arcpy library requires you to be in the official ESRI python environment. So if you can get your Airflow to use the environment packaged with Arc I don't see why not.

[ShaunWalbridge]

It doesn't look like Apache Airflow has support for Windows currently, but they do have an issue for tracking its availability here:  https://github.com/apache/airflow/issues/10388

Without that, you could set up WSL2 to use Airflow via Linux on Windows, but that requires an extra set of skills and technology to use and may not fit into the model you're trying to work with. There are Linux based environments like Server that potentially could work, depending on what you're trying to accomplish.

Cheers, Shaun

[AlexanderDanielPratama]

Hi actually I am right now looking to debug airflow and arcpy 😄

If you probably ask the airflow library compatible with arcpy, in short answer yes. I installed ArcGIS Enterprise 11.2 Linux OS in office VM and then install miniconda3 in the same VM. In miniconda3 I install airflow with its requirements (Installation from PyPI — Airflow Documentation (apache.org)) and arcgis server python (Python 3 runtime for ArcGIS Server on Linux—ArcGIS Server | Documentation for ArcGIS Enterprise)

Keep in mind that, you need arcgis server license to run arcpy.

Currently I have deployed it in my office. But somehow when I create a simple pipeline and ran it  from airflow webserver it stucked for hours. But if I ran one by one arcgis syntax are perfectly fine.

So now I am still debugging the problem, if I have an update maybe I will update this comment.

Cheers.

Notes: @ShaunWalbridge  mention using WSL. It not works actually, because to run arcpy in you need arcgis pro license. WSL is in different environment so it cannot be proceed to next step. The only way is using ArcGIS Server Linux

UPDATE:

Sadly arcpy is not compatible with airflow because airflow does not support multiprocessing outside of its platform (ArcGIS Pro and ArcGIS Enterprise). Airflow have celery and celery is like web framework such as flask, fastapi, djanggo. Arcpy does not allow run directly on web framework maybe due to license violation.

Another way is you can execute arcpy using geoprocessing service or run arcpy python (os.run)

Hopefully it helps