Uploading imagery to Image Services for Real-time Viewing

Blog Post created by EMedina-esristaff Employee on Sep 19, 2018

A long while back, I helped a developer that was working with an Image Service and needed to be able to view uploaded images as soon as possible. I forget the particulars of his project, but aerial imagery was being captured for agricultural use and he had been tasked with creating a Java application which could upload collected images to the Image Service and display them immediately at a small scale - essentially, he needed to illustrate a "snapshot" of the total area that had been captured. In his case, there weren't any GIS Analysts on the ground and updates needed to be monitored from the web via the Image Service.


To gain a conceptual understanding of the process he went through the workflow of adding rasters to a Mosaic Dataset in ArcMap. This mostly went fine, but a major problem in the workflow was that rasters he added to the Mosaic Dataset would not display in the ArcGIS JavaScript viewer (previewing the Image Services at REST) unless he zoomed in very close.


This is where I came in.


Like many people, he wasn't quite sure whether he needed Overviews, Pyramids, both, or something else to achieve the desired result. I explained that for his purposes he wanted to be using Pyramids as Overviews take a while to process and they weren't suited to the dynamic nature of the data. In general, you generate overviews only in the following situations:

  • You want to use the Mosaic Dataset for visualization purposes (at different scales of interest).
  • You have no time constraints when working with large Mosaic Datasets.
  • You are creating a simple Mosaic Dataset with fewer datasets.


I won't get into the specifics, but to view the data at a small scale (i.e., zoomed out) he also needed to override the default MaxPS values: Mosaic dataset attribute table—Help | ArcGIS Desktop / Cell size ranges in a mosaic dataset—Help | ArcGIS Desktop - we arbitrarily chose a value of 300.


To allow for uploading new imagery, he needed to ensure this capability was turned on in the Image Service and a dynamic workspace was set: Preparing image services—Documentation | ArcGIS Enterprise.


With the Image Service published, we turned our attention to which REST operations he would need to use in his Java Application:


Here's where we pause.


The above was enough to get him on his way, but I myself never got the satisfaction of seeing the whole thing through! Not surprisingly, when I chanced upon this article by Michael Cho on how to use the watchdog Python package to monitor for file directory changes my mind was immediately drawn back to this project.


Now, the use of watchdog is probably more practical for say dropping a .csv into a folder and using that to add new features to a Point Feature Service but I've gone ahead and used it to complete the project as I understood it. I say that it may not be practical for Image Services because a) I don't know how common such a workflow is and b) You end up with copies of the data in the arcgisuploads and dynamic workspace directories and for large raster datasets this might put a strain on the Image Server.


Anyhow, I hope just seeing the code inspires others to use it for whatever purposes they see fit. Some things to note on use:

  • This was written to work with both non-secured and secured Image Services. To use it with secured Image Services run it with option "--s": python watcher.py --s
  • You set a folder to watch for changes and when:
    • you drop a raster dataset into the watched folder, it will upload and add to the specified Image Service.
    • you remove a raster dataset into the watched folder, it will remove the image from the Image Service.
  • Of course, you'll need to set the folder to watch and update the URLs to match your environment.


Other information/limitations:

  • This was written for use with Python 3.
  • You'll need have watchdog · PyPI  and requests · PyPI  installed.
  • The code could certainly be simplified - if you have any comments those are greatly appreciated.
    • Currently, for secured services the script will request a token per operation. This is not efficient but it should only take a bit of logic to streamline. I might get around to this in my free time!
  • Datasets are added without consideration of overlapping extents.
  • Deletion works in memory so it'll only remember what you added in the same session. You could probably fix that by writing events to a file or a database table.
    • I wrote this on a Linux environment - if you need to run this script in the background and have it watching a folder at all times you could easily create a systemd (or whatever your distribution's service manager is) service. I'm not certain what the best way to do this is on Windows but I imagine there's more than one way to approach it - pywin32 looks to be a good option!


Everything else, I hope, is covered in the script comments...


To get a preview of the script, please see the gifs below:


Dropping rasters into a directory...

Dropping rasters into a directory - the resulting json responses from the Image Server...



As images are added/removed from the folder, the Image Service is updated accordingly...