topic Re: Displaying blobs in ArcGIS Viewer for Flex Questions

Displaying blobs

DustinWiersma2 — Fri, 10 May 2013 19:41:57 GMT

Hello All,

I have a mobile project running using ArcGIS mobile for Windows. The project is being used by Trimble Junos - in order to allow the Trimbles to collect more than one picture in one location, I named the fields in the DB "Picture_1", ect. This works great as it prompts the camera to open in the Trimble or add a photo that is already stored on the Trimble. The problem I am having is adding the pictures directly into a Flex map so the Flex map is populated with data "live" from the field. My current work flow is:

-Copy the feature class to a personal geodatabase (since it is a Microsoft Access database)
-Run a script that extracts the blobs as jpegs to a folder
-Have the Flex map call to that folder to display the images

The problem with this is that the data does not feed live to the Flex map. Any ideas?
Thanks!

-Dustin Wiersma, GIS Analyst

Re: Displaying blobs

BjornSvensson — Fri, 10 May 2013 20:53:29 GMT

I have a mobile project running using ArcGIS mobile for Windows...pictures

That won't work because the server doesn't return BLOBs using the REST API (which is how Flex and other Web APIs connect to ArcGIS Server).

Instead of storing the images in BLOBs, you would need to use attachments (or files in folders like your batch script is doing now). I don't know enough of the "trimble/mobile" aspect how to make that happen though...
http://resources.arcgis.com/en/help/main/10.1/index.html#//01m90000000r000000

Re: Displaying blobs

DustinWiersma2 — Fri, 10 May 2013 21:00:30 GMT

Thanks for the reply! That is interesting about the rest point and blobs. Unfortunately, the Trimbles do not use the attachment table for this project - other mobile projects do, such as mobile projects using the esri ArcGIS app. For this project I need to have multiple photos for a single point, otherwise I could just use a field as a type raster. So it sounds like I may need to develop a script that automates my current workflow.

Re: Displaying blobs

RhettZufelt — Mon, 13 May 2013 15:19:45 GMT

I don't do mobile, but FLEX has issues accessing a local folder. You normally need to set up a virtual directory on your server that points to the folder you are saving the jpegs in. You can then access them using the URL in the flex viewer.

R_

Re: Displaying blobs

DustinWiersma2 — Mon, 13 May 2013 15:31:11 GMT

Rhett,
Thanks for the reply! That will be the work flow I will try to automate - have a script that extracts the blobs to jpegs in a folder on the server and have the Flex map point to that folder.

Re: Displaying blobs

RhettZufelt — Mon, 13 May 2013 16:14:36 GMT

Dustin,

I have a similar workflow, with photos stored in BLOB fields and one to potentially many photos per feature. The attachemnts feature doesn't really support 1:many so I have a python script that extracts the jpeg photos from the blob and saves them in a folder. As it is writing the blobs, I capture the path and filename that I am saving them to (as well as my feature ID) and append to a list, then at the end of my script, I insertCursor the list to populate a FGDB table. Now I not only have the jpeg stored local in a folder, I also have a ready-made table with all my paths ready to join/calculate to the feature class data that can be used for my link field.

Of course, I had to make a virtual directory on the server pointing to the folder I saved all my jpegs in, then had to go into IIS and, for that virtual folder, allow directory browsing as I often have many photos per site (if just one photo, my "path" actually points to the jpeg file, if more than one, it points to the folder.

This was kind of cluttery, especially for the sites with multiple photos, as IE doesn't give thumbnail views, so, all you see is a list of the somewhat cryptic filenames, and need to click on each photo to view it.

So, I took it one step farther. After my python script extracts and saves the jpegs from the BOLBs, writes the "lookup table", joins/calculates the URL info to my feature class, it then runs a slightly modified version of pyGallery. This creates a gallery for my photos similar to ( http://pygallery.sourceforge.net/samples/ ). So now, if there is one photo, my popup says "Click for Photo", and it takes you direclty to the image viewed in a browser. If there are more than one image, it says "Click for Photos", and if you click, it will take you to the index/thumbnail gallery page for that feature, with all the photos.

So far, not only the best way I've came up with for BLOBs, but also for one:many photos in Flex.

In case this gives you some ideas,

R_

Re: Displaying blobs

DustinWiersma2 — Mon, 13 May 2013 17:28:23 GMT

Wow, that is quite the work flow. That is exactly what I need to do. Do you have the script written so it runs automatically on a time schedule?

Re: Displaying blobs

RhettZufelt — Sun, 12 Dec 2021 16:43:12 GMT

Dustin,

With python and ArcGIS, I have found that it is normally best to execute all python scripts (at least scheduled ones) from a batch file. this way, once it is done with that particular script, since it was spawned in a separate python instance, it will then relsease all the "cryptic" locks it puts on the data and can move on to the next script.

So, if I have many scripts to run, I make one python script with a bunch of os.system(batchfilename) calls, will run the script, close the process, and move on.

each batch file is similar to :

cd C:\Python27\ArcGIS10.1

Python C:\arcgisserver\python\finish.py

pause

Exit

Which sets the path to the python version I want to use, then executes the finish.py script, then exits when done.

To schedule, I just use the windows scheduled tasks (task scheduler in Win 7) to run it on a nightly/weekly basis.

The "pause" in red I normally REM out, but is nice when developing so it doesn't just close the window down if it encounters an error.

R_

Instead of the windows scheduler, I have used pyCRON in the past as well. Works like CRON on 'IX systems.

Re: Displaying blobs

AlexSanders — Mon, 13 May 2013 18:30:34 GMT

Rhett,

Is there any way you can share your python work flow for extracting the BLOB fields from a database to disk?

Alex

Dustin,

I have a similar workflow, with photos stored in BLOB fields and one to potentially many photos per feature. The attachemnts feature doesn't really support 1:many so I have a python script that extracts the jpeg photos from the blob and saves them in a folder. As it is writing the blobs, I capture the path and filename that I am saving them to (as well as my feature ID) and append to a list, then at the end of my script, I insertCursor the list to populate a FGDB table. Now I not only have the jpeg stored local in a folder, I also have a ready-made table with all my paths ready to join/calculate to the feature class data that can be used for my link field.

Of course, I had to make a virtual directory on the server pointing to the folder I saved all my jpegs in, then had to go into IIS and, for that virtual folder, allow directory browsing as I often have many photos per site (if just one photo, my "path" actually points to the jpeg file, if more than one, it points to the folder.

This was kind of cluttery, especially for the sites with multiple photos, as IE doesn't give thumbnail views, so, all you see is a list of the somewhat cryptic filenames, and need to click on each photo to view it.

So, I took it one step farther. After my python script extracts and saves the jpegs from the BOLBs, writes the "lookup table", joins/calculates the URL info to my feature class, it then runs a slightly modified version of pyGallery. This creates a gallery for my photos similar to ( http://pygallery.sourceforge.net/samples/ ). So now, if there is one photo, my popup says "Click for Photo", and it takes you direclty to the image viewed in a browser. If there are more than one image, it says "Click for Photos", and if you click, it will take you to the index/thumbnail gallery page for that feature, with all the photos.

So far, not only the best way I've came up with for BLOBs, but also for one:many photos in Flex.

In case this gives you some ideas,

R_

Re: Displaying blobs

RhettZufelt — Mon, 13 May 2013 20:33:29 GMT

Stuart/Alex,

Here are the nuts and bolts of it:

This is using the python that comes with 10.1 (v2.7.2).

import arcpy, os from arcpy import da  inTable = "C:\\Connection to RCES_P.sde\\SIS_ADM.IMAGES"  #  table with identifier (SITE_CODE), imageID number (IMAGE_ID), and photo in BLOB field (PHOTO)  savepath = "\\\\mcflight01\\WebImages"            #  initial path to save images/sub folders   with da.SearchCursor(inTable,['SITE_CODE','IMAGE_ID','PHOTO']) as cursor:    for row in cursor:          SITE_CODE = str(row[0])   # gets feature identifier from table  (SITE_CODE) as string          Fid = str(int(row[1]))    # gets ImageID from table (IMAGE_ID) as string without decimal point          binaryRep = row[2]        #  get photo from BLOB field as binaryRep variable          filename = (SITE_CODE.replace(':','_')).replace('/','')  + "_" + Fid + ".jpg"  # appends site_code to filename replacing : and /'s          folder = savepath + os.sep + SITE_CODE.replace(':','_')                        # appends site_code to folder replacing : with _            if binaryRep:              if not os.path.exists(folder):                 os.system("mkdir " + '"' + folder + '"')                           # if doesn't already exist, make folder using folder variable              open(folder + os.sep  + filename, 'wb').write(binaryRep.tobytes())    # this actually saves the BLOB stored in binaryRep to filename variable           del binaryRep del row

Re: Displaying blobs

DustinWiersma2 — Mon, 13 May 2013 20:55:08 GMT

Thanks for showing us how you coded this!

Re: Displaying blobs

RhettZufelt — Mon, 13 May 2013 22:48:30 GMT

You bet. Here is where I got the initial information. Also shows how to write to BLOB field as well.

http://anothergisblog.blogspot.in/2012/06/working-with-blob-data-at-101-arcpyda.html

R_

Re: Displaying blobs

MichaelVolz — Tue, 14 May 2013 11:52:37 GMT

Rhett:

Does your script convert BLOB field data for individual records or does it convert BLOB field data for an entire feature class (or however the entire data set is stored)?

This seems like a duplication of file storage as the data is basically being stored in 2 different formats. Is there no tool in FlexViewer that can currently read directly from a BLOB field and display an item such as a jpeg image that is stored in the BLOB format. That seems like a more efficient system.

Re: Displaying blobs

BjornSvensson — Tue, 14 May 2013 12:13:36 GMT

Is there no tool in FlexViewer that can currently read directly from a BLOB field and display an item such as a jpeg image that is stored in the BLOB format.

Like I mentioned before the ArcGIS Server REST API does not support BLOBs, and thus the Flex API cannot get to BLOBs.
"the field values ... can be any of the standard JSON types - string, number and boolean."
http://resources.arcgis.com/en/help/rest/apiref/index.html?feature.html

The preferred solution is to use attachments:
http://resources.arcgis.com/en/help/main/10.1/index.html#//01m90000000r000000

Re: Displaying blobs

MichaelVolz — Tue, 14 May 2013 12:29:36 GMT

Bjorn:

Just to clarify, since the ArcGIS Server REST API does not support BLOBs, that would mean that any of the APIs would also not support BLOBs (e.g. Javascript API, Silverlight API and Flex API)?

Is there any chance that ESRI is looking to enhance these APIs with the ability to directly read/write to BLOB fields? There are many 3rd party GIS software vendors that store data in BLOB fields and it would be very useful to be able to get directly to the data (plus it would reduce duplicated file storage).

Re: Displaying blobs

BjornSvensson — Tue, 14 May 2013 12:37:03 GMT

Just to clarify, since the ArcGIS Server REST API does not support BLOBs, that would mean that any of the APIs would also not support BLOBs (e.g. Javascript API, Silverlight API and Flex API)?

Correct.

And since version 10.0, the best approach for relating images, pdf, word docs, pdf and pther files with a feature is to use "attachments".

Re: Displaying blobs

RobertScheitlin__GISP — Tue, 14 May 2013 12:38:26 GMT

Michael,

You should submit this enhancement request on the ArcGIS Ideas portal then and try to gain support from other to vote it up.

http://ideas.arcgis.com/

Re: Displaying blobs

MichaelVolz — Tue, 14 May 2013 12:55:48 GMT

Thanks Robert. That is what I was planning to do, I just wanted to make sure I was not requesting an enhancement for a widget (tool) that was already available.

Re: Displaying blobs

RhettZufelt — Tue, 14 May 2013 14:06:27 GMT

Rhett:

Does your script convert BLOB field data for individual records or does it convert BLOB field data for an entire feature class (or however the entire data set is stored)?

This seems like a duplication of file storage as the data is basically being stored in 2 different formats. Is there no tool in FlexViewer that can currently read directly from a BLOB field and display an item such as a jpeg image that is stored in the BLOB format. That seems like a more efficient system.

Michael,

My script works on a table (does not have to be a feature class) and "extracts" whatever data is in the BLOB field for that record and saves it to a file. Since I already know that all "objects" stored in my blob fields are jpegs, I name it with a .jpg on the end (could easily be a text file, or whatever in a BLOB, so you need to know what you are extracting). The parts of my script I cut out actually look at what images have "changed", and re-extracts only them. Can easilly add whatever logic you need to only extract what you want.

You are correct, it IS a duplication of data, but, is the best solution I have come up with and really doesn't put much of a dent in my 12 TB. As far as server supporting BLOBs, the only BLOBs I know that are supported are attachments as they are stored as BLOBs in a FGDB. Hopefully in the future, ESRI can extend that to non-ESRI BLOB fields as well.

If you have a simple 1:1 relationship, the attachments is probably the way to go, not to mention they automatically integrate with Flex popups. Still a duplication of files, but they are tied to your data.

You could use something similar to my script to extract the images, then use the attachments script to load them directly into the FGDB as attachments without actually saving them locally. However, the way attachments are currently, if you have a 1:many relationship with multiple features in a class, ALL attachments will be "linked" to the first feature in that class, and none of the others will have any attachments at all. I have many photos, and potentially many features per class, so this is the route I chose.

There are several posts on the ideas page suggesting to improve the attachments tool to support a 1:many as well as to allow linking to a folder somewhere for attachemnts (this is a common one, as many people have their photos/etc. already saved somewhere, and don't want to have to copy them into the FGDB) You may want to promote some of these. I voted your latest one up as well.

R_

Re: Displaying blobs

AlexSanders — Tue, 14 May 2013 15:45:30 GMT

Rhett,

That script worked perfectly that is exactly what we needed! Thank you so much for your help.

Alex Sanders