topic Re: Image Server 11: Deep Learning Studio - Inferencing in Deep Learning Studio Questions

Image Server 11: Deep Learning Studio - Inferencing

SimonSchütte_ct — Fri, 05 Aug 2022 12:59:09 GMT

I set up ArcGIS Image Server in my Enterprise Environment and installed the Deep Learning Framework.
Everything works fine, except the inferencing. When loading a model in the inferencing app, the Start button is supposed to turn active, but it does not:

I don't have a dedicated GPU, but I set the mode to CPU in the settings of the widget above.

Do I miss something? I tried to use the Esri sample deep learning packages and restarted the servers, but no success.
Is the Math Kernel Library required for inferencing in Portal? Configure ArcGIS Image Server for deep learning raster analytics—ArcGIS Image Server | Documentation for ArcGIS Enterprise

https://github.com/Esri/deep-learning-frameworks/blob/master/README.md
states the framework will fall back to the cpu mode when there is no gpu, but also recommends to install the Math Kernel Library, but without more information on how to do it and if it is necessary for use with Portal.

@VinayViswambharan @AkshayaSuresh

Re: Image Server 11: Deep Learning Studio - Inferencing

VinayViswambharan — Fri, 05 Aug 2022 15:23:23 GMT

Hi Simon -

It sounds like a bug that we already fixed (that didnt make its way into the 11 release). If its the same bug, after selecting the model, re-select the imagery layer from the dropdown list should enable the start button. connect with us if you still run into the issue.

Re: Image Server 11: Deep Learning Studio - Inferencing

SimonSchütte_ct — Mon, 08 Aug 2022 10:55:22 GMT

Hi Vinay,

Thank you for providing the information for the workaround, it works.

With my own object detection model I now get the following Error Message after clicking on Start:

Publishing Privilege Check: OK
AddExceptionError
OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A02366D0> returned NULL without setting an error
{"messageCode": "RA_0", "message": "DetectObjectsUsingDeepLearning failed. OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A02366D0> returned NULL without setting an error."}
Failed script DetectObjectsUsingDeepLearning...
Failed to execute (DetectObjectsUsingDeepLearning).

And with the Esri Car detection model I had the following error one time in addition to the one above:

Raster Analytics helper service: https://machine.domain.de/arcgis
Running on ArcGIS Image Server.
Publishing Privilege Check: OK
AddExceptionError
OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A0244030> returned NULL without setting an error
{"messageCode": "RA_0", "message": "DetectObjectsUsingDeepLearning failed. OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A0244030> returned NULL without setting an error."}
C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\envs\arcgispro-py3\Lib\site-packages\urllib3\connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn(
Failed script DetectObjectsUsingDeepLearning...
Failed to execute (DetectObjectsUsingDeepLearning).

Re: Image Server 11: Deep Learning Studio - Inferencing

Anonymous User — Mon, 08 Aug 2022 18:11:00 GMT

The GP service error message seems to point to the output folder when you run the inference. Assuming you have an imagery layer that you want to detect objects from, it will create hosted feature layer that contains these detected objects. For troubleshooting purpose, I would suggest a few steps:

1. Current user is a publisher

2. Check if the current user can create a hosted feature layer on Content page -> Add Item -> Hosted feature layer

3. Check that the image server with the RA role has a fileShare based raster store and there is no other type of raster stores (i.e. cloud raster store or database raster store)

4. Before you click the Start button, click the gear icon to bring up the environment settings. If you have dedicated NVIDIA GPU available on your RA server, you can use either GPU or CPU. Otherwise try switching to use CPU.

One additional palce to check is to make sure you have Deep Learning Libraries installed correctly. If you have been able to use Deep Learning Studio to browse to a model (DLPK) from your portal, it's a sign that DL libraries are installed.

Re: Image Server 11: Deep Learning Studio - Inferencing

Anonymous User — Mon, 08 Aug 2022 20:50:48 GMT

The error message itself seems to be similar to a bug that's already fixed from server side at 10.8. That bug had similar error message about GetOutputCatalogPath of HostedGP object getting NULL value, and it was caused by an RA site with multiple nodes and some nodes were joined later than others and these newly joined nodes are not updated to data store's trusted server list. Is your RA a single node site or a multiple node site? @SimonSchütte_ct

Re: Image Server 11: Deep Learning Studio - Inferencing

SimonSchütte_ct — Tue, 09 Aug 2022 07:30:36 GMT

@Anonymous User Thanks a lot for your help.

- I have a two machine setup: A) ArcGIS Enterprise Base Deployment B) Image Server Advanced with a Local folder Raster Store and installed Deep Learning Libraries. No Additional nodes. It validates as Raster Analysis Server in Portal.

- The current user is the admin

- Hosted feature Layers can be created and are hosted on machine A)
- Image Layers can be created and are hosted on machine B)

- The Image Server has one Raster Store registered only

- Inferencing is run in CPU mode

- I am able to train a model using Deep Learning Studio, so the DL Libraries should be installed correctly

- The Output is the default path: C:\arcgisserver\directories\arcgisoutput

I found this: Solved: Error when running Classify Objects Using Deep Lea... - Esri Community I´ll give it a try.

I opened a support ticket and I will report back when I find the issue. If you have any idea on what the issue could be, your advice would be appreciated.

Re: Image Server 11: Deep Learning Studio - Inferencing

JesseCloutier — Tue, 09 Aug 2022 18:41:27 GMT

The above post was by @JayChen on 8/8/2022.

Re: Image Server 11: Deep Learning Studio - Inferencing

JesseCloutier — Tue, 09 Aug 2022 18:42:09 GMT

The above post was by @JayChen on 8/8/2022.

Re: Image Server 11: Deep Learning Studio - Inferencing

JayChen — Thu, 11 Aug 2022 19:27:57 GMT

Simon - I have no other suggestions at this point of time unless we can reproduce it in house. Let us know if the update of the pb.hba file solved your problem. That's the same fix for the multiple node server joining issue.

Re: Image Server 11: Deep Learning Studio - Inferencing

SimonSchütte_ct — Fri, 12 Aug 2022 08:04:27 GMT

Hi Jay, thank you, I´ll give it a try. If it does not work, I will setup the Image Server again from scratch.

Re: Image Server 11: Deep Learning Studio - Inferencing

BoudewijnBoogaard3 — Sun, 11 Sep 2022 02:48:26 GMT

Hi Simon - Did you find a solution for the output Catalog path error? I'm getting the same error when running the inferencing.

Re: Image Server 11: Deep Learning Studio - Inferencing

SimonSchütte_ct — Wed, 12 Oct 2022 13:48:32 GMT

The issue with the Button is a known bug and can be resolved by choosing another Input imagery source (and revert back to the original). This will activate the button.

If someone runs into issues, setting up Deep Learning Studio, here are the steps:
- install DLS runtime from Github on the Image Server Machine (for CPU-Only this alone will do)
- add File RasterStore to Image Server Data Stores (Server Manager)
- Activate "Raster Analysis Server" in addition to "Image Hosting Server" Role in Portal
(If you do not activate it, you will get a message, that DLS Project could not be created)
- Inferencing Limitation: to use an existing DLPK in the inferencing interface, you need to run a least one training, then open the Inferencing App to change the model. At the moment there seems to be no way to just open the inferencing app. I assume this will be added in 11.1. However, models can be used in the old Map Viewer.
- note the Inferencing Bug mentioned above