Image Server 11: Deep Learning Studio - Inferencing

[SimonSchütte_ct]

I set up ArcGIS Image Server  in my Enterprise Environment and installed the Deep Learning Framework.
Everything works fine, except the inferencing. When loading a model in the inferencing app, the Start button is supposed to turn active, but it does not:

I don't have a dedicated GPU, but I set the mode to CPU in the settings of the widget above.

Do I miss something? I tried to use the Esri sample deep learning packages and restarted the servers, but no success.
Is the Math Kernel Library required for inferencing in Portal? Configure ArcGIS Image Server for deep learning raster analytics—ArcGIS Image Server | Documentation...

https://github.com/Esri/deep-learning-frameworks/blob/master/README.md
states the framework will fall back to the cpu mode when there is no gpu, but also recommends to install the Math Kernel Library, but without more information on how to do it and if it is necessary for use with Portal.



@VinayViswambharan @AkshayaSuresh 

[VinayViswambharan]

Hi Simon -

It sounds like a bug that we already fixed (that didnt make its way into the 11 release). If its the same bug, after selecting the model, re-select the imagery layer from the dropdown list should enable the start button. connect with us if you still run into the issue.

[SimonSchütte_ct]

Hi Vinay,

Thank you for providing the information for the workaround, it works.

With my own object detection model I now get the following Error Message after clicking on Start:

• Publishing Privilege Check: OK
• AddExceptionError
• OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A02366D0> returned NULL without setting an error
• {"messageCode": "RA_0", "message": "DetectObjectsUsingDeepLearning failed. OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A02366D0> returned NULL without setting an error."}
• Failed script DetectObjectsUsingDeepLearning...
• Failed to execute (DetectObjectsUsingDeepLearning).

And with the Esri Car detection model I had the following error one time in addition to the one above:

• Raster Analytics helper service: https://machine.domain.de/arcgis
• Running on ArcGIS Image Server.
• Publishing Privilege Check: OK
• AddExceptionError
• OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A0244030> returned NULL without setting an error
• {"messageCode": "RA_0", "message": "DetectObjectsUsingDeepLearning failed. OutputCatalogPath failed. Error: <built-in method GetOutputCatalogPath of HostedGP object object at 0x00000209A0244030> returned NULL without setting an error."}
• C:\Program Files\ArcGIS\Server\framework\runtime\ArcGIS\bin\Python\envs\arcgispro-py3\Lib\site-packages\urllib3\connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn(
• Failed script DetectObjectsUsingDeepLearning...
• Failed to execute (DetectObjectsUsingDeepLearning).

[Anonymous User]

The GP service error message seems to point to the output folder when you run the inference. Assuming you have an imagery layer that you want to detect objects from, it will create hosted feature layer that contains these detected objects. For troubleshooting purpose, I would suggest a few steps:

1. Current user is a publisher

2. Check if the current user can create a hosted feature layer on Content page -> Add Item -> Hosted feature layer

3. Check that the image server with the RA role has a fileShare based raster store and there is no other type of raster stores (i.e. cloud raster store or database raster store)

4. Before you click the Start button, click the gear icon to bring up the environment settings. If you have dedicated NVIDIA GPU available on your RA server, you can use either GPU or CPU. Otherwise try switching to use CPU.

One additional palce to check is to make sure you have Deep Learning Libraries installed correctly. If you have been able to use Deep Learning Studio to browse to a model (DLPK) from your portal, it's a sign that DL libraries are installed.

[JesseCloutier]

The above post was by @JayChen on 8/8/2022.

[Anonymous User]

The error message itself seems to be similar to a bug that's already fixed from server side at 10.8. That bug had similar error message about GetOutputCatalogPath of HostedGP object getting NULL value, and it was caused by an RA site with multiple nodes and some nodes were joined later than others and these newly joined nodes are not updated to data store's trusted server list. Is your RA a single node site or a multiple node site? @SimonSchütte_ct 

[JesseCloutier]

The above post was by @JayChen on 8/8/2022.

[SimonSchütte_ct]

@Anonymous User Thanks a lot for your help. 

- I have a two machine setup: A) ArcGIS Enterprise Base Deployment B) Image Server Advanced with a Local folder Raster Store and installed Deep Learning Libraries. No Additional nodes. It validates as Raster Analysis Server in Portal.

- The current user is the admin

- Hosted feature Layers can be created and are hosted on machine A)
- Image Layers can be created and are hosted on machine B)

- The Image Server has one Raster Store registered only

- Inferencing is run in CPU mode

- I am able to train a model using Deep Learning Studio, so the DL Libraries should be installed correctly

- The Output is the default path: C:\arcgisserver\directories\arcgisoutput

I found this: Solved: Error when running Classify Objects Using Deep Lea... - Esri Community I´ll give it a try.

I opened a support ticket and I will report back when I find the issue. If you have any idea on what the issue could be, your advice would be appreciated.

[JayChen]

Simon - I have no other suggestions at this point of time unless we can reproduce it in house. Let us know if the update of the pb.hba file solved your problem. That's the same fix for the multiple node server joining issue.

[SimonSchütte_ct]

Hi Jay, thank you, I´ll give it a try. If it does not work, I will setup the Image Server again from scratch.