Creating an FME Web Connection for your Enterprise Portal

[BruceHarold]

A powerful feature of ArcGIS Data Interoperability and 'cousin' FME is the ability to save and share connections to web apps.  Once configured, you can use a web connection to read and write data in any number of workspaces while maintaining secure credentials in only one place.

Portal for ArcGIS is a component of ArcGIS Enterprise I think of as a content management system.  You can start reading about it here.  A portal is a highly capable, single-tenant, secure geospatial infrastructure component where you can create, maintain and share data, maps, scenes and apps.  This blog is about creating and using a portal app to access hosted feature services to be read and written with the ARCGISPORTALFEATURES reader/writer or the ArcGISOnlineConnector transformer.

The starting point is your portal, here is my portal's home page (fake, but you'll get the idea):

https://nonexistent.esri.com/portal/home/index.html 

The first thing to do is create an app to hang the web connection off.

Go to your Content view and click to Add Item:

Choose 'An application' and click Application and fill in the descriptive stuff:

The app will be created and you'll be taken to its home URL, which will look something like this:

https://nonexistent.esri.com/portal/home/item.html?id=50bfc12ef28840d48eac324c8ec67dd0 

In the top right is the Settings view, click on it.

Scroll down (or click on Application beside General top left) and you'll see App Registration:

Click on Registered Info to see details you'll need to create your web connection:

Click on Show Secret to expose the 32-character hex authentication key.  Now you have everything you need to create your portal web app connection for Workbench.  From the Pro Analysis ribbon (or by editing any ETL tool) open the Workbench application and go to Tools>FME Options>Web Connections.  Mine look like this (login obscured):

Click on Manage Services bottom right and in the Manage Web Services dialog use the pulldown bottom left to Create From and pull right and choose Esri ArcGIS Portal (Template).

Now fill in the dialog:

Test and Authenticate, then close the dialog, and you can add a web connection:

...and you are in business!

Restart Workbench and use the new web connection to add a portal feature service reader (login obscured):

Now enjoy your portal features ETL!

Esri ArcGIS Portal (Template).xml.zip

[AsafEvenPaz]

Hi,

I am trying to follow this workflow but cannot find the "Esri ArcGIS Portal (Template)" option. Do you have this in a file which I can import from?

[BruceHarold]

I added an attachment to the post but please also open a support call about the template not being available in your install.  Thank you.

[AsafEvenPaz]

Thanks Bruce.

I added the xml but it has a "password" section there and when I leave it empty I get an error message "

Unable to read file 'C:\Users\asafe\Documents\Metrics\portal.xml'. Most likely cause is an incorrect password.

Encryption Error: AES Decrypt finalization failed."

Thoughts?

[BruceHarold]

Sorry about that, I replaced the attachment with what I think is the right file for you to copy into:

C:\Program Files\ArcGIS\Data Interoperability for ArcGIS Pro\WebServices

[AsafEvenPaz]

Bruce,

Nearly there - I can now see this option and select it but can't find an option to change the authentication information (app_id and password).. 

[BruceHarold]

Are you clicking the Manage Services button and then using the Create From option from the + sign bottom left?

[LoganSuhr_PSE]

@BruceHarold ,

I have successfully connected to my Portal and found the feature service I want to use.  But when I go to generate a list of layers or find branched versions, the wheels just spin.  It never seems to retrieve what I need.  I've tried this on many different feature services.  You help is much appreciated!

[BruceHarold]

Hi, usually this sort of issue is fixed by re-authenticating the web connection, you can do this by going to the Web Connections pane in FME Options and right clicking on the connection, a menu option will be available.

[Jade_Freeman]

I've followed these instructions but either when testing my new web service connection or trying to create a new web connection from my new web service connection, after clicking the Authenticate button, and logging in (via SAML), I get this screen with no way to go any further or save the new web connection.

Any ideas?

[kavi88]

Hello @BruceHarold ,

Thank you so much for this article.

We have several FME Server services that runs nightly to update feature hosted services. This all works well but every month or two the services all fail because they have lost their authentication to AGOL even though there were no account changes. I can re-authenticate the connection and then the services run for a month  and then they lose their authentication again. Is there a way to set up a permanent authentication so that they do not have to be periodically re-authenticated?

Thank you.

[BruceHarold]

Hello Kavi, officially you should contact Safe Software support for an FME Server issue but to be helpful we'll connect with Safe at this end to put together a response.  Please stand by the next few days.

[RahulSharma_SafeSoft]

Hello Kavi, 
The behavior you have described is expected. 
OAuth refresh token does have an expiration time that can be set at the Enterprise level; see the documentation here.

Pleae set  the maximum value set (-1) for the refresh token as shown below.  

To avoid re-authenticating the web connection, I would advise you the following. 

Create a reader with the Esri ArcGIS Portal Feature Service Reader 
If required, restrict the reader to read 1 or 10 features (Maximum Features to Read under Parameters) 
Add a logger and save the workspace 
Publish the workspace  to FME Flow (formerly FME Server) 
Schedule this workspace to run every two weeks(using Automation). This will trigger the web service and generate a refresh token before it expires. 

Rahul Sharma 

[kavi88]

Hello RahulSharma_SfeSoft,

Thank you for your reply, do you think the same procedure works for ArcGIS Enterprise please?

cheers,

Kavi

[AllanLambert]

I followed this set up to access a geocoding service in Portal and it worked perfectly for a day.  I found the way to extend the Auth to 2 weeks / 20,000 minutes.  But I still have to manually Reauthenticate the Web Connection every 2 weeks.  It was great for years when I could specify a service and store a username and pw in the Geocoder. Any ideas or work arounds?  Can I automate the Reauth?

I have to stay on 3.3 because of an integration.  Thank you!!

Pro 3.3 and FME(R) 2023.1.1.1 (20231011 - Build 23636 - WIN64)

[BruceHarold]

Hi, unless I'm missing something, if you change your Authentication Type to Generate Token then you'll embed your username and password into the Geocoder and a new token will be generated on each run.

[AllanLambert]

Code 401.  You are not Authorized to access (geocoder)...Invalid Creds.  I've tried my personal creds and the portal admin.  I've tried all sharing levels for the Portal access app and the geocoder.  ESRI Support blames it on 10.7.1.  I'm upgrading in April or May.  I survive til then.  thanks for answering.

[BruceHarold]

Ah I see, sounds like a portal bug indeed.  You could always shell out to ArcPy in the workspace, assuming the Geocode Addresses geoprocessing tool likes your portal locator, if not then use a file-based locator.