Generate Token for FeatureReader Transformer but all endpoints are read

[DMBowman]

We are moving to short-lived token authentication for ArcGIS Feature Service access, so the workbench was updated to self-generate the token.  The FeatureServer reader does not support passing the self-generated token.  I was able to add a  FeatureReader transformer, but it reads all the endpoints for the REST Service, which is 132 in total.  I only want the transformer to retrieve a subset of endpoints. 

I tried creating individual FeatureReader transformers for the specific endpoints; however, the Initiator port is being called and data are not passing through to the next transformer.  When the FeatureReader transformer calls the entire REST Service, everything passes through the Generic port, although this amounts to millions of source features being read.  Because features are getting stuck at the Initiator port, I added a GeometryExtractor transformer thinking that was needed, but it did not work.

I then tried using the HTTPCaller and GeometryExtractor, but the feature is rejected when I move through the ETL to generate a buffer for the individual layers; these are eventually meant to be written to a fgdbms.

I cross posted on the FME Community, but have not received a response.

[RenatoSalvaleon3]

A couple of questions first: How do you generate your token or what is your authentication type? Are you using AGOL or Enterprise (Portal)?

There's a few options to generate or retrieve that short lived token.

If you want it self generated the most efficient way is to use the FME Hub transformers or the HTTPCaller retrieve a token that is part of your workflow. Follow the next steps. Enable faeture caching first.

1. Add the token generator portion of your workspace. Creator + token generator transformer (like below).

2. Run the workflow to get a token on your translation log. Copy the tken frmo the Translation Log table

3. Add the FeatureReader to your canvass.

4. Configure the transformer for an ArcGIS Server Feature Service reader, like below.

3. On the Constraints Tab --> Feature Typs to Read parameter, click on the ellipsis and you will be prompted with an Unresolved Parameters dialog to paste the token copied from Step 1 above.

Click OK to open the Select Feature Layer to add options.  

Pick one or more feature type/s, and click Ok and you would be prompted again to enter the token created from Step 1 above. 

Once the token is added and Ok is clicked, the feature type/s selected will be added to the FeatureReader transformer with new output port/s. In this case it would be Cities

From this point onwards, as long as the feature type/s don't change, the workspace will be ready for your data updates or other automated workflows.

4. The whole workspace looks like below with a logger on the end. 

The ArcGISPortalTokenGenerator built by Sasha Lockamy can be downloaded from https://hub.safe.com/publishers/sashal/transformers/arcgisportaltokengenerator

You could also build your own portal fenerator using the HTTPCaller transformer and following this GenerateToken api rest endpoint documentation. https://developers.arcgis.com/rest/users-groups-and-items/generate-token/

Please let me know if you have any additional questions. My apologies for the delay.

[DMBowman]

I appreciate the help and I too apologize for a delay in responding to your reply! 

I'm generating a short-lived token (see figure 1), but the Feature Reader is requesting input of a token in the Constraints/Feature Types section in order to generate the subset of endpoints.  An error is thrown with the same token parameter.  Without the token, all endpoints are read. 

I'll download the generator built by Sasha Lockamy to see if that makes a difference.  For the moment, I haven't hit a total roadblock since I am able to use our Enterprise Geodatabase as the data instead of the REST Service.  Although I am concerned the current method may introduce latency once we move from an on-prem geodatabase to the cloud.