topic Re: Custom data feeds samples and documentation in ArcGIS Enterprise Extensibility Questions

Custom data feeds samples and documentation

MarcBateEA — Sat, 29 Apr 2023 16:06:33 GMT

I downloaded the Enterprise SDK v11.1 and only found the command line utility for creating the custom data feeds nodejs project, however, the documentation seems to be from an older version maybe even 10.9.1, and no documentation on custom data feeds, and no samples.

Where are the samples? I don't see them on github either, and the online documentation for the SDK doesn't have anything on custom data feeds either.

Re: Custom data feeds samples and documentation

SimonSchütte_ct — Wed, 03 May 2023 07:28:51 GMT

Check out New in ArcGIS Enterprise 11.1: Custom Data Feeds (esri.com), Extending ArcGIS Enterprise with Custom Data Feeds - Esri Videos: GIS, Events, ArcGIS Products & Industries, Dev Summit 2023: Custom Data Feeds (esri.com)and Esri Videos: GIS, Events, ArcGIS Products & Industries

Re: Custom data feeds samples and documentation

MarcBateEA — Wed, 03 May 2023 14:03:27 GMT

I've seen all that. Didn't answer my question. Those don't go deep enough into creating the provider. The session from DevSummit was closest I could find.

Re: Custom data feeds samples and documentation

SimonSchütte_ct — Wed, 03 May 2023 15:10:58 GMT

I have something, but need to check back for sharing permissions

Re: Custom data feeds samples and documentation

MarcBateEA — Wed, 03 May 2023 15:33:27 GMT

I've been able to get my provider working with the app built using the cdf command line tool. I'm hoping that when deployed to an actual ArcGIS Server 11.1 it will have the same capabilities as any other feature service since this NodeJS app obviously isn't doing all of that. I have run into some issues when adding to a web map or ArcGIS Pro, e.g. exporting to file gdb.

Re: Custom data feeds samples and documentation

SimonSchütte_ct — Thu, 04 May 2023 06:40:13 GMT

Ok, I got the permission to share the link 🙂
Here you can find the documentation from the Beta Program for Custom Data Feeds, including some samples: https://cdf.webgistesting.net/customdatahelp/

Re: Custom data feeds samples and documentation

MarcBateEA — Thu, 04 May 2023 06:44:51 GMT

That's the kind of thing I was looking for. Thank you!

I assume it will be on the Enterprise SDK page soon. Some things I figured out using the Koop documentation and samples that would be nice to include in the Esri help are:

Use the metadata property to:

1) assign field aliases

2) make feature service time-enabled

Also the caching is very important for our provider which doesn't update frequently but can take several minutes to return its results. Once it is cached, it is super fast returning to the client even on my dev setup.

Re: Custom data feeds samples and documentation

MarcBateEA — Fri, 05 May 2023 17:23:03 GMT

I saw you can override the key the cache uses which is helpful because we don't have a ::layer on our service, i.e. it will always be 0 and was causing the plugin source to be called again sometimes.

Model.prototype.createKey = function(req) {
let key = req.url.split('/')[1]
if (req.params.host) key = [key, req.params.host].join('::')
if (req.params.id) key = [key, req.params.id].join('::')
return key
}

How can I also have my provider optionally return a table instead of a feature service that can be added to Field Maps for example as a lookup table for a drop down, or am I pushing it here?

Re: Custom data feeds samples and documentation

MarcBateEA — Sun, 07 May 2023 23:00:56 GMT

I found that if you just exclude the geography from the geojson it returns a table.

it seems like you would create a relationship class to do what I was asking in field maps between a feature class and the table created by cdf. However at least in my local host environment the table can’t be used in the go tool. It just has an x by it nothing else

Re: Custom data feeds samples and documentation

EliasRex — Mon, 22 May 2023 18:40:53 GMT

Were you able to register the custom data provider with the 11.1 ArcGIS Server? I keep receiving an error when trying to register the cdpk: "Node server failed to start"

Re: Custom data feeds samples and documentation

MarcBateEA — Mon, 22 May 2023 19:56:47 GMT

Will be upgrading to 11.1 in the next month and I'll have my provider ready to go, and hopefully you can figure out a solutin and share.

Re: Custom data feeds samples and documentation

AndrewAdamson22 — Tue, 23 May 2023 21:39:04 GMT

Running into the same. Were you able to register?

Re: Custom data feeds samples and documentation

EliasRex — Tue, 23 May 2023 21:42:12 GMT

I was. I made the mistake of installing node packages at the top level of the project. Once I installed the packages at the provider level, I managed to register it with the server.

Re: Custom data feeds samples and documentation

nicogis — Wed, 14 Jun 2023 09:01:47 GMT

Here there is the documentation & samples: - Overview ArcGIS Server Custom Data Feeds

Re: Custom data feeds samples and documentation

EliasRex — Wed, 14 Jun 2023 16:21:57 GMT

Have you found any information or samples that include caching?

Re: Custom data feeds samples and documentation

nicogis — Wed, 14 Jun 2023 16:42:15 GMT

I have seen in create service property set 'true' but I don't know if it is managed this property with CDF

"properties": { "disableCaching": "true" },

Re: Custom data feeds samples and documentation

MarcBateEA — Wed, 14 Jun 2023 16:54:02 GMT

I had to figure it out myself. I allowed a url parameter to be passed called expiresDays so I can use different caching for different projects. I also have a default expiresDays in my config file

let expiresDays = req.query.expiresDays || config.expiresDays
let expires = null
if (expiresDays && expiresDays !== 0) {
    expires = new Date().setDate(new Date().getDate() + expiresDays)
}
// set the fields array where I used aliases, and the array of features, then set ttl
const geoJson = {
    metadata: {
         idField: "OBJECTID",
         fields: fields,
         displayField: Object.keys(allFeatures[0].properties)[0] 
    },
    type: "FeatureCollection",
    features: allFeatures,
    ttl: expires
}

By default it was caching the FeatureServer and the FeatureServer/0 as different things so it was calling my API twice. Since there's only one "layer" in the CDF I removed the part of the key with the layer in it. I was able to override the key it uses for caching in the model.js file to fix that:

Model.prototype.createKey = function (req) {
    let key = req.url.split("/")[1]
    if (req.params.host) key = [key, req.params.host].join("::")
    if (req.params.id) key = [key, req.params.id].join("::")
    //if (req.params.layer) key = [key, req.params.layer].join('::')

    return key
}

Re: Custom data feeds samples and documentation

EliasRex — Wed, 14 Jun 2023 18:48:24 GMT

How do you get access to the key once another request is made?

Re: Custom data feeds samples and documentation

MarcBateEA — Wed, 14 Jun 2023 20:02:49 GMT

You don't have to code anything related to that. It is handled by that nodejs app that the cli tool creates, not your provider

Re: Custom data feeds samples and documentation

EliasRex — Wed, 14 Jun 2023 20:53:38 GMT

Does your getData function re-execute once your ttl time has expired? I have added in the createKey function, but even after the ttl period, I can't get the getData function to execute once again. Not exactly sure what I am missing.