nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Custom data feeds samples and documentation

MarcBateEA

I downloaded the Enterprise SDK v11.1 and only found the command line utility for creating the custom data feeds nodejs project, however, the documentation seems to be from an older version maybe even 10.9.1, and no documentation on custom data feeds, and no samples.

Where are the samples? I don't see them on github either, and the online documentation for the SDK doesn't have anything on custom data feeds either.

Find more posts tagged with

Accepted answers

All comments

s.schuette_ct

Check out New in ArcGIS Enterprise 11.1: Custom Data Feeds (esri.com), Extending ArcGIS Enterprise with Custom Data Feeds - Esri Videos: GIS, Events, ArcGIS Products & Industries, Dev Summit 2023: Custom Data Feeds (esri.com)and Esri Videos: GIS, Events, ArcGIS Products & Industries

MarcBateEA

I've seen all that. Didn't answer my question. Those don't go deep enough into creating the provider. The session from DevSummit was closest I could find.

s.schuette_ct

I have something, but need to check back for sharing permissions

MarcBateEA

I've been able to get my provider working with the app built using the cdf command line tool. I'm hoping that when deployed to an actual ArcGIS Server 11.1 it will have the same capabilities as any other feature service since this NodeJS app obviously isn't doing all of that. I have run into some issues when adding to a web map or ArcGIS Pro, e.g. exporting to file gdb.

s.schuette_ct

Ok, I got the permission to share the link 🙂
Here you can find the documentation from the Beta Program for Custom Data Feeds, including some samples: https://cdf.webgistesting.net/customdatahelp/

MarcBateEA

That's the kind of thing I was looking for. Thank you!

I assume it will be on the Enterprise SDK page soon. Some things I figured out using the Koop documentation and samples that would be nice to include in the Esri help are:

Use the metadata property to:

1) assign field aliases

2) make feature service time-enabled

Also the caching is very important for our provider which doesn't update frequently but can take several minutes to return its results. Once it is cached, it is super fast returning to the client even on my dev setup.

MarcBateEA

I saw you can override the key the cache uses which is helpful because we don't have a ::layer on our service, i.e. it will always be 0 and was causing the plugin source to be called again sometimes.

Model.prototype.createKey = function(req) {
let key = req.url.split('/')[1]
if (req.params.host) key = [key, req.params.host].join('::')
if (req.params.id) key = [key, req.params.id].join('::')
return key
}

How can I also have my provider optionally return a table instead of a feature service that can be added to Field Maps for example as a lookup table for a drop down, or am I pushing it here?

MarcBateEA

I found that if you just exclude the geography from the geojson it returns a table.

it seems like you would create a relationship class to do what I was asking in field maps between a feature class and the table created by cdf. However at least in my local host environment the table can’t be used in the go tool. It just has an x by it nothing else

EliasRex

Were you able to register the custom data provider with the 11.1 ArcGIS Server? I keep receiving an error when trying to register the cdpk: "Node server failed to start"

MarcBateEA

Will be upgrading to 11.1 in the next month and I'll have my provider ready to go, and hopefully you can figure out a solutin and share.

AndrewAdamson22

Running into the same. Were you able to register?

EliasRex

I was. I made the mistake of installing node packages at the top level of the project. Once I installed the packages at the provider level, I managed to register it with the server.

nicogis

Here there is the documentation & samples: - Overview ArcGIS Server Custom Data Feeds

EliasRex

Have you found any information or samples that include caching?

nicogis

I have seen in create service property set 'true' but I don't know if it is managed this property with CDF

"properties": {
        "disableCaching": "true"
    },

MarcBateEA

I had to figure it out myself. I allowed a url parameter to be passed called expiresDays so I can use different caching for different projects. I also have a default expiresDays in my config file

let expiresDays = req.query.expiresDays || config.expiresDays
let expires = null
if (expiresDays && expiresDays !== 0) {
    expires = new Date().setDate(new Date().getDate() + expiresDays)
}
// set the fields array where I used aliases, and the array of features, then set ttl
const geoJson = {
    metadata: {
         idField: "OBJECTID",
         fields: fields,
         displayField: Object.keys(allFeatures[0].properties)[0] 
    },
    type: "FeatureCollection",
    features: allFeatures,
    ttl: expires
}

By default it was caching the FeatureServer and the FeatureServer/0 as different things so it was calling my API twice. Since there's only one "layer" in the CDF I removed the part of the key with the layer in it. I was able to override the key it uses for caching in the model.js file to fix that:

Model.prototype.createKey = function (req) {
    let key = req.url.split("/")[1]
    if (req.params.host) key = [key, req.params.host].join("::")
    if (req.params.id) key = [key, req.params.id].join("::")
    //if (req.params.layer) key = [key, req.params.layer].join('::')

    return key
}

EliasRex

How do you get access to the key once another request is made?

Thomas_Fuhrmann

Thanks @MarcBateEA! Just one quick question - the ttl field should be in seconds, right? I saw that you used the setDate method which returns a timestamp in milliseconds - shouldn't you divide by 1000?

MatejVrtich

Hi,

it seems that caching in the custom data provider is not working at all in version 11.1.

Should be fixed in version 11.2: https://support.esri.com/en-us/bug/the-ttl-property-set-in-the-custom-data-provider-source-bug-000159991

Hopefully the documentation will be expanded with debugging, logging, caching and other useful details for implementing your own data providers.

Matej

MarcBateEA

Thanks for the info. I really would like the CDFs to perform better with the filter manager so the layers can be used with WAB or ExB filter widgets just like a real feature class. The way the filter widget calls the service is too slow for us to use in production environment.

MarcBateEA

If the custom data feed layer is supposed to be just like other feature layers, we want to be able to index its fields so parts of the platform like filter and queries perform the same, and don't take so long you can't use this new technology. 😉.

NurFarahHannanAhmad

Hi Simon,

I've tried to access the link of the sample provider you provided, however the site cannot be reached. Do you have any other sample provider that is accessible at the moment? Appreciate if you can share it. Thanks

MarcBateEA

Just an update - it turns out the ArcGIS Server Custom Data Feed runtime is just NodeJS app running on the ArcGIS Server, and the CDF feature class does not behave just like any other feature service. There are performance issues, esp with the filter widgets in WAB and ExB, and there's a caching bug in 11.1 that will cache it forever if you set any ttl value.

NurFarahHannanAhmad

Hi I've tried following this guide : https://developers.arcgis.com/enterprise-sdk/guide/custom-data-feeds/create-a-yelp-custom-data-feed/ and successfully published the yelp data. I have 2 questions.

1. Is there any way we can get more than 20 feature data? I know by default yelp only provide 20 data for the API.

2. Is there any other sample that is accessible? I'm trying to do a demo for the twitter data but couldn't find any related documentation.

Any update is appreciated.

Thanks.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.