How to handle a case where there are no fields present in a JSON file?

[AndreasHall]

I have a scheduled data pipeline that once a day reads a JSON-file and writes it to a feature service. The file lists weather warnings (SMHI Open Data API Docs - Warnings). Some days there might be no warnings at all present. Then my pipeline fails. Is there any way to handle this use case with ArcGIS Data Pipelines? (If there is no data several days in a row my scheduled pipeline is disabled as it has failed too many times in a row.)

[BruceHarold]

Andreas, I can supply an alternative workflow and have a blog on the topic which I'm publishing November 7th when ArcGIS Pro 3.4 releases - because it uses a new feature.  The subject matter is weather alerts in CAP protocol, which I can see is available in your weather feed.  Empty entries are OK.  I'll let the pipelines team respond on what you are experiencing.

[Sarah_Hanson]

Hi @AndreasHall,

As you noticed, scheduling pauses when a data pipeline fails after too many failed runs (5), and this functionality was built in intentionally to prevent users from paying for the running of a data pipeline that may have a critical issue that needs to be remedied.

We will work to address this in a future release and will reach out to you if we need additional input.

Thank you!

[DuncanMackey · Accepted solution]

Hi Andreas,

The empty JSON file is an interesting case, as Data Pipelines doesn't know the schema, hence the data pipeline failing. We'll look into some possible solutions for a future release. Thanks for bringing this up!

[AndreasHall]

Thanks for the feedback!