topic Re: Issue reading optional field from JSON-file using Arcade in Data Pipelines Questions

Issue reading optional field from JSON-file using Arcade

AndreasHall — Tue, 17 Sep 2024 04:49:49 GMT

I have a scheduled data pipeline that once a day reads a JSON-file and writes it to a feature service. In the pipeline, I do numerous calculate field where I use Arcade to get the information I need from the JSON-file. In these scripts I use $record. to fetch the information I need ($record.approximateEnd in the example below). The issue I am facing is that some of this information is not always present in the input file. When this happens, the arcade script throws an Execution error - Key not found, and the pipeline does not run/fails (no data is updated). I would like to modify my script so that it first tests if $record.approximateEnd exists. If it does not exist it would create a null value. Any other solution that would prevent my pipeline from failing when data is missing would also work.

My input data is SMHI Open Data API Docs - Warnings.

(I already check if $record.approximateEnd is null but I need something to cover the case when there is no $record.approximateEnd at all.)

// Check that EndDate exist, save as Date. if(IsEmpty($record.approximateEnd)){ return null; } else { var MyDate = Split(Left($record.approximateEnd, 10),"-"); var MyTime = Right($record.approximateEnd, 13); MyTime = Split(Left(MyTime, 8), ":"); MyDate = Date(MyDate[0], (MyDate[1]-1), MyDate[2], MyTime[0], MyTime[1], MyTime[2]); } // Check that EndDate is later than StartDate if (MyDate>$record.Start_) { return MyDate; } else { return null; }

Re: Issue reading optional field from JSON-file using Arcade

KenBuja — Tue, 17 Sep 2024 12:45:39 GMT

This code will check if the Feature or FeatureSet contains a field.

function fieldExists(item, fieldName) { //item can be a Feature or FeatureSet var fields = Schema(item).fields; for (var i in fields) { if (Lower(fields[i].name) == Lower(fieldName)) return true; } return false; } iif(fieldExists($feature, "field_name"), "Field Exists", "Field doesn't exist");

Re: Issue reading optional field from JSON-file using Arcade

DuncanMackey — Tue, 17 Sep 2024 16:52:39 GMT

Hi Andreas!

There are two relevant Arcade functions that will likely be helpful here:

So in your example, you could use DefaultValue similarly:

// Set end date to null if the field does not exist. var endDate = DefaultValue($record, "approximateEnd", null)

Hopefully these functions will give your arcade script the flexibility it needs to handle JSON with varying properties. Please let me know if you have any more questions regarding this.

Thank you,

Duncan

Re: Issue reading optional field from JSON-file using Arcade

AndreasHall — Wed, 18 Sep 2024 07:56:36 GMT

Great, thanks a lot!