Feed with multiple rows of data at each timestamp workflow help

[Teresa_Blader]

I'm hitting two API feeds of air quality sensors and the results are not what I'm used to. The data is air quality data from two separate sensor feeds that each detect the same 10 parameter values (pressure, CO, O2, NO, NO2, temperature, humidity, PM1, PM10, PM2.5).

The API returns the data into separate rows for each parameter for a single timestamp, and a few sets of nested JSON that pertain to each parameter type. So the only thing I could think to do is flatten those and then implement a horrendous looking workflow using a real-time analytic (which is mostly horrendous because Join only allows two inputs at a time). I have to do four sets of joins to get everything all back together. Ha!

Does anyone have any suggestions on how to simplify this? I've listed out my steps below: 

Ideally when I'm working with this down the pipeline, it's one row of data for each sensor at a given timestamp, where each parameter has it's own unique field. Funny enough, when I manually download from their software, that is how they format it.

How it looks when not flattened:

Output once flattened and cleaned up:

So I've managed to take that output and push it to one row by: 

1. Map Fields (add lat/long coordinates because the API doesn't have them, rename some fields)
2. Filter by each 'value' 'parameter'
3. Map field (add 'value' and 'quality check' fields to fields with unique names by parameter)
4. Join (two at a time, add unique fields together when sensor ID and timestamp matches)
5. Select Fields (remove the extra join ID fields)
6. Join (getting closer... two at a time, add unique fields together when sensor ID and timestamp matches)
7. Select Fields (remove the extra join ID fields)
8. Join (getting closer... two at a time, add unique fields together when sensor ID and timestamp matches)
9. Select Fields (remove the extra join ID fields)
10. join (one last time.... two at a time, add unique fields together when sensor ID and timestamp matches)
11. Select Fields (remove the extra join ID fields)
12. Merge (now it's time to bring both of these sensors, now just one row each, into the same dataset because the API doesn't have a group sensors ability)

[JeffSilberberg]

@TeresaBlader 

     Unfortunately, I think you have it right.  The only thing you might drop is the Multiple "Select Fields (remove the extra join ID fields)". You can do this in one Node farther down the flow.  Join, Join1, Join2.

    I have a similar flow with Weather Stations and two years in, I have not found an easier solution.  

[Teresa_Blader]

Bummer sauce! 

Thanks for that tip, once I renamed the map fields for those join fields, that got rid of the extra selects. And by merging them at the beginning, I was able to eliminate the duplicate set of filters and joins at least.