https://community.esri.com/t5/data-pipelines-questions/arcgis-data-pipelines-issue-working-with-large/m-p/1612934#M157 <P>Hi Data Pipelines Team,&nbsp;</P><P>Due to the extremely large amount of data in the Living Atlas' iNaturalist observations, we're currently trying to query the data to just our park boundaries to reduce the amount of load time/drawing issues for our wildlife management team.&nbsp;</P><P>We've set up our Data Pipeline to filter by our park's extent then additionally by specific attributes. When working with the setup of this pipeline, we're seeing extreme latency and loading issues. I'm guessing this is due to how large the&nbsp;iNaturalist observations dataset is. Is there a Data Pipeline solution/workflow for working with large datasets like these? For example, I'm not even able to load the fields for the Filter parameters, it just spins:&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Amanda__Huber_0-1746727500540.png" style="width: 400px;"><img src="https://community.esri.com/t5/image/serverpage/image-id/131793iA58959DC572595CC/image-size/medium?v=v2&amp;px=400" role="button" title="Amanda__Huber_0-1746727500540.png" alt="Amanda__Huber_0-1746727500540.png" /></span></P><P>&nbsp;</P><P>When trying to preview, it also just spins.&nbsp;<BR /><BR /></P><P>How does Data Pipelines work with these types of large datasets when trying to configure? Is it trying to load all the data in addition to the fields?&nbsp;</P><P>&nbsp;</P><P>Thanks for any insight!</P><P>Best,&nbsp;</P><P>Amanda Huber</P> Thu, 08 May 2025 18:08:50 GMT Amanda__Huber 2025-05-08T18:08:50Z https://community.esri.com/t5/data-pipelines-questions/arcgis-data-pipelines-issue-working-with-large/m-p/1612934#M157 <P>Hi Data Pipelines Team,&nbsp;</P><P>Due to the extremely large amount of data in the Living Atlas' iNaturalist observations, we're currently trying to query the data to just our park boundaries to reduce the amount of load time/drawing issues for our wildlife management team.&nbsp;</P><P>We've set up our Data Pipeline to filter by our park's extent then additionally by specific attributes. When working with the setup of this pipeline, we're seeing extreme latency and loading issues. I'm guessing this is due to how large the&nbsp;iNaturalist observations dataset is. Is there a Data Pipeline solution/workflow for working with large datasets like these? For example, I'm not even able to load the fields for the Filter parameters, it just spins:&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Amanda__Huber_0-1746727500540.png" style="width: 400px;"><img src="https://community.esri.com/t5/image/serverpage/image-id/131793iA58959DC572595CC/image-size/medium?v=v2&amp;px=400" role="button" title="Amanda__Huber_0-1746727500540.png" alt="Amanda__Huber_0-1746727500540.png" /></span></P><P>&nbsp;</P><P>When trying to preview, it also just spins.&nbsp;<BR /><BR /></P><P>How does Data Pipelines work with these types of large datasets when trying to configure? Is it trying to load all the data in addition to the fields?&nbsp;</P><P>&nbsp;</P><P>Thanks for any insight!</P><P>Best,&nbsp;</P><P>Amanda Huber</P> Thu, 08 May 2025 18:08:50 GMT https://community.esri.com/t5/data-pipelines-questions/arcgis-data-pipelines-issue-working-with-large/m-p/1612934#M157 Amanda__Huber 2025-05-08T18:08:50Z