While Data Pipelines is in beta it is subject to breaking changes. For example, breaking changes could mean your output results change, or certain options are no longer available. In the October 2023 update of Data Pipelines, please be aware of the following updates:
1. When writing to output feature layers and tables, the following field types have changed:
- Big integer fields from input datasets will no longer be written to double fields, they will be written to the new big integer field type.
- Date fields from input datasets will no longer be written to string fields, they will be written to the new date-only type.
Existing data pipelines that use datasets with these fields to update layers will likely need to be updated. You can use Update fields to update the type if it is appropriate for your data.
2. Snowflake table names support more characters but are now case sensitive. If you have existing data pipelines that use inputs from Snowflake, you may need to update the table names to accommodate this change.
3. For GeoJSON file inputs, the parameter Geometry type is now optional, and only needs to be specified if the GeoJSON feature collection contains multiple geometry types. If the feature collection contains only one geometry type, it is no longer valid to set a different type for the Geometry type.
4. For delimited file inputs, the parameter Infer schema is no longer available. Data Pipelines will now infer the dataset schema by default. To review or modify the inferred schema for delimited files, use the new Fields parameter.
If you have questions about changes you observe in Data Pipelines or how to remedy the changes, please reply to this thread or make a post in the Data Pipelines Questions forum.
To learn more about the October 2023 update of Data Pipelines (beta), check out the What's New in Data Pipelines (beta) blog.
