Latest Contributions by BethanyScott

In the February 2024 update of ArcGIS Data Pipelines, the application moved out of beta and is now available for general use. Transitions from beta to general availability are subject to breaking changes. Please be aware of the following changes in the February 2024 update: The results from scheduled data pipelines that were run during beta will no longer be accessible from the scheduling UIs in the Data Pipelines app. Output feature layers or tables that were generated will still be available in your content, and any active tasks will continue to run as scheduled. Results returned after the update will be available in the scheduling UI. The following field types from Google BigQuery are now read differently: BigNumeric fields are now read as type Double. Previously, fields of this type were not supported and were dropped. Datetime fields are now read as type Date. Previously, fields of this type were read as strings. The behaviour for casting numeric fields when using the Update fields tool has changed in the following ways: Casting a decimal value to an integer value now rounds to the nearest integer. Previously the value would have been truncated at the decimal regardless of the decimal value. For example, previously a double value of 2.9 would have been cast to an integer value of 2. Now the value will be cast to a value of 3. Casting a numeric value to a smaller type could now return null instead of the largest supported value for that field. For example, if you cast a big integer value of 9223372036854776000 to an integer, the result was previously 2147483647 (the largest supported value for an integer field). Now the value will be returned as null. Encoded URLs are no longer supported. For example, URLs such as the following which uses encoded values: https://opendata.org/api/geojson?query=issuedate%3A2024&timezone=America%2FLos_Angeles To resolve the encoding issue, you can use a publicly available decoder, such as this one. Just enter your URL with encoded values, click decode, and use the resulting URL as input to the Public URL tool in Data Pipelines. If you have questions about changes you observe in Data Pipelines or how to remedy the changes, please reply to this thread or make a post in the Data Pipelines Questions forum.   To learn more about the February 2024 update of Data Pipelines, check out the What's new documentation and the ArcGIS Data Pipelines is Now Available blog post.  ... View more

We are excited to announce that ArcGIS Data Pipelines is officially out of beta and available for general use in ArcGIS Online. Data Pipelines is a native data integration application in ArcGIS Online that offers fast and efficient ways to ingest, engineer, and maintain data from various sources. The app provides an easy-to-use, drag-and-drop interface for building data integration workflows which reduces the need for coding skills and simplifies your data prep tasks. In addition, Data Pipelines offers built-in scheduling functionality that enables you to automate your data prep workflows and keep your feature layers up to date as the source data grows and evolves. To learn more about ArcGIS Data Pipelines, watch the video below or check out the following resources: ArcGIS Data Pipelines is Now Available blog What’s new documentation     There's more to come in future updates of Data Pipelines, and we want to hear from you on what we can do to improve your data prep workflows. Share your ideas or ask us a question here in Esri Community! ... View more

While Data Pipelines is in beta it is subject to breaking changes. For example, breaking changes could mean your output results change, or certain options are no longer available. In the October 2023 update of Data Pipelines, please be aware of the following updates:  1. When writing to output feature layers and tables, the following field types have changed:  Big integer fields from input datasets will no longer be written to double fields, they will be written to the new big integer field type.  Date fields from input datasets will no longer be written to string fields, they will be written to the new date-only type.  Existing data pipelines that use datasets with these fields to update layers will likely need to be updated. You can use Update fields to update the type if it is appropriate for your data.  2. Snowflake table names support more characters but are now case sensitive. If you have existing data pipelines that use inputs from Snowflake, you may need to update the table names to accommodate this change.  3. For GeoJSON file inputs, the parameter Geometry type is now optional, and only needs to be specified if the GeoJSON feature collection contains multiple geometry types. If the feature collection contains only one geometry type, it is no longer valid to set a different type for the Geometry type. 4. For delimited file inputs, the parameter Infer schema is no longer available. Data Pipelines will now infer the dataset schema by default. To review or modify the inferred schema for delimited files, use the new Fields parameter.     If you have questions about changes you observe in Data Pipelines or how to remedy the changes, please reply to this thread or make a post in the Data Pipelines Questions forum.     To learn more about the October 2023 update of Data Pipelines (beta), check out the What's New in Data Pipelines (beta) blog.   ... View more