topic Re: Parquet & Pro in ArcGIS Pro Questions

Parquet & Pro

Trevor_Hart — Tue, 09 Jan 2024 21:56:45 GMT

Hey All,

Ive created a multi-file connection (mfc) to a directory with some parquet files in Pro 3.1.3

The mfc creates fine and I can preview the layer (see below) but when I add the layer to a map I get no geometries and if I open the attribute table I get an unknown error.

Pro recognizes the geometry column fine according to the layer properties.

Ive tried;

- Different projections (original data is not in wgs84)

- Adding an objectid

- Removing the geometry column

- Removing the compression

But I will get the same error and no geometries.

Re: Parquet & Pro

DanPatterson — Wed, 10 Jan 2024 05:09:11 GMT

related thread which may be of interest

Solved: Using parquet files in an ArcGIS Pro Big Data Conn... - Esri Community

Re: Parquet & Pro

Trevor_Hart — Wed, 10 Jan 2024 08:26:31 GMT

Thanks Dan. I have seen that post. I don’t have an issue creating the connection to the data. It just doesn’t draw and throws an unknown error when I try to open the attribute table. Pro recognizes the data (fields, geometry type etc) and can preview it fine though.

Re: Parquet & Pro

DerekGourley — Thu, 11 Jan 2024 20:07:29 GMT

Hi Trevor,

Thanks for the question.

I see that you are using Pro 3.1.3, which doesn’t have support for adding parquet-based Multifile Feature Connection datasets to the map (including opening the attribute table), visualizing them, or using them directly in tools that are not a part of the GeoAnalytics Desktop tools. The following Pro 3.1 documentation has notes on what is supported: https://pro.arcgis.com/en/pro-app/3.1/help/data/big-data-connections/use-big-data-connections.htm#GUID-943CCC38-C31E-401F-BB79-2FDFA10B75A0

However, with Pro 3.1 you can use them for your analysis or for visualization by first running a GeoAnalytics Desktop tool such as Copy Dataset From Multifile Feature Connection to save the parquet dataset as a shapefile or to a File Geodatabase, which can then be used to do further analysis or visualization.

The good news is that Pro 3.2 added extended support for parquet-based Multifile Feature Connection datasets so that they can be visualized in maps and used as input for most geoprocessing tools. This is covered as a bullet point in the what’s new in Pro 3.2 documentation: https://pro.arcgis.com/en/pro-app/latest/get-started/whats-new-in-arcgis-pro.htm#GUID-954D7B4B-E653-4E12-AA3A-D4AAD24E729C

Re: Parquet & Pro

Trevor_Hart — Thu, 11 Jan 2024 21:09:19 GMT

Thanks for the reply @DerekGourley unfortunately Pro 3.2 is no better even fully patched.

The mfc connection doesnt even list the layers...

The connection under 3.1.3, shows the layers

Pro 3.2 doesnt list anything when you try to expand the connection - notice the arrow is not visible even though I have expanded the connection in Pro

Re: Parquet & Pro

DerekGourley — Fri, 12 Jan 2024 00:49:57 GMT

Hi Trevor,

To help troubleshoot this issue, would you be able to share what happens when you try the following:

When you refresh the MFC in the Catalog, does the arrow appear?

If the arrow does not appear, could you please try right-clicking on the MFC in the Catalog and then selecting the “Show in file explorer” option, followed by opening the .mfc file in a text editor. With the .mfc file open in a text editor, does the file contain a list of datasets or is it an empty list ([])?

The image you shared has two MFC files, was one of them re-created using Pro 3.2? I might suggest also trying a MFC file name that doesn’t have a period in it, such as “Test_32” or “Test_3_2”.

If the MFC file was re-created with Pro 3.2, did the messages from the GP tool (or the MFC UI, depending on how it was created) show that datasets were successfully identified? When you originally created the MFC file with Pro 3.1.3, did it show that the datasets were successfully identified?

If the MFC file was not re-created with Pro 3.2, can you try re-creating it and let me know if it lists any datasets that were successfully identified? Would you be able to share any messages or warnings that you see?

I also wanted to mention a few other things to try / check just in case:

Does the MFC folder structure match the example in the documentation: https://pro.arcgis.com/en/pro-app/latest/help/data/big-data-connections/big-data-connections.htm#GUID-981AC9EC-41D8-4CDA-BD41-E1189BEBF4C2
Is the folder with the parquet files in a location that can be accessed successfully by the computer(s) with Pro? For example, if the MFC folder is in a shared location, can you try copying it locally to the computer(s) with Pro and then creating a MFC file using the local folder to see if that works (if the folder isn’t local already)? Was the same computer used when testing Pro 3.1 and Pro 3.2? If not, the path that the MFC file is looking for might not be the same on another computer. The path can be checked by opening the .mfc file in a text editor and making sure the path listed in the file exists and is correct.
Are the parquet files encrypted? We currently only support unencrypted parquet files: https://pro.arcgis.com/en/pro-app/latest/help/data/big-data-connections/big-data-connections.htm#GUID-EB0994B2-B32A-4C94-B1D0-A0251285E45A
If the MFC datasets still can’t be discovered, our documentation has additional suggestions to help troubleshoot this issue: https://pro.arcgis.com/en/pro-app/latest/help/data/big-data-connections/use-big-data-connections.htm#GUID-F3A5351B-5780-4C23-88ED-0D6B45EE45BD

Re: Parquet & Pro

Trevor_Hart — Tue, 16 Jan 2024 00:36:43 GMT

Hi @DerekGourley

I believe the folder structure is correct and the files are not encrypted. As mentioned they previewed fine in 3.1.3. The 3.1.3 and 3.2 are different machines. All data is local. The MFC connections were recreated in each version of Pro.

For sanity I downloaded the samples from here;

https://www.tablab.app/datasets/sample/parquet

These work fine in Pro 3.2 except for the Iris file which has periods in the column names. If I remove the periods it works fine in Pro.

This is what they look like in Pro

Im going to retry the other files that are not working.

Re: Parquet & Pro

Trevor_Hart — Tue, 16 Jan 2024 01:08:17 GMT

So if I add one of the Parquet files I am testing to that structure

And then try and Sync Pro does this

The MFC has updated though

And the geometry has been recognized

If I close and open the Pro project again Pro just crashes, I have submitted an error report.

If I delete the Parquet file but dont sync the MFC then Pro opens the project fine and the folder is shown in the MFC (but there are no files behind it).

Re: Parquet & Pro

Trevor_Hart — Tue, 16 Jan 2024 03:17:42 GMT

@DerekGourley finally solved (kind of), Pro seems to have issues with the geometry column.

I exported the data as Parquet minus the geometry column and that worked fine in Pro.

If I export and include WKB or WKT I get the error/crash behaviour described above.

What I did instead was calculated a field new field as GeoJSON and exported as Parquet (with no geometry column).

Pro recognizes the GeoJSON field and works as expected - MFC is happy, attribute table is happy and draws in the map.

Do you have any guidance on supported geometry columns?

Pro suggests that WKT and WKB are supported?

Re: Parquet & Pro

DerekGourley — Thu, 18 Jan 2024 00:51:55 GMT

Hi Trevor,

Thanks for following up with all of the additional information and for including the screenshots. I'm going to send you a message with my email to see if it would be possible to share this dataset with us and to see if we can learn more about how the parquet data was generated.

Re: Parquet & Pro

DerekGourley — Mon, 26 Feb 2024 19:37:55 GMT

I was able to work with Trevor to figure out what was going on. Pro 3.2 (and it's patches) were unable to recognize the parquet files that he shared with us. We only see this happen with some parquet files, since Pro 3.2 can recognize other parquet files just fine. As a workaround, we wrote Trevor's parquet files out using Pandas and Pro 3.2 was able to recognize them. This will be fixed in the next release of Pro (3.3 and above). Thank you for reporting this issue Trevor!