topic Re: Pandas and DateTimes in Pro 3.2 in Python Questions

Pandas and DateTimes in Pro 3.2

JoshuaFlickinger — Tue, 20 Feb 2024 16:32:31 GMT

I have a workflow scripted in which I export a Pandas dataframe to a geodatabase table using the method:

df.spatial.to_table(out_table_path)

After upgrading Pro to 3.2, I noticed that my script changed how it was handling dates. After a little digging, I found that my dataframe fields with a datetime64[s] data type were casting to the new ESRI 'Time Only' field during the conversion to a geodatabase table, despite containing both dates and times. This is obviously a deviation from the behavior in older versions of Pro, which had only a single type of field for handling date times.

My Question: Is this the expected behavior, or a bug? Anyone else having problems? I haven't been able to find good documentation on how the new ESRI date time field types work with the arcgis.features module. I did find, via some trial-and-error, that I could get what I needed by forcing my pandas field datatype to datetime64[ns] prior to export, but this doesn't feel like it should be necessary and is far higher resolution than what I need for my data.

Thanks,

Re: Pandas and DateTimes in Pro 3.2

DanPatterson — Tue, 20 Feb 2024 17:00:28 GMT

There were changes to Pandas, perhaps this might provide some extra information

Part 5 - Working with Time Series Data | ArcGIS API for Python

Re: Pandas and DateTimes in Pro 3.2

KimOllivier — Thu, 22 Feb 2024 09:42:57 GMT

I have found other problems with 64 bit Integers now called BigInteger. Integer fields defined in geopackages and spatialite are now cast by default to BigInteger instead of Integer. RelationshipClasses cannot use them which was a surprise because I suddenly had BigIntegers for every primary and foreign key in my database.

I have had to go back and explicitly cast to Integer (32 bit) using FieldMapping on every copy operation so I am not surprised that Pandas needs editing. Just about to do some datetime manipulation, thanks for the heads-up.