topic Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime in Python Questions

ArcPy Data Access Support for numpy.datetime64 and Python datetime

JoshuaBixby — Thu, 16 Apr 2015 21:44:52 GMT

Recent questions/discussions on GeoNet got me thinking about an ArcPy Data Access module limitation I hadn't thought about in a while, specifically the lack of date and time support when working with NumPy arrays. Given where date and time support in NumPy was in 2012 when Esri introduced the ArcPy Data Access module, I can understand why support was missing then, but I think the time is right for them to start supporting it.

I went ahead and created the suggestion on the ArcGIS Ideas site: ArcPy Data Access Support for numpy.datetime64 and Python datetime. I also logged an enhancement request with Esri Support: ENH-000086993: Update the ArcPy Data Access module to include support for numpy.datetime64 and Python datetime. If you would like to see date and time support for the NumPy-related methods in the ArcPy Data Access module, I encourage you to go promote the idea on ArcGIS Ideas and submit your own use case to Esri Support to have your customer number attached the aforementioned enhancement request.

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

DanPatterson_Retired — Thu, 16 Apr 2015 21:55:52 GMT

Done...

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

ShaunWalbridge — Fri, 17 Apr 2015 00:53:23 GMT

Just to reiterate what David Wynne mentioned on the idea, this is fixed in Pro 1.1, and will likely also be included for Desktop 10.4. I agree it's a great feature to have, so that the NumPy array can serve as the common intermediate format for tabular data.

Cheers,
Shaun

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

JoshuaBixby — Fri, 17 Apr 2015 01:48:15 GMT

Shaun Walbridge, good news, indeed.

With Pro having its own numbering/versioning scheme, separate from ArcGIS Desktop, it gets a bit confusing in terms of what will be released when. Outside of ArcGIS 10.3.1 being released shortly, is release information for ArcGIS Desktop 10.3.2, ArcGIS 10.4, and ArcGIS Pro 1.1 publicly documented anywhere?

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

ShaunWalbridge — Fri, 17 Apr 2015 02:38:30 GMT

We announce specific dates for releases that are close to release (like 10.3.1 as you mentioned). For Pro, the releases aren't tied directly to the rest of the ArcGIS release, but generally you should see a Pro release complimentary to each broader release. Note that Pro has an improved mechanism for updating which means that upgrades are more seamless and will likely require less planning around. Pro 1.1 will release in the coming months, and 10.4 and Pro 1.2 will also likely ship in 2015, based on the plenary at DevSummit. Shipping software depends on a number of external factors, and there is naturally some variability in the specifics of the releases.

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

LorneDmitruk — Thu, 18 Jun 2015 21:35:01 GMT

Apart from waiting for 10.4 does anyone have suggestions for dealing with dates not coming across from a numpy array to a table?

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

JamesCrandall — Fri, 19 Jun 2015 14:33:53 GMT

What is your situation/scenario exactly?

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

LorneDmitruk — Fri, 19 Jun 2015 15:32:21 GMT

I'm reading a view from a non geodatabase Oracle table containing a number of date fields. In addition the view field names also contain spaces and parentheses. So I read the table into a pandas data frame using an sql query to get the field names GDB friendly, convert to a numpy record array and from there into a scratch table in a file GDB.

I have played around using table to table but if I use field maps it takes a really long time to process, the table has over 100 fields as that's what the users want. Also the view has a couple of fields that are not required and I strip them out in the sql query. Unfortunately I don't have control over how the view was created and can't get it changed.

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

JamesCrandall — Fri, 19 Jun 2015 17:25:32 GMT

Improper field names sounds like it should be addressed at the database rather than doing a bunch of gymnastics to transform it elsewhere. That's an easy fix, just order the dba's to fix it.

What if you were to do the pandas DataFrame-to-numpy-to-featureclass/table minus the date fields, create a secondary pandas dataframe with just a "joinable" field and the date fields then Table_To_Table() those separately and join them back to the larger featureclass/table.

We do similar processing using cx_Oracle to integrate non-spatial tables.

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

GIS_Spellblade — Wed, 14 Nov 2018 03:42:48 GMT

I'm looking at the documentation in using NumPy in ArcGIS Pro and the only reference I can find to date format is <m8[us]. I'm having issues trying to populate a table using datetime64[ns]; the output type from pandas.to_datetime()

Was this fully implemented? I'm not super familiar with the different types of datetime so this has been fairly hard to research.

Working with NumPy in ArcGIS—ArcPy Get Started | ArcGIS Desktop

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

DanPatterson_Retired — Wed, 14 Nov 2018 04:27:56 GMT

just do a 'help' or 'dir' on these

np.datetime64, np.datetime_as_string, np.datetime_data

most is defined in multiarray

numpy/multiarray.py at master · numpy/numpy · GitHub

If you want to explore the current and backward compatability.

Pandas derives from these

Re: ArcPy Data Access Support for numpy.datetime64 and Python datetime

ShaunWalbridge — Thu, 06 Dec 2018 06:57:07 GMT

In addition to Dan's advice, there's also the ArcGIS API for Python which is included in Pro, and includes a direct conversion to pandas via the spatially enabled data frame: Introduction to the Spatially Enabled DataFrame | ArcGIS for Developers

You can also see this GIS.SE post with some details on conversion: arcpy - NumPyArrayToTable - RuntimeError: unsupported time units. use M8[us] - Geographic Information Systems Stack Exch…