ArcPy Data Access Support for numpy.datetime64 and Python datetime

3772
11
04-16-2015 02:44 PM
JoshuaBixby
MVP Esteemed Contributor

Recent questions/discussions on GeoNet got me thinking about an ArcPy Data Access module limitation I hadn't thought about in a while, specifically the lack of date and time support when working with NumPy arrays.  Given where date and time support in NumPy was in 2012 when Esri introduced the ArcPy Data Access module, I can understand why support was missing then, but I think the time is right for them to start supporting it.

I went ahead and created the suggestion on the ArcGIS Ideas site:  ArcPy Data Access Support for numpy.datetime64 and Python datetime.  I also logged an enhancement request with Esri Support:  ENH-000086993: Update the ArcPy Data Access module to include support for numpy.datetime64 and Python datetime.  If you would like to see date and time support for the NumPy-related methods in the ArcPy Data Access module, I encourage you to go promote the idea on ArcGIS Ideas and submit your own use case to Esri Support to have your customer number attached the aforementioned enhancement request.

11 Replies
DanPatterson_Retired
MVP Esteemed Contributor

Done...

Reply
0 Kudos
ShaunWalbridge
Esri Regular Contributor

Just to reiterate what David Wynne mentioned on the idea, this is fixed in Pro 1.1, and will likely also be included for Desktop 10.4. I agree it's a great feature to have, so that the NumPy array can serve as the common intermediate format for tabular data.

Cheers,
Shaun

Reply
0 Kudos
JoshuaBixby
MVP Esteemed Contributor

Shaun Walbridge​, good news, indeed.

With Pro having its own numbering/versioning scheme, separate from ArcGIS Desktop, it gets a bit confusing in terms of what will be released when.  Outside of ArcGIS 10.3.1 being released shortly, is release information for ArcGIS Desktop 10.3.2, ArcGIS 10.4, and ArcGIS Pro 1.1 publicly documented anywhere?

Reply
0 Kudos
ShaunWalbridge
Esri Regular Contributor

We announce specific dates for releases that are close to release (like 10.3.1 as you mentioned). For Pro, the releases aren't tied directly to the rest of the ArcGIS release, but generally you should see a Pro release complimentary to each broader release. Note that Pro has an improved mechanism for updating which means that upgrades are more seamless and will likely require less planning around. Pro 1.1 will release in the coming months, and 10.4 and Pro 1.2 will also likely ship in 2015, based on the plenary at DevSummit. Shipping software depends on a number of external factors, and there is naturally some variability in the specifics of the releases.

Reply
0 Kudos
JordanCarmona1
New Contributor III

I'm looking at the documentation in using NumPy in ArcGIS Pro and the only reference I can find to date format is <m8[us]. I'm having issues trying to populate a table using datetime64[ns]; the output type from pandas.to_datetime()

Was this fully implemented? I'm not super familiar with the different types of datetime so this has been fairly hard to research.

Working with NumPy in ArcGIS—ArcPy Get Started | ArcGIS Desktop 

Reply
0 Kudos
DanPatterson_Retired
MVP Esteemed Contributor

just do a 'help' or 'dir' on these

np.datetime64,   np.datetime_as_string,   np.datetime_data

most is defined in multiarray

numpy/multiarray.py at master · numpy/numpy · GitHub 

If you want to explore the current and backward compatability.

Pandas derives from these

Reply
0 Kudos
ShaunWalbridge
Esri Regular Contributor

In addition to Dan's advice, there's also the ArcGIS API for Python which is included in Pro, and includes a direct conversion to pandas via the spatially enabled data frame: Introduction to the Spatially Enabled DataFrame | ArcGIS for Developers 

You can also see this GIS.SE post with some details on conversion: arcpy - NumPyArrayToTable - RuntimeError: unsupported time units. use M8[us] - Geographic Informatio... 

Reply
0 Kudos
LorneDmitruk
New Contributor III

Apart from waiting for 10.4 does anyone have suggestions for dealing with dates not coming across from a numpy array to a table?

Reply
0 Kudos
JamesCrandall
MVP Frequent Contributor

What is your situation/scenario exactly?

Reply
0 Kudos