Hi,
We are looking at vehicle tracking data for two of our clients. We have about 1TB of csv data that contains long,lat,time,speed,bearing,linear g,lateral g and ignition status data. We have setup an experimental Hadoop Cluster using the Hortonworks HDP 2.2.6 of 9 data nodes using Ambari. We managed to ingest the data and convert to ORC file in about 2 hours. In total we have just over 1.3 billion points.
We want to identify trips (a consecutive series of points in time for a specific vehicle between ignition on and off) to determine movement patterns over time. This we can do using Hive queries quite easily using the lead and lag functions. My problem however comes in when we want to calculate the length of the trip. All data is stored in WGS84 lat long so I need to use the ST_GeodesicLengthWGS84 in the spatial-framework-for-hadoop. I have managed to build the framework and run some test queries on our hadoop cluster and all seems to be working well. My problem is to convert the lat, long data to a line string that is required as input for ST_GeodesicLengthWGS84. I did come across a modified constructor for ST_LineString in the following fork of the spatial-framework-for-hadoop at https://github.com/cartershanklin/spatial-framework-for-hadoop. The changes that he made were never pulled into the main repo however and it looks like a lot of changes have been made to the main repo.
Has anybody done similar analysis and found a way around this issue.
Regards
Derck
Derck Vonck
Technical Lead
esri South Africa
