topic Re: location_timestamp vs. created_date in ArcGIS Tracker Questions

location_timestamp vs. created_date

JamesCrandall — Wed, 03 Jun 2020 19:14:31 GMT

Need some clarification on the two columns: location_timestamp vs. created_date

If there is a large gap between these two values, does that mean the app user was stationary for that period of time and then at some point the ESRI Tracker service was updated with features later?

For example, we can see in some point features the two date values are quite close (here they are only 2 mins apart):

In other instances the two dates are hours apart:

I'm asking because we are processing tracks at the top of each hour by querying the location_timestamp values from the last known location_timestamp value we maintain. However, if the point features were added at a much later time then they'll be missed if recent tracks have already been processed. That is, we don't look back in time for points added after the top of each hour.

I'm pretty sure I'm assessing this correctly but hoped to get some confirmation so that I can implement something to process the missed features.

Thanks!

Re: location_timestamp vs. created_date

Anonymous User — Wed, 03 Jun 2020 19:26:59 GMT

Hi James,

The location_timestamp field is the date and time when the location was recorded on the device.

The created_date field is the date and time when the location was successfully uploaded to the location tracking feature layer.

Our next doc update at the end this month or early July will provide more information about each of the fields.

Re: location_timestamp vs. created_date

JamesCrandall — Wed, 03 Jun 2020 20:03:07 GMT

Thank you for the quick reply Aaron!

It does sound like my assessment is somewhat correct and point features can be uploaded/added at a much later (hours) time, not necessarily in succession.

We are processing the tracks each hour, enriching them with some other attributes and copying them into another destination feature service that has a similar schema to the tracker service. This is essentially what occurs at the top of each hour:

To identify "new" track features, query the destination feature service to get the most recent feature location_timestamp value for a particular username. Query parameters include: orderByFields': 'location_timestamp DESC', 'resultRecordCount' : '1'
With that location_timestamp value, query the ESRI Tracker service to see if there are any feature >= that value, if so then execute the process on those features.

As you can see, if new point features were created in the ESRI tracker service after the process had executed then subsequent executions would not pickup those in the past.

I'll have to implement a secondary post-processing that identifies these "missed" tracks that were created after the location_timestamp values and pull them into the destination feature service.

Take care,

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 12 Jun 2020 18:26:57 GMT

Aaron -- one last question on this as I've got it worked out to post-process tracks that are created after the location_timestamp that my initial processing misses....

How much later in the future could a Created_date value actually be?

I guess I really just need to know if that is more than a 24 hours / 1 calendar day. This would really help me to solidify recapturing those missed points.

Thanks for any info!

Take care,

Re: location_timestamp vs. created_date

Anonymous User — Fri, 12 Jun 2020 18:49:19 GMT

It really depends on how often your field workers have connectivity. If they come back to the office every night then 24 hours should be more than enough time. If they are out in areas with no connectivity for multiple days, then you'd need to back further in time.

You may want to checkout the Query Top Features API. You could use this to find the most recent track points for each user within a period of time.

Parameters would be something like this:

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 12 Jun 2020 19:06:04 GMT

Good stuff Thanks a bunch!

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 12 Jun 2020 19:18:31 GMT

There is one other item that I cannot understand: Why are there some duplicate ESRI Track points?

I can see that in those after-the-fact times created_date is after the location_timestamp, there may be instances of the same point even though it was created around the location_timestamp. The location_timestamp values are exactly the same.

(edit: it's not a huge issue and I can identify/account for these pretty easily in my post processing tools, but would be good to fully understand.)

Re: location_timestamp vs. created_date

Anonymous User — Fri, 12 Jun 2020 19:34:21 GMT

Hmm that sounds like a bug. Do you have an example of that?

One way this can happen is that an upload is triggered and the points get inserted into the backend database but the response never makes it back to the app, so the app may try to upload those points again. This should be very rare so if you are experiencing this fairly often, we should probably look into it more.

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 12 Jun 2020 20:14:47 GMT

Yep, will find some of those instances for asap.

Thanks again!

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 12 Jun 2020 22:26:24 GMT

Hopefully this will do, I can get you the globalid's and service url if needed too. Bascially, you see there are 2 tracks on top of each other, identical location_timestamp and username, all of the same attributes altogether (except for OID and GlobalID). The difference is the created_date, which seems to show that they were created at later datetimes.

The takeaway is that location_timestamp and username is exactly the same. I couldn't fit the full popup so just pay attention to the (1 of 2). Here are 2 images of point 1:

Here's the second point:

Re: location_timestamp vs. created_date

Anonymous User — Mon, 15 Jun 2020 11:52:37 GMT

Thanks for that info, if you query those two points and look at the returned json, are the location_timestamp values exactly the same down to do the millisecond? What about the geometry are the x,y values exactly the same?

Do you know if this was an Android or iOS device that recorded the tracks?

Was it only those two points in this instance or is there a series of seemingly identical points?

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 10 Dec 2021 20:57:29 GMT

Edit: I will have to determine OS. Also, I cannot find any significant pattern on when this occurs but it can be as much as a few hundred points.

They do look like true duplicates but created at different datetime values, same location_timestamp values in milliseconds. This is from a different instance than in my screenshots above but all the same example in every way,

  "features" : [
    {
      "attributes" : {
        "objectid" : 780399, 
        "location_timestamp" : 1592092364993, 
        "created_user" : "x", 
        "created_date" : 1592092528672, 
        "globalid" : "e8344433-d21e-469e-aa92-59c99df1ae56"
      }, 
      "geometry" : 
      {
        "x" : -80.821906594999973, 
        "y" : 25.762405908000066
      }
    }, 
    {
      "attributes" : {
        "objectid" : 781743, 
        "location_timestamp" : 1592092364993, 
        "created_user" : "x", 
        "created_date" : 1592093778509, 
        "globalid" : "e0e22585-ae08-4406-81cb-243e36fcb829"
      }, 
      "geometry" : 
      {
        "x" : -80.821906594999973, 
        "y" : 25.762405908000066
      }
    }
  ]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I also have a post-processing step that identifies discrepancies after I have moved points from the ESRI tracker service to another hosted feature service that creates local feature classes from the json result of queries against both services. Once they are FC's I run a selectByLocation for identical features as well as using Find Identical GP tool.

When I run the Find Identical GP tool on the ESRI Tracker service results, they are definitely identical in every way except for the Created_date values.

Re: location_timestamp vs. created_date

JamesCrandall — Mon, 15 Jun 2020 16:31:23 GMT

Like I mentioned, I don't have a sense of frequency just yet but I do have some monitoring tools setup. This duplicate issue has only happened about 7 or so times since I started monitoring about 50 ESRI Tracker users collecting roughly 30,000 tracks each 24hrs.

I have a daily summary metrics that I am querying against the ESRI Tracker Service and the Hosted service I have moved points into from the tracker service. In the instances where there's both a "actualDiscrepancies" and "possibleDiscrepancies" values then these are cases where valid ESRI Tracker features were created much later after the copy process ran and these are true discrepancies that I recover those points.

Whenever "actualDiscrepancies" = 0 and "possibleDiscrepancies" > 0 these are all instances where the ESRI track points were created later but already had created a point at the earlier time, hence it's a duplicate feature.

I'm only interested in the actualDiscrepancies and have a process in place to move these into the destination feature service after-the-fact.

(the -1 values are likely due to a missing ">=" somewhere in my programming that queries the feature services but I consider this quite minor and I simply ignore at the moment until I can find time to remedy it).

Re: location_timestamp vs. created_date

JamesCrandall — Fri, 10 Dec 2021 20:57:31 GMT

EDIT: would a single user logged into ESRI Tracker app across multiple devices simultaneously be a source to any of this? I'm still skeptical even that would produce the same features with the exact same location_timestamp values but perhaps there's another process that is being affected by the simultaneous logins.

Aaron,

To reiterate, I'm not worried about the duplicates issue only because I'm handling them in an after-the-fact process that I can identify these instances. But wanted to provide more info to you and your team. Let me know if you'd like me to open a premium ticket to setup a more formal interaction outside of GeoNet, I would be able to remote session with you all if you need.

I now see some triplicates.

[
  {
    "attributes": {
      "objectid": 886296,
      "globalid": "cf208275-3d97-4f07-aa59-89dfe72d8f0a",
      "activity": 0,
      "altitude": -22,
      "app_id": "tracker",
      "battery_percentage": 87,
      "battery_state": 1,
      "course": 96.2351684570312,
      "device_id": "04ba83a0-09b0-4b4a-b87e-2be331924053",
      "floor": null,
      "horizontal_accuracy": 3.90000009536743,
      "location_source": "fused",
      "location_timestamp": 1592434582000,
      "session_id": "7aa745ba-42be-4b23-acc8-11a65d511740",
      "signal_strength": null,
      "speed": 26.8384494781494,
      "vertical_accuracy": 100,
      "created_user": "zzz",
      "created_date": 1592434625398,
      "last_edited_user": "zzz",
      "last_edited_date": 1592434625398
    },
    "geometry": {
      "x": -81.03567099999998,
      "y": 25.857096300000023
    }
  },
  {
    "attributes": {
      "objectid": 886297,
      "globalid": "dbabadd9-9d46-43eb-b42b-05836c94e46e",
      "activity": 0,
      "altitude": -22,
      "app_id": "tracker",
      "battery_percentage": 87,
      "battery_state": 1,
      "course": 96.2351684570312,
      "device_id": "04ba83a0-09b0-4b4a-b87e-2be331924053",
      "floor": null,
      "horizontal_accuracy": 3.90000009536743,
      "location_source": "fused",
      "location_timestamp": 1592434582000,
      "session_id": "7aa745ba-42be-4b23-acc8-11a65d511740",
      "signal_strength": null,
      "speed": 26.8384494781494,
      "vertical_accuracy": 100,
      "created_user": "zzz",
      "created_date": 1592434625398,
      "last_edited_user": "zzz",
      "last_edited_date": 1592434625398
    },
    "geometry": {
      "x": -81.03567099999998,
      "y": 25.857096300000023
    }
  },
  {
    "attributes": {
      "objectid": 886298,
      "globalid": "f7c56f5c-5d20-4772-9f28-2f5ba4e90640",
      "activity": 0,
      "altitude": -22,
      "app_id": "tracker",
      "battery_percentage": 87,
      "battery_state": 1,
      "course": 96.2351684570312,
      "device_id": "04ba83a0-09b0-4b4a-b87e-2be331924053",
      "floor": null,
      "horizontal_accuracy": 3.90000009536743,
      "location_source": "fused",
      "location_timestamp": 1592434582000,
      "session_id": "7aa745ba-42be-4b23-acc8-11a65d511740",
      "signal_strength": null,
      "speed": 26.8384494781494,
      "vertical_accuracy": 100,
      "created_user": "zzz",
      "created_date": 1592434625398,
      "last_edited_user": "zzz",
      "last_edited_date": 1592434625398
    },
    "geometry": {
      "x": -81.03567099999998,
      "y": 25.857096300000023
    }
  }
]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: location_timestamp vs. created_date

JamesCrandall — Wed, 01 Jul 2020 18:58:48 GMT

Hi Aaron -- hope all is well.

Is there any update regarding the documentation on this issue?

We're now seeing a much larger lag/delay in some instances and trying to determine if this is an app setting, device/OS type, connectivity issue or if there's something on the ESRI side of things that may cause this? Recently we noticed a specific instance with a 4 day delay between location_timestamp and created_date:

Re: location_timestamp vs. created_date

Anonymous User — Wed, 01 Jul 2020 19:38:26 GMT

We just updated the doc today:

Use tracks—Tracker for ArcGIS | Documentation

4 days is definitely abnormal. It's most likely a connectivity issue.