Failure on writing fix time causing sync to fail

[JoeHershman]

We have an iOS application (using the Xamarin Forms API).  This uses a custom LocationDataSource to receive data from an external high accuracy GPS device.  We also use the SharpGIS Nmea parser from @dotMorten_esri to parse the incoming NMEA sentences

In general this all works without issue.  Where we do have occasional problems which then results in a failure to sync a table is with writing the ESRIGNSS_FIXDATETIME and I have no idea how this could occur.  But in a high number of the offline *.geodatabases that fail to sync when this data is exported to cvs and opened in Excel the ESRIGNSS_FIXDATETIME comes in as a number and does not get converted to a date.  The reason for bringing in Excel is because the attribute table will not open in ArcGIS Pro and this is the only way to see the bad data.

What the application does in order to send these GPS collection data is to extend the Esri.ArcGISRuntime.Location.Location class and add the additional data

public class NmeaLocation : Esri.ArcGISRuntime.Location.Location
{
	public NmeaLocation(MapPoint position, double horizontalAccuracy, double velocity, double course, bool isLastKnown) : base(position, horizontalAccuracy, velocity, course, isLastKnown)
	{
	}

	public NmeaLocation(DateTimeOffset? timestamp, MapPoint position, double horizontalAccuracy, double verticalAccuracy, double velocity, double course, bool isLastKnown) : base(timestamp, position, horizontalAccuracy, verticalAccuracy, velocity, course, isLastKnown)
	{
	}

	public Gga.FixQuality FixQuality { get; set; }
	public double Hdop { get; set; }
	public double Vdop { get; set; }
	public double Pdop { get; set; }
	public short NumSats { get; set; }
	public string Receiver { get; set; }
}

This object is then sent into the UpdateLocation method of the custom LocationDataSource and I can use the additional properties to update the GPS attributes in the feature class.  Updating the ESRIGNSS_FIXDATETIME is pretty straight forward

if (feature.Attributes.ContainsKey("ESRIGNSS_FIXDATETIME"))
{
	if ( nmeaLocation.Timestamp != null )
	{
		feature.SetAttributeValue("ESRIGNSS_FIXDATETIME", nmeaLocation.Timestamp.Value.UtcDateTime);
	}
}

This does not throw an exception, but it writes the invalid value.

Is there a way to validate this data before writing it to the table?  What else could be done?  While this does not occur frequently, it only needs happen once on a job to break the sync.

@MichaelBranscomb 

Thanks

-Joe

[dotMorten_esri]

>  But in a high number of the offline *.geodatabases that fail to sync when this data is exported to cvs and opened in Excel the ESRIGNSS_FIXDATETIME comes in as a number and does not get converted to a date

I'm not sure which process you use here to do that, but note that the local geodatabase format doesn't actually support dates and uses an epoch number instead to store dates. It's the schema of the table that tells us that the number should be interpreted as a date.

> This does not throw an exception, but it writes the invalid value.

I'm not sure what you mean by "the invalid value" ? What makes it an invalid value? In your check for whether the timestamp is null or not, can't you also validate what value it contains?
Also I'm not entirely sure why you're converting the timestamp to DateTime? Attributes should generally be using DateTimeOffset instead of DateTime. So you can replace

 nmeaLocation.Timestamp.Value.UtcDateTime

with

 nmeaLocation.Timestamp.Value

[JoeHershman]

When I say invalid it is a value, not null, that makes no sense

If I were to convert this in DB Browser:

SELECT datetime(ESRIGNSS_FIXDATETIME) From Vertices where objectid = 60

The result is:

So the update worked ok writing from the API to the sqlite data table.  But when this tries to sync because it is not a valid Date, it does not sync.  We can bring the table into Pro and the point seems to be way out in nothing space.  However, the Lat/Lon which come from the MapPoint are valid and where they should be.  While probably inefficient the Lat/Lon coordinates are projected to a Web Mercator point before calling UpdateLocation and before writing to the table the point is projected back to WGS 84 to get Lat/Lon.  Because of this it would seem the shape geometry itself is ok, but because of the other issue it shows a point out in the middle of nowhere

I should note that when this occurs many of the GNSS fields are null, so that data may not be picked up from the receiver

[dotMorten_esri]

It's not clear to me when you say this problem occurs. Does the problem occur with the location you get from the NMEA Parser, does it occur when you insert, or does it occur after it got into the database?

What did you do to validate that the location coming from the NMEA parser is valid before adding it to the table?

I wouldn't read much into what the sqlite database functions interpret that number value as. It's not what we use, and it doesn't appear like the datetime() sqlite function takes a double, but an integer I think.

> I should note that when this occurs many of the GNSS fields are null, so that data may not be picked up from the receiver

You should probably validate that the location is valid and a proper fix and timestamp before using it?

[JoeHershman]

Like I mention, because the value of lat/lon in the database are correct, it seems to me that point itself is valid.  If not how would it have projected correctly.  We also do a significant number of checks to validate the point prior to writing to the database.  Not sure what else we could do to validate geometry

protected async Task<bool> IsValidatePoint()
{
	if ( CurrentLocation == null )
	{
		await DialogService.DisplayAlertAsyncAudio...
		return false;
	}

	if ( CurrentLocation.IsLastKnown )
	{
		return false;
	}

	if ( double.IsNaN(CurrentLocation.Position.X) || double.IsInfinity(CurrentLocation.Position.X) )
	{
		return false;
	}

	if ( double.IsNaN(CurrentLocation.Position.Y) || double.IsInfinity(CurrentLocation.Position.Y) )
	{
		return false;
	}

	if ( Math.Abs(CurrentLocation.Position.X) < 0.1 || Math.Abs(CurrentLocation.Position.X) < 0.1 )
	{
		return false;
	}

	if ( CurrentLocation.Position.IsEmpty )
	{
		return false;
	}

	return true;
}

For now we are just going to not write the Fix time to the database.  Not the optimal solution, but better than not being able to sync.  May revisit this later if it is decided by the client it is needed for auditing the data

[dotMorten_esri]

> Like I mention, because the value of lat/lon in the database are correct, it seems to me that point itself is valid. 

I'm a little confused here. You mentioned the issue was with the time stamp you're writing, not the geometry. Have you (in addition to validating the geometry), validated that the time stamp you're getting from your custom NMEA datasource is actually correct?