topic Re: creating flat structure GeoEvent Definition from Verizon Networkfleet in ArcGIS GeoEvent Server Questions

creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Tue, 05 Feb 2019 17:31:16 GMT

So I am completely new to GeoEvent Server. I am trying to understand how to map verizon's networkfleet message to a flat structure. The appendix page that comes with the instructions on how to install the connector is not very informative for a newbie like me. I need some more guidance. Is there anymore detailed info out there that shows how to do this?

I have all the definitions that get created when you add the adapter. I am confused about how you get this:

"GPSFix.Speed[0].Value"

"GPSFix.Speed[1].Value"

"GPSFix.Speed[2].Value"

I am not sure what the "value" part of this is, or the number in the brackets.

Also confused about the appropriate outputs to use. I have a feature service created. Just trying to figure out how I can get my feature service updated with the location of our vehicles. The end result I want is to have one point representing each vehicle in our fleet that shows their location every so often based off our contract with verizon. Don't need any elaborate set up. just a simple output showing locations with vehicle label.

Any help with where to find more documentation, or even an example of someones Geoevent service structure that they have created, would be greatly appreciated from a newbie such as myself

Thanks.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

EricSpangenberg — Tue, 05 Feb 2019 17:47:25 GMT

Erik - we are in the same situation here. Total newbie to GeoEvent Manager and Verizon Networkfleet. I have been reading the training modules and like you, I just want to see how to create the flat structure that the Verizon document noted.

My plan here was to create the output and eventual geoevent service as a streaming service, but if there is a better option I would welcome any input.

Like you, just want to show where the assets (snow plows) are in the city.

I'll share here if I find more

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Tue, 05 Feb 2019 18:59:19 GMT

Eric

I appreciate that. I will do the same.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

NathanKemphues — Mon, 18 Feb 2019 20:17:38 GMT

Hi Eric and Erik

Thanks for asking this, it has made me finally document our primary GeoEvent service! With GeoEvent, even a simple sounding task often winds up being quite convoluted.

For your main question the type of output connector you will need is Update a Feature. This will require that you have created a featureclass in an SDE geodatabase and published it with ArcServer (or Portal?) with at least query, add, and update access.

The bit about "GPSFix.Speed[0].Value" has to do with the data coming in as an array (or series of arrays, if I understand correctly). Hooking your input up directly to a text output (JSON) will let you see what the raw data looks like. In this case, you have two levels of array: GPSFix has several values (VIN, MessageTime, Latitude, Speed, etc.) Thus you have "GPSFix.Speed." However, there are three different speed readings: instantaneous, average, and maximum (since the last message, I think). Thus "GPSFix.Speed[2].Value" will give you the number for the maximum speed; you could also ask for "GPSFix.Speed[2].Units to learn that this is in MPH.

I'm attaching a screenshot of our service and another of a JSON. I have more I can share if you like.

One word of advice: you will want to pay close attention to GeoEvent Definitions. Some processors work best if you have created the definition yourself (say, when you are using the Field Mapper to ensure that you output matches the schema of your feature class), while other processors will want to create their own (like the field enricher).

Good luck!

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

EricSpangenberg — Wed, 20 Feb 2019 18:07:07 GMT

Nathan -

Thank you very much for sharing your GeoEvent service workflow, more questions on that later.

My struggle early was that I kept trying to create our output as a streaming service, nothing worked. Early last week I decided to do what you just shared – I created an Output “Add Feature to the Spatiotemporal Big Data Store”

With that done we were able to create a GeoEvent Service – this is where the Field Mapper Processor comes into play in order to ‘flatten’ the Verizon hierarchical field structure. We created our own GeoEvent definition to match the desired fields as outlined in the Verizon training document.

In the original Source Fields for the field mapping processor the values were all GPSFix.Speed.Value. update those source fields with the respective [0], [1], [2] doing so appeared to resolve correctly.

Now, after reading Nathan's post, I’m wondering if what we need to do is create a new service that will Update the feature ?

The existing Add a Feature appears to be working in that we have points on the map now… a lot of points.

Nathan, would you be willing to share the details in the various processors you use in your GeoEvent service? The Current Time, Add Fleet Data, Pass Through…all of those have me very curious about what more we can do with this.

I know I need to read the GeoEvent Tutorials, but I’d just like to get something up for our street department.

Thank you for your response – very much appreciated.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

NathanKemphues — Wed, 20 Feb 2019 18:28:32 GMT

You are correct Update a Feature is what you want. If you only want to show trucks that are actually on the road, it has an option to delete records after a given time.

"Current Time" is a field calculator to ensure that we always have at least one field that displays in the correct timezone. "Add Fleet Data" is the 'field enricher' to processor to add data to make it more readable (our truck numbers, operators, etc.)

"Pass through" is merely a placeholder in case we make some changes ; )

Also note the filter "GPS"- verizonnetworkfleet sends out several different types of message (i.e. "Diagnostic") with different definitions. Selecting only those messages with the correct definition ensures that you are working with a consistent set of incoming data.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

MikeSchonlau — Wed, 20 Feb 2019 18:32:26 GMT

Hello Folks

We have been using Verizon NetworkFleet inputs with GeoEvent for about the past 18 months. I'm happy to share what we've learned (mostly the hard way). We currently have public snow plow maps displaying live vehicle locations around the City. Here a few notes that we have discovered along the way:

The Verizon GPS payload is really the only one we care about
For adding attributes like a vehicle id, vehicle group, or driver names, we found using the Field Enricher (feature service) is the most reliable way to go
It is a good idea to come up with a data retention policy for AVL data because it can grow large fast and it will ultimately slow your feature service down if you don't truncate/archive it

Here is some information I recently shared with another user about we setup our GeoEvent services to add vehicle id's and other custom attributes:

To solve the vehicle labeling problem, I first published a hosted feature service in ArcGIS Online (you could also publish this to ArcGIS Server). This is my lookup table that contains the VIN's and their corresponding Vehicle ID's and Vehicle Groups. Here is the schema:

And this is how that attribute table is populated:

I needed our Public Works folks to create this table for me and then I loaded their table into this feature service. The geometry of the records is not important and not used, so I just gave all of the records a geocodable address.

Now in GeoEvent, here is the service I created:

I think the data stream I am getting from Verizon is actually just the GPS payload data, so I probably don't need the filter. The Field Mapper looks like this:

You can see I'm creating a new GeoEvent Defintion that includes the new fields I want to populate.

Now I send those GeoEvents to the Field Enricher, which looks like this:

The Enrichment Fields are the new fields I want to populate using info from the lookup hosted feature service. The join is happening on the VIN field from both the incoming GeoEvents and my lookup hosted feature service.

I am then sending the resulting GeoEvents to my output feature services - one for the current location only and another for the cumulative locations:

I hope this is helpful. Feel free to contact me if you have other questions.

Mike

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

EricSpangenberg — Wed, 20 Feb 2019 19:29:17 GMT

Wow, that tidbit of information about the "GPS" filter was huge. I applied that just now and a whole series of log errors just went away!

Earlier this morning I asked our street crew to provide a file in anticipation of using something like field enricher - your comment and Michael's below are a huge help.

Thank you very much!

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

EricSpangenberg — Wed, 20 Feb 2019 19:35:55 GMT

Michael,

Thank you very much! We are learning the hard way here as well, although with responses like yours and Nathan's above, probably not has hard as yours!

I appreciate your willingness to take the time to share your experience.

One thing, the images you shared in the post above - those didn't come through, the links do not seem to show? The url is mail.google.com/mail.....

Would you be willing to send direct to me (espangenberg at lccountymt.gov)

Thank you again. I hope this has been as helpful to Erik Blake as it has been for me!

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

MikeSchonlau — Wed, 20 Feb 2019 22:38:52 GMT

I think I fixed the images. Let me know if you still can't see them.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

EricSpangenberg — Wed, 20 Feb 2019 22:46:53 GMT

Fixed - thank you! This is very helpful!

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Thu, 21 Feb 2019 13:38:41 GMT

Thanks Nathan for all your help and sharing of your workflow. Much appreciated.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Thu, 21 Feb 2019 13:39:07 GMT

Thanks again Mike

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Mon, 25 Feb 2019 20:13:18 GMT

Nathan

After I apply my GPS filter it would make sense that my In/Out counts are no longer the same, right? my In count is higher then my out count.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

NathanKemphues — Mon, 25 Feb 2019 22:03:32 GMT

Yes, exactly. The other messages (diagnostic etc) are coming in, but not

making it past the filter.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Tue, 12 Mar 2019 15:39:42 GMT

Nathan

I had a quick question about the "delete old features" option under the advance section of the update feature output.

Does this actually delete your vehicle feature displayed on the map, or just the log info that gets sent to display the feature. Sorry if this sounds confusing. I guess I don't know how to really ask the question. My point that I am trying to get at is, what is the best practice or recommended setting to have so that our service doesn't start to get sluggish or bogged down. Thanks.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

NathanKemphues — Fri, 22 Mar 2019 17:57:37 GMT

If you are using the Update Features output, then the "Delete Old" option will clear off vehicles that haven't moved in a while. Your service shouldn't get bloated with this setup: you will never have more records than you have trucks. However, if you use the "Add Feature" output, you can get a new feature record every time the GPS sends a signal. In that case, you will have a historic record, however it can turn into a huge dataset, so you may wish to limit the length of time you store them, which would be the value of "Delete Old."

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Tue, 26 Mar 2019 17:40:45 GMT

Thanks Nate

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

ErikBlake — Thu, 18 Jul 2019 12:14:47 GMT

Hey Michael, Good morning. So we just started experiencing our feature service really slowing down like you stated in your previous reply. I tried to compress our database and unregister our dataset to drop the delta tables. But it had grown so large that even trying to compress it and let it run over night didn't work, it just ran and did nothing. So I started from scratch built a new enterprise database and all that. And have now started to compress nightly or every other day. this seems to keep our feature service and the labeling performing pretty well. I guess my question fro you is what do you do to keep it from getting bogged down.

Re: creating flat structure GeoEvent Definition from Verizon Networkfleet

MikeSchonlau — Thu, 18 Jul 2019 21:02:40 GMT

Hi Erik

We typically use two methods to control this data. 1) The Advanced settings of your Geoevent output enables you to Delete Old Features and set the age of those features to truncate. 2) We make regular backups of the feature classes, so we don't ever completely lose that old data.

A specific scenario might be snow plows. We typically get snow between November and March. We create daily backups of the snow plow AVL feature classes during that period by copying them into file gdb's and zipping those gdb's (might be overkill - could get away with less frequency) using Python scripts. Then we set the Advanced properties on our Geoevent output to only keep the data we need to use for public snow plow maps and internal analysis. We can always go back to an archived copy of the data if we need to use it.

Actually, since my earlier messages we are now writing all of our NetworkFleet data to hosted feature services on ArcGIS Online and it works well. We then publish this data as hosted feature views, where we can control which fields are visible and other filters. Hope this helps.