Skip navigation
All People > BBaldwin-esristaff > GeoEnabled

GeoEnabled

6 posts

I recently put together a simple demonstration to show how easy it is to integrate IoT sensors from Disruptive Technologies with ArcGIS. Within just a few hours, you can start getting real-time updates from field devices (cabinets, transformers, server rooms, etc) into your maps and dashboards. Some of the use cases for the integration included getting real-time alerts around potentially hazardous situations (open cabinets/manholes) and assisting with operations by providing real-time alerts on non-SCADA devices.  If you missed the recent demonstration video, you can find it here.

 

IoT Dashboard

 

This blog is going to walk through the configuration that was demonstrated in the video (with help from Niklas of Disruptive Technologies... thanks Niklas!). It will take you step-by-step through the setup and provide the configuration files for getting an ArcGIS/Disruptive Technologies environment configured.

 

The first thing we are going to work on, is getting GeoEvent Server configured.

 

If you are new to Esri’s GeoEvent for ArcGIS, please take a look at some of the Getting Started material. The most basic definition of GeoEvent, is that it is a hub for real-time data that can be associated with a map. If you want to visualize real-time temperature, watch status changes on a device, or monitor weather feeds, then GeoEvent is the tool that can be used to connect A to B. 

 

In GeoEvent, you basically need to get 3 things to get the integration in place with the Disruptive Technologies sensors: define an input, create a GeoEvent service, and then create an output. We will be walking through all 3 steps.

GeoEvent - Defining the Input

The first step in GeoEvent, is to build out your input. In this case, we are going to be using the ‘Receive JSON on a REST Endpoint’ input. This is installed out of the box, so you just need to select it to get started.

 

Create input connection

 

After you select ‘Receive JSON on a REST Endpoint, you need to configure it. The only ‘gotcha’ is to ensure that you DO NOT fill in the ‘JSON Object Name’ parameter. You can see I left it blank in the configuration below. Lastly, provide a ‘Name’ and then provide a name for the ‘GeoEvent Definition’ (we will change this later on, so the name doesn’t matter).

 

Create input

 

After you hit ‘Save’, go back and edit the input you just created. You should now see a URL that was generated. It should look something similar to this:

 

https://MYSERVER.com:6143/geoevent/rest/receiver/disruptive-tech

 

This will be the URL that you will provide to Disruptive Technologies to create your Adapter. So, copy it down.

 

This newly created input is now just waiting for data/messages. Now, we need to go over to Disruptive Technologies Studio to set up an Adapter, which will send messages to GeoEvent.

Disruptive Technologies Studio

Disruptive Technologies is a company based out of Norway, which also has offices in the US. They create some of the world’s smallest IoT sensors and have a companion software platform that can be used to manage all of the sensors. You can learn more about them at their website: https://www.disruptive-technologies.com/ 

 

All of Disruptive Technologies sensors communicate, via gateways called Cloud Connectors, back to the Disruptive Technologies cloud. All data is end-to-end encrypted, the Cloud Connectors are plug-n-play with built-in cellular and the data is available both via a web application and standard APIs.

 

The web application used to manage your fleet of sensors, set up integrations and give users access, is called Studio.

Creating a Data Connector

The standard cloud-to-cloud integration between Disruptive Technologies and third-party cloud services is done via a real-time push integration called a Data Connector. 

 

For a quick introduction to Data Connectors, watch the video Data Connector in 50 seconds.

 

With GeoEvent set up to receive sensor events, it is now time to set up the Data Connector that will send the data over to GeoEvent.

 

The first step is to go to your project in Disruptive Technologies’ Studio, navigate to Data Connectors and press “Create new”.

 

New connector

 

Next, fill in a name and, most importantly, the Endpoint URL which will be the URL that you saved from GeoEvent earlier.

 

Save endpoint

 

We only want to send the events which contain sensor data, so only check these event types, and press save.

 

Event data

 

All sensor data from this Project will now be sent to GeoEvent. To verify that data is being sent and received correctly, you can open your new Data Connector and take a look at the “Activity Last 24 Hours”.

 

Enabled

 

Now, lets head back over to GeoEvent to finish defining our input and get some data on a map!

GeoEvent - Creating the GeoEvent Definition

We have our input defined in GeoEvent, we have the Data Connector created in Studio… now we just need to define the message format that GeoEvent will receive in the GeoEvent Definitions.

 

Back in the GeoEvent Manager, you should now see that you are receiving messages on your input. These should be visible on the Monitor pane. 

 

At the top of the GeoEvent Manager, select Site, and then select GeoEvent Definitions. You should see all of your input and output schemas defined here. For this blog, I created a sample ‘input’ schema that you can use which contains all possible fields (can download this at the bottom of the blog).

 

Click ‘Configuration Store’, and then select Import Configuration.

 

Configuration store

 

After you import the configuration, go back to GeoEvent Definitions, and you should see the new GeoEvent Definition that was just imported. Lastly, you need to go back to your input and change your setting for ‘Create GeoEvent Definition’ to NO and then select the name of the definition you just imported.

Defining the Output

After you get your input created, you need to send the data somewhere! With GeoEvent, you have a few different options available. You can either archive your data in the Spatiotemporal Big Data Store, or you can Update Features to provide a ‘real-time snapshot’ of your input data. 

 

For this demonstration, I created a basic feature service in ArcGIS Online. Here are the fields I included:

 

Fields

 

I could have included fields to capture the signal strength, transmission mode, etc, but my main goal was to capture the edit time, temperature, and a boolean for the object present sensor.

Creating the GeoEvent Service

The last step is stringing it all together. When you create a GeoEvent Service, you are basically creating the logic that connects your input to your output.

GeoEvent service

 

In this GeoEvent Service, I created a Field Mapper processor, to link the Source and Target fields together. 

 

The one caveat with this, is that the input is not sending any geographic data or coordinates as part of the message. So, you need to manually place the features in your map or edit their position after they get created. Now, whenever the sensors sends data, it will update the target fields you defined.

 

Now you just need to build some awesome maps, apps, and dashboards!

 

IoT Dashboard

Real-time IoT and GIS

If you have any questions around the configuration, please do not hesitate to reach out. The ability to integrate the Disruptive Technologies API with GeoEvent Server was incredibly seamless and honestly fun to do. I can’t wait to see how customers continue to deploy this technology.

 

Brian Baldwin - bbaldwin@esri.com

Niklas Norin - niklas.norin@disruptive-technologies.com

I just had a great question from a customer: "Whenever we publish or update a map service, it's a real pain to manually click 'enable pop-up' for each layer, never mind getting all of the right attributes set up and configured again".

 

If you are constantly adding the same map service to a map and basically want the same pop-up configuration each time, you should not manually recreate the same configuration...and you don't have to.

 

I have experienced the same pain and honestly, never thought about looking into making this a whole lot easier on myself. But, in looking for an answer, I stumbled across a fantastic tool for the job that is put out by Esri: ArcGIS Online Assistant (it also includes some great other functionality for copying items, updating urls, etc). If you have never visited the link, now is the time.

 

After you log in to your organization, click I Want To, and then select View an Item's JSON.

All of the pop-up configuration is saved on the feature layer level. So, select a web map that has a layer with some pop-up configurations that you never want to manually rebuild again. On the right hand side of the screen you will be presented with all of the web map's JSON. For this little exercise, scroll down to the Data section. Now, you can see all of the operational layers and also, any of the layers, which include a popupInfo parameter (if pop-ups were setup).

 

 

Now, just copy the entire Data field and save it out to a local text editor. Or, select specific operational layers to save. You can now restore or 'build' those custom pop-ups that you painstakingly built out at a moments notice!

 

As noted in the ArcGIS Online Assistant tool, the JSON Editing feature is a 'Do At Your Own Risk' exercise and you should certainly use a JSON validator prior to updating any code... so proceed with caution and make sure you understand the risk.

 

In my test, I saved the entire Data field. Next, I went back to ArcGIS Online and created a new web map with the same operational layers, but without any pop-ups enabled. After saving this new web map and going back to the AGO Assistant tools, I began editing the Data field (As always... PLEASE SAVE OUT ANY JSON YOU ARE GOING TO OVERWRITE FIRST TO ENSURE THAT YOU CAN RECOVER IF THINGS GO AWRY) and simply pasted the configured JSON in and saved it. After returning back to the new web map... all of the pop-ups were configured and ready to roll.

We can all agree, drones are pretty cool. Yet, as the 'ooo's' and 'aahhh's' start to diminish after you pull it out of the box, people start to wonder how it can provide value to your organization.

 

Just west of LA, is an area called the Cajon Pass and it plays a critical role in Southern California. Not only does a major rail corridor and interstate travel through the pass, but there are some major transmission lines travelling through as well. Just last year (2016), there was a big wildfire (Blue Cut Fire) that ended up burning 37,000 acres. The interstate ended up getting closed, rail was impacted, and the fire burned right under 500 kVa conductor. The fire was incredibly disruptive, but could have been even more so if any of the transmission lines had been taken out.

 

So, a few of us decided to take a road trip over to a small section of the Cajon Pass that didn't burn during the fire, which would provide a chance to see how vegetation could impact the transmission line in a future fire. Along with monitoring the vegetation, we wanted to accomplish a couple of things:

 

  1. Conduct tower inspections in the field using Drone2Map
  2. Build out orthomosaics to show the right of way (ROW)
  3. Create point clouds of the transmission towers
  4. Determine where vegetation could threaten the conductor

 

The goal was also to accomplish all of this with out of the box Esri tools. Recent software releases from Esri now provide customers with the ability to process, analyze, share, and view ortrhomosaics, point clouds, and inspections within core software.

 

Inspections

The first flight that we conducted was a free flight, to capture some of the detail on the transmission tower. We wanted to see how a drone could be used to get more detail than conducting a ground based visual inspection.

 

One of the cool things about Drone2Map, is the ability to use it as a field application. After landing the drone, we pulled out the SD card and started a new 'Inspection' project in Drone2Map. 

 

Drone2Map takes a few minutes to pull in all of the imagery, builds out the flights lines, and places all of this over the World Imagery Service provided by Esri. There is also an inspection viewer that opens up to let users see the detail in their photos.

 

Drone2Map tower inspectionIncludes the ability to create mark-up on the photos

 

From the ground, it would have been impossible to discern whether there were any occupants in the couple of nests that were located on the tower. Using Drone2Map in the field would have been a great way to capture avian protection information.

 

Avian nests

Ca-Caw!

 

Imagery

The next flight we conducted, was a grid. We wanted to capture photos shooting straight down at the ground, while flying over the conductor and towers. This would help us build out orthomosaics and some 3D datasets. Drone2Map provides a couple different options for how a user can build out their imagery and also provides a rapid setting so that you can see how your coverage looks while in the field. Rather then running the rapid mode, we ended up processing our imagery back in the office, so that I could build out all of the output options (DTM, DSM, and imagery).

 

Flight paths before processing
Flight paths, with ArcGIS Online World Imagery

 

Flight paths after processing
Flight paths with an orthomosaic processed in Drone2Map

 

While Drone2Map stitches together the images to build orthomosaics, it also builds a point cloud from the imagery, which is used to create the DSM and DTM. While the point cloud is not displayed in Drone2Map, users can take that LAS output and bring it into other applications (ArcGIS Pro).

 

Point Clouds

From the previous flight, Drone2Map built out an unclassified point cloud which we will use in ArcGIS Pro in a minute. But, we also wanted to conduct a 3rd flight that orbited just the towers, to see what kind of visualization we could build out for the structures.

 

Tower orbit flight with Drone2Map

Orbit flight path

 

Drone2Map point cloud in ArcGIS Pro

Point cloud from the orbit flight in ArcGIS Pro

 

I also just want to make sure everyone is clear that this point cloud was not collected via a Lidar scan or with a Lidar sensor, this is built from the imagery that was flown.

 

Vegetation

Lastly, we wanted to take some of the data that was built out using Drone2Map and use ArcGIS Pro to determine the impact that vegetation could have on the conductor.

 

By using the orthomosaic that Drone2Map built out, we can use the imagery tools in ArcGIS Pro to segment and classify the imagery. This segmented image can then be used to classify our point cloud. So, now we know where all of the vegetation is...

 

Vegetation in Drone2Map

 

Then, we can use the 3D Analyst tools in ArcGIS Pro to see what actual vegetation is within a certain distance of our conductor.

 

Classified vegetation by distance

 

Close

While drones are really cool, there are a number of great applications for there use in the utility space. In just one short morning, we were able to conduct field inspections, build orthomosaics, and create 3D data for both visualization and vegetation management. 

 

Recent software releases from Esri now provide customers with the ability to process, analyze, share, and view ortrhomosaics, point clouds, and inspections within core software.

 

Lastly, all of these outputs can now be shared to the web via ArcGIS Online or Portal! So, any of the point clouds or imagery that you produce can be shared with users across your organization.

We frequently get questions from customers about how to to style their ArcGIS Online/Portal homepages, similar to the demonstration sites that we designed for our Electric and Gas sites. This blog is going to walk through the way that we styled our homepages, as well as providing all of the source code and imagery.

 

Westerville Homepage

Westerville Power & Light Homepage, our fictitious electric utility demonstration site. 

 

Currently (as of July 2017) the ability to customize and configure an ArcGIS Online/Portal homepage feels pretty limiting. Yet, the fact that ArcGIS Online/Portal lets you override the default CSS, apply your own styles, and link out to external content gives you the ability to start with a completely blank slate. Rather than simply hiding all elements and starting from scratch, you can restyle many of the elements that already exist, which I will be walking through in this post.

 

First things first, please go and download the included CSS/HTML that was attached to this post. Also, brief caveat that I am no web designer by trade, so this CSS could certainly be improved upon syntactically (feel free to roll your eyes at any point while looking at the samples). Lastly, I prefer to use a 3rd party app (Notepad++) because as the script is pasted into ArcGIS Online/Portal, all formatting is removed.

 

So, let's start with a blank slate:

 

Start

 

This is what pretty much every organization homepage starts out looking like. Eek. So, how do we make this look better? Honestly, with a couple of code snippets and images, it takes about 5 minutes.

 

After you open the attached .ZIP file, you will find 2 HTML files: Organization Description and Banner. The sections are titled for the sections of the ArcGIS Online/Portal settings that this code can be pasted into. The standard homepage is composed of roughly 3 different elements: 1 - Top Banner, 2 - Featured Content, and 3 - Organization Description. Open up the file titled Banner.html and we will work on styling Top Banner, Featured Content, and the background.

 

Sections

 

If you are looking at the 'Banner.html' code, the first thing you will note is the ability to override many of the Esri default styles. Everything in the <style> section is pretty much overriding defaults.

 

#bottomContent  {
    background: none;
    box-shadow: none;
    border-bottom: none;
}

#bottomContent #organizationSection .top-nav {
    border-bottom: none !important;
}

 

A lot of these declarations are just overriding the box shadows, etc that Esri applies to elements. Play around with the Chome or Firebug Developer tools, and you can look at all of the default elements to see what other elements you may want to tweak. For example, you could hide the entire FeaturedMaps element and create a fully custom page if you so desired. 

 

In the code, the last two elements you will see in the <style> section are the #topBanner and #resback. For both of these, you can see that we are simply applying a background image to the DIV and then placing the element. #topBanner is placing the header banner image and #resBack is placing the background image at a fixed position.

 

In ArcGIS Online, navigate to My Organization, Edit Settings, Home Page, and set the Background Image to No Image. Then, set the Banner to 'custom design', click the code view icon, and paste in the Banner.html contents.

 

Banner

 

Looks pretty good already!

 

Now, on to the Organization Description. Open up the OrganizationDescription.html file. All of the styles associated with this file are just applying style to custom DIV elements that we can paste into the Organization Description. So, rather than just using text, we can create custom content and styles in this space.

 

What we are doing here, is just taking some basic images and applying a little bit of CSS style to get a rounded corner and a hover on a mouse over. All of these images then link out to ArcGIS Online content in the organization, or any other content you may want to link to. I have also seen folks embedding StoryMaps, Dashboards, and other content directly into the Organization Description. You can get as creative with this section as you want!

 

.customContent {       
     margin-bottom: 35px;
    text-align: center;    
}       

.groupItem img {
     background-color: rgba(119,119,119,.8);
     -webkit-border-radius: 5px 5px 5px 5px;      
     border-radius: 5px 5px 5px 5px;
     padding: 8px;
     margin: 0 10px 0 10px;         
}

.groupItem:hover img {      
     background-color: rgba(210,224,255,.4);
     opacity: .95;  
}       
<div class="customContent">       
     <p class="groupItems">      
          <a class="groupItem" href="" target="_blank" title="Distribution">  
               <img alt="Distribution" src="distribution.png" />   
          </a>      
          <a class="groupItem" href="" target="_blank" title="Transmission">  
               <img alt="Transmission" src="transmission.png" />
          </a>      
          <a class="groupItem" href="" target="_blank" title="Generation">  
               <img alt="Generation" src="generation.png" />
          </a>         
     </p> 
</div>

Now, take all of the OrganizationDescription.html code and navigate to Organization, Edit Settings, General, Organization Description, click the code icon, and paste everything in. After you hit save and go to Home, you'll end up with this great finished product:

 

Done

 

If anyone comes up with some great examples, please feel free to post them in the comments! I would love to see what people come up with. 

 

Shout out to Bernie for the initial #resBack code sample.

Gallery Applications for Curated Content Access

ArcGIS Online offers users the ability to create maps and applications for an entire enterprise. As the number of named users increases across a utility or company, the amount of content can increase rapidly and there is a stronger need for content management to ensure that users can find and access the maps and applications they need.

 

View a video of the demonstration.

 

One of the tools that can help with content dissemination and end user experience is the Public Gallery Application.  Using your group settings, the application lets administrators control how their users interact with and discover content.  In the screenshot below, a utility has Gallery Applications for all of their departments.  As a user clicks on the Operations application, it launches a Gallery Application view of the Operations group with content filtered to hide feature services, maps, and other content.  Users are no longer overwhelmed with hundreds of maps and services, just a curated list of useful tools.

1.PNG

 

Groups

While Groups in ArcGIS Online offer a structure for sharing content and controlling access, they do not provide the best framework for providing users direct access to that content.

 

Very quickly, groups can become unruly as you share services, maps, documents, apps, and more to those groups.  Asking novice users to navigate through these groups could cause confusion or frustration. The screenshot below is from an Operations group with over 60 items.

 

Groups still provide the foundation for what content users have permission to access, but they do not provide the best user interface for navigating through that content.

2.PNG

 

Gallery Applications to the Rescue

One way to control the types of content that users have access to is by creating a Public Gallery web application from the group.  Basically, using a Gallery Application allows users to create a curated view of a specific group based on item types or tags.

 

While you are in the group you want to work with, click the Share button, select Create a Web App, and choose Public Gallery.  This will create a new application for you.

 

There are a number of options to configure the look and feel of the Gallery Application, but the most important settings are in the Gallery Options tab.  Here, you can control the types of items that will appear in the application, the number of items, the way they will appear, or you can use a search string to limit what items are shown.  Using the search string option, you could place a ‘Featured’ tag on all items in a group that would act as a filter, as shown in the screenshot below.

 

In the Enable Features tab, one setting to note is the “Show a list allowing users to filter by items type”. With this setting enabled, users have the ability to filter what they are seeing to maps, apps, dashboards, or other item types.  This combined with the tag search string provides powerful functionality for providing a curated, filtered view of items to your users.

 

After you Publish your application, it’s sharing can be controlled just like any other organizational item.  As depicted in the first screenshot, you could share your Gallery Application to the Featured Content group to create a list of all departments.  Or, you could start nesting Public Gallery applications in other groups, to create a Gallery Application of Gallery Applications.   

Closing

The Gallery Applications offer a range of possibilities for how administrators can begin to create custom views of content shared into groups.  Rather than forcing users to navigate through content heavy groups, they can be provided with a clean user interface that lets them quickly discover the content they need.

 

If you have any questions, please feel free to contact me at: bbaldwin@esri.com

On the official Esri Blog's, there was an article pushed out almost 2 years ago that discussed the way users could begin using a barcode scanner with Collector to automatically populate a field.  Pretty awesome.

 

The one limitation of the article were some of the details that were missing.  "Why do I need an external scanner?" "Can I just use the device camera?" "Will the keyboard still work?"  So, what are the answers, and why 2 years later?

 

I am currently working with a few other individuals on putting together a solution for "Tracking and Traceabilty", a regulation that will be coming to gas distribution customers, most likely in the next year (read this StoryMap for more details).  In brief, the regulation requires users to capture the barcode information from any assets that are installed or updated (think gas pipes).  What better way to handle this regulation than with Collector?  Users can capture the coordinates of the feature, the barcode, installation date, and then sync all of this data directly with their production GIS, all with one device.  This was the impetus for revisiting the Collector barcode scanning capability and seeing what was possible today.

 

With the new release of Collector for ArcGIS, there is the ability to use a high-accuracy GPS and capture metadata.  So, users now have the ability to get sub-meter accuracy from Collector and view the real-time metadata of the GPS accuracy during a collection.  This is a huge enhancement, but what about the barcode scanning?  While the high-accuracy GPS pairs with and works across all devices (hardware), there were some minor differences with the barcode scanning.  I only tested 2 different types of devices for this workflow (sorry Collector for Windows).

 

Apple Devices

Using an iPad Air, I paired a Bad Elf GNSS Surveyor (thanks for providing Bad Elf!) as the external GPS unit (a separate Esri blog discusses this in detail).  The Collector Beta version (full release in June 2016) displayed the accuracy and full metadata streaming in from the unit, pretty cool.  Next, I paired a Socket Mobile CHS 8Ci as the external barcode scanner.  Out in the field (walking around the campus), I was able to collect barcodes using the trigger on the barcode scanner (using a test piece of PVC) and click on the scanner to disable it momentarily to use the keyboard to fill in other values.  All told, it worked really well.  One limitation I had, was with the type of scanner I was using, which only reads 1D barcodes (horizontal type).  The same company produces a 1D and 2D scanner, which could be used for QR barcodes and traditional, but I was not able to test with it.  Lastly, the scanner needed to be in HID mode, which basically replaces the keyboard, hence the need to click it and momentarily disable if you need to enter data via the keyboard.

 

This led me to a question, "Why not just use the camera to capture the barcode?"  While there are countless QR and barcode scanning apps, none of these integrate with the Apple keyboard, meaning, you could capture values in a separate app, copy the value, re-open Collector, and paste in the values, but that would be a pain.  Enter Android.

 

Android Devices

On a Samsung Galaxy tablet, I located an app available through a 3rd party that replaces the default keyboard on the device.  This keyboard has a 'scanner' button as part of the keyboard, allowing values to be entered directly into Collector without leaving the app and using the on-board camera.  Simply search the Google Play Store for "barcode scanner keyboard" and there are 2 apps by 'Tec-It' (make sure you install the 'keyboard/scanner' apps).  There may be other apps with the same functionality, but this is the only one I tested.  Also, I didn't test the paid version, but it looks like it can scan both 1D and 2D barcodes.  Using Collector, I was able to scan a barcode and then continue entering values using the keyboard right within Collector.  It worked really well and it was nice to only have the external GPS (Bad Elf) unit, rather than 2 extra devices I was carrying around.  Yet, to be fair, they probably weigh 6 ounces.

 

Closing

The one caveat to the Android scanner app, would be the quality of the scan.  Using the on-board camera will not give you the same consistency/quality as a laser based scanner built for one purpose only, to scan barcodes.  Keep this in mind when thinking of a solution.  Obviously, the ability to have one less piece of hardware is a benefit, but comes with a drawback in terms of consistency.

 

Both the iPad and Galaxy had their pros and cons, but either could be a great solution.  It was great to see how Collector could be deployed with both a barcode scanner and high-accuracy GPS unit to extend the types of data that could be collected and how it could be used for the Tracking and Traceability regulation.