Ah summer is here! Which means going to the beach, attending parties, staying cool, and of course, squashing bugs and enhancing features in ArcGIS for Power BI. Oh and I think the Esri User Conference is coming up!
As organizations look to improve efficiencies by leveraging robotic process automations, many are adopting Microsoft’s Power Automate to support their automated workflows and data pipelines. We have started building the ArcGIS Connectors for Power Automate to help extend the Flows to leverage your geospatial content.
The biggest consideration to be aware of when starting to build Flows in Power Automate using the ArcGIS Connectors is record count. Record count is important for a few reasons:
HTTP request time limits in Flows
Maximum of 5000 iterations the ‘Do until’ control can handle
Getting consistent results
Power Automate has HTTP request limits and timeouts for connector operations that take longer than two minutes. Geospatial can have thousands of records and when working with complex geometry, the size of the data grows, and processing can take a while. To prevent the connectors from timing out, the ArcGIS and ArcGIS PaaS connectors have been built with 150 record count limits to help prevent timeouts from occurring.
As we know, many datasets we work with have more than 150 features, but the ‘Do until’ control allows us to iterate through the data in 150 record chunks.
There are a few steps to getting the ‘Do until’ control set up to step through your dataset:
Initialize a variable to track the offset as the Flow iterates through the records
Wrap the logic for each record in a ‘Do until’ control
Update the offset variable with the current offset of the record you are working with
This pattern will allow the flow to iterate through the records with less concern of the control timing out.
For those new to programming concepts, below are a few resources to provide some additional understanding:
One thing to be aware of, the ‘Do until’ control does have a maximum number of iterations that it will run. By default, the control is set to run 60 times. The count is configurable and can be set to run a maximum of 5000 times. For other configuration options and design considerations, check Microsoft’s documentation.
Finally, record count is an important consideration for consistent of results. In our testing, we have seen discrepancies in the result counts when working with large datasets and have found working with data that is 5000 records or less has much more consistent results.
Record count and data size is a VERY important consideration when building your Flows in Power Automate.
Use the ‘Do until’ control to facilitate the iteration through the individual records.
Configure the ‘Do until’ count limit to support the number of records you expect to work with each time the Flow is run.
We've received a number of questions regarding how and when Power BI data is sent to Esri when Power BI users interact with the ArcGIS Maps for Power BI visual. I'd like to share these answers with the community so everyone is aware of when their data is - and isn't being sent to Esri for geoprocessing.
Q: There are data going to and from the ArcGIS site. I am assuming this is just address data for mapping purposes only. The question is what data is sent between Power BI and ArcGIS? And is this data being stored by ArcGIS?
A: Correct, only the data needed to geocode the address (ie, fields placed in the Location field well) are passed to Esri servers. These data are only used to generate the information used to place the locations on the map and they are not stored by Esri servers.
Q: Does ArcGIS offer a service where this data is not collected for internal use or shared with any partners?
A: As mentioned, we do not store this data. We also do not share it with partners. If data being passed to Esri is a concern, you can use the latitude and longitude field wells to map their point data instead of using the Location field well. When using the latitude and longitude field wells, no data is passed to Esri for mapping – but it does require you to have performed the geocoding process some other way.
Q: Does ArcGIS maps geo-coding have a limit on the number of waypoints that can be sent? If so, what is the limit? Is there a limit per day?
A: Limits to geocoding are on a per-map basis and can be found in our documentation here.
Q: Is it possible our address list or search history is never shared for marketing purposes to any partner of ArcGIS?
A: Address list and search history information from ArcGIS Maps for Power BI is never shared for any purpose to any partner of Esri.
Today we are excited to announce integration between ArcGIS Online and ArcGIS Maps for Power BI. This update is a part of the Microsoft Power BI Desktop May 2018 update and allows you to include feature layers and custom basemaps from your ArcGIS Online organization in your Power BI reports and dashboards. To date, only public feature layers and Esri default basemaps have been available to use in your ArcGIS Maps visuals in Power BI. Now, Level 1 or Level 2 named users in ArcGIS Online can sign into ArcGIS Maps for Power BI and start using secured content that is only available within their ArcGIS Online organization.
Maximize value by using ArcGIS and Power BI together
This is an exciting new capability for organizations that use both Microsoft Power BI and ArcGIS Online. Now it is as easy as adding your feature layers to a map visual on a shared Power BI report to distribute them throughout your organization, even to people who have no prior knowledge of GIS. (Keep in mind when sharing secured layers that report viewers will need to have the proper ArcGIS credentials to view secured content.)
Many organizations have standardized on Microsoft Power BI as their business intelligence tool-of-choice, and the ability to use authoritative location content from ArcGIS Online inside your business intelligence system will take your organization to the next level.
A new way to buy Plus subscriptions
We are also launching a new Plus subscription purchasing option with the May update. Plus subscriptions for ArcGIS Maps for Power BI are intended for Power BI users who need more capabilities than are included in the free map visual, but who don’t have or need to create private feature layers in ArcGIS Online. Since November, Plus subscriptions have been available for purchase individually, but this isn’t ideal when sharing reports with large numbers of people. With each person who views a report with Plus content needing their own Plus subscription, it can be unwieldy to have them all buying their own individual accounts.
Today we are launching the ability to purchase Plus subscriptions for your entire organization in one transaction. This will allow anyone in your organization that uses Power BI to view and share map visuals with Plus content. To find out full details about what’s included with a Plus subscription, or to contact Esri about purchasing a Plus subscription for your organization – head on over to ourproduct page.
For more details about ArcGIS Online integration or Plus subscriptions for your organization, take a look at theMicrosoft blogwe co-authored with Senior Program Manager Lukasz Pawlowski from Microsoft.
To review detailed system requirements and get our full documentation on the ArcGIS Maps for Power BI visual, head over to ourhelp site.
Scott and the entire ArcGIS Maps for Power BI team
Our recent webinar "Five Ways to Use Maps to Boost Your Business Intelligence" showcased ArcGIS Maps for Power BI in Microsoft Power BI and featured ways map visualizations helped boost BI for better decision making. Lots of great questions were submitted during our live broadcast and we've compiled them below, grouped into topics for easy reference.
Q: Can you clearly define what is necessary for an ESRI subscription and a Power BI subscription so that there is a seamless connection to use ArcGIS while using Power BI?
A: You do not need a Power BI paid subscription to be an Esri Plus subscriber. You can be a free user of Power BI and a Plus subscriber for ArcGIS Maps for Power BI. You DO need to be signed in to Power BI for Esri to properly track identity — but again, you can be signed in to Power BI with a free account.
Q: Can you consume ArcGIS map/feature services?
A: ArcGIS Maps for Power BI can use publicly-shared feature services from ArcGIS Online as reference layers in your maps.
Q: Is it possible to insert webmaps from ArcGIS Online in Power BI? A: Currently it is not possible to use webmaps with ArcGIS Maps for Power BI.
Q: Can you connect to data in SDE or an internal Enterprise web map service? A: Currently the only supported reference layer types are publicly-shared features services on ArcGIS Online.
Q: Is ESRI maps supported by Power BI embedded? A: Not at this time.
Q: What is needed for users of Power BI to publish custom maps to ArcGIS online? A: To publish a custom layer to ArcGIS, you would need an ArcGIS Online organizational account with publishing privileges.
Q: For ArcGIS Online, can you use feature layers for analysis or only visualization? And can you save/publish from Power BI to ArcGIS.com? A: Reference layers can be used for visualization and to make selections from your Power BI data. And no, you cannot publish back to ArcGIS.
Q: Whenever I embed my ArcGIS maps in Power BI Server, I am no longer able to select the edit functionality and change the map features. Has ESRI been working with Power BI to implement this functionality? Is there a timeline on that functionality? A: We are working to enable ArcGIS Maps for Power BI in embedded scenarios but we have not announced a timeline for that capability.
Q: Is "ArcGIS" only ArcGIS Online? If so, only public layers? Can we log into our own ArcGIS Online site and use ours? A: Currently ArcGIS Maps for Power BI only connects to ArcGIS Online. Only publicly-shared feature services are searchable at this time. It is our goal to allow ArcGIS customers (both Online and Enterprise) to sign in and use their own private basemaps and reference layer. We anticipate this functionality in 2018.
Q: What features are exactly available with a plus subscription to ArcGIS Maps for Power BI? A: A handy matrix comparing Plus and standard capabilities can be found at esri.com/powerbi
Q: When will I be able to use custom ArcGIS maps and not only public? We have our own maps we would like to use in Power BI. A: We are targeting 2018 to enable the capability to sign in and use private basemaps and reference layers from your existing ArcGIS organization.
Q: Can Power BI access ArcGIS Online data that is private? A: Not at this time — this functionality will be coming in 2018.
Q: Can we use our own basemaps published from our ArcGIS Server? A: Not at this time.
Q: Can you use existing GIS server services like vector and cached basemaps? A: Not at this time.
Q: Can you bookmark layers and pins? A: No, but that's an interesting idea!
Q: Would I be able to change the RGB of the color like in ArcGIS? A: No, this visual is designed to be easy to use by people of all skillsets so we have reduced some of the complexity found in tools like ArcGIS.
Available Data and Custom Layers
Q: Do you have maps that show Election Districts for each state? A: I just did a quick search and it appears we have Population Density by Congressional District in the Living Atlas. There may be many other election district layers in the "ArcGIS" reference layer tab, or you could even publish your own to ArcGIS if what you're looking for isn't there.
Q: Is there demographic information on things like hotels, manufacturing locations, schools, etc.? A: There are thousands of infographics variables available to use, including hotels, manufacturing, schools, and more. Specific availability of variables is country dependent (not all variables are available for all countries), so I suggest you sign up for a Plus subscription (which includes a free 60-day trial) and check it out for yourself!
Q: Will we ever be able to create our own custom regions (e.g. Power Grid Operator regions i.e. multiple US states) with ArcGIS Maps for Power BI? A: You can create custom regions in ArcGIS Online and publish them for use in ArcGIS Maps for Power BI today. The catch is they must be shared publicly. Private reference layers will be tied to ArcGIS named-user sign-in, coming in 2018.
Q: Will we also learn how to create our own layer, e.g. sales territories? A: That's beyond the scope of this webinar, but it's a great idea for an advanced webinar in the future!
Q: Are demographic layers part of ArcGIS Maps for Power BI? Or is a plus subscription required? A: 10 US-based demographic layers are included in ArcGIS Maps for Power BI for free. A Plus subscription will give you access to additional reference layers from Esri's Living Atlas.
Q: For drive time, where is the ND service coming from? A: All services used by ArcGIS Maps for Power BI are Esri geoprocessing services.
Q: Are drive times smart enough to figure out 5:00 pm drive time vs. 11:00 am drive time? A: Not at this time.
Q: Can you do 60-minute drive times? A: No, drive times are limited to 30 minutes.
Q: Can you have multiple pins? Multiple drive time areas? Show multiple layers, i.e. dealers and customers? Publish sales territories to Esri? A: Yes, you can have multiple pins and drive time areas. You can have two layers at a time — one from your Power BI data and one reference layer. ArcGIS Maps for Power BI only consumes content from ArcGIS, it does not publish anything back to ArcGIS Online.
Q: Will the drive time feature be available through the Power BI server? A: We haven't commented on if or how we will provide support through the Power BI Report Server.
Q: Is it possible to overlay more than one drive time? For example, a 30-minute range and 1-hour range? A: Not at this time.
Q: When will the drive-times become non-persistent? If I change locale focus, I have to readjust the drive time on my map. A: Drive times are designed to be persistent features on the map. If you have other ideas on how you'd like to see drive times behave, submit an idea in our community: geonet.esri.com/groups/powerbi
Power BI Embedded
Q: Is there a timeline for the embedded functionality? A: We are anticipating support for embedded functionality in 2018.
Q: Will these services be coming to embedded? If so when can we expect these changes? A: We are anticipating support for embedded functionality in 2018.
Q: Why do we get failed to locate 2 features error when we use this visual and how do we fix it? A: We'll need more details to help you out. Come join us in our community at geonet.esri.com/groups/powerbi and
we'll try to figure out what's going on.
Q: Do we need a license to geocode? A: No license necessary to perform geocoding. Out of the box you'll have access to 1,500 geocodes per map; with a Plus subscription you'll have access to 5,000 geocodes per map.
Q: How does the 5,000 geocodes per map compare to the MS standard map visual? A: Out of the box with ArcGIS Maps for Power BI you'll have access to 1,500 geocodes per map; with a Plus subscription you'll have access to 5,000 geocodes per map.
Q: Does Esri offer a service that converts addresses to lat/long? Where do we find pricing information on this? A: Esri does provide capabilities to convert addresses to latitude/longitude. It is a basic capability of ArcGIS Online (www.arcgis.com) or if you want to do it programmatically as part of a workflow, you can use ArcGIS developer API's (developers.arcgis.com). The process of converting addresses to latitude/longitude is called 'geocoding', so search for that on those pages.
Q: Scott mentioned a 10,000 limit per map of lat/long location pairs. Is there a way to exceed this limit? A: No, 10,000 is a hard limit at this time.
Q: Is the Geocodes (1,500) also available for Portugal? The same question for post codes. A: Yes, we provide global address geocoding and boundary matching (and that includes Portugal).
Q: Let’s say I have a map with 100,000 points for a state, I understand that 1,500 will show up. However, if I ZOOM into say, a specific city where only 200 points exist in the dataset, will the 200 become visible? A: Zoom level will not affect which data is loaded onto the map, but you can use slicers and other visuals to restrict the dataset by county / ZIP code / etc. to more fully show the data in a smaller geographic area.
Q: If you cannot connect to feature classes, can you connect to SQL Server tables with a column of type geometry or geography? A: ArcGIS Maps for Power BI does not support geometry or geography types stored in SQL Server.
Q: Can you use WKT in the location field? A: No, I don't believe WKT is supported.
Q: Can we use our own geocoding service to get around the number of records limitation (1,500 or 5,000)? A: Not at this time.
Q: Is the 1,500 Geocode daily limit limited by a 24-hour period or a calendar date period? For example, if we run 1,500 at 3 pm, can we run another 1,500 at midnight or will we have to wait until 3 pm the next day? A: The limit is a per-map limit, not a per-day limit.
Q: How do I know all my addresses were mapped? What if I have some bad or brand new addresses in my data? A: If some records were not able to be mapped, an error message will display along the bottom of the map area telling you how many records were not mapped. As your data changes, Power BI broadcasts new data to the visuals you are using. When ArcGIS Maps for Power BI receives new data from Power BI, we attempt to map it (as long as no limits have been exceeded).
Q: How many points does the heatmap geocode? If I have a set with 100,000 unique addresses, does the heatmap process all of them or only 1,500? In other words, does the density consider all the locations? Or only 1,500? A: The heatmap and cluster map themes work with the data that is on your map. If you add 10,000 locations via latitude and longitude, the heatmap will work upon those 10,000 addresses. If you attempt to load 100,000 addresses and only 1,500 of them are returned on your map — the heatmap map theme will only work on the 1,500 locations on the map. The heatmap is only applied to the locations that were actually mapped.
Q: I've been asked to map loyalty card members around our stores. A typical well-established store has 20,000 or more members. I used lat/long and my map still got cut off. Are there any plans to increase the number of data points we can use? I will be interested in learning the tips and tricks, as there may be some segments we could identify to break it up. A: 10,000 is the maximum number of points we can display with good performance. I would suggest using slicers or other visuals on your report to segment the customers into smaller sets, if possible.
Q: Is there also a service to derive lat/long from our address data, so that we can use those next time instead of geocoding every time? A: Esri does provide capabilities to convert addresses to latitude/longitude. It is a basic capability of ArcGIS Online (www.arcgis.com) or if you want to do it programmatically as part of a workflow, you can use ArcGIS developer API's (developers.arcgis.com). The process of converting addresses to latitude/longitude is called 'geocoding', so search for that on those pages.
Q: Does the order of address info matter? A: For US addresses, we expect Address, City, State, ZIP fields to be joined together into a new field in that order.
Q: Is there a way to map addresses around specific locations — so perhaps clients' and employees' surrounding office locations? A: If you only want certain parts of your dataset to be mapped at any given time, you can use other visuals such as slicers to filter what is shown on the map. For example, you could set up a slicer on a ZIP code field and filter to show only specific ZIP codes around the stores you're interested in.
Q: Do you support three-word addressing? A: Not at this time.
Q: The content provided is available for other countries than US, like Portugal? A: Plus subscriptions provide access to global infographics and the Living Atlas which also contains global reference layers.
Q: Does this default behavior work for Canadian addresses as well? A: Yes, the location field well handles global addresses or boundaries.
Q: Is ArcGIS Maps for Power BI limited to the US or does it work worldwide? A: ArcGIS Maps for Power BI works worldwide.
Q: What do we do with International addresses where there is no zip code? Also, how do we get latitude/longitude? A: International addresses should be formatted according to local custom. Our backend world geocoding service will process them appropriately. If you want to pre-process your addresses to turn them into latitude and longitude coordinates, you can upload your dataset to ArcGIS Online (requires an account) and export the coordinates. More info here: https://doc.arcgis.com/en/arcgis-online/reference/geocode.htm
Q: How can you show the legend for the underlying data in the reference maps? A: Map legends are not currently a part of ArcGIS Maps for Power BI. The interactive nature of the map visual allows you to interact with each location to find out more information about it if you choose.
Q: How do we make the legend types visible on report so that we know blue color is for car and red is for another vehicle? A: Map legends are not currently a part of ArcGIS Maps for Power BI. The interactive nature of the map visual allows you to interact with each location to find out more information about it if you choose.
Q: Is there an increase in performance in the Plus subscription? A: There isn't a performance increase, but you do get more geocoding per map with a Plus subscription.
Q: Is the Plus subscription included in with Power BI Pro license? A: No, Plus subscriptions and Power BI Pro subscriptions are completely separate.
Q: How much is the Plus subscription? A: Plus subscriptions are $5 per month per user.
Q: Is it possible to load polygon Esri shapefile? With no zip field? A: It is not possible to load a shapefile directly into ArcGIS Maps for Power BI.
Q: Is there a way to map polylines for roads? A: Currently it is not possible to map to polylines. Thanks for the feedback, we are evaluating a few options to enable mapping to custom geometries.
Q: If you have the Plus subscription, then are you able to share BI reports via the web to users that don't have the Plus subscription? I am assuming the answer is yes? A: To author maps with Plus content, you need a Plus subscription. To view maps with Plus content, you need a Plus subscription.
Q: Is it possible to insert layouts and structures in map (polylines)? A: If you publish polylines to an ArcGIS Online feature service and share it publicly, you can add it as a reference layer.
Q: When you have your data displayed on the map and you hover over a point or polygon, is there a way to display more than one attribute in the popup? We want users to be able to view multiple attributes within each data point. A: Yes! I didn't cover it in the webinar but there is an additional field well called 'Tooltips'. Any field you drag into the Tooltip field well will be displayed in each pop-up.
Q: Can I use your product without Power BI? We are looking for a "Mapping" app but can't use Power BI. A: ArcGIS Maps for Power BI requires Power BI, but Esri has many ways to enable mapping. If you use Excel, for example, we have an app called ArcGIS Maps for Office that enables you to create maps from within your spreadsheets: http://www.esri.com/software/maps-for-office
Q: Our maps from our Cityworks program is in State Plane vs Lat/Long. For state plane eastings and northings to show up, they have to be converted using R script. Is there another solution? A: Our visual currently only accepts coordinates in latitude/longitude format.
Q: Can hosted REST services be used in analysis or is it just for reference? A: With this visual, what you see is what you get. There are no extensibility points for further analysis capabilities.
Q: Can I use your product within SAS or we must use Power BI? A: The ArcGIS Maps for Power BI visual only works in Power BI.
Q: Can you publish this to SharePoint site? A: Not at this time.
Q: Can we share our maps within a secured environment so that only people within our organization can view them? A: Yes. Map visuals are part of Power BI reports and security of those reports is handled by Power BI and your organization. You get to choose who sees the reports you create.
Q: Would you add the Spatial Join in future? A: It is something we are considering.
Q: Can you connect to feature classes in a SQL Server database? A: No. The only ways to add your location data to the map are through the Location field well, the latitude and longitude field wells, or by adding a reference layer from ArcGIS.
Q: Is there a way to use Esri-supported Global Transportation Information (traffic, speed limits, turn restrictions, etc.) in Power BI or does the data need to be purchased for an additional fee? A: No, there is currently no way to use the Esri Global Traffic Information inside ArcGIS Maps for Power BI.
Q: What kind of information can be accessed in the transportation layer, i.e. railways, highways, waterways? A: I'm not sure which transportation layer is being referenced in this question — feel free to stop by our community on GeoNet to discuss further! community.esri.com/groups/powerbi
Q: Is it available in the Azure Government Cloud? A: Yes, Microsoft confirmed that it should be available in the Azure Government Cloud already.
With the ArcGIS Maps for Power BI visual Esri has delivered great value into Microsoft Power BI. Starting today you can get even more content, capabilities, and value with a Plus subscription.
When you become a Plus subscriber, you'll gain access to:
More basemaps (including satellite imagery, topographic, and more for a total of 12 basemaps you can use in the visual).
More geocoding. Plus subscribers can geocode 5,000 addresses per map (up from 1,500 for everyone else).
Esri's Living Atlas, a browse-able collection of global reference layers.
Thousands of global infographics showing demographics and spending patterns.
What happens when I share a map with Plus features?
A Plus subscription is required to author and view maps with Plus content. If you share a map with Plus content with someone who is not a Plus subscriber, they will not be able to view the map until they also become a Plus subscriber. When authoring a map, keep an eye out for the Plus icon - it will help you know if you're adding content that can only be seen by other Plus subscribers.
How do you get it?
You'll notice a new icon in your map visual starting today. Click on the 'Sign up' button inside this new icon to go to the Esri site where you can enter your contact and credit card information. After successfully signing up for Plus you will get 60 days free. Your new Plus subscription will be $5/month after the free 60 day period.