UserR! 2020 is coming to St Louis in July and this year will have a specific track and activities for the geospatial community. If you've done something interesting with R and ArcGIS please join is in St Louis to share your work in a 5 minute lightning talk or 20 minute presentation. The deadline for abstracts is February 3rd. This is the week before the annual Esri User Conference in San Diego, so you're coming from overseas you can go straight to San Diego after St Louis. Again this year at UC we will hold a full day hands-on preconference workshop on using R with ArcGIS.
Jonathan L Wilson, PhD is partner and Chief Data Scientist for Retail Scientifics with over a decade of experience delivering best-in-class custom predictive models for a wide variety of clients. Zack Garza is the Sr. Math Wizard for Retail Scientifics specializing in Python, R, and mathematically intensive projects.
When it comes to solving the problems we face in the world today, we need the best available tools. Often, that means needing to integrate multiple platforms together. Unfortunately, this task can be unnecessarily time-consuming and complicated, especially when specific pieces do not ‘play nicely’ together. Additionally, with the ever-increasing size of data available to analyze and the demand for automated methods to generate the latest results instantly, analysts and data scientists alike are finding their time increasingly stretched thin. ArcGIS seeks to help mitigate these issues by providing a framework for integration between platforms at multiple levels.
The free and open-source R ecosystem is one of the most widely used statistical programming languages. It provides a large collection of predictive machine learning algorithms for data scientists and analysts to leverage, along with an active and vibrant support community producing high-quality documentation and resources. As such, R tends to be on the forefront of cutting-edge predictive analytics methods and novel field-specific statistical algorithms. The GIS-focused software ArcGIS, offers a powerful suite of vetted spatial analysis methods combined with an extensive platform to ease a variety of tasks that utilize its rich spatial data, dynamic mapping, visualization capabilities, and integration with Python.
Here we present one such example of an integrated solution designed by Retail Scientifics to leverage the power of R from within the ESRI ecosystem using ESRI’s Web AppBuilder framework as a web-based front-end application. This approach allows data scientists to develop high quality technical models within R, while simultaneously allowing non-technical users to employ these models via a user-friendly, spatially-enabled interface.
Example: Retail Site Selection
Predictive modeling in areas such as sales forecasting, marketing, and operations analysis is a necessity to thrive in today’s business environment. A significant competitive advantage can be gained by combining such modeling with spatial data. For example, a common scenario in the retail world is a desire for a retailer to expand by opening new locations. As this typically requires significant financial investments, predicting future performance is a key way of prioritizing which locations might yield the most potential revenue. With such revenue forecasts in hand, one can be much more confident about the possible relative performance of a location before making high-risk, capital-intensive decisions.
Approaches to revenue forecasting have evolved considerably over time, from simple summaries of population and income data, to computationally-intensive ensemble modeling techniques that incorporate historical trends and performance. In the presence of big data however, what might have been possible for a traditional analyst to accomplish with simple techniques, such as linear regression, now requires considerably more programmatic horsepower to obtain accurate forecasts.
In support of the more contemporary approaches, each potential retail location can be enriched with thousands of demographic variables, as well as a host of other spatial data such as expenditures and location attributes -- all of which can be used in the construction of a predictive model. It is a complex task to ensure that the model is built correctly, particularly when working with a large volume of data that is typically highly correlated, but such models can yield highly accurate forecasts that can quantitatively inform enterprise decisions and create immense value.
The front-end interface using ESRI’s WebApp Builder marries up easily to use cloud-based services with the high-powered R-based models developed for predictive modeling. This creates a useful bridge allowing for data scientists to build complex models which can be leveraged by non-technical individuals within the business.
Building a Model
Examples of powerful algorithms commonly used for forecasting in enterprise applications (such as revenue prediction, marketing applications, operation analysis, pricing optimization, and more) include, but are not limited to, regularized generalized linear models, neural networks, and spatial regression. While a number of these algorithms are integrated into the ArcGIS platform, niche libraries tailored for specific prediction types are easily available as R packages, which can be installed with a single line of code.
For those interested in building models directly, please see the Github repository for a more detailed view of how such models can be constructed and used to generate forecasts. For demonstrative purposes we utilize one of the simplest type of models in this example: an ordinary linear regression. However, for actual applications and greater predictive accuracy, it is advised to test the application of more contemporary algorithms and techniques.
Using the Model
To solve the challenge of delivering a complex and computationally intensive predictive model to non-technical users, Retail Scientifics has developed an easy to use web-based front-end built on Web AppBuilder, which calls a cloud-based API that references R code for the model. Once a model is constructed and properly calibrated by a skilled data scientist, this approach allows the model to be leveraged by a broad range of users through a simple form interface.
We present an example of this integration below for a multi-outlet sales forecasting application, along with a live demonstration that can be accessed here. For a user to obtain a new prediction, they simply click on the map to identify a location to execute the model, enter a few key site characteristics, and click the “Run Estimate” button.
The R model is then called behind the scenes via the API and returns structured data back to the ESRI front-end, which includes the dynamically generated estimate from the predictive model. In production deployments, the output is often quite customized, including more spatial and client-specific data, which we then structure into various charts and tables to yield a comprehensive report:
The above example highlights how powerful functionality can easily be accessed in an approachable and user-friendly format. Such a workflow can also be expanded to utilize the R-ArcGIS bridge. The arcgisbinding package, for example, offers the ability to easily convert between a variety of ArcGIS data types, including file-based geodatabases, layer files, hosted feature layers, raster layers, and more. This package extends the functionality of existing spatial R packages, with the added benefit of being able to perform custom data manipulations like subsets, selections based on SQL queries, and reprojections, all within the same function call.
Integration across multiple platforms allows for the development of analytical tools that are not only powerful in their methodology, but also easy to share and approachable for users. This is a great example of how integration helps drive us forward to solve bigger problems in new ways. For further discussions about what the R-ArcGIS bridge can do for your workflows, check out our GeoNet community or feel free to send an email. If you have questions or would like to learn more about this solution, do not hesitate to get in touch with Retail Scientifics.
About Retail Scientifics
Retail Scientifics is a boutique data science consulting firm. Many of our clients leverage our custom-built spatial analytics models and tools through ESRI platforms. We are experts in modeling, data collection and analysis, and are the market leaders in prediction accuracy. We have expertise in combining spatial and statistical analytics, particularly in the retail and restaurant domains.
Esri’s 2018 UC was a spectacular event for the R-ArcGIS bridge. Not only did the bridge unveil its support for raster data this year, along with the release of multiple new resources, but also, session turnout was at an all-time high. 2018 marked the first year the R-ArcGIS bridge team offered a full-day, hands-on, preconference seminar on the bridge, Statistical Spatial Data Analysis with R and ArcGIS, which featured all the latest regarding using the bridge within R, ArcGIS, Jupyter notebooks, and alongside Conda and Python. As the thrill from UC transitions back into daily routines, there is no better time to catch-up with the latest developments on the bridge and to build it into your workflows. Let the R-ArcGIS bridge help you expand your analyses by bringing in the latest statistical and field-specific methods from R and by making sharing your results easy.
If you have already been working with the R-ArcGIS bridge and are eager to dive into the latest advancements, make sure you update your arcgisbinding package to the latest version to checkout several new functions. We have also updated our documentation included in R/RStudio to provide even more details and examples on how you can utilize the bridge in your workflows. To learn how you can do this and to access this documentation, see our latest resource on installing and setting-up the R-ArcGIS bridge, which includes details for every different installation option, along with information on how to update the bridge.
The bridge represents an exciting frontier into the full integration of ArcGIS’ spatial analysis power with novel, and field-specific statistical analyses from R. Stay tuned for more posts with details on new ways to make this integration even more versatile with support for big data and with the ability to work exclusively in R.
The R-ArcGIS bridge was recently featured in the live training seminar, Go Deeper with Data Analytics Using ArcGIS Pro and R. The topics covered included, how to easily transfer data between ArcGIS Pro and R, an open-source programming language for statistical analysis, and how to access R’s powerful statistical functions from within ArcGIS Pro. This functionality of the bridge allows you to perform an analysis unique to R in complement with ArcGIS tools, and enables streamlined workflows and simple sharing of advanced workflows. Additionally, R users were shown how they can use the bridge to easily access geospatial data and to take advantage of the advanced visualizations and geoprocessing capabilities of ArcGIS Pro.
This seminar isfreely availablefor all to view until September 30th. Additionally, all the resources used in the seminar have been made available for you to access and to learn from. These materials can be found by navigating to theGitHub home page for the R-ArcGIS Communityand clicking on the r-sample-tools box. It is here that you will find the data, scripts, and some documentation for the live training seminar.
For those looking for extra examples, additional sample scripts and data to be used with the bridge for tools like model based clustering and semiparametric regression can also be foundhere. In addition to toolscreated and shared by the R-ArcGIS bridge user communityfor others to use and learn from. We are always excited to feature community contributions to the R-ArcGIS bridge project, so as you create your own script tools, consider contributing to the GitHub project for the bridge for others to benefit from as well.
Finally, if you are considering getting started working with the bridge for the first time, check out our new installation instruction videos for your current version of ArcGIS Pro or ArcMap.
ArcGIS Pro 2.0+
This video is currently being processed. Please try again in a few minutes.
At this year's UC, the R-ArcGIS bridge continued to revolutionize the way we think about our workflows by showcasing its ability to incorporate novel analytical methods and to streamline the process of integrating R functionality into ArcGIS.
The power of the bridge goes beyond being able to easily transfer data between ArcGIS and R and vice versa. The bridge allows us to wrap R functionality directly into ArcGIS geoprocessing script tools which enables concise analysis processes and the ability to share R functionality with anyone in your organization. Additionally, the bridge eases certain tasks when working with GIS data in R. The R-ArcGIS bridge offers R users the ability to work with all types of spatial data, everything from shapefiles, geodatabases, tables, feature services, and more, along with a simplified process for reprojecting data and even the ability to generate data from specific statistical distributions and directly write it into ArcGIS.
Here is one of the demos given at the UC that shows off some of this functionality.
Support for learning about additional functions included in the R-ArcGIS bridge's R package,arcgisbinding, can be found in thepackage's vignette. If you are interested in learning how to get started with the bridge for other versions of Pro or ArcMap, you can find installation information on thebridge's website. If you are looking for guided resources on using the bridge's basic functionality, check out ourLearn Lessonon the bridge, ourintroductory web course, or join us for ourlive training seminaron the R-ArcGIS bridge on August, 31st. Finally, if you are interested in more advanced functionality, like creating script tools, check out oursecond web courseon this topic.
Happy bridging and stay tuned for more updates on the bridge coming in Pro 2.1!
As we close in on our 2 year anniversary in San Diego next week, we have a lot of new things to share with you and plenty of opportunities for you to learn how ArcGIS users can leverage the analytic capabilities of R.
This past month, the R-ArcGIS bridge showcased its capabilities on the main stage. At Esri’s 2017 DevSummit, the bridge was demoedalong with ArcGIS’ conda-based python integration to show users the power and possibilities the two provide.
To provide our users with easier access to the R-ArcGIS bridge and all of its capabilities, we have been working hard on multiple new resources. Our goal is to not only enable users to quickly get up and running with the bridge and all of its features, but also to inspire users with new ways the bridge can extend their workflows.
The first of these resources is a new Learn ArcGIS lessonand video called Analyze Crime Using Statistics and the R-ArcGIS Bridge. The lesson walks you through installing the R-ArcGIS bridge, getting everything up and running, as well as the process of doing an analysis where you’ll seamlessly move between R and ArcGIS, depending on the analytical methods you want to employ. By the end, you’ll be ready to tackle your own analyses harnessing the power of both R and ArcGIS in new and exciting ways.
Additionally, two new web courses have been created. The first of which, “Using the R-ArcGIS Bridge“, is designed to show you the basics of installing the bridge, transferring data back and forth between ArcGIS and R, and features a simple analysis utilizing R. The second course, “Integrating R Scripts into ArcGIS Geoprocessing Tools“, specifically focuses on how to use the bridge to wrap R functionality and create script tools that can be used in ArcGIS, just like any other tool.
Check them all out and please let us know of any feedback you may have!
It’s been about a year and half since we released the R-ArcGIS Bridge on GitHub, and we just passed 6,000 downloads. Thanks to all who’ve provided feedback to help us continually improve the project.
Last year, Esri joined the R Consortium, a coalition of software companies focused on supporting the R community in its efforts to maintain, distribute, and improve R software. Esri’s R Consortium membership will allow us to better support the needs of the community and further strengthen our relationships with other consortium partners, including Microsoft and IBM, that actively support the R open source project. Esri’s membership made the list of biggest R stories of 2016 and was also announced in a Computerworld article.
On the technical side, there are a few new things in the works worth mentioning. Using the new ArcGIS API for Python, you can now use R in Jupyter notebooks along with your other scripting. The bridge also works with R tools for Visual Studio (RTVS) and with the high-performance Microsoft R Open.
There’s also a new toolbox project and tutorial available developed by Francesco Tonini called CHANS (Coupled Human Natural Systems Tools). It uses the FactoMineR and missMDA R packages presented as ArcGIS geoprocessing tools to perform factor analysis on mixed quantitative/qualitative data.
If you have a project such as a sample toolbox and tutorial that you’d like to share with the community, reach out to us on GitHub or GeoNet. We would be happy to help you polish and publish your project.
The most requested enhancement for the R-ArcGIS project has been raster support, which is the number one development priority for the project in 2017.
Soon we’ll release the first in a series of new online training materials to help beginners get going and also training to help advanced tool developers build robust integrated solutions.
We look forward to seeing many of you at the Developer Summit in Palm Springs next month and hearing about the interesting work you’re doing.
Esri has been teaching and promoting integration with R at the User Conference and Developer Summit for several years. During this time we have seen significant increase in interest, and received useful feedback from our ArcGIS users and R users about a variety of needs and techniques for integrating ArcGIS and R. Based upon this feedback, we are working with ArcGIS and R users to develop a community to promote learning, sharing, and collaboration. This community will include a repository of free, open source, R scripts, geoprocessing tools, and tutorials.