What Is Holistic Testing?
Holistic testing is part of the Esri software development process that tests software releases using real-world workflows and datasets. By implementing user workflows as an integral part of Esri’s software testing strategy, software issues can be identified early and fixed prior to releasing the final product. Throughout the process of discovering software issues based on user workflows, Esri will also gain knowledge and understanding of customer GIS workflows and scenarios. This is a path for Esri to capture user requirements and build those requirements into the software to ensure customer software needs are met.
How does it work?
Esri will schedule a holistic testing session based on the changes in the functional areas of the software. The holistic test team will work directly with a functional development team or an industry manager to identify and coordinate a week of software testing based on feature function or a specific set of workflows for an industry. The test sessions are scheduled about 1 month in advance to ensure all logistics can be coordinated, from preparation of the software to be tested to travel and lodging considerations. Testing participants are invited to the location of the testing event, which will be at Esri Headquarters in Redlands, CA or another specified location for the duration of testing. The holistic team has 2 physical labs, at Esri Headquarters in Redlands, CA, where most test sessions will be held. Each lab is equipped with powerful machines including quad core processors with 16+GB of RAM that are pre-configured with daily builds of the Esri software and any other 3rd party software required support testing for the week. These machines are on an internal Esri network managed using a secure Windows network strategy that allows for file sharing between machines and testing participants.
*Please note: We can host test sessions at various locations based on availability at the desired location. (i.e. Esri R&D Centers; Esri Regional Offices; International Distributors)
For the week of testing itself, it will kick-off with an opening presentation co-delivered by the functional development lead/industry manager sponsoring the week and holistic team.
This kick-off presentation includes:
- Week agenda
- Lab Resources
- Software Overview
- New Features/Functionality in the software
- Features/Functionality that have changed
- Features/Functionality that have been remove
- Features/Functionality that we are interested in being tested
- How feedback will be collected
- How to track status of feedback collected
Throughout the week, testers will be supported by the functional development team sponsoring the test week and the holistic test team. This is done to ensure that an exchange of information regarding technical issues and workflows occurs. Testing participants will be working alongside the developers who helped create the software that is being tested. Testing participants will be able to get the clarity needed on how tools are designed and why, so they can be better prepared, in the future, when moving to the latest Esri software release.
Feedback is collected through a web-based system to capture all of the discrete technical issues that are found through the test session. Additionally, an end of the day wrap-up session is held to discuss all issues captured via the web-based system. The wrap-up session is held daily, in a group format so all testing participants can hear all feedback, reported from all testers, throughout the testing. Members of the development team are present to comment on the issues that have been reported. This is a collaborative environment. Our expectation is that testing participants will openly discuss the issues they have reported and share their findings with the entire group. Our development team will ask questions to ensure there is a complete understanding of the issue reported and also ask if other testers have run into the same issue while testing. This will create a conversational atmosphere where an exchange of process and technique will occur.
When does it occur?
Test sessions will occur weekly throughout the year and the software development cycle. At certain times in the software development cycle, sessions will be scheduled to validate design decisions and at other times, sessions will be scheduled to ensure the final functionality will meet the needs of the user and is free of critical technical issues. As Esri evolves in its methods of software development, the testing schedule will change and adapt to meet the needs of the testing requirements. As more web-based solutions are introduced, test sessions will be held more frequently to ensure we are capturing as much feedback as possible, prior to final release.
Who is involved?
Esri customers, Esri Staff, Esri Partners and Esri International Distributors can all be involved in any one or multiple test sessions. Our mission is to test Esri software products from a user-workflow point of view. The above mentioned GIS users represent unique communities who have GIS projects with different goals and different deliverables. We are looking for GIS users who have well-defined workflows, new GIS projects or unique workflows that push the limits of the Esri software. For any GIS users interested in participating these testing sessions, the holistic team will work with testing participants to understand the focus of the workflow that is to be tested. We will help to create a test plan for the week of testing to ensure a two-way exchange of information between the testing participants and Esri. All feedback is valuable to Esri to ensure we are building software that will meet user requirements. We want to make sure this program provides a mechanism to get that feedback into Esri so we can take an action.
What this means as a testing participant, is that at any test session, the testing participant can be working alongside an Esri staff member, a business partner that the company has worked with in the past, or company in the same industry as the testing participant.
Overall, the lab experience will be collaborative where all testing participants are sharing information and their findings and exchanges ideas for best practice. Testing participants can participate in a testing session in two different ways:
Attending an in-house Holistic Testing event at Esri headquarters in Redlands, California. As mentioned above, the Holistic Testing Lab accommodates users from around the world who use and work with Esri software and functionality that is currently under development.
Remote interaction with members of the Holistic Testing Team from your office. If travel to the Holistic Testing Lab is not possible, we can work with your team to setup remote participation.
How to get involved?
If you would like to participate in or schedule a holistic testing session for the next Esri software release, please email HolisticTesting@esri.com for more information.