agile in a regulated environment

the challenge

Agile software development is about delivering software in a continuous process of incremental improvement. Delivering in small increments requires tight integration between the users and the developers who write the software. "Agile" attempts to achieve this by getting software in front of the users quickly and obtaining their feedback for the next round of development. In doing so, it dispenses with the process of gathering extensive requirements into large documents, developing and testing software against the requirements, and releasing a large set of functionality in a monolithic bundle of software.

Regulated environments don't tend to function in an "agile" fashion, because regulation requires checks and controls. Software delivery draws regulatory oversight because software drives today's business and poor software delivery endangers the predictable execution of that business. Countless examples exist of software leading to major problems in the areas of:

Regulators respond to these perennial problems in typical fashion: they require the implementation of rules around software delivery. These rules typically instantiate processes that are the opposite of agile: heavy on step-by-step checks; lots of paperwork; roles such as "release manager" that typically act as gate-keepers; and perhaps most importantly–sign-off by the users prior to release.

analysis

The problem is one that's not that dissimilar to other compliance problems: reconciling the company's desire to move quickly and with great latitude with the externally-imposed slow-and-steady pace of the regulator. This is frequently described as a clash of cultures, but when it comes to agile software delivery it's the clash of cadence that I believe is the issue. You simply can't be agile and regulated if you take regulated to mean something like "adjusted for proper functioning".

For software teams, the product must be delivered at a pace that lets the team get code to a user community quickly because they're working on so much that they simply have to get things off their plate in a timely fashion. Also, they need immediate feedback or they'll lose their familiarity with the code they produced for the round of review. But software can't be delivered to production in regulated environments without having been through "proper review", so the pace of developer must be throttled for things like:

When marrying two functions that happen with different cadence, you need some kind of synchromesh that can keep both functions happening at their own pace.

building the synchromesh

The way through this problem was to develop an internal "voice of the client" in a strong product ownership role. This person, in addition to having a solid understanding of the client's needs, had to run a QA function that did more than simply test against the required functionality. The QA function had to ensure that all of the external considerations had gone into the software's development. They also had to act as a collator to allow for the continuous build-up of functionality and fixes while the client was unable to accept regular small releases. This was accomplished by advising the user community on developments, and by sharing staged releases of new functionality in a special environment for user acceptance testing. There, the client testers could verify the work done by the internal QA team, and ensure that all of the controls had been considered as well.

It was imperfect. For one thing, the user community would inevitably find things that the internal QA team could not. But because they were also doing testing that the end users couldn't (such as security and performance/scaling tests), the two groups complemented each other very well.

In this way, the development team stayed "agile", and the regulated client group was able to function within the guidance of the regulators and the internal stakeholders that enforced the regulations, such as compliance and internal audit.

 

©2019 m. werneburg.  michael@werneburg.ca  +1 647.896.3850