SDLC in a regulated environment

the challenge

Regulated industries cannot tolerate failed software releases. For a software vendor in a regulated industry, the challenge is to deliver software with new functionality, yet not impact the stability of the existing system. It’s the problem of “changing the wheels on a moving bus”. New functionality must be delivered without error, and new releases cannot impact the stability or functionality of the existing system in whole.

Enhancements may can be part of the core product, or custom developed for a single client. Changes frequently must be migrated to other client builds, where they can differ significantly. The issue has many angles:

• Development issues:
  • Business analysts are responsible for obtaining a proper spec (or a "story", under agile).
  • Developers are charged with coding to the spec (only).
  • Automated unit testing systems need to a) be deployed but b) must also integrate with version control and release management software
  • Security controls must be integrated as the code is being developed, not after the fact.
  • Availability and integrity components must be baked in during development as well.
• Testing issues:
• Software quality assurance function is separate from development.
• Test environments must be kept separate from production and development.
• Testers test to the specification, not the code.
• Version control issues:
• Multiple developers working at once raises the specter of multiple changes happening to a single code base simultaneously.
• Software delivered to regulated industries tends to require a great number of internal controls and features such as reporting. A complex architecture with many dependencies makes code management much more complex.
• Custom and core developments can occur simultaneously.
• Package release issues:
• Developers can’t release code to test or production.
• According to the audit controls, package release team shouldn’t know the code, data, or dependencies.
• Release instructions must be accurate for rollout and rollback.
• Scheduled &emergency releases often follow different paths from development to production.
• Audit issues:
• All releases are to be audited for compliance with stated procedures and standard controls from sources such as COBIT, including:
• Quality of release notes.
• Presence of code review.
• Approval—prior to release—by change control committee and client.
• Ensuring that the code released from development into test was binary identical to the code released to production.
• Ensuring that findings from prior application vulnerability scans had been put in place.
• Ensuring that common sources of vulnerabilities such as cross-site scripting and SQL injection are considered during the design process and during coding of all solutions.
• Separating responsibilities and access for three groups: those who create the assets; those who approve their release; and those who commit the release.

the software firm

A software firm that specializes in retail brokerage compliance was experiencing rapid growth as the compliance market matured. Wholly responsible for delivering and managing the software stack used by its regulated clients, the firm was responsible for not only accurate and secure software releases, but for ensuring that many risk controls remained in place with every release.

implementation

Through a review of all of the required controls, and an understanding of the process of applying fixes for results application vulnerability scans, several things were done.

separate SDLC stages

First, logically separate environments were created for development, systems integration testing, user acceptance testing, and production, as follow. This ensured that different activities could be carried out on distinct code bases. This prevented admixture of different code bases and prevented developers from altering code that was already released to test. It also gave the testers somewhere to work with release candidate without having to make exceptions for ongoing development. It further made it possible to port fixes from one environment to the next in the sequence in a controlled fashion.

virtualization

The deployment of these separate environments was accomplished using virtualized servers. This allowed the firm to quickly rollout multiple instances of each level of the SDLC, as needed, to facilitate needed upgrades to the software stack (such as operating system and database engine), and to keep costs down while enjoying this flexibility. Moreover, it allowed for the scrubbing of environments if they were damaged by a release – virtualization allowed us to effect a rollback in minutes.

release ticketing

All releases were tracked by tickets in the service desk software. These tickets were configured with common features such as:

• name of the requester
• the project (for time tracking and tracking release efficiency &throughput)
• target client system(s)
• the assets being released
• the software version number
• the scope of the release (major, minor, emergency)
• sign-off for each stage in the release
• rollout and rollback descriptions
• failure state impact estimations

The tickets became the foundation for all release management activities. When a developer needed to create a code branch for a future release, she first created a ticket. From that point on, the release was tracked by its ticket number—the branch number was even labelled with the ticket number inside the version control system. From that point on, all details about the release were recorded in one place, the ticket. This included any documents that had to be attached to the ticket, such as

gatekeeper roles

The SQA Manager was gatekeeper to the systems integration testing (SIT) environment. They ensured that the developers had obtained or prepared:

• A proper spec from the business systems analysts.
• A completed release ticket with attached release note.
• Release instructions that work with the version control and release management software.

The Production Manager was the gatekeeper for all production release candidates. This was not a technical role, but rather one of oversight:

• Runs the change control meeting.
• Reports on metrics.
• Investigates failed releases; tracks remedial actions.

The package release team then delivered all new software, schema changes, and configuration changes. They:

• Verified that they had authorization from the Production Manager prior to any release.
• Verified that they had good release instructions.
• Use the version control and release management software packages to roll out and roll back releases.

outcomes

A relatively simple set of rules allowed the firm to enjoy considerable success in software delivery. They delivered value-added functionality in a reliable fashion (1 error in 557 releases after the first review). The dependable processes so established also freed the time of PortfolioAid SME’s and senior management from participating in what had become "business as usual". And finally, the auditors were happy. The clients typically asked for the auditors' reports annually, and those reports reflected the software company's ongoing ability to adhere to this release process.

my role

In building this governed software release strategy, I:

• Gathered and analyzed the governance/audit requirements.
• Mapping the governance framework to business strategy.
• The writing (and substance) of process manuals.
• Managing the auditors. I have worked with American, Japanese, and Canadian auditors. (I hold a Certified Internal Auditor designation).
• Led the discussion and design of the SDLC.
• Managed the process to its fulfillment as a one-time project (I am a PMP).