As an IT auditor at a software company, I discovered that security vulnerabilities in our bespoke product had not been getting released to clients on a timely basis. We had been doing penetration tests for years, but obtaining the penetration test report had not translated to the fixes being released to the users. Our clients remained exposed to known vulnerabilities, a situation that meant my employer was assuming all potential liability for the situation.
There were, it turned out, many things that slowed delivery of the fixes. Some factors were organizational and some were technical. I address the organizational challenges of client resistance and lack of internal commitment in my recent Journal article. But I will offer an additional insight for readers of Practically Speaking on overcoming technical complexities in patching a bespoke software product.
The technical complexities in the environment were nearly endless. Some penetration test findings applied only to certain versions of the software. Some fixes were beyond the capabilities of the development platform and required extensive software workarounds. Other fixes required a minimum browser level that was beyond a client’s reach. Sometimes a peculiarity of the client environment prevented one fix or another from working, e.g., a home-brewed single sign on or bespoke antivirus configurations could hamper the rollout of bug fixes. These complexities prevented the patch bundle from each year’s penetration test report from being deployed into the production environment.
The problem turned out to be both the vendor and the client believing each year’s findings could be resolved via a monolithic patch. By bundling the fixes, we greatly increased the likelihood that some complexity would render the patch unsuitable or undeliverable to a given client. Working with the developers, we devised a matrix that broke down each year’s penetration test results, with a row for each distinct finding in each report. We worked with the product owners to understand which finding applied to which version of the software. When a software fix was created, we recorded the version control branch number for the fix against the relevant finding. When a release was scheduled for a client, we worked with the project management office to ensure that the relevant fixes got scheduled.
It was messy and labor-intensive, but it worked. Supported by a strong version control system and a documented package release program, a reliable program for tracking penetration test fixes to production was put into effect. In time, we eradicated client exposure to known vulnerabilities, resolved our employer’s potential liability and were ready for each year’s fresh batch of findings.
"Addressing Shared Risk in Product Application Vulnerability Assessments" ISACA Journal, volume 5, 2017.