Tag Archives: Pentest

Can Penetration Testing Trigger Change in Development Teams?

Software penetration testing has become a common practice in software companies. Penetration testers apply exploratory testing techniques to find vulnerabilities, giving developers feedback on the results of their security efforts—or lack thereof. This immediate benefit is mostly uncontested, although it comes with the caveat that testers find only a fraction of the present vulnerabilities and their work is rather costly.

Security experts commonly recommend that developers should respond to the results of a penetration test not merely by fixing the specific reported vulnerabilities, but rather analyze and mitigate the root causes of these vulnerabilities. Just as commonly, security experts bemoan that this does not seem to happen in practice. Is this true and what prevents penetration testing from having an impact beyond defect fixing?

Studying Software Developers

We studied this question by observing a product group of a software company over the course of a year. The company had hired security consultants to test one of its products for security defects and subsequently train the developers of this product. We wanted to know how effective this intervention was as a trigger for change in development. To this end we conducted two questionnaires, observed the training workshop, analyzed the contents of the group’s wiki and issue tracker, and interviewed developers and their managers.

The product group in our study comprised several Scrum teams, development managers, and product management. Scrum is a management framework for agile software development. Scrum defines three roles—Development Team, Product Owner, and Scrum Master—and artifacts and ceremonies for their collaboration. The Development Team is responsible for and has authority over technical matters and development work; the Product Owner is responsible for requirements and priorities; and the Scrum Master facilitates and moderates their collaboration.

The participants of our study appreciated the penetration test results as feedback and the training workshop as an opportunity to learn. They managed to address the particular vulnerabilities reported by the consultants. They also felt that security needed more attention in their development work. Yet, after addressing the vulnerabilities uncovered by the penetration test, they returned to their familiar ways of working without lasting change.

Analyzing our observations through the lens of organizational routines, we found three key factors inhibiting change in response to the penetration test and training: successful application of existing routines, the organizational structure of roles and responsibilities, and the overall security posture and attitude of the company.

(1) Existing Routines

To address the immediate findings of the penetration test, our developers used an existing bug-fixing and stabilization routine. Defect reports arrive asynchronously and sometimes require quick response; developers therefore commonly dedicate some—variable—percentage of their working time to defect fixing in response to reports. The penetration test fed the team dozens of new defects at once, but developers hat a routine to deal with it. Moreover, management tracked the number of open defects, so that the sudden increase raised attention and created pressure on the team to get this number down.

Feature development, on the other hand—where change would have to occur—remained mostly unaffected. Feature development followed the process of Scrum and the penetration test neither had a specific impact here nor did it feed requests or ideas into this routine.

(2) Organizational Structure

Following the ideas of Scrum and agile development, a strong division of labor and responsibilities characterized the organizational structure in the setting of our study. Product management and product owners were responsible for the direction of the development work, whereas development teams enjoyed a certain autonomy in technical questions. This structure worked as a social contract: managers expected developers to take care of security as a matter of quality and developers were keen to deliver the features requested by management. However, the penetration test had little impact on the manager’s priorities beyond the pressure to reduce the number of open defects. The developers thus found themselves in a situation where security was not explicitly required and additional security work could not be justified.

(3) Business Role of Security

Finally, security had limited perceived importance for the business of the studied company, which thus far had not experienced any public security disaster and did not actively sell security. The company therefore lacked a security narrative that could have been used to justify security efforts beyond defect fixing. This, together with the inherent low visibility of security and insecurity, shaped priorities. Product managers knew that features sell their products—new features are easy to show and explain, whereas security improvements are not. Security was perceived as contributing little to the success of the product and the company, making it a low-priority requirement.

Our study points to some of the complexities of managing software development and of triggering change by interventions. While it would be tempting to assign blame to a single factors, such as agile development or negligent management, the problem is really more complicated. Organizational structures and routines exist and they are shaped by business needs. Scrum, for example, is highly beneficial for the studied company. One might even ask whether the company’s dealing with security is a problem in the first place. Are they perhaps doing the right thing for their market and customers?

Our Paper:

A. Poller, L. Kocksch, S. Türpe, F. A. Epp; K. Kinder-Kurlanda: Can Security Become a Routine? A Study of Organizational Change in an Agile Software Development Group. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW’17), February 25–March 1, 2017, Portland, OR, USA.
[DOI: 10.1145/2998181.2998191, BibTeX, slide deck]

The Point of Penetration Testing

I’m glad to see that the community starts to acknowledge what penetration testing really means. We never encountered the empty hands issue as we never sold the let’s-see-if-we-can-hack-it kind of a penetration test. We carry out vulnerability assessments based on whichever information we can get about a system or product, the more the better, and use testing as a means of getting more, or clarifying information. The point of a penetration test is not to find and demonstrate one or a handful of arbitrary attacks, the point of a penetraion test is to identify vulnerabilities that need attention. Sure, it is fun to give an uberhacker demo after the test, and we do that, too, if we’ve found gaping security holes. But more important in a penetration test is good coverage of the potential vulnerabilities, and a sound assessment of their importance, relative to the purpose and risk profile of a system. I even remember pentesting projects where we found only a few relatively minor issues in a low-risk website – and made our clients happy by telling them just that. There is really no need to be disappointed if a penetration test ends without a root shell. There is no point in withholding information from testers, reducing their efficiency. And there is no point in testing just the infrastructure but not the applications.

Praktische IT-Sicherheit 2010

Das Rheinlandtreffen “Praktische IT-Sicherheit” findet in diesem Jahr früher als gewohnt am 31.8. und 1.9. in der Hochschule Bonn-Rhein-Sieg statt. Ich bin mit einem Vortrag zum produktionssicheren Testen dabei, dessen Inhalt im Wesentlichen unserem Artikel in der <kes> 2010#2 folgt. Sicherheitstests sind ein Schwerpunkt der Veranstaltung, aber nicht das einzige Thema. Das komplette Programm gibt’s auf der Website bei unseren Kollegen vom Fraunhofer INT, wo man sich auch anmelden kann.