What Norwegian Developers Want and Need From Security-Directed Program Analysis Tools

Code enforcing access control policies often has high inherent complexity, making it challenging to test using only classical review and testing techniques. To more thoroughly test such code, it is strategic to also use program analysis tools, which often can find subtle, critical bugs going unnoticed to humans. These powerful tools are however rarely used in software consultancy practice, due to factors such as bad usability or unsatisfactory non-functional characteristics. To encourage wider adoption of such tools, more must be learned about how to design them to the preferences of software consultants. Towards this goal, we conducted a survey of Norwegian software consultants. Among our findings is a positive relation between preference for soundness over completeness in tools and preference for annotation-based over automated tools. 51% of the developers surveyed prefer soundness over completeness when detecting access control vulnerabilities, while only 37.5% view completeness as the more important characteristic. Qualitative responses illuminate concerns regarding usability, soundness, completeness, and performance.