Challenging Challenge Questions: An Experimental Analysis of Authentication Technologies and User Behaviour

To authenticate human users to systems, challenge questions based on personal information are often used, typically when a primary authentication credential, such as a password, is forgotten. This ought to be a trustworthy mechanism, that is both reliable and accurate: personal information should be inherently memorable and not known to others. However, concerns have been raised recently about these assumptions: for example, some commonly used questions may be based on information that is available publicly. A possible improvement, then, is to allow users to choose their own questions. Here we report on an experiment which gathered user chosen questions and a subsequent security and usability analysis of them. Our experiment itself follows a novel method which is designed to engender the trust of participants, so they participate honestly. This methodological innovation demonstrates that it is possible to perform ethical authentication experiments where sensitive information does not have to be collected from users. Our experiments revealed some surprising results. Although subjects sometimes seemed aware of the need for security, they often ‘missed the mark’ by a wide margin; similarly, there are serious concerns over the usability of freely chosen questions with free‐form answers. These results should raise some serious questions for those setting the policy agenda for either testing or building authentication solutions for Internet applications.