Premium
Relevance in cyber‐physical systems with humans in the loop
Author(s) -
Gopalakrishna Aravind Kota,
Ozcelebi Tanir,
Lukkien Johan J.,
Liotta Antonio
Publication year - 2016
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.3827
Subject(s) - cyber physical system , relevance (law) , computer science , context (archaeology) , metric (unit) , machine learning , artificial intelligence , performance metric , class (philosophy) , perception , physical system , human in the loop , engineering , paleontology , operations management , physics , management , quantum mechanics , neuroscience , political science , law , economics , biology , operating system
Summary In cyber‐physical systems such as intelligent lighting, the system responds autonomously to observed changes in the environment. In such systems, more than one output may be acceptable for a given input scenario. This type of relationship between the input and output makes it difficult to analyze machine learning algorithms using commonly used performance metrics such as classification accuracy (CA). CA only measures whether a predicted output is right or not, whereas it is more important to determine whether the predicted output is relevant for the given context or not. In this direction, we introduce a new metric, the relevance score (RS) that is effective for the class of applications where user perception leads to non‐deterministic input–output relationships. RS determines the extent by which a predicted output is relevant to the user's context and behaviors, taking into account the variability and bias that come with human perception factors. We assess the performance of a number of machine learning algorithms, using different datasets, including data from an intelligent lighting pilot. We find that using RS instead of CA is appropriate to analyze the performance of conventional machine learning algorithms, particularly for the class of non‐deterministic multiple‐output problems. Our method may be applied to other scenarios in which cyber‐physical systems involve humans in the control loop. Copyright © 2016 John Wiley & Sons, Ltd.