Can structured data fields accurately measure quality of care? The example of falls

By automating collection of data elements, electronic health records may simplify the process of measuring the quality of medical care. Using data from a quality improvement initiative in primary care medical groups, we sought to determine whether the quality of care for falls and fear of falling in outpatients aged 75 and older could be accurately measured solely from codable (non-free-text) data in a structured visit note. A traditional medical record review by trained abstractors served as the criterion standard. Among 215 patient records reviewed, we found a structured visit note in 54% of charts within 3 mo of the date patients had been identified as having falls or fear of falling. The reliability of an algorithm based on codable data was at least good (kappa of at least 0.61) compared with full medical record review for three care processes recommended for patients with two falls or one fall with injury in the past year: orthostatic vital signs, vision test/eye examination, and home safety evaluation. However, the automated algorithm routinely underestimated quality of care. Performance standards based on automated measurement of quality of care from electronic health records need to account for documentation occurring in nonstructured form.