Process industries rely on effective decision-making by control room operators to ensure safety. Control room operators acquire various inputs from the displays, interpret them, make a prognosis, and respond through appropriate control actions. In order to perform these effectively, the operator needs to have appropriate mental models of the process. Novice operators develop such mental models during training. Hence, a holistic assessment of operators' training is essential to ensure process safety. Traditional approaches to assessing operators' training are primarily based on task measures, operator's actions, expert judgment, and notions of success/failure. These assessment schemes, however, ignore constructs of human cognition that lead to a particular performance such as the difficulties faced by the operator in information acquisition, reasoning about the abnormality, and/or decision making. Moreover, most of the existing approaches do not provide quantitative metrics to assess the operator’s performance during the training program. This work addresses the critical need to understand and evaluate the cognitive performance of control room operators using eye-tracking. Quantitative measures were developed using Fixation Transition Entropy (FTE), Correspondence Analysis (CA), and Hidden Markov Model (HMM) to objectively evaluate operator performance. The FTE provides information about the operator’s eye gaze transitions on the Human-Machine Interface (HMI). The operator's attention on the HMI is quantitatively estimated by the FTE, but it does not provide information about the variables that are attended by the operator. The measures developed using correspondence analysis of eye gaze– association and salience metrics – capture which variable(s) are attended by the operator during process abnormalities. Finally, to describe the mental models of the operators during process abnormalities, Hidden Markov Model-based formalism was proposed. This is the first study of its kind to mathematically characterizes the control room operators’ mental models rather than rely on qualitative descriptions. Two axioms of learning are derived from the proposed HMM-based mental models. These axioms help evaluate operator training in terms of understanding causal relationships and proactive monitoring strategies. The potential of the proposed approaches is demonstrated on an in-house chemical process simulator to which operators interacted via HMI. During the experiment, process information (process variable trends), alarm details (occurrence and clearance), operator actions (sequence), and eye gaze data were collected. Various case studies and statistical analysis revealed that the proposed operator performance measures could effectively capture the extent of operators' training and their mental models as they progress from being novices to a stage characterized by high performance and correct mental models. The HMM based operator’s mental models is able to identify the distinguishing features in the mental models of an expert operator who is well-versed with the process and clearly contrasts those who do not have an adequate understanding of the process causality. The quantification offered by the metrics precludes subjective factors in the benchmarking of operators, with the concomitant benefits of objectivity. Therefore, the proposed methodologies fares excellently on the desired attributes of a robust operator training assessment procedure, including consistency, repeatability, and neutrality.


Keywords – Process industry; Operator training; Eye-tracking; Hidden Markov Model; Correspondence analysis; Gaze transition entropy; Mental models