Affiliations: School of Information Sciences, University of Pittsburgh, USA | Department of Informatics, University of Piraeus, 80 Karaoli & Dimiitriou St., Piraeus 185 34, Greece
Correspondence:
[*]
Corresponding author: Tomasz D. Loboda, School of Information Sciences, University of Pittsburgh, USA. E-mail: [email protected]
Abstract: Users presented with a system equipped with many advanced features and vast collections of searchable documents often have problems getting to the information they seek. Our studies with intelligence analysts using the Rosetta Web-based information access system revealed inefficiencies in the way they used the tool. Motivated by that observation, we implemented a prototype of an intelligent software agent that monitors the behavior of users with a goal of generating real-time suggestions to improve their performance. To evaluate the agent's ability to predict the changes in the analysts' performance, we compared it to the judgment of human domain experts. The results revealed evidence of substantial disagreement among human judges. We found the level of agreement among the judges and the agent to be much higher. The agent's predictions were similar to those of an average judge.
Keywords: Evaluation methodology, sep user study, inter-rater reliability, user interface agent, Bayesian networks, information access system
