Evaluation is an essential activity in conducting rigorous recommender systems research as it allows assessing the system’s ability to provide personalized recommendations. Particularly, evaluating the user experience is an important goal and there are a number of criteria that have to be met to ensure a satisfying user experience. However, the evaluation procedures employed have mostly focused on one single evaluation method and also, on one single quality objective (e.g., prediction accuracy). One single method, though, is not able to comprehensively assess all the important aspects of user experience. We argue that employing a multi-method evaluation integrating a number of single methods allows for getting a more integrated and richer picture of user experience and quality drivers of recommender systems. We characterize the methods eligible and reason that a multi-method evaluation may improve not only the evaluation procedure, but ultimately, the user experience with recommender systems.
Eva Zangerle (http://www.evazangerle.at/) is a Senior Postdoc Researcher at the University of Innsbruck, Austria, at the Department of Computer Science. Her main research interests are within the fields of recommender systems, social media analysis, and information retrieval. Particularly, she investigates music recommender systems based on data retrieved from social media platforms, aiming to exploit new sources of information for recommender systems. Her passion lies in advancing context-aware music recommender systems and RecSys in collaborative environments. She was awarded a Postdoctoral Fellowship for Overseas Researchers for a research stay at Ritsumeikan University in Kyoto from the Japan Society for the Promotion of Science.