Interactive Modeling of Concept Drift and Errors in Relevance Feedback

In exploratory search tasks, users usually start with considerable uncertainty about their search goals, and so the search intent of the user may be volatile as the user is constantly learning and reformulating her search hypothesis during the search. This may lead to a noticeable concept drift in the relevance feedback given by the user. We formulate a Bayesian regression model for predicting the accuracy of each individual user feedback and thus find outliers in the feedback data set. To accompany this model, we introduce a timeline interface that visualizes the feedback history to the user and gives her suggestions on which past feedback is likely in need of adjustment. This interface also allows the user to adjust the feedback accuracy inferences made by the model. Simulation experiments demonstrate that the performance of the new user model outperforms a simpler baseline and that the performance approaches that of an oracle, given a small amount of additional user interaction. A user study shows that the proposed modeling technique, combined with the timeline interface, made it easier for the users to notice and correct mistakes in their feedback, resulted in better and more diverse recommendations, allowed users to easier find items they liked, and was more understandable.