Official URL: http://link.springer.com/article/10.1007/s10791-01...

Since its inception in 2013, one of the key contributions of the CLEF eHealth evaluation campaign has been the organization of an ad-hoc information retrieval (IR) benchmarking task. This IR task evaluates systems intended to support laypeople searching for and understanding health infor- mation. Each year the task provides registered participants with standard IR test collections consisting of a document collection and topic set. Partici- pants then return retrieval results obtained by their IR systems for each query, which are assessed using a pooling procedure. In this article we focus on CLEF eHealth 2013 and 2014's retrieval task, which saw topics created based on pa- tients' information needs associated with their medical discharge summaries. We overview the task and datasets created, and the results obtained by par- ticipating teams over these two years. We then provide a detailed comparative analysis of the results, and conduct an evaluation of the datasets in the light of these results. This two-fold study of the evaluation campaign teaches us about technical aspects of medical IR, such as the e�ectiveness of query expansion; the quality and characteristics of CLEF eHealth IR datasets, such as their reliability; and how to run an IR evaluation campaign in the medical domain.