Goeuriot, Lorraine and Jones, Gareth J.F. and Kelly, Liadh and Leveling, Johannes and Lupu, Mihai and Palotti, Joao and Zuccon, Guido
(2018)
An Analysis of Evaluation Campaigns in ad-hoc Medical Information Retrieval: CLEF eHealth 2013 and 2014.
Information Retrieval Journal.
ISSN 1386-4564
Abstract
Since its inception in 2013, one of the key contributions of the
CLEF eHealth evaluation campaign has been the organization of an ad-hoc
information retrieval (IR) benchmarking task. This IR task evaluates systems
intended to support laypeople searching for and understanding health infor-
mation. Each year the task provides registered participants with standard
IR test collections consisting of a document collection and topic set. Partici-
pants then return retrieval results obtained by their IR systems for each query,
which are assessed using a pooling procedure. In this article we focus on CLEF
eHealth 2013 and 2014's retrieval task, which saw topics created based on pa-
tients' information needs associated with their medical discharge summaries.
We overview the task and datasets created, and the results obtained by par-
ticipating teams over these two years. We then provide a detailed comparative
analysis of the results, and conduct an evaluation of the datasets in the light of
these results. This two-fold study of the evaluation campaign teaches us about
technical aspects of medical IR, such as the e�ectiveness of query expansion;
the quality and characteristics of CLEF eHealth IR datasets, such as their
reliability; and how to run an IR evaluation campaign in the medical domain.
| Item Type: |
Article
|
| Academic Unit: |
Faculty of Science and Engineering > Computer Science |
| Item ID: |
9452 |
| Depositing User: |
Liadh Kelly
|
| Date Deposited: |
08 May 2018 16:01 |
| Journal or Publication Title: |
Information Retrieval Journal |
| Publisher: |
Springer Netherlands |
| Refereed: |
Yes |
| URI: |
|
| Use Licence: |
This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available
here |
Repository Staff Only(login required)
 |
Item control page |
Downloads per month over past year
Origin of downloads
Altmetric
Altmetric