Cole, Michael J. and Zhang, Xiangmin and Liu, Jinging and Liu, Chang and Belkin, Nicholas J. and Bierig, Ralf and Gwizdka, Jacek (2011) Are self-assessments reliable indicators of topic knowledge? Proceedings of the American Society for Information Science and Technology, 47 (1). pp. 1-10. ISSN 00447870

Preview

Download (239kB) | Preview

Abstract

Self-assessment of topic/task knowledge is a human metacognitive capacity that impacts information behavior, for example through selection of learning and search strategies. It is often used as a measure in experiments for evaluation of results and those measurements are taken to be generally reliable. We conducted a user study (n=40) to test this by constructing a concept-based topic knowledge representation for each participant and then comparing it with the participant judgment of their topic knowledge elicited with Likert-scale questions. The tasks were in the genomics domain and knowledge representations were constructed from the MeSH thesaurus terms that indexed relevant documents for five topics. The participants rated their familiarity with the topic, the anticipated task difficulty, the amount of learning gained during the task, and made other knowledge-related judgments associated with the task. Although there is considerable variability over individuals, the results provide evidence that these self-assessed topic knowledge measures are correlated in the expected way with the independently-constructed topic knowledge measure. We argue the results provide evidence for the general validity of topic knowledge self-assessment and discuss ways to further explore knowledge self-assessment and its reliability for prediction of individual knowledge levels.

Item Type:	Article
Keywords:	User studied; methodology; knowledge representation;
Academic Unit:	Faculty of Science and Engineering > Computer Science Faculty of Science and Engineering > Research Institutes > Hamilton Institute
Item ID:	15218
Identification Number:	https://doi.org/10.1002/meet.14504701285
Depositing User:	Ralf Bierig
Date Deposited:	11 Jan 2022 12:46
Journal or Publication Title:	Proceedings of the American Society for Information Science and Technology
Publisher:	ASIST
Refereed:	Yes
URI:	Publisher
Use Licence:	This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

Repository Staff Only(login required)

Item control page

Downloads

Origin of downloads