Using Learning Analytics to evaluate the quality of multiple-choice questions: a perspective with Classical Test Theory and Item Response Theory

Azevedo, José Manuel; Oliveira, Ema; Beites, P. D.

Publication

Using Learning Analytics to evaluate the quality of multiple-choice questions: a perspective with Classical Test Theory and Item Response Theory

2019Journal article

dc.contributor.author	Azevedo, José Manuel
dc.contributor.author	Oliveira, Ema
dc.contributor.author	Beites, P. D.
dc.date.accessioned	2020-02-07T16:47:52Z
dc.date.available	2020-02-07T16:47:52Z
dc.date.issued	2019
dc.description.abstract	Purpose The purpose of this paper is to find appropriate forms of analysis of multiple-choice questions (MCQ) to obtain an assessment method, as fair as possible, for the students. The authors intend to ascertain if it is possible to control the quality of the MCQ contained in a bank of questions, implemented in Moodle, presenting some evidence with Item Response Theory (IRT) and Classical Test Theory (CTT). The used techniques can be considered a type of Descriptive Learning Analytics since they allow the measurement, collection, analysis and reporting of data generated from students’ assessment. Design/methodology/approach A representative data set of students’ grades from tests, randomly generated with a bank of questions implemented in Moodle, was used for analysis. The data were extracted from a Moodle database using MySQL with an ODBC connector, and collected in MS ExcelTM worksheets, and appropriate macros programmed with VBA were used. The analysis with the CTT was done through appropriate MS ExcelTM formulas, and the analysis with the IRT was approached with an MS ExcelTM add-in. Findings The Difficulty and Discrimination Indexes were calculated for all the questions having enough answers. It was found that the majority of the questions presented values for these indexes, which leads to a conclusion that they have quality. The analysis also showed that the bank of questions presents some internal consistency and, consequently, some reliability. Groups of questions with similar features were obtained, which is very important for the teacher to develop tests as fair as possible. Originality/value The main contribution and originality that can be found in this research is the definition of groups of questions with similar features, regarding their difficulty and discrimination properties. These groups allow the identification of difficulty levels in the questions on the bank of questions, thus allowing teachers to build tests, randomly generated with Moodle, that include questions with several difficulty levels in the tests, as it should be done. As far as the authors’ knowledge, there are no similar results in the literature.	pt_PT
dc.description.version	info:eu-repo/semantics/publishedVersion	pt_PT
dc.identifier.doi	10.1108/IJILT-02-2019-0023	pt_PT
dc.identifier.uri	http://hdl.handle.net/10400.6/9146
dc.language.iso	eng	pt_PT
dc.peerreviewed	yes	pt_PT
dc.relation.publisherversion	https://www.emerald.com/insight/content/doi/10.1108/IJILT-02-2019-0023/full/html	pt_PT
dc.rights.uri	http://creativecommons.org/licenses/by/4.0/	pt_PT
dc.subject	Analytics	pt_PT
dc.subject	e-assessment	pt_PT
dc.subject	Item Response	pt_PT
dc.title	Using Learning Analytics to evaluate the quality of multiple-choice questions: a perspective with Classical Test Theory and Item Response Theory	pt_PT
dc.type	journal article
dspace.entity.type	Publication
oaire.citation.endPage	341	pt_PT
oaire.citation.issue	4	pt_PT
oaire.citation.startPage	322	pt_PT
oaire.citation.title	The International Journal of Information and Learning Technology	pt_PT
oaire.citation.volume	36	pt_PT
person.familyName	Oliveira
person.familyName	Damas Beites
person.givenName	Ema Patrícia de Lima
person.givenName	Patrícia
person.identifier.ciencia-id	0910-B8EE-A6F6
person.identifier.ciencia-id	591A-8D9A-73CE
person.identifier.orcid	0000-0003-3341-1757
person.identifier.orcid	0000-0003-0266-7055
person.identifier.rid	K-5772-2017
person.identifier.scopus-author-id	37041170800
rcaap.embargofct	Copyright cedido à editora no momento da publicação	pt_PT
rcaap.rights	closedAccess	pt_PT
rcaap.type	article	pt_PT
relation.isAuthorOfPublication	b380b9d9-6d31-48e7-8750-5ddc091ef68c
relation.isAuthorOfPublication	9f86873f-5364-4413-a23d-ceab6f2372a3
relation.isAuthorOfPublication.latestForDiscovery	b380b9d9-6d31-48e7-8750-5ddc091ef68c