Assessing oral presentations and interactions: From a systematic to a salient-feature approach

Publications: Contribution to bookChapterPeer Reviewed

Abstract

Most rating scales for performance assessment distinguish between different levels by systematically replacing abstract qualifiers such as some, many, or most at each band (the systematic approach). Less frequently, distinctions are based on concrete aspects of performance characteristic of the band concerned (the salient-feature approach). This chapter presents a study which compares and contrasts the two approaches. The main aim was to evaluate whether rating scales featuring salient aspects of performance are more reliable for the purpose of assessing academic presentation and interaction skills in the context of an undergraduate speaking course than rating scales which distinguish between the levels systematically. Both qualitative and quantitative methods were employed to evaluate the effectiveness of the scales. In phase one, the scores of 60 live-exam performances rated on the basis of systematic scales were compared to the scores of 84 mock-exam performances based on salient-feature scales. The latter had two formats, first as six-point scales with every band (except for the lowest) being defined by descriptors and then as ten-point scales with unworded bands in between. Many-facet Rasch analysis showed that the salient-feature scales are generally superior in terms of rater reliability and criteria separation. However, raters were unable to distinguish as many as ten bands reliably, although, according to interview data, raters find undefined intermediate levels very useful. The results have implications for scale revision, rater training, and future scale development.
Original languageEnglish
Title of host publicationDeveloping advanced English language competence
Subtitle of host publicationA research-informed approach at tertiary level
EditorsArmin Berger, Helen Heaney, Pia Resnik, Angelika Rieder-Bünemann, Galina Savukova
Place of PublicationCham
PublisherSpringer
Pages297–321
Number of pages25
ISBN (Electronic)978-3-030-79241-1
ISBN (Print)978-3-030-79240-4
DOIs
Publication statusPublished - 2022

Publication series

SeriesEnglish Language Education
Volume22

Austrian Fields of Science 2012

  • 602007 Applied linguistics
  • 602008 English studies
  • 503007 Didactics

Keywords

  • Descriptor formulation
  • Group interviews
  • Many-facet Rasch analysis
  • Performance assessment
  • Rating scale development and validation

Cite this