Rating EFL written performance / Katalin Bukta; managing editor, Anna Borowska; language editor, Carl Becker.

Format:

Book

Author/Creator:

Bukta, Katalin, author.

Contributor:

Borowska, Anna, editor.

Becker, Carl, editor.

Series:

Versita discipline. Language, literature.

Versita Discipline: Language, Literature

Language:

English

Subjects (All):

English language--Study and teaching--Foreign speakers.

English language.

Physical Description:

1 online resource (424 pages).

Edition:

1st ed.

Place of Publication:

London, England : Versita, Versita Limited, 2013.

Language Note:

English

Summary:

The study focuses on the investigation of the process during which raters of EFL written performance make their decisions. It consists of a pilot and a main study, each of which concentrates on assessment of writing. The rationale is to detect the decision-making processes that raters follow, which can be used for training raters, and with which the reliability of rating can be improved. The pilot study is based on data collected during a large-scale language proficiency assessment of two age groups from learners of English and German languages. Raters were asked to think aloud during the rating task. Data was then transcribed and analysed. The participants in the main study were novice raters, who produced verbal protocols. 37 EFL teacher trainees took part in rater training and practised think-aloud protocol production. Then, they evaluated ten compositions written by EFL learners and verbalised their thought processes. The verbal protocols served as a basis for data collection. The analysis of data resulted in the conclusion according to which more reliable and objective assessment is possible when evaluating written performance.

Contents:

Intro

Introduction

Overview of the Book

PART I

An Overview of the Literature on L2 Writing and Assessment

Chapter 1

Writing Ability and L2 Writing Instruction

1.1 The Writing Skill in L1 and L2

1.1.1 Theoretical Models of Written Text Production

1.1.2 Theoretical Frameworks of Communicative Competence

1.2 Writing Ability in L1

1.3 Writing Ability in L2

1.3.1 The Relationship Between L1 and L2 Writing Ability

1.3.2 Skilled and Unskilled L2 Writers

1.4 L2 Writing Instruction

1.4.1 Controlled Composition Approach to Writing

1.4.2 Process-Oriented Approach to Writing

1.4.3 The Role of Writing in Communicative Language Teaching

1.5 Conclusion

Chapter 2

Assessing Language Ability

2.1 Assessing Language Ability

2.1.1 History of Language Testing

2.1.2 Recent Developments in Language Testing Research

2.2 L2 Ability Assessment

2.2.1 Language Ability and Language Performance

2.2.2 Characteristics of Performance Assessment

2.3 Test Features

2.3.1 Test Purpose

2.3.2 Testing Methods

2.4 Qualities of Language Tests

2.4.1 Reliability

2.4.2 Validity

2.4.3 Authenticity

2.4.4 Impact

2.4.5 Washback

2.4.6 Practicality

2.5 Test Method Characteristics

2.6 Assessing the Four Language Skills

2.7 Scoring Methods

2.7.1 Characteristics of Rating Scales

2.7.2 Rater Variables

2.7.3 Score Interpretation

2.8 Test-Taker Characteristics

2.9 Language Test Construction

2.10 Conclusion

Chapter 3

Assessing Writing Ability

3.1 Written Performance Assessment

3.2 Alternative Ways of Writing Ability Assessment

3.3 Nature of Written Performance Assessment

3.3.1 Validity of Writing Assessment

3.3.2 Task Characteristics

3.3.3 Definition of Audience

3.3.4 Test Taker Characteristics.

3.4 Conclusion

Chapter 4

Rating Written Performance

4.1 Scoring Procedures

4.2 Rater Variables

4.3 Rating as a Problem-Solving Activity

4.4 Frameworks of Scoring Processes

4.4.1 Milanovic, Saville and Shuhong's Framework of the Scoring Process

4.4.2 Wolfe's Framework of the Scoring Process

4.4.3 Lumley's Framework of the Scoring Process

4.4.4 Cumming, Kantor and Powers' Framework of the Scoring Process

4.5 Rater Stability

4.6 Rater Training

4.7 Conclusion

Chapter 5

Verbal Protocol Analysis as a Research Method in Written Performance Assessment

5.1 Research in L2 Acquisition Studies

5.2 Introspective Reports as Data Collection

5.3 Categorisation of Verbal Protocols

5.4 Concurrent Think-Aloud Protocols in Written Performance Assessment

5.5 Verbal Protocol Analysis Procedure

5.5.1 Data Preparation and Collection Procedures

5.5.2 Verbal Report Procedure Preparation

5.5.3 Data Transcription

5.5.4 Segmenting the Protocols for Coding

5.5.5 Developing a Coding Scheme

5.5.6 Analysis of the Coded Data

5.6 Advantages and Limitations of Verbal Protocol Analysis as Research Methodology

5.7 Conclusion

PART II

Investigating Raters' Decision-Making Processes and Awarded Scores in Rating Hungarian Efl Learners' Compositions

Chapter 6

A Pilot Study: Tracing Raters' Decision-Making Processes

6.1 Background to the Pilot Study on Assessment of Written Performance

6.2 Research Questions

6.3 Research Design

6.3.1 Participants: the Raters

6.3.2 Procedures for Data Collection

6.3.3 Test of Written Performance: the Task

6.3.4 The Assessment Scale

6.3.5 The Coding Scheme

6.4 Results and Discussion

6.4.1 Distribution of Comments During Rating

6.4.2 Comments on Rating Technicalities.

6.4.3 General Comments on the Scripts

6.4.4 The Way Raters Arrived at a Score

6.4.5 Assessment of the Communicative Goal

6.4.6 Assessment of Vocabulary

6.4.7 Assessment of accuracy and spelling

6.4.8 Assessment of the Text Organisation

6.5 The Rating Process

6.6 Conclusion

Chapter 7

The Main Study: Processes and Outcomes in Rating L2 English Written Performance in a Pre-Service Tefl Course

7.1 Background to Main Study

7.2 Design of the Study

7.3 Research Questions

7.4 Participants

7.5 Data Collection Instruments

7.5.1 The Scripts

7.5.2 The Rating Scale

7.5.3 The Rating Task Assignment Package

7.6 Data Collection Procedures

7.6.1 Script Characteristics

7.6.2 Script Selection

7.6.3 Training Raters

7.6.4 The Rating Task

7.7 Processing Verbal Data

7.7.1 Transcript Segmentation

7.7.2 Coding Scheme Production

7.8 Rater Characteristics: Grouping Raters

7.9 Summary

Chapter 8

Features of Raters' Rating Patterns

8.1 Raters' Gender Distribution

8.2 Language of the Verbal Protocols

8.3 Sequencing the Scripts for Rating

8.4 Length of Verbal Protocols

8.5 Raters' Rating Sequences

8.6 Raters' Rating Patterns

8.7 Conclusion

Chapter 9

Raters' Rating Focus

9.1 Raters' Rating Foci

9.1.1 Management Focus: Management Strategies

9.1.2 Rating Focus: Rating Strategies

9.1.3 Reading Focus: Reading Strategies

9.1.4 Raters' Own Focus: Other Comments

9.2 Conclusion

Chapter 10

Raters' Focus on the Four Rating Criteria

10.1 Raters' Focus on the Four Rating Criteria

10.2 Raters' Focus When Rating Task Achievement

10.3 Raters' Focus When Rating Vocabulary

10.4 Raters' Focus When Rating Grammar

10.5 Raters' Focus When Rating Organisation

10.6 Conclusion

Chapter 11.

Raters' Script Interpretation on the Weakest and Top Script

11.1 Rating Script N2: Benchmarks and Total Scores

11.1.1 Comments on Script N2 in the Pre-Scoring Stage

11.1.2 Rating Task Achievement of Script N2

11.1.3 Rating Vocabulary of Script N2

1.1.4 Rating Grammar of Script N2

11.1.5 Rating Organisation of Script N2

11.2 Ratings of Script N6: Benchmarks and Total Scores

11.2.1 Comments on Script N6 in the Pre-Scoring Stage

11.2.2 Rating Task Achievement of Script N6

11.2.4 Rating Grammar of Script N6

11.2.5 Rating Organisation of Script N6

11.3 Conclusion

Chapter 12

Raters' Perception of the Rating Task and Thinking Aloud

12.1 The Feedback Sheet

12.1.1 Raters' Feedback on the Language-Testing Course

12.1.2 Raters' Feedback on Training for Rating Written Performance

12.1.3 Raters' Feedback on the Rating Task

12.2 Conclusion

General Conclusions

Background to Written Performance Assessment

Features of Rating Processes

Raters' Rating Foci

Interpretation of the Rating Criteria

Raters' Script Interpretation

Raters' Perceptions of the Rating Task

Placing Empirical Findings on the Rating Processes into Theoretical Frameworks

Verbal Protocol Analysis as Data Collection Method

Implications of the Findings

Limitations of the Study

Further Research

References

Appendices

Appendix 6.1 The letter writing task in the pilot study

Appendix 6.2 The rating scale in the pilot study

Appendix 6.3 The coding scheme for the pilot study

Appendix 6.4 Number of utterances made during the rating process in the pilot study

Appendix 6.5 A sample from EngR1 transcript in the pilot study (translated from Hungarian)

Appendix 7.1 The writing task in the main study

Appendix 7.2 The rating scale in the main study.

Appendix 7.3 The letter to the students in the main study

Appendix 7.4 The ten scripts in the main study

Appendix 7.5 The score sheet in the main study

Appendix 7.6 The feedback sheet in the main study

Appendix 7.7 The course description for the elective seminar course in Testing in ELT

Appendix 7.8 A sample verbal protocol transcript in the main study

Appendix 7.9 The coding scheme in the main study

Appendix 7.10 A sample of a coded protocol in the main study

Appendix 7.11 Competent raters' total scores and rankings of the ten scripts in parentheses in the main study

Appendix 7.12 Proficient raters' total scores and rankings in parentheses in the main study

Index.

Notes:

Bibliographic Level Mode of Issuance: Monograph

Includes bibliographical references and index.

Description based on online resource; title from PDF title page (ebrary, viewed May 31, 2015).

ISBN:

3-11-039928-8

OCLC:

947127904