The Construction of MTI Translation Evaluation Index System

[a]English Department， Xi’an Shiyou University， Xi’an， China.

[b]Mathematics Department， Xi’an Shiyou University， Xi’an， China.

*Corresponding author.

Received 20 November 2014; accepted 8 January 2015

Published online 26 January 2015

Abstract

Translation evaluation was essential to MTI training. In order to standardize， guide， diagnose， and regulate the process， the study firstly examined researches regarding translation competency both at home and abroad from 1970s till now; Then， based on the latest understanding of translation competency， with Analytic Hierarchy Process （AHP）， a MTI Translation Evaluation Index System （MTI TEIS） was developed， aiming to digitally represent and assess the gradual progress of translator’s comprehensive translation abilities on an objective basis. The MTI TEIS was translator-oriented and it emphasized individual difference. It may not only apply to MTI teachers who were in charge of evaluating and controlling the whole translation training process， but also to MTI candidates who expected to conduct self-evaluation regularly.

Key words： Index evaluation system; MTI; Translation competency; AHP

Dong， M.， & Cai， H. J. （2015）. The Construction of MTI Translation Evaluation Index System . Canadian Social Science， 11（1）， <Page>-0. Available from： http：///index.php/css/article/view/5377

DOI： http：///10.3968/5377

INTRODUCTION

Since the implementation of the Master of Translation and Interpreting （MTI） project in 2007 in China， 170 universities have been approved to set up MTI education centers by 2014. However， few of them have a digitalized system to assess the gradual progress of translation competency of MTI candidates （Wang， 2010）. Most of the existed evaluation systems are of qualitative nature of MTI mentors giving each candidate an overall score from 1 to 100 based on their impression over their translation works. Such kind of assessment falls into the category of outcome evaluation， in which the subjective and personal impression of the evaluators counts a lot. Thus， naturally， the so-called halo effect cannot be avoided.

Though some schools have tried to design the so-called translation evaluation index systems， no agreement has reached yet over how to define the evaluation constructs， namely， translation competency， which should be the key in any kinds of translation assessment. Literature study shows that most current definitions of translation competency are expert-focused. They cannot be used to describe the status quo of translation-beginners， such as those MTI candidates. They are over-static and cannot reflect the dynamic progression of a translator’s translating skills （Li， 2011）. All in all， how to define translation competency has become a bottleneck on the way to developing a practical and objective translation evaluation index system.

However， the definition offered by Professor Li Ruilin sheds some lights over this issue. Professor Li synthesizes researches associated with translation capabilities and put forward a brand new construct---Translator Attainments （hereafter abbreviated as TA）. The construct is able to reflect the dynamic and gradual development of a translator’s translating abilities. Besides， it can also differentiate the individual difference between translators over their translating capabilities. What is worth mentioning is that it is more operable and can be employed as the theoretical basis of the MTI TEIS.

In order to make the MTI TEIS more hierarchical and objective， AHP method is adopted to offer weights to each level of indices， which makes the system both quantitative and qualitative in nature. Detailed information will be stated in the following sections.

1. THEORETICAL STUDY

TA is a construct coined by Professor Li Ruilin on the basis of the concept of Translator Competency （hereafter abbreviated as TC）. It is more comprehensive than the latter （Li， 2011）. With TC as its core， TA actually includes a host of physiological， psychological， social and linguistic capabilities which are essential for a translator to complete his/her translation tasks.

TC refers to a series of higher-order thinking skills that are necessary for translators to solve Problems related to translation， which may help translators to analyze， synthesize， assess and create cognition regarding translation as well as interact with translation situation based on the existing static knowledge and experience of a translator.

According to Professor Li， in the process of translation， higher-order thinking skills are employed when defining translation problems， selecting translating strategies， evaluating translation works and constructing cognitive schemes， etc..

As TC is always in a state that is constantly changing and restructuring， generally， a translator’s TA roughly goes through three stages， namely， the initial， intermediate and ultimate stages. In its initial stage， translators are only equipped with some static and discrete knowledge about translation. And their translation works are more like symbol converting， which are possibly full of negative transfer of source language and not very communicative. In medium stage， translators attempt to solve translation problems in a trial-and-error manner. Thus， it is exactly this period that translation teaching intervention may achieve its best results. In the ultimate stage， translators are able to gain in-depth understanding of translation and start to restructure their knowledge via higher-order thinking skills. Their translating activities are with more elasticity and tension， which characterizes their accomplishment in TA. Right in this stage， both the innovativeness and intelligent incremental level of a translator have reached the expert level （Li， 2011）.

Professor Li points out that TA can be mainly indicated by a translator’s accomplishments in language， knowledge， appreciation， strategy， as well as digital literacy and social achievement as well.

Analysis on Professor Li’s explanation about TA reveals that language accomplishment refers to the abilities via which translators may conduct meaningful activities such as explaining， negotiating and expressing the target language based on source text in different contexts. Knowledge accomplishment is associated with the capabilities via which translators may obtain， process and restructure information regarding translation tasks. Strategy accomplishment can be reflected by the capabilities of identifying ill-structured problems， creating innovative solution and selecting appropriate translation strategies， etc.. Digital literacy may be represented by a translator’s ability of employing modern digital technologies to complete translation tasks. Appreciation accomplishment can be seen from translators’ capabilities to evaluate their own as well as others’ translation works. Social accomplishment can be shown as a translator’s capabilities to manage， coordinate and coordinate a translation project as well as their social and professional responsibilities and self-positioning within a translation team. Professor Li’s effort in this respect has set a sound theoretical foundation for the establishment of indices at different levels in the MTI TEIS.

2. THE CONSTRUCTION OF MTI TEIS

2.1 The Establishment of Indices at Various Levels & the Hierarchical Structure of MTI TEIS

MTI TEIS is hierarchical in structure and consisted of three levels with level one being located at the top class and each lower level being subordinated to the higher level and logically related to it. Besides， each level has its particular indices. Since TA may better indicate the gradual and dynamic progression of translation competence of translators， it is decided that the key components of TA including language， knowledge， strategy， and appreciation accomplishments as well as digital literacy and social accomplishment are selected as level-one indices in MTI TEIS. When choosing level-two and level-three indices， the SMART principle is strictly observed. The acronym of SMART stands for specific， measurable， attainable， realistic and time-bound.

In addition， when designing level-two and level-three indices， the following information are also put into consideration， which is the training objectives of MTI， questions commonly encountered by translators， classical translation evaluation standards and related linguistic knowledge， etc..

To a large extent， the establishment of level two and three indices is aimed to be in line with actual translation situations and better reflect the creativity， problem-solving， decision-making and critical-thinking capabilities of MTI candidates. The established hierarchical structure of MTI TEIS is presented in Diagram 1 （see Appendix 1）.

2.2 The Construction of Judgment Matrix and Consistency Test

A five-member panel was set up. It consisted of experts majoring in translation. Firstly， each expert was given a printed copy of Table 1， Weighing standards based on the relative importance of indices. Then， they were explained about how to weigh each index according to Table 1. Secondly， the five experts started to give weight to each index. The results were compared and discussed by the panel. Any dispute would lead to repeating discussion among experts， until consensus was reached.

Table 1

Weighing Standards Based on the Relative Importance of Indices

Implications Weight

Xi is as important as Xj 1

Xi is a bit more important than Xj 3

Xi is more important than Xj 5

Xi is clearly more important than Xj 7

Xi is much more important than Xj 9

The importance of Xi over Xj is between every two levels mentioned above 2， 4， 6， 8

Based on the weight given by the 5 experts， 25 judgment matrixes were formed （see Appendix 2）. WithMatlab7.0， the consistency of each matrix was tested. Due to the existence of subjective elements， the matrix consistency in this study was hard to achieve. Because of this， C.R. was introduced. C. R. was the quotient of C.I. （Consistency Index） and R. I. （Random Index）. If C.R. of a matrix was less than 0.1， it indicated that all the indices within the matrix were consistent with each other in logic.

In this case， the relative weight of importance of the matrix could be gained. If not， the matrix had to be reconstructed.

2.3 The Complete MTI TEIS

After going through consistency test， the relative weight of importance of each index within the 25 matrixes were calculated， based on which a complete MTI TEIS was obtained （see Table 2）.

Table 2

The Complete MTI TEIS

Level one index Level two index Level three index

Language accomplishment

0.0673 Pragmatics

0.8333 Negative transfer of source language 0.2722

Translationese 0.2722

Source text comprehension 0.0308

Grammatical mistake in translation 0.2722

Conciseness 0.0497

Accurate diction 0.0380

Excellent translation 0.0648

Discourse

0.1667 Cohesion and coherence 0.25

Sign of restructuring in translated text 0.75

Knowledge

accomplishment

0.0673 Cross culture knowledge

0.1250 Successfully solve cultural conflict

1

Knowledge of related disciplines

0.8750 Know about technologies of related disciplines

0.25

Good at conducting multi-discipline learning 0.75

Strategy accomplishment

0.2611 Identification of ill-structured problems and putting forward corresponding solutions

0.1095 Identifying hidden translation problems 0.25

Creativity of the translation solution

0.75

Selection of translation strategies

0.3090 Selecting appropriate translating strategy 0.75

Creativity of translating strategy 0.25

Implementation of translation project

0.5816 Translation brief 0.8333

Vocabulary list 0.1667

Digital literacy

0.2818 Search engine employment

0.0572 Kinds of search engine 0.75

Selection of key words 0.25

Corpus employment

0.1594 Kinds of corpus 0.75

Familiarity with the corpus 0.25

Translation software employment

0.2839 Kinds of software 0.75

Familiarity with the software 0.25

Employment of digital technologies

0.4994 Kinds of digital technology 0.0841

Familiarity with the digital technology 0.2109

Make reasonable decision based on higher-order thinking skills and wise judgment 0.7049

Appreciation accomplishment

0.3025 Appreciating translation works

0.8333 Reasonableness when commenting on translated works 0.1250

Make reasonable decision based on higher-order thinking skills and wise judgment 0.8750

Self-assessment

0.1667 Reasonableness when commenting on self-translated works 0.75

Reasonableness when commenting on works translated by others 0.25

Social accomplishment

0.0201 Ethic managing competency

0.8750 Love the motherland， people and support the communist party 0.75

Like to serve others 0.25

Managing social network

0.1250 Collaborative skills 0.1666

Project management capabilities 0.7396

Appropriate positioning within a project

team 0.0938

CONCLUSION

MTI TEIS is a multi-source evaluation system. It is relatively more objective and reliable. It may reflect the development of a translator’s translation attainments at different stages. With the help of AHP， system may control the subjective impact on translation evaluation process to a large extent， and be more fair and fine-grained. In addition， with TA as its evaluation construct， MTI TEIS may better fit the reality of MTI teaching practice in China. It is hoped that MTI TEIS could be widely adopted and trialed in various MTI training centers. Constructive suggestions regarding the improvement of system are welcomed whole-heartedly.

REFERENCES

Colina， S. （2003）. Translation teaching： From research to the classroom. Boston， MA： McG raw-H ill.

Li， R， L.， & He， Y. （2011）. Research on translation project learning mode from the perspective of learning sciences. Foreign Language Education， 32（1）， 94-98.

Li， R. L. （2011）. From translation competency to translator’s attainments. Chinese Translators Journal， 33（1）， 46-50.

Lin， L.， & Lin， G. （2006）. The application of fuzzy mathematics and AHP in performance evaluation. China Management Informationization， 9（11）， 13-16.

Risku， H. A. （2010）. Cognitive scientific view on technical communication and translation： Do embodiment and situatedness really make a difference？ Target， 22 （1）， 94-111.

Rothe-Neves， R. （2007）. Notes on the concept of translator competence. Quaderns Traduccio， （4）， 125-138.

Wang， S. H.， & Wang， W. P.， （2010）. Developmental translation teaching evaluation model. Journal of PLA University of Foreign Languages， （3）， 76-81.

Aappendix 1

The Hierarchical Structure of MTI TEIS

Level one index Level two index Level three index

Language accomplishment A Pragmatics

A1 Negative transfer of source language A11

Translationese A12

Source text comprehension A13

Grammatical mistake in translation A14

Conciseness A15

Accurate diction A16

Excellent translation A17

Discourse

A2 Cohesion and coherence A21

Sign of restructuring in translated text A22

Knowledge

accomplishment

B Cross culture knowledge

B1 Successfully solve cultural conflict

B11

Knowledge of related disciplines

B2 Know about technologies of related disciplines

B21

Good at conducting multi-discipline learning B22

Strategy accomplishment

C Identification of ill-structured problems and putting forward corresponding solutions

C1 Identifying hidden translation problems C11

Creativity of the translation solution

C12

Selection of translation strategies

C2 Selecting appropriate translating strategy C21

Creativity of translating strategy C22

Implementation of translation project

C3 Translation brief C31

Vocabulary list C32

Digital literacy

D Search engine employment

D1 Kinds of search engine D11

Selection of key words D12

Corpus employment D2 Kinds of corpus D21

Familiarity with the corpus D22

translation software employment

D3 Kinds of software D31

Familiarity with the software D32

Employment of digital technologies

D4 Kinds of digital technology D41

Familiarity with the digital technology D42

Make reasonable decision based on higher-order thinking skills and wise judgmentD43

Appreciation accomplishment

E

Appreciating translation works

E1 Reasonableness when commenting on translated works E11

Self-assessment

E2 Make reasonable decision based on higher-order thinking skills and wise judgment E21

Reasonableness when commenting on self-translated works E22

Social accomplishment

Social accomplishment

F

Ethic managing competency

F1 Reasonableness when commenting on works translated by others F11

Love the motherland， people and support the communist party F12

Managing social network

F2 Like to serve others F21

Collaborative skills F21

Project management capabilities F22

Appendix 2

Judgment Matrix of Level-One Indices

A B C D E F

A 1 1 5 6 7 1/8

B 1 1 5 6 7 1/8

C 1/5 1/5 1 1 1 1/9

D 1/6 1/6 1 1 1 1/9

E 1/7 1/7 1 1 1 1/9

F 8 8 9 9 9 1

Judgment Matrix of Level-Two Indices Subordinate to A

A1 A2

A1 1 1/5

A2 5 1

Judgment Matrix of Level-Two Indices Subordinate to B

B1 B2

B1 1 7

B2 1/7 1

Judgment Matrix of Level-Two Indices Subordinate to C

C1 C2 C3

C1 1 3 5

C2 1/3 1 2

C3 1/5 1/2 1

Judgment Matrix of Level-Two Indices Subordinate to D

D1 D2 D3 D4

D1 1 3 5 8

D2 1/3 1 2 3

D3 1/5 1/2 1 2

D4 1/8 1/3 1/2 1

Judgment Matrix of Level-Two Indices Subordinate to E

E1 E2

E1 1 1/5

E2 5 1

Judgment Matrix of Level-Two Indices Subordinate to F

F1 F2

F1 1 1/7

F2 7 1

Judgment Matrix of Level-Three Indices Subordinate to A1

A11 A12 A13 A14 A15 A16 A17

A11 1 1 1/9 1 1/6 1/7 1/4

A12 1 1 1/9 1 1/6 1/7 1/4

A13 9 9 1 9 2 1 2

A14 1 1 1/9 1 1/6 1/7 1/4

A15 6 6 1/2 6 1 1 1

A16 7 7 1 7 1 1 2

A17 4 4 1/2 4 1 1/2 1

Judgment Matrix of Level-Three Indices Subordinate to A2

A21 A22

A21 1 3

A22 1/3 1

Judgment Matrix of Level-Three Indices Subordinate to B1

B11

B11 1

Judgment Matrix of Level-Three Indices Subordinate to B2

B21 B22

B21 1 3

B22 1/3 1

Judgment Matrix of Level-Three Indices Subordinate to C1

C11 C12

C11 1 3

C12 1/3 1

Judgment Matrix of Level-Three Indices Subordinate to C2

C21 C22

C21 1 1/3

C22 3 1

Judgment Matrix of Level-Three Indices Subordinate to C3

C31 C32

C31 1 1/5

C32 5 1

Judgment Matrix of Level-Three Indices Subordinate to D1

D11 D12

D11 1 1/3

D12 3 1

Judgment Matrix of Level-Three Indices Subordinate to D2

D21 D22

D21 1 1/3

D22 3 1

Judgment Matrix of Level-Three Indices Subordinate to D3

D31 D32

D31 1 1/3

D32 3 1

Judgment Matrix of Level-Three Indices Subordinate to D4

D41 D42 D43

D41 1 3 7

D42 1/3 1 4

D43 1/7 1/4 1

Judgment Matrix of Level-Three Indices Subordinate to D5

D51 D52

D51 1 3

D52 1/3 1

Judgment Matrix of Level-Three Indices Subordinate to E1

E11 E12

E11 1 7

E12 1/7 1

Judgment Matrix of Level-Three Indices Subordinate to E2

E21 E22

E21 1 1/3

E22 3 1

Judgment Matrix of Level-Three Indices Subordinate to F1

F11 F12

F11 1 1/3

F12 3 1

Judgment Matrix of Level-Three Indices Subordinate to F2

F21 F22 F23

F21 1 5 1/2

F22 1/5 1 1/7

F23 2 7 1