See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/257036314

Adaptation of the Rating Scale Mental Effort

(RSME) for use in Indonesia

Article in International Journal of Industrial Ergonomics · January 2013

DOI: 10.1016/j.ergon.2012.11.003

CITATIONS READS

4 954

3 authors:

Ari Widyanti Addie Johnson

Bandung Institute of Technology University of Groningen
6 PUBLICATIONS 19 CITATIONS 70 PUBLICATIONS 1,270 CITATIONS

SEE PROFILE SEE PROFILE

Dick de Waard
University of Groningen
115 PUBLICATIONS 3,115 CITATIONS

SEE PROFILE

All in-text references underlined in blue are linked to publications on ResearchGate, Available from: Addie Johnson
letting you access and read them immediately. Retrieved on: 16 November 2016
 International Journal of Industrial Ergonomics 43 (2013) 70e76

Contents lists available at SciVerse ScienceDirect

International Journal of Industrial Ergonomics

journal homepage: www.elsevier.com/locate/ergon

Adaptation of the Rating Scale Mental Effort (RSME) for use in Indonesia
Ari Widyanti a, b, *, Addie Johnson a, Dick de Waard a
a
Department of Psychology, University of Groningen, Grote Kruistraat 2/1, 9712 TS Groningen, The Netherlands
b
Department of Industrial Engineering, Bandung Institute of Technology (ITB), Bandung, Indonesia

a r t i c l e i n f o a b s t r a c t

Article history: The Rating Scale Mental Effort (RSME) is a unidimensional instrument used to measure subjective mental
Received 7 April 2012 workload. The RSME consists of a line with a length of 150 mm marked with nine anchor points, each
Received in revised form accompanied by a descriptive label indicating a degree of effort. The RSME has been widely used in
11 October 2012
Western countries (e.g., in Europe and North America), but, when translated to Indonesian, has been
Accepted 6 November 2012
Available online 7 December 2012
shown to be relatively insensitive to changes in mental effort among Indonesians, raising the question of
whether the insensitivity is related to national culture or to the translation of the scale. To investigate
whether the relative insensitivity of the RSME might lie on the translation of the instrument, a new,
Keywords:
RSME
seven-anchor point, version of the RSME was created to better reflect how the anchor points are
Mental workload interpreted by Indonesians. The new version of the RSME was compared with the existing, nine-anchor
Subjective measures point version of the instrument in an experiment in which 100 participants performed a search task of
Self-reports different levels of difficulty. Half of the participants used the original RSME and half the adapted
Culture instrument. The adapted instrument appeared to be more sensitive to changes in load than the original
RSME. However, a comparison with Dutch participants who performed the same task suggests that
culture, and not only properties of the scale used to measure mental effort, influences the measurement
of subjective mental workload.
Relevance to industry: As industrialization increases in developing countries, so does the need for low-
cost, easy to implement, and valid mental-workload instruments. We adapted the Rating Scale Mental
Effort (RSME) for use by Indonesians. The adapted RSME shows improved sensitivity, but cultural
differences in reporting load are still apparent.
Ó 2012 Elsevier B.V. All rights reserved.

1. Introduction low cost, and because their use is relatively unobtrusive in work
situations (De Waard, 1996; O’Donnell and Eggemeier, 1986).
An increasing emphasis on safety, health, and comfort in the Although some subjective measures of mental workload have
workplace has made the evaluation of operator mental workload in been developed recently (e.g., Mental Workload Index; Pretorius
the workplace a key issue (DiDomenico and Nussbaum, 2011). and Cilliers, 2007), the most frequently used subjective measures
Mental workloadddefined as the difference between the cognitive of mental workload are The NASA-Task Load Index (NASA-TLX; Hart
resources demanded by the task and the cognitive resources and Staveland, 1988), the Subjective Workload Assessment Tech-
available to and assigned by the worker (Gopher and Donchin, nique (SWAT; Reid and Nygren, 1988), and the Rating Scale Mental
1986)dis often measured with subjective, self-report measures, Effort (RSME; Zijlstra and Van Doorn, 1985; Zijlstra, 1993; for
either in isolation or together with performance measures (e.g., a review see Rubio et al., 2004). Both the NASA-TLX and the SWAT
reaction time), or physiological measures such as heart-rate (Cain, are multidimensional scales, whereas the RSME is a unidimensional
2007; O’Donnell and Eggemeier, 1986), eye movement data (Di scale. The SWAT, for example, contains the scales time load, mental
Stasi et al., 2011), optical brain measures (Ayaz et al., 2012) or effort load, and psychological stress and takes about 45 min to
combined physiological measures (Ryu and Myung, 2005). complete, not including scoring (European Organization for the
Subjective measures are popular because of their ease of use and Safety of Air Navigation, 2003). The NASA-TLX includes six scales
(mental demand, physical demand, temporal demand, perfor-
mance, effort, and frustration level) and takes about 10 min of the
* Corresponding author. Department of Psychology, University of Groningen,
Grote Kruistraat 2/1, 9712 TS Groningen, The Netherlands. Tel.: þ31 50 363 3845;
operator’s time to complete. The RSME, which measures only
fax: þ31 50 363 6304. “mental effort”, can be completed in less than a minute. An
E-mail address: [email protected] (A. Widyanti). important question is whether multidimensional rating scales

0169-8141/$ e see front matter Ó 2012 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.ergon.2012.11.003
 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76 71

provide benefits as compared to a simple, unidimensional scale. increasing levels of workload because doing so would assert their
One reason to prefer multidimensional rating scales is their diag- own sense of effort and set them apart from the group.
nosticity. That is, because different aspects of load are rated, it is The finding that Indonesians and the Dutch use rating scales of
possible to determine, to some extent, which factors contribute to mental workload differently was to some extent predicted because,
workload (Dey and Mann, 2010; Rubio et al., 2004; Wierwille and as noted by Nachreiner (1999), subjective measures are likely to be
Eggemeier, 1993). However, it has been argued that unidimen- language and culture dependent, and it has been argued that
sional rating scales can be as useful and as sensitive to changes in implementation of subjective measures such as the RSME in places
mental workload as are multidimensional rating scales in many other than the countries where they were developed can be subject
conditions (Veltman and Gaillard, 1996), and, in fact, the RSME has to problems of interpretation (Van de Vijver and Poortinga, 1997).
repeatedly been shown to be sensitive to changes in mental Before concluding that cultural factors are responsible for differ-
workload in both laboratory (e.g., Mulder et al., 2009; Wilson et al., ences in how subjective ratings of mental workload are made, it
2006) and applied (e.g., De Waard, 1996; Lin and Cai, 2009) settings. must be ruled out that the properties of the scales used across
The RSME consists of a 150-point, 15-cm long vertical line groups of comparison are responsible for these differences. In
marked at 1-cm intervals, and containing nine anchor points with general, instruments developed within one culture and used in
descriptive labels ranging from “absolutely no effort” (close to the another culture can be subject to construct, method, or item bias
0 point on the 0e150 point scale), through “rather much effort” (Van de Vijver and Poortinga, 1997). Construct bias is said to occur
(about 57 on the scale), to “extreme effort” (about 112 on the scale). when there is incomplete overlap in the definition of the construct
Responses are made by marking the line at the point corresponding across cultures or an incomplete coverage of the construct. Method
to the amount of mental effort taken to complete a task. The bias can result from different response styles, lack of comparability
development of the RSME was based on magnitude estimates of of samples (e.g., differences in education, age, and gender compo-
effort made by students (n ¼ 39) and bus drivers (n ¼ 25) in the city sition), different familiarity with response procedures, or inter-
of Groningen, The Netherlands. Of the 17 labels that were originally viewer effects. Finally, item bias may occur when items are
rated, nine labels, selected to mark approximately equal intervals inadequately translated or formulated.
on the 150-point scale, were selected for inclusion in the final Although mental workload should be assessed with different
version of the scale (see Van Doorn and Zijlstra, 1988; Zijlstra and techniques (e.g., physiological and performance measures), in
Van Doorn, 1985, for details). In previous work, we (Johnson and practice, low-cost, easy to implement methods are the most used.
Widyanti, 2011) reported how the RSME was translated to Indo- We therefore focus on the optimization of the RSME. The purpose of
nesian and used to assess mental workload along with an Indo- the present study was to determine whether a more sensitive
nesian translation of the NASA-TLX. The major finding in that study Indonesian version of the RSME than that used by Johnson and
was that the range of ratings of subjective measures of mental Widyanti (2011) could be developed by eliminating any construct
workload imposed by a demanding search task (as measured by the or item bias which may have been present due to the descriptive
RSME and the NASA-TLX) was more restricted among Indonesian labels and anchor points used in the original Indonesian version of
participants than among the Dutch participants that they were the RSME.
compared to. The range restriction in the Indonesian as compared First, the descriptive labels of the anchor points of the English-
to the Dutch ratings is of concern because sensitivitydthe capa- language RSME were newly translated into Indonesian using
bility to detect changes in the levels of workload imposed by task back-translation (Brislin, 1970; see Fig. 1). A group of 80 Indonesian
performancedis an essential property of measures of mental students was then asked to match the descriptive labels to the
workload (O’Donnell and Eggemeier, 1986). Range restriction in original nine anchor points of the RSME. Three of the descriptive
scale use thus limits the effectiveness of the instruments. labels (“rather much”, “considerable”, and “great” effort) were
We (Johnson and Widyanti, 2011) interpreted the differences we found to be inconsistently assigned to the anchor points. These
found in how the RSME and NASA-TLX were used by Indonesian vs. three descriptors were thus collapsed into one label capturing the
Dutch participants in terms of culture. For example, because common meaning of the original three labels (the Indonesian
subjective mental workload depends on the ability and willingness equivalent of “moderate” effort) and a separate group of 80 Indo-
of participants to report their mental state, differences in how rating nesian students was asked to place the descriptive labels on
scales are used may be influenced by the fact that the expression of a version of the RSME with only the 150-point scale marked, and no
thoughts, preferences, and feelings is not encouraged in Eastern separate anchor points. The anchor points derived from these data
cultures, such as that of Indonesia, whereas such expression is were used to create a new version of the RSME (the “adapted
commonly valued in Western cultures (i.e., in cultures with RSME”). The sensitivities of the original RSME and the adapted
a Western European cultural heritage, including North America). RSME were then compared in an experiment in which participants
We also suggested that the relatively high sense of “conformity” (the performed a search task in which visual displays were searched for
restraint of actions, inclinations, and impulses likely to upset or target items held in memory (Schneider and Shiffrin, 1977; Shiffrin
harm others or to violate social expectations or norms; Schwartz and Schneider, 1977) with different levels of difficulty (as indicated
et al., 2001) that characterizes Indonesians may cause them to be by changes in reaction time and error rates) that has been shown to
relatively conservative in rating mental workload and that a rela- reliably induce different levels of mental workload (e.g., Aasman
tively high belief in “fate control” (a belief that life events are pre- et al., 1987; Johnson and Widyanti, 2011; Rubio et al., 2004).
determined; Leung et al., 2002) in Indonesian participants also may Groups of 50 Indonesian students each rated the subjective mental
lead them to underestimate changes in mental effort, as self-effort is workload associated with each condition of the search task using
not ascribed a large role in influencing performance. Finally, the either the original or the adapted RSME.
high power distance (the extent to which the less powerful
members of institutions expect and accept that power is distributed 2. Materials and methods
unequally; Hofstede and Hofstede, 2005) and collectivism (the
extent to which individuals are integrated into strong, cohesive in- 2.1. Development of the adapted RSME
groups groups as opposed to being expected to act individually and
care for themselves; Hofstede and Hofstede, 2005) that characterize Fig. 1 gives an overview of the steps taken to create the
Indonesia may make Indonesians less willing to admit experiencing adapted RSME. The English-language version of the RSME
72 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76

Fig. 1. An overview of the steps taken to create the adapted RSME.

(Zijlstra, 1993) was newly translated into Indonesian following nine anchor points of the RSME. The order of the descriptive
a procedure similar to that used by Johnson and Widyanti (2011). labels was randomized and then presented in the same,
First, ten bilingual master’s students of the Industrial Engineering randomized order to all participants on a sheet of paper together
Department of Bandung Institute of Technology (ITB; TOEFL with the 0e150 point (15-cm) RSME scale. Participants were
scores 475 or higher) performed independent forward trans- asked to draw a line from each descriptive label to one of the nine
lations of the nine descriptive labels of the RSME, presented in anchor points, using each anchor point exactly once. Data from
order of increasing effort, into Indonesian. Following this, two nine participants were excluded from further analysis due to
bilingual ITB faculty members (TOEFL scores 575 or higher and at failure to follow the instructions (e.g., matching one descriptive
least 4 years residence in the USA) compared the ten translations label to more than one anchor point), leaving data from 71
and reached consensus on the words used and their order. An participants for analysis.
EnglisheIndonesian dictionary (Wojowasito and Poerwadarminta, Of the 71 participants, 43 participants (60.6%) positioned each
1991) was consulted where necessary. Two Ph.D. students descriptive label of the RSME to the same anchor points as in the
(TOEFL 475 or higher) then translated the scale back into English English version of the scale. Eleven of the 28 participants trans-
for comparison. The two faculty members then resolved any posed the descriptive labels “rather much effort” and “considerable
remaining differences in the translation. Once the RSME had been effort”. Seven participants reversed the order of “considerable
translated into Indonesian, the adequacy of the translation was effort” and “great effort”. One participant reversed the order of
checked by asking 80 students (mean age ¼ 19.48 years, “some effort” and “rather much effort”. Five participants made two
SD ¼ 1.19, 38 female) from the Industrial Engineering Department or more transpositions of three labels. The remaining four partici-
of ITB to match the Indonesian descriptive labels to the original pants transposed four or five descriptive labels.
 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76 73

We next investigated whether collapsing the confusing labels

and allowing participants to choose different anchor points would
result in more consensus as to the order of the anchor points. The
descriptive labels that were subject to the most transpositions
(“rather much effort”, “considerable effort”, and “great effort”) were
collapsed into one label that was close to the meaning of the three
confusing labels (“cukupbesar” in Indonesian; “moderate effort” in
English), such that seven labels remained. A new group of 80 ITB
students (mean age ¼ 20.9 years, SD ¼ 0.8, 37 female) was asked to
place the seven descriptive labels on a version of the scale used in
the RSME that did not contain anchor points. The descriptive labels
were presented in the same, randomized order to all participants
on a sheet of paper together with the 0e150 point scale. Descriptive
labels could be placed at any position and in any order on the scale
by drawing a line from each descriptive label to a point on the scale.
One participant was excluded from analysis due to failure to follow
the instructions, leaving data from 79 participants for analysis.
The means and standard deviations of the anchor points chosen
by the 79 participants are shown in Table 1. The means were used as
the anchor points for a new Indonesian version of the RSME, which
we refer to as the adapted RSME (see Fig. 2).

2.2. Comparison of the original and adapted RSME

To determine whether the sensitivity of the adapted RSME

would be better than the original RSME, we tested Indonesian
students in a search task (Schneider and Shiffrin, 1977; Shiffrin and
Schneider, 1977) with four conditions that differed in level of
difficulty. Participants rated the subjective mental workload asso-
ciated with each condition by filling out either the original RSME or
the adapted RSME upon completion of each condition.

2.2.1. Participants
One hundred students of the Industrial Engineering Department
of ITB (mean age ¼ 20.48 years, SD ¼ 1.72, 62 female) participated
in the experiment; 50 used the original RSME to rate subjective
mental effort and 50 used the adapted RSME. Four participants Fig. 2. Adapted Rating Scale Mental effort (RSME).
were excluded from further analysis due to poor task performance,
leaving 48 in the group who used the original RSME and 48 in the
group who used the adapted RSME. there were two items in each search frame; in Condition 2 there
were two items in the memory set and search frames; in Condition
2.2.2. Task 3 there were two items in the memory set and four items in the
Participants performed a search task in which one, two, or four search frames; and in Condition 4 there were four items in both the
target items had to be held in memory while searching a visual memory set and search frames. The appropriate version of the
display of two or four items for the targets held in memory RSME was presented on the computer screen after each of the four
(Schneider and Shiffrin, 1977; Shiffrin and Schneider, 1977; see conditions and was marked by clicking a point on the scale using
Johnson and Widyanti, 2011, for details). the computer mouse.
The stimuli were presented in four conditions, varying in diffi-
culty. In Condition 1 there was one item in the memory set and
3. Results
Table 1
The anchor points of the original and the adapted RSME. 3.1. Subjective mental workload
Descriptive labels Anchor points Mean values of anchor
of the original RSME points given by Indonesian Ratings of subjective mental workload were compared using
participants (standard deviations a mixed ANOVA with group (original vs. adapted RSME) as
in parentheses) a between-subjects factor and condition (1e4, in order of difficulty)
Extreme effort 112 137 (18.6) as a within-subjects factor. As shown in Fig. 3, the effect of condi-
Very great effort 102 115 (17.6) tion was significant (F(3, 282) ¼ 36.93, p < .001, MSE ¼ 151.83), with
Great effort 85 89 (17.6)
increasing task demands being reflected in higher ratings on the
Considerable effort 71 e
Rather much effort 57 e RSME. The mean scores were numerically higher (70.76 vs. 64.06)
Moderate effort e 52 (16.0) and the range of scores numerically greater (22.19 vs. 14.67) for the
Some effort 37 e adapted version of the RSME than for the original RSME, but neither
A little effort 25 34 (13.9) the effect of group (F(1, 94) ¼ 3.265, p ¼ .074, MSE ¼ 1321), nor the
Almost no effort 13 16 (8.3)
Absolutely no effort 2 1 (2.3)
Condition  Group interaction (F(3, 282) ¼ 1.65, p ¼ .179,
MSE ¼ 151.83) were significant.
74 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76

Fig. 3. Mean RSME scores as a function of group (group using the original RSME vs.
Fig. 4. Mean RSME scores as a function of group (Dutch participants from study of
group using the adapted RSME) and condition (in increasing order of difficulty). Error
Johnson and Widyanti, using the original RSME vs. Indonesian participants from the
bars represent the standard error of the mean.
present study, using the adapted RSME) and condition (in increasing order of diffi-
culty). Error bars represent the standard error of the mean.
3.2. Between-study comparisons
Incorrect responses (4.03% of trials) and RTs less than 150 ms and
The experimental task and procedure of the current study were greater than the condition mean plus 2.5 standard deviations,
identical to those used by Johnson and Widyanti (2011), making it computed separately for target-present and target-absent trials
possible to compare the present data with those obtained in that (6.37% of correct trials), were excluded from the RT analyses.
study. This comparison allows us to determine how stable the Accuracy and RT were analyzed separately using mixed ANOVAs
results of the original RSME are for Indonesian participants and to with group (group using original vs. group using adapted RSME) as
get an indication of how the results obtained for Indonesians using a between-subjects factor and condition (1e4, in order of difficulty
the adapted RSME compare to those of Dutch participants using the due to increasing memory-set and/or search-frame size), and target
original RSME. presence (target present vs. target absent) as within-subject factors.
The range of values obtained with the original version of the As shown in Fig. 5, performance on the search task was similar
RSME was very similar to that of the Indonesian participants in for both groups. The main effect of group was not significant
the Johnson and Widyanti (2011) study (mean values of 57e71 vs. for either accuracy (F(1, 94) ¼ 1.272, p ¼ .262, MSE ¼ 0.006) or RT
56e70 for the present and 2011 studies, respectively), and was (F(1, 94) ¼ 0.125, p ¼ .725, MSE ¼ 70,958). The effect of condition
smaller than the range found for Dutch participants in the 2011 was significant for both accuracy (F(3, 282) ¼ 88.40, p < .001,
study (mean values of 39e70). Although cross-experiment MSE ¼ 0.002) and RT (F(3, 282) ¼ 906.32, p < .001, MSE ¼ 14,829),
comparisons must be interpreted with caution, it is noteworthy with RT increasing and accuracy decreasing as a function of
that the Condition  Group interaction (F(3, 534) ¼ 3.35, p ¼ .027, increasing task difficulty. There were also significant effects of target
MSE ¼ 139.35) was significant when the data from the Indonesian presence on accuracy (F(1, 94) ¼ 250.27, p < .001, MSE ¼ 0.003) and
participants in the 2011 study (N ¼ 84) were combined with those RT (F(1, 94) ¼ 618.21, p < .001, MSE ¼ 10,494), reflecting that accu-
of the present study and a mixed ANOVA with group (group using racy was higher and RTs were faster on target-present trials than on
original vs. group using adapted RSME) as a between-subjects target-absent trials. The Condition  Target Presence interaction was
factor and condition (1e4 in order of difficulty) as a within- significant for both accuracy (F(3, 282) ¼ 102.40, p < .001,
subjects factor was conducted, suggesting that the adapted Indo- MSE ¼ 0.002) and RT (F(3, 282) ¼ 312.54, p < .001, MSE ¼ 3749),
nesian RSME is more sensitive to changes in workload than is the reflecting a greater effect of condition on target-absent than on
original version of the RSME. A cross-experiment comparison of the target-present trials. No interactions involving group were signifi-
Indonesian participants from the current study who used the cant (all ps’ >.15).
adapted RSME and the Dutch group (N ¼ 82) from the 2011 study
by means of a mixed ANOVA with group (Indonesian group using 4. Discussion
adapted RSME vs. Dutch group using original RSME) as a between-
subjects factor and condition (1e4 in order of difficulty) as a within- To evaluate whether the relative insensitivity of the RSME
subjects factor revealed a marginally significant Condition  Group previously found for Indonesian as compared to Dutch participants
interaction (F(3, 384) ¼ 2.57, p ¼ .060, MSE ¼ 207.97; see Fig. 4), could be due to properties of the scale, itself, a version of the RSME
suggesting that the sensitivity of the RSME is greater for Dutch than with new anchor points at different locations than those of the
for Indonesian participants, even when the adapted version of the original RSME was developed. The original and adapted scales were
RSME is used by the Indonesians. then compared in an experiment in which participants performed
a search task under four conditions of varying difficulty.
3.3. Search-task performance First, it was found that there was indeed some confusion about
the ordering of the descriptive labels of the original RSME scale,
Search-task performance was analyzed to ensure that the especially in the middle range of the scale. The reason for this is
performance of the two groups was similar so that differences in that the Indonesian words for “considerable”, “rather much”, and
estimates of mental effort could be attributed to differences in the “great effort” have similar meanings in Indonesian. To mitigate the
scales, and not differences in expended effort. The dependent effects of these confusing labels, the three labels were collapsed
variables for the search task were accuracy (proportion correct) and into one label, “cukupbesar” (in English, “moderate effort”),
reaction time (RT). All responses that exceeded the time limit of 5 s and the seven labels were assigned to anchor points in a manner
(0.03% of trials) were excluded from the analysis of accuracy. similar to the original RSME. It was found that a broader range of
 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76 75

Fig. 5. Mean accuracy (upper panels) and mean reaction time (lower panels) as a function of target presence (target present or absent), condition (in increasing order of difficulty),
and group (group using original vs. group using adapted RSME).

the 150-point scale was used in the adapted than in the original view that culture must be taken into account in administering and
RSME (1e137 vs. 2e112 points for the adapted and original RSME, interpreting measures of mental workload (Nachreiner, 1999). For
respectively). Because a broader range of the scale was covered by example, in cultures in which conformity is valued and where
the anchor points, we thought it possible that the adapted RSME power distance is high, better results may be obtained when test
might provide a more sensitive measure of mental workload for results are anonymous, as such individuals prefer not to stand out
Indonesians than the original RSME. Although the results of from the crowd and feel uncomfortable in expressing emotions or
a search experiment showed no significant differences in the opinions to superiors (Hofstede and Hofstede, 2005). Thus, in
sensitivity of the original and adapted RSME, an examination of the addition to taking care to equate samples as much as possible on
data from the present experiment combined with the data reported relevant variables, it seems advisable to measure cultural values
by Johnson and Widyanti (2011) suggest that the adapted instru- that influence the sensitivity of mental workload measures (e.g.,
ment may be more sensitive than the original version. power distance, conformity) using instruments such as the Portrait
The evidence that the adapted version of the RSME is more Values Questionnaire (PVQ; Schwartz et al., 2001) and the Hofstede
sensitive than the original Indonesian version of the RSME, Value Survey (Hofstede et al., 2008), or to use published data where
although mixed, together with the suggestion (by comparison with available (e.g., http://www.geert-hofstede.com/) to determine the
data from Johnson and Widyanti, 2011) that Dutch participants use need to adapt procedures to measure mental load.
a larger range of the RSME than do Indonesians even when the Our adaptation of the RSME was carried out in accord with
RSME has been adapted to avoid problems of translation, suggests International Test Commission (ITC) guidelines (see Hambleton,
that the differences in the sensitivity of the mental workload 2001) for test development and adaptation. However, the study
instrument as a function of cultural group are not related only to was limited in that only students served as participants and only one
the properties of the scale. In other words, even though the adapted experimental task was used to measure mental workload. More
RSME appears to be an improvement on the original RSME for insight into how mental workload is reported would likely be gained
Indonesian participants, Indonesians still use the scale differently if worker populations and a broader range of tasks were used.
than do the Dutch. The importance of taking into account not only
language, but also culture when adapting measures of subjective 5. Conclusions
mental workload was alluded to by Hart (2006) with respect to
another popular measure of subjective mental workload, the NASA- The adapted RSME with a reduced number of descriptive labels
TLX. However, Hart reported no data regarding how culture may appears to be used differently than the original RSME. Because
influence subjective mental workload ratings. a broader range of values is used, the adapted instrument may be
Johnson and Widyanti’s (2011) suggestions that cultural more sensitive to changes in demand than the original one.
dimensions reflecting conformity (Schwartz et al., 2001) and fate However, Dutch participants in an earlier study (Johnson and
control (Leung et al., 2002) moderate the effectiveness of subjective Widyanti, 2011) appear to use an even larger range of the RSME
workload measuresdand that it is reasonable to assume that than do Indonesians who use either the original or the adapted
workers in countries with a high power distance (Hofstede et al., Indonesian RSME, which seems to point to fundamental differences
2008) and low individualism (Hofstede et al., 2008) are likely to in how Dutch and Indonesian participants rate experienced work-
be more reluctant to report increases in workloaddare supported load. This fact suggests that not only scale properties, but also
by the current results. The finding of a relation between individual culture has an influence on the measurement of subjective mental
and cultural values and reports of mental workload supports the workload in Indonesia.
76 A. Widyanti et al. / International Journal of Industrial Ergonomics 43 (2013) 70e76

References Nachreiner, F., 1999. International standards on mental workload e the ISO 10075
series. Industrial Health 37, 125e133.
O’Donnell, R.D., Eggemeier, F.T., 1986. Workload assessment methodology. In:
Aasman, J., Mulder, G., Mulder, L.J.M., 1987. Operator effort and the measurement of
Boff, K.R., Kaufman, L., Thomas, J.P. (Eds.), Handbook of Perception and Human
heart rate variability. Human Factors 29, 161e170.
Performance. Cognitive Process and Performance, vol. II. Wiley, New York,
Ayaz, H., Shewokis, P.A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B., 2012. Optical
pp. 42-1e42-9.
brain monitoring for operator training and mental workload assessment.
Pretorius, A., Cilliers, P.J., 2007. Development of a mental workload index: a systems
NeuroImage 59, 36e47.
approach. Ergonomics 50, 1503e1515.
Brislin, R.W., 1970. Back-translation for cross-cultural research. Journal of Cross-
Reid, G.B., Nygren, T.E., 1988. The subjective workload assessment technique:
Cultural Psychology 1, 185e216.
a scaling procedure for measuring mental workload. In: Hancock, P.A., Mesh-
Cain, B., 2007. A Review of the Mental Workload Literature [online]. (Report).
kati, N. (Eds.), Human Mental Workload. Elsevier, Amsterdam, pp. 185e218.
Available from: http://www.dtic.mil/cgi-bin/GetTRDoc?AD¼ADA474193
Rubio, S., Diaz, E., Martin, J., Puente, J.M., 2004. Evaluation of subjective mental
(accessed 20.12.11.).
workload: a comparison of SWAT, NASA-TLX, and workload profile methods.
De Waard, D., 1996. The Measurement of Drivers’ Mental Workload. PhD thesis.
Applied Psychology: An International Review 53, 61e86.
University of Groningen, Groningen, The Netherlands.
Ryu, K., Myung, R., 2005. Evaluation of mental workload with a combined measure
Dey, A., Mann, D.D., 2010. Sensitivity and diagnosticity of NASA-TLX and simplified
based on physiological indices during a dual task of tracking and mental
SWAT to assess the mental workload associated with operating an agricultural
arithmetic. International Journal of Industrial Ergonomics 35, 991e1009.
sprayer. Ergonomics 53, 848e857.
Schneider, W., Shiffrin, R.M., 1977. Controlled and automatic human informa-
DiDomenico, A., Nussbaum, M.A., 2011. Effects of different physical workload
tion processing: I. Detection, search, and attention. Psychological Review
parameters on mental workload and performance. International Journal of
84, 1e70.
Industrial Ergonomics 41, 255e260.
Schwartz, S.H., Melech, G., Lehmann, A., Burgess, S., Harris, M., Owens, V., 2001.
Di Stasi, L.L., Antolí, A., Gea, M., Cañas, J.J., 2011. A neuroergonomic approach to
Extending the cross-cultural validity of the theory of basic human values with
evaluating mental workload in hypermedia interactions. International Journal
a different method of measurement. Journal of Cross-Cultural Psychology 32,
of Industrial Ergonomics 41, 298e304.
519e542.
European Organization for the Safety of Air Navigation, 2003. Review of Workload
Shiffrin, R.M., Schneider, W., 1977. Controlled and automatic human information
Measurement, Analysis and Interpretation Methods [online]. (Report). Available
processing: II. Perceptual learning, automatic attending and a general theory.
from: http://www.eurocontrol.int/integra/gallery/content/public/documents/
Psychological Review 84, 127e190.
expt_def_wp2_final.pdf (accessed 20.12.11.).
Van de Vijver, F.J.R., Poortinga, Y.H., 1997. Towards an integrated analysis of bias in
Gopher, D., Donchin, E., 1986. Workload e an examination of the concept. In:
cross-cultural assessment. European Journal of Psychological Assessment 13,
Boff, K.R., Kaufman, L., Thomas, J.P. (Eds.), Handbook of Perception and Human
29e37.
Performance. Cognitive Process and Performance, vol. II. Wiley Interscience,
Van Doorn, L., Zijlstra, F.R.H., 1988. Variation in response functions complicates the
New York, pp. 41-1e41-49.
evaluation of scales. In: Saris, W.E. (Ed.), Variation in Response Functions:
Hambleton, R.K., 2001. The next generation of the ITC test translation and adap-
a Source of Measurement Error in Attitude Research. Sociometric Research
tation guidelines. European Journal of Psychological Assessment 17, 164e172.
Foundation, Amsterdam, pp. 87e97.
Hart, S.G., 2006. NASA-Task Load Index (NASA-TLX); 20 years later. In: Proceedings
Veltman, J.A., Gaillard, A.W.K., 1996. Pilot workload evaluated with subjective and
of the Human Factors and Ergonomics Society 50th Annual Meeting. Human
physiological measures. In: Brookhuis, K.A., Weikert, C., Moraal, J., De Waard, D.
Factors and Ergonomics Society, Santa Monica, CA, pp. 904e908.
(Eds.), Aging and Human Factors. Proceedings of the Europe Chapter of
Hart, S.G., Staveland, L.E., 1988. Development of NASA-TLX (Task Load Index):
the Human Factors and Ergonomics Society Annual Meeting 1993. University
results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N.
of Groningen, Traffic Research Centre, Haren, pp. 107e128. Retrieved from:
(Eds.), Human Mental Workload. Elsevier Science Publisher, North-Holland,
http://www.hfes-europe.org/books/agibk.htm.
pp. 139e184.
Wilson, M., Smith, N.C., Chattington, M., Ford, M., Marple-Horvat, D.E., 2006. The
Hofstede, G., Hofstede, G.J., 2005. Cultures and Organizations: Software of the Mind,
role of effort in moderating the anxiety e performance relationship: testing the
second ed. McGraw Hill, New York.
prediction of processing efficiency theory in simulated rally driving. Journal of
Hofstede, G., Hofstede, G.J., Minkov, M., Vinken, H., 2008. Value Survey Module
Sports Sciences 24, 1223e1233.
2008 Manual [online]. Available from: http://www.geerthofstede.nl/research–
Wierwille, W.W., Eggemeier, F.T., 1993. Recommendations for mental workload
vsm/vsm-08.aspx (accessed 20.12.11.).
measurement in a test and evaluation environment. Human Factors 19,
Johnson, A., Widyanti, A., 2011. Cultural influences on the measurement of
263e281.
subjective mental workload. Ergonomics 54, 509e518.
Wojowasito, S., Poerwadarminta, W.J.S., 1991. KamuslengkapInggeris-Indonesia
Lin, Y., Cai, H., 2009. A method for building a real-time cluster-based continuous
Indonesia-Inggerisdenganejaan yang disempurnakan (English-Indonesian,
mental workload scale. Theoretical Issues in Ergonomics Science 10, 531e543.
Indonesian-English dictionary). Hasta, Bandung.
Leung, K.L., Bond, H.M., De Carrasquel, S.R., Munoz, C., Hernandez, M., Murakami, F.,
Zijlstra, F.R.H., 1993. Efficiency in Work Behavior: a Design Approach for Modern
Singelis, T.M., 2002. Social axioms: the search for universal dimensions of
Tools. Doctoral thesis. Delft University of Technology, Delft, The Netherlands.
general beliefs about how the world functions. Journal of Cross-Cultural
Retrieved from: http://repository.tudelft.nl/view/ir/uuid%3Ad97a028b-c3dc-
Psychology 33, 286e302.
4930-b2ab-a7877993a17f/.
Mulder, L.J.M., Dijksterhuis, C., Stuiver, A., De Waard, D., 2009. Cardiovascular state
Zijlstra, F.R.H., Van Doorn, L., 1985. The Construction of a Scale to Measure Perceived
changes during performance of a simulated ambulance dispatchers’ task:
Effort. Technical Report. Delft University of Technology.
potential use for adaptive support. Applied Ergonomics 40, 965e977.