Predicting Portuguese Steam Review Helpfulness Using

Artificial Neural Networks

Jardeson L. N. Barbosa Raimundo Santos Moura Roney L. de S. Santos

Universidade Federal do Piauí Universidade Federal do Piauí Universidade Federal do Piauí
Teresina, Piauí Teresina, Piauí Teresina, Piauí
[email protected] [email protected] [email protected]

ABSTRACT rithms that build a model from a user’s past behavior as

Online consumer reviews are currently available in most e- well as similar decisions made by other users, and then use
Commerce websites. By discussing products features, re- this model to predict items that the user may have an in-
views help consumers during the search stage of the buy- terest in. But even though those systems have acceptable
ing process. However, from thousands of reviews published to good performance, they are still less efficient than per-
daily, it is not always clear which ones consumers should sonal recommendations that come from trustable sources,
read. To help with that, consumers can vote user reviews like family members and friends [21].
as helpful or not helpful. This paper proposes an automatic Goldenberg et al. (2001) [8] showed that the decision mak-
method to analyze the helpfulness of online consumer re- ing process is heavily influenced by the word of mouth com-
views. This analysis was made on reviews collected from munication (WOM). Over the internet, online social net-
Steam using Multilayer Perceptron Artificial Neural Net- works concentrate a good portion of the WOM communi-
work. We found out indication that certain features of re- cations because everyone can share their own experiences
views affect the perception of helpfulness and we discuss with the world without much effort. Consumers can express
applications and future researches. their opinions about products through reviews by writing
their own recommendations or discussing, liking and shar-
ing recommendations from other consumers [12]. They can
Keywords also use those reviews during the decision making process.
extraction and treatment of social data; neural networks; From online reviews, consumers have access to informa-
modeling of user behavior tion about the products they are interested in through sev-
eral review features, such as the number of stars or the tex-
1. INTRODUCTION tual description [10]. However, given the high amount of
spam, low quality texts and false information on the inter-
With the rise of the global Internet usage, a big number net, consumers can be misled and may need a huge amount
of business transactions started using the Internet. One of of time to find helpful reviews, reducing their efficiency.
those services is the e-Commerce. The e-Commerce is usu- To solve that problem, a vast majority of websites use a
ally a more comfortable way to buy and sell products, and measure called review helpfulness, introduced by Amazon2 .
for that reason, it became gigantic. It is estimated that there In websites like Amazon, consumers can vote in reviews that
are 12 to 24 million e-Commerce stores selling products on they consider helpful or not helpful during the buying deci-
the Internet around the world today1 . Nevertheless, while sion process, so future consumers are aware of the best-voted
the huge amount of websites and products available on the reviews and can avoid the low-quality ones.
Internet is good for consumers, the searching stage of the Despite the success of that measure, it presents its prob-
online buying process becomes a wearing activity as con- lems. One of the problems is that reviews need a good
sumers are subject to multiple choices, making it difficult to amount of votes in order to be read, since reviews with-
take decisions [24]. out votes will not be considered helpful. That is a special
To make the buying decision process easier for consumers, problem for new reviews that may never be read, since they
some websites make use of automatic recommendation sys- may never get enough votes [16]. In that scenario, reviews
tems. Those so called “recommender systems” use algo- that are not very good can be considered the most helpful
1
http://blog.lemonstand.com/just-how-big-is-the- ones just because they were published first.
ecommerce-market-youll-never-guess To relieve that and other related problems, this paper
presents a study on review helpfulness through analysis of
Permission to make digital or hard copies of all or part of this work for personal or
textual descriptions and non-textual information present in
classroom use is granted without fee provided that copies are not made or distributed reviews. It proposes a model capable of automatically pre-
for profit or commercial advantage and that copies bear this notice and the full cita- dict the review helpfulness of online user reviews using an
tion on the first page. Copyrights for components of this work owned by others than Artificial Neural Network Multilayer Perceptron (ANN MLP).
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-
publish, to post on servers or to redistribute to lists, requires prior specific permission This investigation is based on reviews collected from Steam
and/or a fee. Request permissions from [email protected]. Store, a digital store and social network with more than 125
WebMedia ’16, November 08-11, 2016, Teresina, PI, Brazil million users that distributes digital computer video games,

c 2016 ACM. ISBN 978-1-4503-4512-5/16/11. . . $15.00
2
DOI: http://dx.doi.org/10.1145/2976796.2976871 https://www.amazon.com.br/

287
digital music, computer software and digital movies. Steam 2.2 Formative features of review helpfulness
was chosen because of its innovative review system and com- Evidences support the theoretical conceptualization of re-
munity model. view helpfulness as a formative construct [16]. Thus, it is
The main contributions of this work are a model for brazil- understood that there are a number of features that influ-
ian portuguese reviews and the investigation on the impact ence the perception review helpfulness.
of social features, present in social networks like Steam, in Some past works identified the direct influence of the re-
the perception of review helpfulness, along with features al- view content into its helpfulness perception. With the help
ready studied in previous works, such as linguistic features. of a supervised machine learning, Kim et al. (2006) [11]
The rest of this paper is organized as follow: Section 2 found that text size, its unigrams and the product type are
presents a literature review. Section 3 presents the proposed important features to measure the perception of review help-
model. Section 4 shows the results of the experiments. Sec- fulness. Danescu-Niculescu-Mizil et al. (2009) [6] conducted
tion 5 presents the conclusion and final considerations. a study on the Amazon corpus and identified that a review is
perceived as more helpful when its rating is close to the av-
erage rating for the same product. Otterbacher (2009) [18]
2. LITERATURE REVIEW analized data from the website Amazon and determined that
the perception of review helpfulness can be manifested by
2.1 The perception review helpfulness the relevance of the topic, readability and the author’s cred-
In a voting system as the one proposed by Amazon, the ibility and objectivity. Furthermore, it was discovered a
review helpfulness can be defined as strong relationship between the chronological order of the
reviews their helpfulness. Schindler and Bickart (2012) [19]
np
H= (1) examined the content of the review features and found that
np + nn the text size and the amount of statements positively influ-
where np represents the number of positive votes and nn ence the perception of helpful reviews, but only to a certain
represents the number of negative votes [11]. While this extent. Using neural networks, Lee and Choeh (2014) [15]
approach is simple and good enough, it presents some un- found that the text size and the number of words with only
favorable trends, as the lack of votes for new reviews [16] one letter are good indicative of the review helpfulness per-
and the fact that not everyone that reads reviews actually ception.
votes on them [11]. For that reason, the most-voted reviews Some scholars also explored the influence of the declara-
are not necessarily an accurate representation of the most tion of authorship on the perception of reviews helpfulness.
helpful ones. Connors et al. (2011) [4] studied some basic factors asso-
Another problem of that approach is that it does not take ciated with review helpfulness. They realized that the au-
into account the amount of votes received by each review. thor’s expertise, i.e. the knowledge of the author about the
For example, it is impossible to distinguish two reviews with product domain, has positive influence on the perception of
helpfulness 0.8 (or 80%), even if one of them has 10000 votes reviews helpfulness. Forman et al. (2008) [7] discovered that
and the other one has just 10. Mathematically, this prob- the presence of descriptive information about the identity of
lem was already solved in 1927 by Edwin B. Wilson using the author of online reviews has a positive impact on the
the lower bound of Wilson score confidence interval for a perception of review helpfulness.
Binomial parameter [1] Another point of interest is the type of product being re-
viewed and the impact it causes in the perception of review

zα2 q  helpfulness. Sen and Lerman (2007) [20] investigated how
p̂ + 2n/2 ± zα /2 [p̂(1 − p̂) + zα2 /2 /4n]/n consumers evaluate the review helpfulness of positive and
H= (2) negative reviews and found that the product type moderates
(1 + zα2 /2 /n)
the effect of review valence. It was noted that, for utilitarian
products, readers are more likely to attribute the negative
where p̂ represents the percentage of observed positive votes,
opinion expressed in the review to the author’s external rea-
n is the total of received votes and z quantile of the standard
sons. As to hedonic products, negative assessments are as-
normal distribution.
signed to internal reasons. In the same topic, Mudambi and
This equation considers the amount of votes of a review
Schuff (2010) [17] found that, for experience goods, reviews
as a sample of a hypothetical population where all reviews
with extreme ratings are less useful than reviews with mod-
are equally voted. Assuming a set confidence level (95% in
erate ratings. The depth of the review also has a positive
this work), it is determined a new success probability for
influence on the perception of usefulness, being stronger to
the analyzed distribution. Some websites such as Reddit 3
search goods.
and Yelp 4 already use this equation to order their topics,
This paper presents a different approach by analyzing of
comments and reviews.
the helpfulness of reviews written in Brazilian Portuguese
In this paper, review helpfulness is the percentage of pos-
and by exploiting different review features from previous
itive votes of a review based on Equation (2). Formally, the
works, such as the consumer engagement on the online com-
review helpfulness measure can be described as the extent to
munity and past reviews published by the same consumer.
which consumers perceive their capability to facilitate judg-
These features can be easily extracted thanks to the virtual
ment or purchase decisions [16].
community model proposed by Steam.
3
http://www.redditblog.com/2009/10/reddits-new-
comment-sorting-system.html
4
http://officialblog.yelp.com/2011/02/the-most-romantic- 3. PROPOSED MODEL
city-on-yelp-is.html For a given review, the objective of this work is to find the

288
measurement of this review helpfulness, called H, that can
also be defined by the Equation (2), where H is a real num-
ber between 0 and 1. To develop a model that predicts this
measurement, it was defined a set of formative features of
review helpfulness. Based on this study, it was hypothesized
that there are three important groups of review features that
model the review helpfulness: features based on the author-
ship, textual features and features based on review meta-
data. The proposed model is given by a regression function
that has as input a X vector containing those features and
as output a scalar H that is the perception of review help-
fulness measurement. It was chosen an ANN MLP to ap-
proximate this function because MLP are universal function
approximators. An ANN for function approximation can be
described as follow:

n1
X
y(x1 , x2 , ..., xn ) = λi gj (ui ) (3) Figure 1: Example of Steam user profile
i=1

n
X
ui = Wij xj − θi (4)
j=1

where the function y to be mapped by the ANN will be con-

structed by superimposing the logistics type activation func-
tions, represented by the terms gj (ui ), which are weighted
by factors λi [5]. The logistics function is defined as
1
g(u) = (5)
1 + e−β∗u Figure 2: Example of Steam user review
where β is a real constant associated with the steepness of
the curve front of his turning point.
The universal approximation theorem applied to the MLP was carried out automatically using a scraper 5 written in
provides the necessary basis for defining the structural con- Python using the Scrapy Framework6 . It was defined as
figurations of these networks in order to map algebraic func- the population of interest all reviews of digital games from
tions [5]. In addition, the ANN model allows any functional Steam written in Brazilian Portuguese, with at least one
form that relates the independent variables to the dependent helpfulness vote and posted by users who have public pro-
variable need not be imposed by the model, that is, you can files. Since Steam reviews are ordered so that reviews with
capture non-linear relationships in the data without the need few votes or many not helpful votes are difficult to access,
for prior specification [15]. the collection was concentrated only in the 100 most played
To define the characteristics of the proposed model, it was games on Steam on 26th February, but not limited to re-
analyzed the data from user reviews about digital games views published that day. This list is updated daily on the
from Steam described in the following subsections. site itself7 .
Of the 100 most played games on 26 February, five had
3.1 Data collection no reviews in Brazilian Portuguese with at least one vote.
Unlike a big number of e-commerce websites, Steam is also Therefore, the initial collected sample had 10,168 reviews.
an online community. Steam users have a profile where they To avoid asymmetry and trend in the data, it was decided
can add a personal picture; have friends; exchange private to remove reviews with less than three votes, since most
and public messages; publish screenshots, texts and videos reviews had very few votes.
and engage in groups. An example of Steam user profile is The data cleaning consisted in removing all the texts with
shown in Figure 1. correctness under a certain threshold, defined as the mean
Steam has a detailed review system that includes, in addi- of the original set (0.76). This was made because some re-
tion to the user text, several other information. The Figure 2 views are not written in Brazilian Portuguese or may employ
shows a typical Steam review in Brazilian Portuguese. The a colloquial language hardly understood by machines. The
final rating is presented at the top, above the review text correctness of the text is modeled as the number of correct
and can be Recommended or Not Recommended. Other in- words divided by the total number of words in a text. To
formation include username, number of products the user determine if the word belongs or not to the Brazilian Por-
has in its account, number of previously published reviews, tuguese, we used the dictionary br.ispell developed by the
the amount of hours the user has played the game and the
helpfulness vote system with a third option besides Helpful 5
Computer software technique of extracting information
and Not Helpful called Funny that is not taken into consid- from websites
6
eration to calculate the perceived helpfulness of reviews. http://scrapy.org/
7
The data collection for the model training and testing http://store.steampowered.com/stats/

289
 can be analyzed using the Flesh-Kincaid readability test, a
Table 1: Linguistic patterns mathematical method that evaluates how readable is a text
NOUN ADV? V? (ADV? ADJ | ADV V)*
by its average of words per sentence and average of syllables
ADJ NOUN
per total words [13]. An adaptation to the Portuguese was
ADV ADJ
published by Squarisi and Salvador (2005) [23]
 
nw
+ np ∗ 0.4 (6)
University of Sao Paulo8 . After the data cleaning, it was ns
obtained a final sample containing 5,823 reviews that were
published between November 2013 and February 2016. where nW is the total number of words of text, ns is the
total number of sentences and np is the total number of
3.2 Review features polysyllables. The constant 0.4 is the average number of
letters of the word in the phrase in Portuguese. The higher
3.2.1 Features concerning authorship the score, the less readable is the text. A text score 1 can
be easily read by anyone.
There are two important features regarding authors: rep-
Based on discoveries of Kim et al. (2006) [11] and Lee
utation and expertise. Studies suggest that reviews submit-
et al. (2014) [15] , it was modeled the text size from the
ted by authors with positive history are seen as more help-
number of words, number of sentences and the number of
ful [25]. The reputation incorporates aspects of the author’s
monosyllabic text, as the Portuguese has not large amount
credibility and expertise indicates the level of knowledge of
of semantically relevant words with only one letter.
an author about the discussed topic.
The author reputation is modeled from three variable: the 3.2.3 Review Metadata
average number of reviews votes (total number of votes from
The final evaluation and the posting date of the review are
past reviews divided by total number of reviews), the aver-
considered metadata. Final evaluation refers to the binary
age number of positive votes (total of positive votes from
evaluation of product present in each review, which may be
past reviews divided by the total number of votes in past
“recommended” or “not recommended”. It is important to
reviews); and the number of friends a user has in the Steam
note that readers may be influenced by the average ratings of
community. The expertise is modeled by the amount of
the product. This idea was explored by Danescu-Niculescu-
hours a review author played the analyzed game. For ex-
Mizil et al. (2009) [6].
ample, an author who played only two hours of a particular
Considering an evaluation review as being 1 for “recom-
game may have less proficiency in the topic than one that
mended” (or positive) and 0 for “not recommended” (or neg-
played 20 hours.
ative), the evaluation is modeled as an expression where p is
3.2.2 Features regarding the textual content of re- the percentage of reviews with positive evaluations (in dec-
views imal representation) for a product and x is the individual
evaluation review for the same product.
Fundamentally, previous researches dedicated themselves
to two types of textual analysis in online reviews: analysis of
semantic features and analysis of the stylistic features of the xp + (1 − x) ∗ (1 − p) (7)
text [3, 11]. By exploring the textual content of a review, it
can be attentive to the opinions and sentiment expressed in Starting from Otterbacher (2009) [18] discoveries, it was
the text. According to the Oxford Dictionary, an opinion is modeled the review’s posting time as the difference between
a view or judgment formed about something, not necessarily the release date of the product and the review posting date
based on fact or knowledge. in days.
It is hypothesized that online reviews with a large amount 3.2.4 Validation
of opinions are more informative. In this approach, opinions
should be modeled numerically. So it is interesting only The ANN MLP for function approximation is composed
know the amount of opinions expressed in every review. To of three layers: an input layer, just one hidden layer and
this end, holds up a process of extracting opinions accord- an output layer. The input layer maps each of the input
ing to an adaptation of the model proposed by Sousa et al. variables studied. The output layer has only one neuron
(2015) [22]. that maps the output variable H, that represents the review
In the extraction of opinions, the texts are analyzed ac- helpfulness. To set the most appropriate number of neurons
cording to the phrasal structure of sentences. The stan- in the hidden layer it was used the Cross-validation pro-
dard adopted was “subject + verb + predicative of subject”, cess and for the neurons on the hidden layer, It was used
where the core of the subject is the qualified feature and the the logistic function as activation (5). For the output layer,
predicate core is the qualifying word. For example, “esse jogo it was used a linear activation function (3), since the out-
é divertido” (this game is fun) is extracted as (jogo, diver- put neuron performs only a linear combination of logistic
tido) or (game, fun). These linguistic patterns were defined activation functions implemented in the hidden layer neu-
by a manual analysis performed on another sample of 385 rons [5]. Therefore, after the MLP network training pro-
reviews collected on the same website (confidence level 95% cess, the weights matrix relating to the output neuron will
and error 5%). The result is shown in Table 1. correspond to the own λi parameters of the Equation (3),
The stylistic elements, the text readability and the text i.e. λi = W1,i . The illustration of this network is shown in
size were considered. Readability is the ease in which a text Figure 3.
can be understood. The readability of a text in English The input variables defined in the hypotheses specification
are: i) the number of reviews votes by the total amount of
8
http://www.ime.usp.br/ ueda/br.ispell/ reviews of a user; ii) the ratio of positive votes by the total

290
 Figure 3: MLP for functional approximation

number of votes of a user; iii) the number of the user’s friends described as follows:
in the Steam community; iv) the amount of time (in hours) krcf
that the author dedicated to game analysis; v) the amount of Ms = p (9)
k + k(k − 1)rf f
opinions expressed in the text; vi) the readability of the text;
vii) the size of text in words; viii) the number of sentences where Ms is the heuristic “merit” of a feature subset S con-
in the text; ix) the number of monosyllabic words in the taining k features, rcf is the mean feature-class correlation
text; x) the difference between the average evaluation of the (f ∈ S) and rf f is the average feature-feature intercorrela-
product and the user evaluation; xi) the difference between tion [9]. The numerator of Equation (9) can be thought of
the date of release of the product and the date of publication as providing an indication of how predictive of the class a set
of the review in days. The expected output variable is the of features are; the denominator of how much redundancy
review helpfulness. there is among the features [9].
The relationship between each input variable and the out-
put variable can be calculate using the Relative Strength 4. RESULTS
(RS) measure [15] [26]: The most appropriated topology for an MLP to map an
specific problem is specified empirically [5]. The topology
n
X selection process is usually given by trial and error [15]. To
(Wki Wjk ) select the best topology between the candidates, it was used
RSji =
k=0
(8) the cross-validation k-fold. In that process, the total sample
m X n
X is divided in k folds, being (k − 1) of them used to com-
|Wki Wjk | pose the training subset while the the remaining partition
i=0 k=0
is the subset test. In this paper, k was defined being 10.
The overall performance of each candidate topology is ob-
where Wki denotes the weight of the k-th hidden unit and tained according to the average of individual performances
the i-th input unit. Wjk denotes the weight between the j-th observed when applying the k partitions.
output unit and the k-th hidden unit. RSji calculates the The RNA implemented in this study has random values
relative weight between the i-th input variable and the j-th as initial weights, learning rate and 0.1 and momentum 0.3.
output variable. The numerator RSji calculates the ratio We opted for small values to ensure that convergence occurs,
between the i-th input variable and the j-th output variable even if the process requires more times. To avoid overfitting,
and can be either positive or negative, depending on the each topology was trained until it was found the least mean
weights. Since the denominator relationship calculates the square error of the square (RMSE) local or when it was
total of all input and output variables. reached 104 epochs, whichever occurs first.
To validate the chosen features, it was used the Correlation- Figure 4 presents a graphic with the RMSE for each of the
based Feature Selection (CFS) [9]. CFS takes the subset tested topologies. We begin with a topology of one neuron
evaluation approach. The most suitable features are selected in the hidden layer and increase the layer in one neuron to
taking into account the correlation between all the features each new topology. We note that the error increases with
in the model. A good model has features that are not cor- every new neuron included in the hidden layer, but begins
related with each other, but are correlated with the class. to decrease as soon as it reaches five neurons. The topology
Features with high correlation with the class variable are chosen was the one that has the lowest number of neurons
highly relevant. CFS’s feature subset evaluation function is in the hidden layer and the lowest RMSE. The final RNA

291
MLP model chosen is a network of three layers consisting of The expertise of the author also seems to be a relevant fea-
11 input nodes, five nodes in the hidden layer and one output ture. In the case of Steam, the amount of hours played by
node. Given the complexity of performing predictions based an author at the time of publication of the review has a posi-
on human behavior, the RMSE of 0.1929 was considered tive impact on the perception of usefulness, but is correlated
acceptable. with another feature. The amount of user’s friends in the
community also has a positive impact on the perception of
review helpfulness.

5. CONCLUSIONS AND FUTURE WORK

This study aimed to explore ways to quantify and predict
the perception of helpfulness of online user reviews. The
results of this study were positive and give clues about what
are the relevant features and what can be explored in the
future. For the e-Commerce, these results provide clues as
to which website structures may be used so that the best
reviews are read by consumers and therefore maximize sales.
Features that were expected to be important, such as the
linguistic patterns, actually underperformed. That may be
Figure 4: Variation of the RMSE as the number of
caused by the colloquial language used on Steam or the use
neurons in the hidden layer increases.
of internet memes [2].
To assert the applicability of our work, the experiments
Table 2 shows the relative weight of review helpfulness
present in this work will be replicated in different domains,
determinants using the Equation (8). Values above 0.02
using real world data, such as TripAdvisor (hospitality in-
are good indicators of impact on the perception of review
dustry), Foursquare (location-based social networks), Ama-
helpfulness [15].
zon (search goods) and AdoroCinema (movies and tv).
For future work, we intend to replace the use of linguistic
Table 2: Relative weight of the variables. patterns for more robust NLP techniques, such as Latent
Variables Weights Semantic Analysis (LSA). LSA is an extraction technique
average number of votes per user 0.14272 and representation of contextual use of words using statis-
user average rating 0.06112 tical computations for a large corpus. The principle of LSA
amount of timed played 0.02507 is that the aggregate of all words contexts in which a given
number of friends 0.01743
word appears or does not appear derives a set of mutual con-
linguistic patterns -0.00442
readability 0.00789 straints which determines in large part the similarity of the
number of words 0.02794 meaning of words and groups of words [14]. It is expected
number of sentences 0.01224 that techniques that are more contemporary can identify
difference between user rating and average rating 0.01302 situations that could not be analyzed in our model, as the
number of monosyllabic words -0.00962 occurrence of hidden subject and humor in texts.
difference between product release date and posting date 0.00122
This work will also be expanded to include a compari-
son between our model and some variations of our model
Applying the Correlation-based feature selection for dis- regarding the dataset and the machine learning algorithm.
crete data algorithm and using the hill climbing as search Finally, we intend to explore the impact of humor in the
method, four features were selected: the average number of perception of review helpfulness. The very Steam allows
votes per use, the number of friends of the author, the word users to rate reviews as funny, an option rarely used, but
count of the review and the difference between the user rat- that can serve as a basis for future studies.
ing and the average rating. In that case, those features are
enough to fully represent our model. Running the neural
network with this subset of inputs yields a slightly better 6. REFERENCES
RMSE than before, of 0.1915.
The results of this analysis confirm earlier researches con- [1] A. Agresti and B. A. Coull. Approximate Is Better
ducted by Danescu-Niculescu-Mizil et al. (2009) [6], Schindler than ”Exact” for Interval Estimation of Binomial
et al. (2012) [19] and Kim et al. (2009) [11]. It was noted Proportions. The American Statistician,
that the user rating and the text size are characteristics that 52(2):119–126, May 1998.
positively impact the perception of review helpfulness. On [2] N. Baym. Personal Connections in the Digital Age.
the other hand, the date of posting of the reviews does not DMS - Digital Media and Society. Wiley, 2010.
seem to have strong impact on the perception of review help- [3] Q. Cao, W. Duan, and Q. Gan. Exploring
fulness in online communities such as Steam, which contra- determinants of voting for the “helpfulness” of online
dicts findings made by Otterbacher (2009) [18]. One of the user reviews: A text mining approach. Decision
possible explanations could be that Steam hold regular sales Support Systems, 50(2):511 – 521, 2011.
that revitalize pages of old products. [4] L. Connors, S. M. Mudambi, and D. Schuff. Is it the
Among the new discoveries, it was found that the author’s review or the reviewer? a multimethod approach to
reputation is a relevant factor in the perception of review determine the antecedents of online review
helpfulness. It was found that authors with positive historic helpfulness. In In 44th Hawaii International
of votes in past reviews have best new evaluated reviews. Conference on System Sciences, 2011.

292
 [5] I. N. da Silva, D. H. Spatti, and R. A. Flauzino. Redes [21] R. R. Sinha and K. Swearingen. Comparing
Neurais Artificiais: para engenharia e ciências recommendations made by online systems and friends.
aplicadas, chapter Redes Perceptron Multicamadas. In DELOS Workshop: Personalisation and
Artliber, 2010. Recommender Systems in Digital Libraries, 2001.
[6] C. Danescu-Niculescu-Mizil, G. Kossinets, [22] R. F. d. Sousa, R. A. L. Rabelo, and R. S. Moura. A
J. Kleinberg, and L. Lee. How opinions are received by fuzzy system-based approach to estimate the
online communities: A case study on amazon.com importance of online customer reviews. IEEE
helpfulness votes. 18th International Conference on International Conference on Fuzzy Systems
World Wide Web,, 2009. (FUZZ-IEEE), 2015.
[7] C. Forman, A. Ghose, and B. Wiesenfeld. Examining [23] D. Squarisi and A. Salvador. A arte de escrever bem:
the Relationship Between Reviews and Sales: The Role um guia para jornalistas e profissionais do texto. pages
of Reviewer Identity Disclosure in Electronic Markets. 50–52. Editora Contexto, 2005.
INFORMS, 2008. [24] K. D. Vohs, R. F. Baumeister, B. J. Schmeichel, J. M.
[8] J. Goldenberg, B. Libai, and E. Muller. Talk of the Twenge, N. M. Nelson, and D. M. Tice. Making
network: A complex systems look at the underlying choices impairs subsequent self-control: A
process of word-of-mouth. Marketing Letters, Volume limited-resource account of decision making,
12, Issue 3, pages 211–223, 2001. self-regulation, and active initiative. Motivation
[9] M. A. Hall. Correlation-based Feature Subset Selection Science, Vol 1(S), pages 19–42, 2014.
for Machine Learning. PhD thesis, University of [25] C. N. Wathen and J. Burkell. Believe it or not:
Waikato, Hamilton, New Zealand, 1998. Factors influencing credibility on the web. Journal of
[10] N. Hu, L. Liu, and J. Zhang. Do online reviews affect the American Society for Information Science and
product sales? the role of reviewer characteristics and Technology, 53(2):134–144, 2002.
temporal effects. Information Technology and [26] Y. Yoon, T. Guimaraes, and G. Swales. Integrating
Management Vol. 9 No. 3, pages 201–214, 2008. artificial neural networks with rule-based expert
[11] S.-M. Kim, P. Pantel, T. Chklovski, and systems. Decis. Support Syst., 11(5):497–507, June
M. Pennacchiotti. Automatically assessing review 1994.
helpfulness. In D. Jurafsky and r. Gaussier, editors,
EMNLP, pages 423–430. ACL, 2006.
[12] Y. A. Kim and J. Srivastava. Impact of social
influence in e-commerce decision making. In
Proceedings of the Ninth International Conference on
Electronic Commerce, ICEC ’07, pages 293–302, New
York, NY, USA, 2007. ACM.
[13] J. P. Kincaid, R. P. Fishburne, R. L. Rogers, and B. S.
Chissom. Derivation of New Readability Formulas
(Automated Readability Index, Fog Count and Flesch
Reading Ease Formula) for Navy Enlisted Personnel.
Technical report, Feb. 1975.
[14] T. Landauer, P. Foltz, and D. Laham. An introduction
to latent semantic analysis. Discourse processes,
25:259–284, 1998.
[15] S. Lee and J. Y. Choeh. Predicting the helpfulness of
online reviews using multilayer perceptron neural
networks. Expert Syst. Appl., 41(6):3041–3046, 2014.
[16] M. Li, L. Huang, C.-H. Tan, and K.-K. Wei.
Helpfulness of online product reviews as seen by
consumers: Source and content features. International
Journal of Electronic Commerce, 17:101–136, 2013.
[17] S. M. Mudambi and D. Schuff. What makes a helpful
online review? a study of customer reviews on
amazon.com. MIS Quarterly, 34(1):185–200, 2010.
[18] J. Otterbacher. ’helpfulness’ in online communities: A
measure of message quality. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems, CHI ’09, pages 955–964. ACM, 2009.
[19] R. Schindler and B. Bickart. Perceived helpfulness of
online consumer reviews: The role of message content
and style. Journal of Consumer Behaviour,
11:234–243, 2012.
[20] S. Sen and D. Lerman. Why are you telling me this?
an examination into negative consumer reviews on the
web. Journal of Interactive Marketing, 21:76–96, 2007.

293