What Makes Your Opinion Popular? Predicting The Popularity of Micro-Reviews in Foursquare

 

What Makes your Opinion Popular? Predicting the Popularity of Micro-Reviews in Foursquare Marisa Vasconcelos, Jussara Almeida e Marcos Gonçalves {marisav,jussara,mgoncalv}@dcc.ufmg.br Universidade Federal de Minas Gerais, Belo Horizonte, Brasil

ABSTRACT Location based social networks (LBSNs), such as Foursquare, allow users to post micro-reviews, or tips, about the visited places (venues (venues)) and to “like” “like” previ previously ously posted reviews. reviews. Tips may help attracting future visitors, besides providing valuable feedback to business owners. In particular, the number of “likes” a tip receives ultimately reflects its popularity or helpfulnes helpf ulnesss among users. In that context, context, accurate predic predic-tions of which tips will attract more attention (i.e., “likes”) can drive the design of automatic tip filtering and recommendation schemes and the creation of more effective marketing keti ng strategie strategies. s. Previo Previous us efforts to automatically automatically assess the helpfulness of online reviews targeted mainly more verbose and formally structured reviews often exploiting textual features. How However ever,, tips are typi typically cally much shorter and contain cont ain more informal content. content. Thus, Thus, we here propose and evaluate new regression and classification based models to predict the future popularity of tips. Our experimental evaluation shows that a linear regression strategy, using features of both the user who posted the tip and the venue where it was posted as predictors, produces results that are at least as good as those produced by more sophisticated approaches and/or by using each set of features in isolation.

Categories and Subject Descriptors H.3.5 [Onli H.3.5 Online ne Info Informat rmation ion Serv Services ices]: ]: Web-bas eb-based ed services; J.4 [Computer [Computer Application Applicationss]: Social Social and behav behav-ioral sciences

Keywords Popularity Prediction, Content Filtering, Location-based Social Networks, Micro-reviews

1.

INT INTRO RODUC DUCTIO TION N

The success of online social networks and GPS (Global Positioning System) based applications have led to an abundance of data on personal interests and activities shared on Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial commercial advantage advantage and that copies copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. SAC 2014,  March 24–28, 2014, Gyeongju, Republic of Korea

Copyright 2014 ACM 978-1-4503-2469-4/14/03 ...$15.00. http://dx.doi.org/10.1145/2554850.2554911.

the Web. Web. Indeed, Indeed, the so-called so-called location location-base -based d social networks (LBSNs), such as Foursquare and Google+1 ,  as well as social networks that have incorporated location-based services (e.g., Facebook and Yelp) allow users to share not only where they are but also location-tagged content, including personall reviews about visited places. Our present persona present focus is on Foursquare, where users can post micro-reviews, or  tips , with their opinions about registered venues. Tipss can serv Tip servee as fee feedba dback ck to hel help p oth other er users users ch choose oose pla places ces to visit. visit. For examp example, le, a tip recom recommen mendin ding g a hot hotel el or disapproving the service of a restaurant may guide others in their choices. Tips nourish the relationship between users and real businesses, being key features to attract future visitors, besides providing valuable feedback to business owners. Unlike check ins, which are visible only to the user’s friends, tips are visible to everyone, and thus may greatly impact online information sharing and business marketing. Users can also evaluate evaluate previo previously usly posted tips by “likin “liking” g” them in sign of agreem agreement ent with their content. content. The number number of likes can be seen as feedback from other users on the helpfulness, or   popularity   of the tip. Th Thus, us, it is expecte expected d that both users and venue owners may have particular interest in retrieving tips that have already received or have the potential of receiving a large number of likes, as they reflect the opinion of more people and thus should be more valuable. However, finding and retrieving those tips may be a challenge given the presence of an increasing amount of spam content [18 content  [18], ], unhelpful and possibly misleading information. Moreover, Moreo ver, Foursquar Foursquaree displ displays ays the tips posted at a given given venue sorted by the number of likes received2 ,  which contributes to the rich-get-richer effect  effect   [13], [13], as well well-ev -evaluat aluated ed tips are more visible and may keep receiving more feedback, while recently posted tips may have difficulty to be seen. Thus,, predi Thus predicting cting,, as early as possible, the futur futuree popularpopularity of a tip may help determining its potential as valuable information, which enables the design of effective tip recommendation and filtering tools to better meet the needs of both users and venue owne owners. rs. For example, venu venuee owners can benefit with a rapid feedback about more widespread opinions on their businesses and associated events (e.g., promotions)   [11] motions) [11],, whi while le users, users, who often often acc access ess LBSNs using using small screen mobile devices, might enjoy an interface that favo favors rs tips that are more like likely ly to beco become me popular. popular. The identification of potentially popular tips may also drive the design of cost-effective marketing strategies for the LBSN

1 2

http://www.foursquare.com   https://plus.goo http://www.foursquare.com https://plus.google.com gle.com

This is the only option for mobile users.

 

environment, and help minimizing the rich-get-richer effect, as those tips may receive greater visibility in the system. Previous efforts towards predicting the popularity or helpfulness of online reviews focused mostly on textual features [11, features [11, 13 13,,   15]. 15]. Howe However ver,, com compar pared ed to review reviewss in other systems systems (e.g., TripAdvisor and Yelp), tips are typically much more concise (restricted to 200 characters), and may contain more subjectivee and informal conten subjectiv contentt (e.g. “I love this airpor airport” t”,, “Go Yankees”). Moreover, users, often using mobile devices, tend ten d to be mor moree direct direct and brief brief,, avoid avoiding ing many detai details ls about specific characteris characteristics tics of the venue venue.. Thu Thus, s, some as-

model as a classifier to retrieve only reviews having a predicted helpfulness higher than a threshold. These prior studies are based mostly on content features, which are suitable for more verbose and ob jective jective reviews. Such featu features res may not be adequate for predicting the popularity of Foursquare tips, which tend to be more concise and subjective. Other related studies focused on designing models for information diffusion in blogs   [7]   and Twitter  Twitter   [2, [2,   10], 10], often exploiting features of the social graph, like the number of  friends and their probability of influence   [1]. [1]. Text extual ual fea fea-tures extracted extracted from the messages (e.g., hashtags) hashtags) or the

pects exploited by existing solutions, especially those related to the textual content, may not be adequate to predicting the popularity of shorter texts such as tips. Moreover, unlike in other systems, Foursquare does not let users assign unhelpfulnes unhelpfulnesss signals to tips. Thus, Thus, the lack of a “lik “like” e” does not imply that a tip was not helpfu helpfull or interesting, which makes the prediction of tip popularity much harder. Foursquare also enables the analysis of aspects that are specific to online social networks, where properties of the social graph may impact content popularity [1] popularity [1].. Finally, the prediction of tip popularity is also inherently different from previous analyses of information diffusion on Twitter [10] [10] and news sites [2] sites  [2]  which exploited mainly aspects related to the user who posted the informat information ion.. A tip is ass associa ociated ted with a venue and tends to be less ephemeral. Thus its popularity may also be affected by features of the target venue. In this context, we here introduce the problem of predicting the future   popularity   of Foursquare oursquare tips, estimating estimating a tip’s tip’s popularit popularity y by the nu numbe mberr of likes it receiv receives. es. Mor Moree specifically, we analyze various aspects related to the user who posted the tip, the venue where it was posted and its content, and investigate the potential benefits from exploiting these aspects to estimate the popularity level  of a tip in the future. To that end, we propose and evaluate evaluate sever several al regression and classification models as well as various sets of  user, venue and tip’s content features as predictor variables. Our evaluation on a large dataset, with over 1,8 million users, 6 million tips and 5 million likes, indicate that a simpler multivariate linear regression strategy produces results that are at least as good as those obtained with the more sophisticated Support Vector Regression (SVR) algorithm and Support Vector Machine (SVM) classification algorithm [8 [8]. Moreover, using sets of features related to both the tip’s au-

topic of the message, and user features (e.g., number of followers) have also been used to predict content popularity in these systems [2, systems  [2, 10] 10].. Unlike blog posts and tweets, tips are associated with specific venues, and tend to remain associated with them (and thus visible) for longer. longer. Thu Thus, s, a tip’s popularity may be affected by features of the target venue. Finally, we are not aware of any previous study of popularity ulari ty predicti prediction on of tips or other micro micro-revi -reviews. ews. The only previous analysis of tips and likes on Foursquare was done in [18],, where the authors uncovered different profiles related [18] to how users exploit these features. In comparison with that work, we here focus on a different (though related) problem - the prediction of tip popularity.

thor andaccurate the venue where it than was posted as predictors to more predictions exploiting features ofleads only one of these entities, while adding features related to the tip’s content leads to no further significant improvements.

other numbers of levels (e.g., three) and range definitions, finding, findin g, in genera general, l, similar results. Altern Alternativ atively ely,, we tried to group tips using various features (discussed below) and clustering techniques (e.g., k-means, x-means, spectral and density-based clustering). However, However, as previously observed for quality prediction in question answering services services [12]  [12],, the resulting clusters were not stable (i.e., results varied with the seeds used). used). This may be due to the lack of discriminativ discriminativee features specifically for the clustering process , which does not imply that the features cannot be useful for the prediction task, our main goal here. As in  [11, 13, 15]   we exploit both regression and classification techniques in the design of our tip popularity prediction models. We experiment with two types of regression algorithms, which produce as output a real value, which is rounded to the nearest integer that represents a popularity level, as done for predicting the helpfulness of movie reviews in [13] in  [13].. Our first approach is an ordinary least squares (OLS OLS)) multivariate linear regression model. It estimates the

2.

REL RELA ATED WORK WORK

The task of assessing the helpfulness or quality of a review is typi typically cally addressed by emplo employing ying classifica classification tion or regression-based methods[11, methods[11, 15, 13] 13].. For examp example, le, Kim   et  al.  [11]   used SVR to rank reviews according to their helpfulness, exploiting features such as the length and the unigrams of a review and the rating score given by the reviewers. ers. O’Maho O’Mahony ny   et al. al.   [15]  proposed a classificati classification-b on-based ased approach to recommend TripAdvisor reviews, using features related to the user reviewing history and the scores assigned to the hotels. Similarly, Liu et al.  [ [13] 13] proposed  proposed a non-linear regression model that incorporated the reviewers’ expertise, the review timeliness and its writing style for predicting the helpfulnes helpf ulnesss of movie movie reviews. The authors also used their their

3.

POPUL POPULARIT ARITY Y PREDICTIO PREDICTION N MODELS MODELS

Our goal is to dev develo elop p mode models ls to pre predic dictt the popula popularrity level level of of a tip giv given en futur futureeoftime. tim e. received, We esti estimat mate e the popularity a tipatbya the number likes and we perform predictio predictions ns at the time the tip is p osted. To that end, we categorize tips into various  popularity levels , defined based on the number of likes received up to a certain point in time p after   p  after  they   they were posted. As in [2, in  [2, 10 10,,  13]  13],, we choose to predict the popularity level instead of the number of likes because the latter is harder, particularly given the skewed distribution of number of likes per tip [18] tip  [18],, and the former should shoul d be good enough for most purpose purposes. s. We consider   p equal to 1 mont month h and two popula popularity rity leve levels: ls: low and high. Tips that receive at most 4 likes during the first month after postage have low popularity , whereas tips that receive at least 5 likes in the same period have  high popularity . We note that the avail availabili ability ty of enoug enough h examp examples les from each class for learning the model parameters impacts the definition of the popularity levels. We also experimented with

 

popularity level of a tip  t  at a given point in time p ,  R (t, p), as a linear function of   k   predictor variables   x1 ,   x2   · · ·   xk , i.e.,   R(t, p) =  β 0  +  β 1 x1  +  β 2 x2  +  · · · β k xk . Model par parameameters  β 0 ,  β 1   · · ·   β k  are determined by the minimization of the least squared errors [8 [8] in the training data (see Section 5.1 Section 5.1). ). We also consider the more sophisticated Support Vector Regression (SVR) algorithm  algorithm   [8], [8], a state-of-the-art method for regression regression learning. learning. Unlik Unlikee the OLS model, SVR does not consider errors that are within a certain distance of the true value (within the   margin ). It also also allows allows the us usee of  different differ ent kernel functio functions, ns, which which help solving a larger set of problems problems,, com compar pared ed to linear linear regre regressi ssion. on. We use both linear and radial basis function (RBF) kernels, available in the LIBSVM package   [3], [3], as the latter handles non-linear relationshi relati onships. ps. We also experimen experimentt with the Support Support Vector Machine (SVM) algorithm  algorithm   [8]  [8]   to  classify  tips into popularity levels. The goal of SVM is to find an optimal separating hyperplane in the feature space that gives the largest minimum distance to the training examples. Like with SVR, we use both linear and RBF kernels in our experiments. We compare the accuracy of the OLS, SVM and SVM models against that of a simple (baseline) strategy that uses the median number of likes of tips previously posted by a user   u  as an estimate of the number of likes that a new tip posted by  u  will receive (and thus its popularity level). Having presented our models, we now turn to the predictor variables   x1 ,   x2 ,   · · · ,   xk . These variab variables les are features features of the tip  t  whose popularity level we are trying to predict. We here consider   k  = 129 features related to the user who posted  t , the venue where  t  was posted, and  t ’s textual content. Our selected features are grouped into three sets:   3 User features: features: includes features related to the user’s activities (e.g., number of tips, number of likes received/given) and her social networ network k (e.g., number number of friends/fo friends/follo llowers wers,, percentage of likes coming from her social network, numbers of tips posted and likes given by her social network). We consider two scenarios when computing user features: (1) all tips posted by the user, and (2) only tips posted by the user at venues of the same category of the venue where t  was posted. We refer to the latter as user/cat as  user/cat features. features. Venue features: features: this set of features captures the activities at the venue where   t  was posted or its visibilit visibility y. Examples Examples are the number of tips posted at the venue, the number of  likes received by those tips , the numbers of check ins and unique visitors, and the venue category. : this set contains contains features features,, proTip’s Content Features Features: posed in [4, in  [4, 5, 16 16,,   14], 14], related to the structure, syntactics, readabilit reada bility y and sen sentimen timentt of the tip’s content. content. It includes the numbers of characters, words, and URLs or e-mail addresses dress es in the tip, readabilit readability y metric metricss and features based on the Part-Of-Speech tags such as percentages of nouns, and adjectives adject ives.. It also inclu includes des three scores - p ositi ositive, ve, negative negative and neutral - that capture the tip’s sentiment. These scores are computed using SentiWordNet [6 [6], an establ established ished sentiment lexical for supporting opinion mining in English texts. In SentiWordNet, each term is associated with a numerical score in the range [0, 1] for positive, negative and objectivity (neutral) sentiment information. We compute the scores of a tip by taking the averages of the corresponding scores over over all wor words ds in the tip that appear in Sent SentiW iWordNe ordNet. t. To handle negation, we adapted a technique proposed in [17] [17] 3

A complete set of all features used is available at  https:

//sites.google.com/site/sac2014sub/

Table 1: Overview of Selected Features Type

User

Venue

Content

Feature # posted tips # likes received median # likes rec. mean # likes rec. so ci cia l net (SN) s iz ize % likes from SN # posted tips # likes received median # likes # check ins # visitors # words # chR aL rascters #U

Min 1 0 0 0 0 0 1 0 0 0 0 1

Mean 3.72 3.13 0.48 0.58 44.79 0.71 2.13 1.80 0.45 217.33 87.35 10.25

Max 5,791 208,619 657.0 858.10 318,89 0 1 2,419 7,103 390 484,683 167,125 66

CV 3.25 63.40 2.77 2.70 15 ..9 95 0.43 2.23 11.60 2.81 6.18 5.87 0.78

1 0

50 9..0 72 8

209 0

0 8..7 25 7

that reverses the polarity of words between a negation word (“no”, “no”, “didn’t” “didn’t”,, etc.) and the next punctuation mark.

4.

FEA FEATURE TURE ANAL ANALYSIS

In this section, we first describe the dataset used in our experiments experim ents (Sectio (Section n   4.1), 4.1), and then briefly briefly analy analyze ze som somee selected features (Section 4.2 (Section 4.2). ).

4.1 4.1

Da Data tase sett

We crawled Foursquare using the system API from August to October 2011, collecting data of more than 13 million users. We believe that this represents a large fraction of the total user population at the time, which, reportedly, varied from 10 to 15 million between June and December 20114 . Our complete dataset contains almost 16 million venues and over 10 milli million on tips. How However ever,, to avoid introducing introducing biases towards very old or very recent tips, we restricted our analyses to tips and likes created between January 1st 2010 to May 31st 2011. This filter left us with over 6 million tips and almost 5,8 million likes, posted in slightly more than 3 million venues by more than 1,8 million users. We note that around 34% of the tips in our analyzed dataset received, during the considered period, at least one like.

4.2

Feat Feature ure Characteriz Characterization ation

Table 1 Table  1 presents  presents statistics of the selected features for users and venues venues with at least one tip in our dataset. The table shows, for each feature, minimum, mean, maximum values and coefficient of variation (CV), which is the ratio of the sta standa ndard rd devia deviatio n tovariations the mean. mean. WeCV’s), not notee that tha t mos mostttheir fea fea-tures have verytion large (large reflecting 5 heavy tailed distributions distributions ,  as observed in in [18]  [18].. In par partic ticuular, the CVs of the numbers of likes received by tips previously posted by the user and at the venue are very large. Indeed, most users posted very few tips and/or received few likes while most tips and likes were posted by few users. For example, 46% and 48% of the users posted only one tip and received only one like, while only 1,499 and 2,318 users posted more than 100 tips and received more than 100 likes, respectively respecti vely.. These heavy tailed distr distributi ibutions ons sugge suggest st that tips may experience the rich-get-richer effect. Moreover, the median number of likes per user is, on average, only 0.48. Th Thus, us, many use users rs hav havee this this fea featur turee equ equal al to 0. Thi Thiss will will impact our prediction results, as discussed in Section 5. Section  5. 4

https://foursquare.com/infograp https://foursqu are.com/infographics/10million hics/10million

and   http://www.socialmedianews.com.au/foursquarereaches-15-million-users/ 5

We omit these distributions due to space constraints.

 

We find that the Spearman’s correlation coefficient (ρ), which is a non-parametric measure of statistical dependence between two variables   [19], [19], computed over the numbers of  tips and likes received by each user is moderate (ρ=0.54), and between the numbers of tips and likes given by each user is even lower ( ρ=0.37). Thus, in general, users who tip more do not necessarily receive or give more feedback. We also find that the social network is quite sparse among users who post tips: a user has only 44 friends/followers, on average, while 37% of the users have at most 10 friends (followers), although the maximum reaches 318,890. Moreover, the fraction of likes coming from the user’s social network tends to be reaso reasonably nably large large (70% on averag average). e). Thus Thus,, the user’s social network does influence the popularity of tips. Regarding venue features, there is also a heavy concentration of activities (check ins, visitors, tips, and likes) on few venues. ven ues. The maximum maximum number of lik likes es per ven venue ue exceeds 7,000,, but is only 1.8 on average 7,000 average.. The correla correlation tion between between either the number of check ins or the number unique visitors and the number of tips of the venue is moderate (around 0.52). Thus Thus,, to some extent, extent, popular venues venues do tend to attract more tips, although this is not a strong trend. The correlation between the number of tips and the total number of likes per venue is also moderate (ρ=0.5). Thus, in general, tipping tends to be  somewhat  effective   effective in attracting visibility (likes) to a venue. The same correlation exists between the total number of likes and the number of check ins (ρ=0.5), but we cannot infer any causality relationship as temporal information is not available. However, the correlation between the median number of likes and the number of  check ins is weaker ( ρ=0.3), implying that not all tips posted at the same venue receive comparable number of likes. This is probably due to various levels of interestingness of those tips, but migh mightt also reflect the ric rich-geth-get-rich richer er effect. Finally, regarding content features, we find that most tips are very short, with, on average, around 60 characters and 10 words, and at most 200 characters (limit imposed by the application) and 66 words. Moreover, the vast majority (98%) of the tips carry no URL or e-mail address, and we find no correlation between the size of the tip and the number of  likes received by it (ρ<0.07), as one might expect.

5.

PREDICTION MODEL EVALU EVALUA ATION

We now evaluate the models proposed to predict, at the time a tip   t  is posted, the popularity level   t   will achieve  achieve   1 later. We discuss our experim experimental ental setup (Sectio (Section n month   later. month 5.1 5.1), ), our main results (Section 5.2 (Section  5.2), ), and the relative importance of the features for prediction accuracy (Section 5.3 (Section  5.3). ).

5.1

Experi Experimen mental tal Setup Setup

Our methodology consists of dividing the available data into training and test sets, learning model parameters using the training set, and evaluating the learned model in the test set. We split the tips chronologic chronologically ally into trainin training g and test sets, rather than randomly, to avoid including in the training set tips that were posted after tips for which predictions will be performed. We build 5 runs, thus producing 5 results, by sliding the window containing training and test sets. To learn learn the models, models, we consider consider the most most recen recentt tip posted by each user in the training set as candidate  for popularity prediction, and use information related to the other tips to compute the featur features es used as predictor predictor variables variables.. All tips in the test set are taken as candidates for prediction,

and their feature values are computed using all information (e.g., tips, like likes) s) avail available able until the moment when the tip was posted (i.e., the prediction is performed). For each candidate in both training and test sets, we compute the feature values by first applying a logarithm transformation on the raw numbers to reduce their large variability, normalizing the results results between between 0 and 1. Mor Moreo eove ver, r, in order to have enough historical data about users who posted tips, we consider only users who posted at least 5 tips. To control for the age of the tip, we only consider tips that were posted at least 1 month before the end of training/test sets. We also focus on tips written in English, since the tools used to compute some textual features are available only for that language. After applying these filters, we ended up with 707,254 tips that are   candidates  for prediction. The distribution of candidate tips into the popularity levels is very skewed, with 99.5% of the tips with low popularity. Such imbalance, particularly in the training set, poses great challenges to regression/classification accuracy. Indeed, initial results from using all available data were very poor. A widely used strategy to cope with the effects of class imbalance in the training data is under-sampling   [9]. [9]. Sup Sup-pose there are  n  tips in the smallest category in the training set. We produce produce a balanc balanced ed training training set by rand randoml omly y selecting equal sized samples from each category, each with n   tips. Note that under-s under-sampli ampling ng is p erfor erformed med only in the tra traini ining ng set. The test set remai remains ns unch unchang anged. ed. Bec Becaus ausee of  the random selection of tips for under-sampling, we perform thi thiss operati operation on 5 tim times es for each slidin sliding g win windo dow, w, thus thus pro pro-ducing 25 different results, in total. The results reported in the next sections are averages of these 25 results, along with corresponding 95% confidence intervals. OLS model parameters are defined by minimizing the least squared errors of predictions for the candidate tips in the training traini ng set. This can be done as their popularity popularity level (i.e., nu numbe mberr of likes one month after after postage postage)) is known known.. Bes Bestt values for SVM and SVR parameters are defined in a similar manner, using cross-validation within the training set. We assess the accuracy of our prediction prediction models using recall, precision, macro-avera macro-averaged ged precision, and macro-averaged recall. The  precision  of class   i  is the ratio of the number of  tips correctly assigned to   i  to the total number of tips predicted as   i, whereas the   recall   is the ratio of the number of tips correctly classified to the number of tips in class   i. The (recall) is the average precision macro-averaged (recall) computedprecision across classes.

5.2

Predic Predictio tion n Result Resultss

Figures 1 Figures  1-a) -a) and and 1  1-b) -b) show macro-averaged precision and recall, along with 95% confid recall, confidence ence interv intervals, als, for 30 strategies for predicting a tip’s popularity level. These strategies emerge from the combination of five prediction algorithms with alternative sets of predictor variables. In particular, for OLS, SVR (linear and RBF kernels) and SVM6 algorithms, we consider consider the follo following wing sets of predi predictors: ctors: only user features, only venue features, only content features, all venue and user featur features, es, all user, ven venue ue and content content features features.. We also consider only user features restricted restricted to the category of  the venue where the tip was posted (user/cat features). For predictions using the median number of likes of the user, here referred to as median strategy, we compute this num6

We show only results for the linear kernel as they are similar for the RBF kernel.

 

(a)   Macro-Averaged Precision

 

(b)   Macro-Averaged Recall

 

(c)  Recall for High Popularity Tips

Figure 1:   Effectiveness of Alternative Prediction Models and Sets of Predictor Variables ber over all tips of the user and only over the tips posted at venues of the same (target) category7 .   The significance of  the results was tested using two statistical approaches (oneway ANOVA and Kruskal-Wallis [19] Kruskal-Wallis  [19])) with 95% confidence. Figure 1 Figure  1-a) -a) shows that there is little difference in macroaveraged averaged precision across the predi prediction ction algorithms algorithms,, except for SVR with linear kernel using only content features, which performs perfor ms worse. The same degradation degradation of SVR is observ observed ed in terms of macro-averaged recall (Figure   1-b -b). ). For that that metric, the superiority of SVM, SVR and OLS over the simpler median median strategy is clear. For any of those strategies, strategies, the best macro-averaged recall was obtained using user and venue features, with gains of up to 64% over the other sets of predictors. Comparing the different techniques using user

recall of that class, and is thus preferable over exploring only user features. features. Note also that including con content tent features features as predictors lead to no further (statistically significant) gains. Comparing OLS, SVR and SVM with user and venue features as predictors, we find only limited gains (if any) of the more sophisticated SVR and SVM algorithms over the simpler OLS strategy. strategy. In particular, particular, SVR (with RBF kernel kernel)) leads to a small improvement (2% on average) in the recall of the high popularity level, being statistically tied with SVM. However, in terms of precision of that class, OLS outperforms SVR by 13.5% on average, being statistically tied with SVM. We observe that SVR tends to overestimate the popularity of a tip, thus slightly improving the true positives of the high popularity class (and thus its recall) but also in-

and features, we see small gains recall venue for SVM and OLS over SVR (up in tomacro-averaged 1.15% and 0.84%, respectively respecti vely)) while OLS and SVM produc producee statis statisticall tically y tied results. resul ts. How However ever,, we note that OLS is much much simpler than both SVM and SVR, as will be discussed below. We note that recall results are higher than precision results in Figures 1 Figures  1-a,b). -a,b). This is because the precision for the smaller class (high popularity) is very low even with a high recognition rate (large number of true positives). Recall that under-sam under -sampling pling is appli applied ed only to the train training ing set, and thus class distribution distribution in the test set is still ver very y unbalance unbalanced d and dominated domin ated by the larger class (low populari popularity). ty). Thus Thus,, even small false negative rates for the larger class results in very lo low w pre precis cision ion for the other other (smal (smaller ler)) class. class. For that reareason, towards a deeper analysis of the prediction strategies, we focus prima primarily rily on the recall metric and discu discuss ss separate results for each class, with especial interest in the high pop-

creasing theintroduces false negatives lowinto popularity class, which ultimately a lotofofthe noise the predictions for the high popularity popularity class (hurting its precisio precision). n). Note that the limited gains in recall of SVR come at the expense of a 30 times longer model learning process, for a fixed training set. Thus, the simpler OLS model produces results that, from a practical practi cal perspective perspective,, are very competitiv competitivee (if not better) to those obtained with SVM and SVR.

ularity class. The focus on recall is parti ularity particular cularly ly interesting interesting for tip filtering filtering and recomm recommendati endation on tools where users (regular users or venue/business owners) may want to retrieve most of the potentially popular tips at the possible expense of some noise (tips in the low popularity class). Figure 1 Figure  1-c) -c) shows average recall of tips for the high popularit ularity y level. level. Not Notee tha that, t, for any predict prediction ion model model,, the there re are no significant differences in the  recall  of   of high popularity tips across the various sets of predictors, provided that user features featur es are inclu included. ded. Moreo Moreover ver,, the use of venue featur features es  jointly with user features improves the  precision  of   of the high popularity class (omitted due to space constraints) in up to 46% (35% for the OLS algorithm). algorithm). That is, adding ven venue ue features reduces the amount of noise when trying to retrieve potentially popular tips, with no significant impact on the

of the number of likes received by tips posted by the user. Thus, the feedback received on previous tips of the user is the most important factor for predicting the popularity level of her future tips. tips. Figur Figuree   2a, which 2a,  which shows the complementary cumulative distributions of the best of these features (average number of likes) for tips in each class, clearly indicates that it is very discriminative of tips with different (future) (futur e) popularity popularity levels. Simil Similar ar gaps exist between the distributions of the other two aforementioned features. Features related to the social network of the tip’s author are also import important. ant. The number of friends/f friends/follo ollowers wers of the author and the total number of likes given by them occupy the 4th and 9th positions of the ranking, respectively. Moreover, we find that authors of tips that achieve high popularity tend to have more friends/followers. The best venue feature, which occupies the 6th position of the ranking, is the number of unique visitors (Figure 2b (Figure  2b). ). Moreover, total number of check ins (7th position), venue category (8th position) and total number of likes received

7

The figures results of the median prediction model only onl y for the show user user and use user/c r/cat at sets set s of predic pre dictor tor vari ariabl ables es since they are the same for the other sets.

5.3

Fe Featu ature re Import Importanc ancee

We now analyze the relative importance of each feature for prediction accuracy by using the Information Gain feature selection technique  technique   [5] [5] to rank features according to their discriminative capacity for the prediction task. Due to space constraints, we focus on the most discriminative features. The three most important features are related to the tip’s author. They are the average, total and standard deviation

 

(a)  Best user feature

  (b)   Best venue feature   Important Features for Tip Popularity

Figure 2:

statistically as good as (if not better than) those obtained with the more sophisticated SVM and SVR methods, particularly for tips in the high popularity level but also when considering average performance across both popularity levels analyzed. This work is a first effort to identify the most relevant aspects for predicting the popularity of Foursquare tips. tips. Thi Thiss is a ch chall alleng enging ing problem problem which which,, in com compar pariso ison n with previous related efforts, has unique aspects and inherently different nature, and may depend on a non-trivial combination binati on of va various rious user, venu venuee and content content featur features. es. Nevertheless, the knowledge derived from such effort may bring valuable insights into the design of more effective automatic tip filtering and ranking strategies.

7.

ACKNOWLED CKNOWLEDGMEN GMENTS TS

This research is partially funded by the Brazilian National Institute of Scien Science ce and Tec Technology hnology for the Web (MCT/CNPq/ (MCT/CNPq/ INCT grant number 573871/2008-6), CNPq, CAPES and FAPEMIG.

8.

Figure 3:   Recall for tips in the high popularity level when removing one feature at time.

by tips posted at the venue (11th positio position) n) are also very discriminative, appearing above other user features, such as number of tips (19th position) in the ranking. Finally, we extend our evaluation of the importance of different sets of features by assessing the accuracy of the OLS strategy as we remove one feature at a time, in increasing order of importance given by the Information Gain. Figure 3   shows the impact on the average recall of the high popularity class as each feature is removed, starting with the complete set of user, venue and content features. For example, the second bar shows results after removing the least discriminative feature (number of common misspellings per tip). Note that the removal of many of the least discriminative features has no significant impact on recall, indicating that these features are redundant. Accuracy loss is observed only after we start removing features in the top-10 positions. Among those, the largest losses are observed when the number of check ins at the venue and the size of the user’s social network netw ork are removed, removed, which reinf reinforces orces the importance importance of  venue and social network features for the prediction task. Similar results are also obtained for macro-averaged recall (omitt (om itted). ed). In sum, using the top 10 most most impo importa rtant nt features produces predictions that are as accurate as those of  using the complete complete set of features features.. We note that we did test whether multicollinearity exists among different predictors, which could affect the accuracy of the OLS model. We found that despite the strong correlations between some pairs of  predictors, removing some of these variables from the model does  not  improve accuracy.

6.

CONCLUSIO CONCLUSIONS NS AND FUTURE FUTURE WORK

We have tackled the problem of predicting the future popularity ulari ty of micro-rev micro-reviews iews (or tips) in Foursquare. oursquare. To that end, we proposed and evaluated classification and regressionbased strategies exploring different algorithms and various sets of features as predictors. Our experimental results showed that the simpler OLS algorithm produces results that are

REFE REFERE RENC NCES ES

[1] E. Bakshy Bakshy,, I. Rosenn, Rosenn, C. Marlow, Marlow, and L. Adamic. Adamic. The Role of Social Networks in Information Diffusion. In  WWW , 2012. [2] R. Bandari, S. Asur, and B. Huberman Huberman.. The Pulse of News in Social Media: Forecasting orecasting Popularit Popularity. y. In  ICWSM , 2012. [3] C. Chang and C. Lin. LIBS LIBSVM: VM: a Library for Support Support Vector Machines, 2001. [4] B. Chen, J. Guo, B. Tseng, and J. Yang. Use Userr Reputation in a Comment Rating Environment. In  KDD , 2011. [5] D. Dali Dalip, p, M. Gon¸ calves, calves, M. Cristo, and P. Calado. Automatic Assessment of Document Quality in Web Collaborative Digital Libraries.  JDIQ , 2(3), 2011. [6] A. Esuli and F. Sebastiani. Sebastiani. Sen SentiW tiWordNe ordNet: t: A Publicly Av Available ailable Lexical Resource for Opini Opinion on Mining. Mining. In   LREC , 2006. [7] D. Gruhl, R. Guha, D. Liben, and A. Tomkins. Tomkins. Information Information Diffusion Diffusi on Through Through Blogsp Blogspace. ace. In  WWW , 2004. [8] T. Hastie, R. Tibshiran Tibshirani, i, and J. Friedman. riedman.  The Elements of  Statistical Learning Statistical Learning:: Data Mining, Inference and  Prediction . Springer, 2008.

[9] H. He and E. Garcia. Learning from Imb Imbalance alanced d Data. IEEE TKDE , 21(9), 2009. [10] L. Hong, O. Dan, and B. Davison. Predicting Predicting Popu Popular lar Messages in Twitter. In  WWW , 2011. [11] S.-M. Kim, P. Pantel, T. Chklovski, Chklovski, and M. Pennacc Pennacchiotti. hiotti. Automatically Assessing Review Helpfulness. In  EMNLP , 2006. [12] Predicting B. Li, T. Jin, M. Lyu, I. King, B. Mak. Question Analyz Analyzing ing and Question Quality in and Community Answering Services. In  WWW , 2012. [13] Y. Liu, X. Huang, A. An, and X. Yu. Modeling and Predicting The Helpfulness of Online Reviews. In  ICDM , 2008. [14] E. Momeni and G. Sageder. Sageder. An Empirical Analysis Analysis of  Characteristics of Useful Comments in Social Media. In WebSci , 2013. [15] M. O’Mahony O’Mahony and B. Smyth. Smyth. Learning to Recommend Recommend Helpfull Hotel Reviews. In   RecSys, 2009. Helpfu [16] M. O’Mahony and B. Smy Smyth. th. Using Readability Readability Tests to Predictt Helpfu Predic Helpfull Product Product Reviews. Reviews. In  RIAO , 2010. [17] B. Pang, L. Lee, and S. Vaithyana aithyanathan. than. Thumbs Thumbs Up? Sentiment Classification Using Machine Learning Techniques. In  EMNLP , 2002. [18] M. Vasconcelo Vasconcelos, s, S. Ricci, J. Almeida, F. Benevenut Benevenuto, o, and V. Almeida. Tips, Dones and Todos: Uncovering User Profiles in Foursquare. In   WSDM , 2012. Standard Probability  [19] D. S. Kokoska. Kokosk a.   CRC Chapman & Hall, 2000. andZwillinger Statistics.and