In this project we first established a systematic way of assessing the features of tweets, accounts, and account activity that enabled the creation of a credibility metric. Second, we found that this metric applied to 2017 tweets by accounts believed to be tied to the Russian IRA identifies that high engagement is correlated with credible content. Thus, the tweets by Twitter accounts suspected of being from Russia’s IRA are a post-truth achievement in the sense that:
At first glance, most tweets look credible.
Credibility scores are related to the retweet count.
Most credible-looking tweets circulated the most.
Twitter is an important social network site that users increasingly use as a news source (Kwak, Lee, Park & Moon, 2010). A major area of concern is that alongside information from traditional news sources, mis/dis-information is being spread on Twitter (Lazer, Baum, Benkler, Berinsky, Greenhill, Menczer et al., 2018). One recent example is the case of the disinformation campaigning by Russia during the run-up to the election of U.S. President Donald Trump – a topic of significant public discourse (Allcott & Gentzkow, 2017; Knight Foundation, 2019). Concurrently the term 'fake news’ (used to describe a variety of materials, including both misinformation as well as deliberate hoaxes) has become mainstreamed (Phillips, 2018). In response, counter-campaigns that range from fact-checking to media literacy have been increasingly implemented at national and grassroots levels. Yet as Marwick (2018) notes, these approaches have not stemmed the flow of so-called 'fake news' and its ilk on Twitter. Given this context, the question arises of what makes tweets credible? What is meant by credibility? And how can it be assessed?
What does credibility look like in the context of a disinformation campaign? In this project, we were concerned with developing a credibility metric specific to our data set of Twitter accounts associated with Russia’s Internet Research Agency (IRA), accused of propagating 'fake news' and disinformation internationally (Twitter, 2018). We assumed that as successful disinformation agents must also play by Twitter’s rules, they may even 'game' the algorithm in a way that technically makes them ‘real’ or ‘authentic’ and arguably, credible. We identified features, and signals at both the account and tweet level, that successful disinformation efforts may have in common. Our hope was to come one step closer to a potential near real-time tool to scan for disinformation campaigns more reliably on Twitter in our heightened post-truth climate.
Our approach drew upon the work of Gupta, Kumaraguru, Castillo & Meier, (2014) who to assess the credibility of twitter content developed TweetCred; a tool attributing credibility scores based on various characteristics. We took a similar approach, but built a credibility metric from a specific set of misinformation data.
The data set used in this project came directly from Twitter:
On 17 October Twitter, in a blog post entitled ‘Enabling further research of information operations on Twitter’, released data sets containing: “3,841 accounts affiliated with the IRA, originating in Russia, and 770 other accounts, potentially originating in Iran. They include more than 10 million Tweets and more than 2 million images, GIFs, videos, and Periscope broadcasts, including the earliest on-Twitter activity from accounts connected with these campaigns, dating back to 2009.”
RQ1: How can the credibility of a tweet and user be determined?
RQ2: To what extent can qualitative and quantitative features of tweet text, users, and account activity predict how much a tweet is retweeted?
We accessed the data through the Digital Methods Initiative’s Twitter Capture and Analysis Toolset (TCAT). Although the data goes back to 2009, in this project we focused on the data from 2017. This was a period of emerging revelations about misinformation campaigns, when users would theoretically be more on guard. Yet the IRA accounts continued to successfully propagate content on the platform perceived as credible.
To make our sample, we ordered users during this period by retweet count and then selected 15 users to study: five from among the top of the list (the most successful), five from the middle, and five from the bottom. The top most three and bottom most three where deliberately omitted to minimize the outlier effect. Our 15 user accounts comprised 13,371 tweets from which a random sample of 498 tweets was then chosen for further manual analysis.