[Cross-post from Medium https://medium.com/misinfosec/who-handles-misinformation-outbreaks-e635442972df]

Misinformation attacks— the deliberate and sustained creation and amplification of false information at scale — are a problem. Some of them start as jokes (the ever-present street sharks in disasters) or attempts to push an agenda (e.g. right-wing brigading); some are there to make money (the “Macedonian teens”), or part of ongoing attempts to destabilise countries including the US, UK and Canada (e.g. Russia’s Internet Research Agency using troll and bot amplification of divisive messages).

Enough people are writing about why misinformation attacks happen, what they look like and what motivates attackers. Fewer people are actively countering attacks. Here are some of them, roughly categorised as:

Journalists and data scientists: Make misinformation visible
Platforms and governments: Reduce misinformation spread
Communities: directly engage misinformation
Adtech: Remove or reduce misinformation rewards

Throughout this note, I want to ask questions about these entities like “why are they doing this”, “how could we do this better” and “is this sustainable”, and think about what a joined-up system to counter larger-scale misinformation attacks might look like (I’m deliberately focussing on sustained attacks, and ignoring one-off “sharks in the streets” type misinformation here).

Make Misinformation Visible

If you can see misinformation attacks in real time, you can start to do something about them. Right now, that can be as simple as checking patterns across a set of known hashtags, accounts or urls on a single social media platform. That’s what the Alliance for Securing Democracy does, with its dashboards of artefacts (hashtags, urls etc) from suspected misinformation trolls in the US (Hamilton68) and Germany (Artikel38), what groups like botsentinel do with artefacts from suspected botnets, and sites like newstracker do for artefacts from Facebook.

In almost real-time, there are groups tracking and debunking rumours: Snopes and similar fact-checking organisations, small-scale community groups like no melon and Georgia’s Myth Detector, and state-level work like the European External Action Service East Stratcom Task Force’s EU vs Disinfo campaign and US State Department’s polygraph.info.

For some events, like US elections, there are many groups collecting and storing message-level data, but that work is often academic, aimed at learning and strategic outputs (e.g. advocacy, policy, news articles). Some work worth watching includes Jonathan Albright’s event-specific datasets; Kate Starbird’s work on crisis misinformation; Oxford Internet Institute’s Computational Propaganda project (e.g. their 2016 USA elections work) and DFR Lab’s #electionwatch and #digitalsherlocks work.

Real-time misinformation monitoring is costly to set up accurately, and is usually tied to a geography, event, group of accounts or known artefacts, and a small number of platforms. That leaves gaps (were there any dashboards for the recent Ontario elections?) and overlaps, and there’s a lot of scope here for real-time information-sharing (oh god did I just volunteer for yet another data standards body?) across platforms, countries and topics, because that’s where the attackers are starting to work.

Simple things like checking a new platform for known misinformation artefacts, e.g. the US House Intelligence Committee’s datasets [“Exhibit A”] of Russia-funded Facebook advertisements, and checking for links to known fake news sites have already unearthed new misinformation sources. We’re already seeing multi-platform attacks and evolution in tactics to make detection and responses harder, and really good techniques (the evolutions of early persona management software into large-scale natural language posts) haven’t really been used yet. When they do, our tracking will need to get broader and more sophisticated, and will need all the artefacts it can get.

More tracking projects are listed in Shane Greenup’s article, and it’s worth watching Disinfo Portal, which reports on other anti-misinformation campaigns.

Reduce misinformation spread

Large-scale misinformation is a pipeline. It starts in different places and for different reasons, but it generally needs a source, content (text, images, video etc), online presence (websites, social media accounts, advertisements, ability to write comments or other user-generated content in sites and fora etc), reach (e.g. through botnet or viral amplification), and end targets (community beliefs, people viewing advertising etc) to succeed.

The source is a good place to start. I’ve had success in the past asking someone to remove or edit their misinforming posts, but that’s not our use-case here: we’re looking at persistent and often adversarial content. Attacking a creator’s intent, or using other less-savory methods to dissuade them is perhaps something from a bad spy novel, but these have also been known to happen in real life. The creators of fake news sites are often there to make money, and either removing the promise of cash (see below) or making it riskier to obtain that money (e.g. by penalising the site creators) might work. The creators of social media-based misinformation attacks often have other incentives, getting quickly into the realm of politics and diplomacy (also see below).

The next place is the platforms that host misinformation. These are typically websites (e.g. “macedonian fake news sites” ) and social media platforms, but we’ve also seen misinformation coordinated across other user generated content hosts, including comment sections and payment sites.

Websites typically use internet service providers, domain name providers and advertising exchanges (for monitisation). Internet service providers could and have removed sites that violate their terms of service (e.g. GoDaddy removed the Daily Stormer and other sites have been removed for stealing content), but we haven’t seen them remove reported “fake news” sites yet (we’d love to be corrected on that). Sites can be removed for domain squatting or names similar to trademarks, but most fake news sites steer clear of this. Advertising exchanges and their customers do keep blacklists of sites that include fake news sites — that’s covered in more detail below. It’s also possible to disrupt views to fake news sites, by changing search engine results using sites with similar text, squatting on domains with similar names etc.

Misinformation on social media can be original content (text, images etc), pointers to fake news sites, or amplification of either of these through repetition, use of popular hashtags and usernames, or other marketing techniques to get sources and messages more attention. One area of much interest recently has been amplification of misinformation by trolls, bots and brigades (e.g. coordinated 8chan users); we’re starting to see these adapt to current detection techniques and are looking at ways they’re likely to evolve as artificial intelligence techniques improve (that’s for a different post).

Social media platforms have tried a variety of ways to stop misinformation spreading.

Early work focussed on the reader of misinformation, e.g. adding credibility markers to messages, adding fact-check buttons and notifying people who engaged with known bots/trolls (this is included in Facebook’s list of possible actions). Whilst reader education is important in combatting misinformation, user actions are difficult to obtain (which is why advertisers set a high monetary value on clicks and actions) and some actions (e.g. Twitter check marks) added to reader confusion.
The most-discussed platform actions are identifying and removing fake accounts. In an adversarial environment, this needs to be done quickly, e.g. remove botnets whilst misinformation attacks are happening, not after, and that speed increases the potential for collateral damage including misclassification and platform friction for users. Removing accounts is also not without risk to the company: Twitter suspended 70 million accounts recently, which had very little effect on the active botnets being tracked (most of the accounts removed were inactive), but did damage Twitter’s share price: Twitter, like many social media platforms, makes most of its revenue from advertising, and also has to manage perception of the company (a good example of this is the drop in transactions after AppNexus’s 2015 bot cleanout).
People are good pattern and nuance detectors: users can and do flag fake accounts and hate speech in messages to platforms, but this both creates a queue to be managed, and a potential for abuse itself (several of my female friends have had their social media accounts reported by abusers online). Abuse reports appear to go into ‘black holes’ (many of the well-documented botnet accounts are still active), and misinformation messages are often carefully crafted to create division without triggering platform hate speech rules, but there may still be some merit in this.
A softer way to reduce the spread of misinformation is to limit the visibility of suspected misinformation and its promoters, and increase the operating cost (in terms of time and attention) for accounts thought to be parts of botnets. We’ve seen shadowbans (making content invisible to everyone except its creator) and requests for account verification (e.g. by SMS) on bot accounts with specific characteristics recently: whilst reverifying every account in a 1000-bot network can be done, it takes time and adds friction to the running of a misinformation botnet.
The end point of misinformation is its target demographic(s). Since reach (the number of people who see each idea or message) is important, anything that limits either reach or the propagation speed of misinformation is useful. There are spam filter-style tools at this end point (e.g. BlockTogether) that highlight suspicious content in a user’s social media feeds, but platforms don’t yet have coordinated blocking at the user end point.
Large-scale misinformation attacks happen across multiple platforms, any one of which might not have a strong enough signal for removing messages on its own. There are meetings but not much visible response coordination across platforms yet, and the idea of an EFF-style watchdog for the online providers that misinformation flows through is a good one if it’s backed up with coordinated real-time response. This is where standards bodies like the Credibility Coalition are valuable, in helping to improve the ways that information is shared.

Stopping misinformation at the platform level can be improved by borrowing frameworks and techniques from the information security community (which yes, is also a post in its own right), including techniques for adversarial environments like creating ‘red teams’ to attack our own systems. Facebook has, encouragingly, set up a red team for misinformation; it will be interesting to see where that goes.

Stopping misinformation at the platform level also needs new policies, to provide a legal framework for removing offenders, and to align misinformation with the business interests of the platforms. One solution is for platforms to extend their existing policies and actions for hate speech, pornography and platform abuses. Removing misinformation comes with financial, legal and social risks to a platform, and if it’s to get into policies and development plans, it needs strong support. This is where governments can play a large part, in the same way that GDPR data privacy regulations forced change. There are already regulations in Germany, and similar activity in other countries that look very similar to the pre-GDPR discussions on data privacy and consent; unfortunately, misinformation regulations are also being discussed by countries with a history of censorship, making it even more important to get these right. A sensible move for platforms now is to create and test their own policies and feed into government policy, before policy is forced on them from outside.

Reducing misinformation rewards

Misinformation attacks usually have goals, ranging from financial profit to creating favorable political conditions (approval or confusion: both can work) for a specific nationstate and its actions.

Online advertising is a main source of funding for many misinformation sites. Reducing misinformation advertising revenues reduces the profits and economic incentives of “fake news” sites, and the operating revenue available to misinformation campaigns. Adtech companies keep blacklists of fraudulent websites: although very few of them (e.g. online advertisers in Slovakia) explicitly blacklist misinformation sites, misinformation sites and bots are often blacklisted already because they’re correlated with activities like click fraud, bitcoin mining and gaming market sentiment (e.g. in cryptocurrency groups). Community campaigns like Sleeping Giants have been effective in creating adtech boycotts of political misinformation and hate speech sites. Other work supporting this includes collection and analysis of fake news sites.
Political misinformation is part of a larger information warfare / propaganda game: fewer politicians have played it better than Macron’s election team creating false information honeypots and responses to their disclosure before the election blackout started in France. Social media is now a part of cyberwarfare, so we’re likely to see (or not see) more skillful responses like this online.

Ultimately when we respond to misinformation attacks by reducing rewards, we’re not trying to completely eradicate the misinformation — as with all good forms of conflict reduction, we’re trying to make it prohibitively costly for any but the most determined attackers to have the resources and reach to affect us.

Engaging misinformation online

Even if platforms work to remove misinformation botnets and limit the reach of trolls, misinformation will still get through. It’s not clear who’s responsible for countering misinformation attacks at this point: it’s made it past the platforms’ controls, but still has the potential to damage belief systems and communities (increasingly in real life, as evidenced by trolls setting up opposing protests at the same locations and times).

At this point, it’s appropriate to engage with the material and its creators, limiting the reach of bots and trolls whilst being mindful of personal and community safeties. This engagement can happen at most points in the misinformation pipeline, and this is another area where the infosec mindset of creatively adapting systems can be useful. Some recent notable examples include:

Lithuanian and Latvian ‘Elves’: roughly 100 people who respond to Russian-backed misinformation with humour and verified content, in “Elves vs. Trolls skirmishes”.
VOST work (in Spanish) on tagging disaster misinformation with links to verified updates, and retweeting misinformation with “Falso” stamps.
Overwhelming misinformation hashtags with other content (beautiful recent examples: a right-wing hashtag suddenly filled with news about a swimming contest, and an Indian guru using the Qanon hashtag).
Using search engine optimisation and brigading to move other stories above misinformation pages and related search terms in search and news results.

Engaging with bots and trolls has risks, including doxxing, which should be carefully considered if you’re planning to do this yourself.

The ecosystem, in general

Many of the players countering misinformation are working on a small scale. This can be effective, but we’re looking at an issue that will probably be part of the internet for the foreseeable future, and from my experience running crisismapping deployments, I know that without more support, these efforts might not be sustainable in the long term.

The social media platforms have a great deal of leverage on this problem — they could shut down misinformation almost overnight, but that would be at great potential cost to them, both in audience and in financial cost (e.g. the cost to share price of ad-supported platforms removing ad viewings). Reducing money supplies (e.g. adtech) and other misinformation incentives is another good avenue of approach, and we’ve seen some working community responses but they might be difficult to scale over time.

Misinformation attacks aren’t going away. Today they’re nation-state backed and personal attacks; tomorrow those skills could be scaled with AI and applied to companies, groups and other organisations. We need to think about this as an ecosystem, using similar tools and mindsets to information security. We also need to create more joined-up, cross-platform responses.

Thanks to connie moon sehat (hide).