How Extremist Groups Navigate the Online Ecosystem: A Q&A with SIPA Professor Tamar Mitts

Image

When extremist groups face restrictions on major social media platforms, where do they go? And more importantly, how do they manage to thrive despite widespread content moderation efforts? These questions drove Assistant Professor of International and Public Affairs Tamar Mitts, who is also a member of the Data Science Institute, the Saltzman Institute of War and Peace Studies, and the Institute of Global Politics, to investigate how over one hundred extremist organizations navigate the digital landscape.

Her findings, published in her recent book Safe Havens for Hate: The Challenge of Moderating Online Extremism, challenge conventional wisdom. In this SIPA News Q&A, Mitts explains why platforms’ fragmented moderation policies actually enable extremist resilience and what it would take to create more effective solutions to the worsening problem of online hate.

What inspired you to write Safe Havens for Hate, and what was the core message you wanted to convey to readers about the challenge of moderating online extremism? 

When researching extremist movements on social media, and the efforts that tech companies put in to limit their reach on their platforms, I noticed a very consistent pattern in the way these movements adapted. That made me curious to look more closely at their adaptation to content moderation. 

What I found was that adaptation strategies are not only consistent across groups, but also are closely tied to the structure of the online information ecosystem – an ecosystem shaped by the strategic interactions between national governments and tech companies. The book tells the story of how the competing incentives of these actors produce an information environment that actually makes it easier for harmful content producers to build resilience to content moderation.

You define “online extremism” in three ways, the first one being “a set of ideas, beliefs, or preferences that diverge from the views of the majority of the population.” This definition seems quite broad – couldn't this potentially lead to labeling minority viewpoints as “extreme” simply because they differ from mainstream opinion? How do you balance addressing genuinely harmful content without enabling a “tyranny of the majority” in content moderation decisions?

This is a great question. In the book, I offer three definitions to show the analytical approaches scholars have previously taken when defining extremism – one of which defines the concept as a set of ideas or preferences that represent the extreme ends of a spectrum not shared by the majority. This definition can seem overly broad, but it’s important to understand that this is precisely the conceptualization that national governments and tech companies – whose behavior is the focus of my book – often rely on when making decisions about what constitutes harmful content.

In the book, I take a broad approach to this concept not because I necessarily endorse it, but because I want to study how those in power operationalize it. And yes, in cases where those who decide what stays online are aligned with the majority, then minority views that fall outside that consensus may be sanctioned under this logic. That’s a central tension I explore.

If you look at the empirical cases in the book – particularly Chapter 3, which shows how governments grapple with this question – you’ll see that this isn’t the only definition in play. In fact, the most commonly invoked definition is the one that ties extremism to calls for physical harm against people or property. In these cases, groups are labeled extremist not only because their views are out of step with the majority, but also because they promote or engage in real-world violence.

In your research, you found that extremist groups exploit gaps between different social media platforms. Could you explain how this happens and give some examples?

Yes, and this is actually one of the core arguments of the book. A common assumption is that moderation just pushes extremists from one platform to another in a kind of whack-a-mole dynamic. But what I found is that their migration isn’t random, but is highly strategic. Extremist groups tend to gravitate toward specific platforms that allow them to maximize both authenticity and reach. They use smaller, less moderated platforms (places with weaker enforcement and less public scrutiny) for internal communication, mobilization, and organizing. But they don’t give up on the big platforms. They continue to use platforms like Facebook, X, or YouTube to circulate more innocuous content and reach broader audiences.

This behavior reflects the structure of the online environment itself. Larger platforms, especially those based in democratic countries, are under much more pressure from governments and civil society to moderate harmful content. Smaller platforms often face less of that pressure and operate under different incentive structures, so they’re more permissive. What emerges from this is a fragmented ecosystem, where extremist groups can strategically allocate different types of content to different platforms, depending on the affordances and constraints of each.

So rather than being pushed around by moderation, these groups are actively navigating a tiered ecosystem in ways that let them maintain both operational capacity and public visibility. That’s a big part of why moderation efforts so often fall short – because the problem isn’t confined to any one platform.

Your book shows that this lack of convergence in content moderation across platforms enables extremist resilience. While perfect convergence may not be realistic, what mechanisms or initiatives could help platforms create more effective solutions?

Throughout the book, I show what happens when platforms diverge in their policies to combat harmful content, and what happens when they converge (at least to some extent). Of course, perfect convergence is not realistic, but greater coordination or resource sharing can certainly help. This is partly what the Christchurch Call for Action [The Christchurch Call for Action to Eliminate Terrorist and Violent Extremist Content Online, organized after the terrorist attacks of March 15, 2019 against mosques in Christchurch, New Zealand] tried to do. Third-party initiatives such as ROOST [Robust Open Online Safety Tools, an initiative incubated at SIPA’s Institute of Global Politics and launched earlier this year] for example, which aim to provide open-source software and shared databases to help platforms better identify and remove harmful content known to incite real-world harm, can also help.

How does the global nature of the internet complicate efforts to combat extremism, especially when considering differing laws and cultural norms in different regions?

One of the most interesting dynamics we’ve seen in the past several years is the effort by many national governments to regulate social media platforms. As you can probably imagine, different governments define what counts as harmful content in very different ways, and these pressures make it difficult for tech companies to create a unified set of policies for addressing harmful content across their platforms.

What you see in practice is that platform policies are sometimes tailored to specific jurisdictions (to comply with local regulations), while other policies apply more globally. The downstream consequences of this variation in how harmful content is defined – and in how moderation is implemented across regions – is exactly what I’m exploring in the book.

You use data from a variety of extremist groups in writing this book. Were there any surprising patterns or findings that emerged when analyzing the strategies these groups used to stay active online?

The most interesting pattern was how similar these groups are in the way they adapt to content moderation – even across very different ideologies, cultures, and regions. When they face restrictions, their responses tend to follow the same playbook. What I suggest in the book is that this consistency doesn’t stem from the groups themselves, but from the structure of the online environment. The way platforms are governed, the kinds of pressure they face, and the lack of coordination across them all shape how extremist actors – regardless of background – behave online.

Could you describe some of the technological solutions, such as AI or machine learning, that are being used to detect and remove extremist content?

Since social media platforms, when trying to address harmful content, are dealing with huge volumes of information, automation has been a central part of the process from the start. The most commonly used tools are machine learning classifiers that can automatically detect content platforms define as harmful – or flag it for human review – so it can be removed or sanctioned if it violates policy. But classifiers are never perfect, and their errors can sometimes lead to backlash, especially when users feel their content was wrongly removed or their account unfairly suspended.

What do you hope readers, especially policymakers, will take away from your book? Are there specific actions you would recommend to address the issue at both the micro and macro levels?

At the most basic level, I hope the book helps readers understand that online extremism isn’t just about bad actors posting harmful content, but it’s about how the structure of the information ecosystem enables those actors to persist, adapt, and thrive. For policymakers, that means moving away from reactive, platform-by-platform approaches and instead treating this as a systemic problem. So on the macro level, we need more coordination: more shared standards, more transparency, and more support for smaller platforms that often lack the resources to enforce their own rules. On the micro level, the focus should be on closing those enforcement gaps – for example, investing in better tools, making sure platforms actually use them, and helping individual users better understand the problem.

What’s your next research project?

I have many ongoing projects related to this work, but the one I’d highlight is my research on the strategic use of information in conflict zones. I’m especially interested in how armed actors use social media and other digital tools to shape the beliefs and behavior of different audiences during wartime – whether to signal strength, suppress resistance, or influence international opinion. This builds on some of the core themes in the book, but shifts the focus to conflict settings where the stakes around information are even higher.