Signals of harm

Signals of harm

An indicator-based framework to improve identification of risk factors for CT and P/CVE online monitoring

Among the most challenging of questions that any given society must confront is when should the state intervene before a crime has been committed. This dilemma is particularly pronounced in the context of terrorism and violent extremism, where the consequences of inaction can be catastrophic, yet premature or excessive intervention risks undermining the very rights and freedoms that democratic societies seek to protect. 

This challenge takes on an added level of complexity when considering how radicalisation is no longer confined to physical spaces or through identifiable networks. Increasingly, it unfolds online through fragmented communities, algorithmically curated content and encrypted-private channels.  

In this sense, individuals can encounter extremist narratives, form ideological affiliations and deepen their commitment to violent causes with nary a peep on the radar of the authorities. Policymakers are therefore confronted with a difficult balancing act: how can legitimate security concerns be addressed without eroding fundamental rights or creating disproportionate powers of surveillance? 

This question cannot be answered through simplistic binaries that frame security and liberty as mutually exclusive objectives. Effective policy in this regard requires a more nuanced approach that equally weights safety measures that are proportionate, accountable and anchored in the rule of law with the protection of individual rights without ignoring genuine threats to public security. 

This paper seeks to contribute towards identifying ways to unpack this seeming binary. By examining the pathways of radicalisation and developing a framework to guide intervention, it seeks to identify when heightened scrutiny may be justified and what safeguards should accompany such actions. In doing so, it offers a pathway towards a more calibrated approach to preventing violent extremism – one that strengthens security while preserving the privacy, liberties and legal protections that underpin a democratic society. 

Datuk Prof Dr Mohd Faiz Abdullah
Executive Chairman

• This paper examines indicators of online extremism and proposes a structured framework to support consistent risk assessment, early intervention as well as whole-of-government and whole-of-society responses to preventing violent extremism and terrorism.
• Social media and online platforms have been exploited by extremist and terrorist groups to disseminate propaganda, spread ideological narratives and disinformation, facilitate recruitment and influence online users. They are expanding their reach and potential impacts, aided by emerging technologies.
• Gaps in Malaysia’s counterterrorism and preventing or countering violent extremism efforts include difficulties in collecting legally defensible online evidence without clear thresholds, a primary regulatory and enforcement focus on clear legal violations and enforceable online behaviour as well as limited integration between regulatory and security-led online monitoring.
• The increasing complexity of online radicalisation requires a shift away from reactive enforcement towards a preventive, proportionate and evidence-based approach. One nuance is recognising that not all harmful or concerning online content necessarily meet evidential or legal standards for criminal enforcement.
• The main recommendation is to operationalise an indicator-based framework to support online monitoring. A proposed matrix classifies indicators by risk levels, matched with proportionate and recommended responses. It is complemented by a tiered risk classification system, with the option of incorporating the RAG+ and Channel systems.
• Further recommendations include reforming legal provisions, positioning restorative justice as a pre-crime intervention model, adopting a “trusted flaggers” programme, developing an accessible reporting platform with a single point of contact and strengthening cross-functional coordination.
• Policy considerations include balancing between human rights and public safety, deploying hybrid detection approaches combining artificial intelligence-assisted pre-screening with human expertise, capacity-building and clearly identifying thresholds in online content governance and criminal justice approaches.

The rapid growth of online platforms has created significant challenges for regulators and enforcement agencies in identifying and responding to harmful or extremist digital content. While some online content clearly incites terrorism or promotes violent extremism, other forms may be still lawful, yet concerning.

This situation raises several important regulatory and legal questions. How can regulators and enforcement agencies distinguish between online content that incites terrorism or promotes violent extremism, and that which is legally permissible but which may still pose risks towards users and public safety? What evidence-based indicators can help authorities identify early warning signs of risk and intervene before they escalate into real-world violence? When do online behaviours cross the legal threshold for prosecution? How should authorities decide whether criminal enforcement is necessary, or whether preventive or non-criminal interventions are more appropriate?

These questions reveal a gap within Malaysia’s digital safety, counterterrorism (CT) and preventing or countering violent extremism (P/CVE) framework, one which quietly shapes the national security landscape. Exposure to violent, extremist or incendiary content, whether blatant or subtle, can draw individuals towards dangerous ideas, self-radicalisation or affiliations with groups that nurture such impulses. Extremist networks understand this vulnerability well. They manipulate online spaces with calculated intent, especially by reaching the children and adolescents who form a significant portion of Malaysia’s online community.¹ Only 57.5% of these young users receive parental guidance (where parents check their child’s social media accounts or browser history), leaving many to navigate digital risks unprotected.²

Developing evidence-based indicators and thresholds centred upon CT and P/CVE strengthens governance, enables proactive interventions and equips civil society organisations (CSOs) and the public with tools to hold institutions accountable. In the Malaysian context, establishing common terms of reference is essential for ensuring consistent interpretation and preventing regulatory overreach, since the boundaries between harmful content, extremist expression and legally protected speech are often complex and contested. Existing legal frameworks – including provisions under the Penal Code 1936 (Act 574) relating to terrorism offences, such as Sections 130J(1)(k), 130JB and related provisions, as well as Section 233 and related provisions in the Communications and Multimedia Act (CMA) 1998 – provide mechanisms to address the dissemination of harmful or threatening content.³

However, the absence of clearly defined indicators for identifying early signals of extremist risk may create uncertainties for regulators and enforcement agencies. An indicator can be defined as a “behaviour that suggests an individual has likely already radicalised to violent extremism and may require more timely intervention”.⁴ This absence can lead to inconsistent enforcement, potential encroachment upon legitimate expression or delayed intervention in situations where online content may contribute to radicalisation or violence. Establishing structured, evidence-based indicators would therefore support more proportionate, transparent and legally defensible regulatory responses.

Internationally, the challenge of countering violent extremism online remains unresolved, with the European Union (EU) currently leading global discussions on thresholds, indicators and intervention strategies. These efforts are reflected in its regulatory instruments, notably the Digital Services Act and the Regulation on Addressing the Dissemination of Terrorism Content Online, or Regulation (EU) 2021/784.⁵ These regulations apply across all member-states and impose accountability and risk-mitigation obligations upon online platforms, including requirements to swiftly remove and report identified terrorism-related content.

Against this background, this research examines Malaysia’s position within evolving global discourse, while accounting for its distinct social realities and legal complexities. It seeks to examine existing approaches and identify practical considerations for strengthening policy and regulatory responses to online violent extremism. As a disclaimer, it focuses primarily on content, rather than platform governance systems. Research employed a mixed-methods approach, combining doctrinal and non-doctrinal research with desk research and a review of academic literature, policy reports, legislation, jurisprudence and annual reports of online regulators, as well as semi-structured interviews and discussions, expert workshops and qualitative content analysis (see also the Appendix below). This provided the foundation for identifying indicators and developing an online indicator matrix, tiered risk classification system as well as reporting and flagging mechanisms. Due to time constraints and limited engagement with some stakeholders, it was not possible to conduct interviews and discussions with all the stakeholders approached. All research participants are anonymised.

“Terrorism” and “violent extremism” are pervasive terms in global security discourse and often experienced as remote concerns that gain heightened public attention during periods of media reportage on major incidents. However, there are no universally agreed-upon definitions of both terms, leading to a diversity of definitional approaches developed at the national, regional and international levels.

In Malaysia, Section 130B(2) of the Penal Code defines terrorism as “any act that is done with the intention of causing death or serious bodily injury to any person; or causing extensive destruction to a place or property; or causing serious disruption of any essential service, facility or system; or creating a public emergency.”⁶ The most common understanding of violent extremism, meanwhile, includes “acts or support for violence to achieve social, political or legal outcomes or in response to specific political or social grievances”.⁷ In contrast, there has always been little discussion of CT policy, which is primarily limited to the scholarly field.

When a public policy is introduced, it often undergoes changes and modifications over time. Hall’s three “orders” of policy changes – the routine adjustments to policy settings, changes in policy instruments and shifts in overarching policy goals – can illustrate transitions in Malaysian laws regarding CT and P/CVE.⁸ A third-order policy change in the CT and P/CVE framework occurred when the Internal Security Act (ISA) 1960, previously used to counter terrorism, was repealed in 2011 and replaced by the Security Offences (Special Measures) Act (SOSMA) 2012, which now serves as the procedural law for arrest, investigation and trial. Building upon this legal shift, Malaysia saw a series of first- and second-order policy changes. These included the introduction of Chapter VIA: Offences Relating to Terrorism in the Penal Code, which expanded the scope of terrorism-related offences (such as terrorist group membership, support, financing and engagement in recruitment and training for terrorist acts), and the Anti-Money Laundering, Anti-Terrorism Financing and Proceeds of Unlawful Activities Act (AMLA) 2001, which reinforced provisions on terrorism financing, freezing assets, seizure procedures and reporting requirements for financial institutions.

The factors shaping CT policymaking include interests, ideas, institutions and events.⁹ Over the past 20 years, following the 11 September 2001 attacks in the United States of America and to counter the threat posed by the so-called “Islamic State” (Daesh), substantial efforts and resources have been dedicated to policymaking globally. This led to different CT responses, including in Malaysia, that required another paradigm shift in policymaking.¹⁰ Malaysia’s criminal justice system enhanced preventive detention and enforcement powers through the Prevention of Terrorism Act (POTA) 2015 and the Special Measures Against Terrorism in Foreign Countries Act (SMATA) 2015, alongside other broad measures introduced as part of the evolving CT and P/CVE framework. This includes the launch of the Malaysian Action Plan of Preventing and Countering Violent Extremism (MyPCVE).¹¹ The latest measure, the Online Safety Act 2025, adds a new protective layer for online users by strengthening regulatory oversight of digital platforms, emphasising protections for children and managing online harm, by imposing obligations upon platforms that host or distribute user-generated content.¹² Although the Act requires platforms to identify, manage and mitigate risk, as well as provide reporting and user support mechanisms, there is still room for harmful content (as defined in the Act) to be shared, since it does not extend to private one-to-one messaging.

Overall, Malaysia’s CT framework has evolved from an executive-based era of preventive detention to a criminal justice system characterised by legal restructuring.¹³ The criminal justice system offers an institutionalised balance between the rights of the accused and society’s right to security.¹⁴ A significant turning point has also emerged in both policy and discourse, with the term “terrorism” being used less frequently, replaced by “violent extremism”. These changes reflect a broader shift in focus within Malaysia’s CT and P/CVE approach.

However, the comprehensive procedures for investigating and prosecuting terrorism as a criminal offence have come to be regarded as inadequate for the current digital age. This is because the nature of terrorism is evolving, and its threats are now different. Therefore, the way the government responds needs to change accordingly.

The prevailing ambiguity in current definitions under the CT and P/CVE framework reflects their evolving and multifaceted nature, the absence of universally accepted definitions as well as variations in interpretation across jurisdictions. It is further compounded by conceptual overlaps and varying usage of terminology in existing policy and academic literature. As such, this definitional framework intends to establish operational definitions to guide analysis and ensure the consistent use of key terms.

Harmful content: Based on the Online Safety Act 2025, this is any content specified in its First Schedule. For CT and P/CVE purposes, such content is specifically defined as that which may incite violence or terrorism. Malaysia also outlines a separate definition for “priority harmful content”.¹⁵

Internet crimes: Crimes committed on or facilitated by the use of the internet. This term is often used interchangeably with “cybercrime”, which is a much broader term including any “criminal offence that has been created or made possible by the advent of technology or a traditional crime which technology’s use has transformed”.¹⁶ While this term is frequently overused and includes many internet crime categories (including hacking, fraud and cyberterrorism), this paper mainly focuses on social media-enabled terrorism.

Lawful but concerning content: Also known as “lawful but awful” content or “borderline” content, it “is legal but widely considered to be socially reprehensible”. Similarly to the vaguely distinguished terms “terrorism” and “violent extremism”, borderline content remains an equally convoluted concept.¹⁷

Online content: Any digital information serving as a medium, such as comments, pictures, videos, files, posts, links, chatroom chats, blogs or messages.

Radical content: Generally, content that can shift a person’s thinking and behaviour to become significantly different from how most of members of their society and community view social issues and participate politically.¹⁸ Since radical content can be an overly broad category and may include lawful dissent, for CT and P/CVE purposes, it should be: distinguished from radical content for activism; and defined as content that creates readiness to engage in illegal and violent political action.¹⁹

Terrorist content: This definition should not depart from the definition of a terrorist act. For example, Regulation (EU) 2021/784 defines terrorist content as “material that solicits someone to commit or to contribute to terrorist offences, or to participate in activities of a terrorist group, incites or advocates terrorist offences, such as by glorification of terrorist acts, or provides instruction on how to conduct attacks.”²⁰

Violent extremist content: Violent extremist content can be defined as content that is “conceptually weaker than the term terrorism, which has an identifiable core”. The Shanghai Convention includes a target requirement, while Australia and the United States include a motive requirement (such as political, religious or ideological purposes).²¹ The Australian government requires three characteristics: the intention to advance a political, religious or ideological cause; supporting or facilitating serious violence; and including elements of intimidation.²²

There are at least four routes through which online conduct evolves into real-world harm or a terrorist attack. The first is a stepwise progression through distinct stages. The second holds that not all terrorists necessarily go through distinct stages. The third holds not all terrorists go through all these stages. The fourth holds that a terrorist may not go through stages in a particular order.

Regardless of their route, individuals can engage in terrorist attacks when they have the right intent, motivation and resources: financial, physical and material (for example, access to weapons and explosives) capability; as well as research and planning (for example, hostile reconnaissance).²³ Hostile reconnaissance is understood as purposeful observation (for example, of people, vehicles, buildings, places, spaces, whether onsite or online) to inform the planning of attacks against specific targets, sometimes involving rehearsals where one or more elements of a plan are practised.²⁴

Traditionally, ideology has been a significant factor in the formation of terrorist and extremist groups, by binding followers to a shared goal, providing a sense of belonging and justifying struggle. The role of the internet in radicalising individuals to violence and facilitating violent attacks through a shared ideology is not a new challenge.

One current problem is re-emerging threats of right-wing extremist ideology and religious extremist ideology (including but not limited to Islamist, Buddhist, Hindu and Christian extremisms).²⁵ Even white supremacist ideology has found its way to Southeast Asia in the form of “multicultural white supremacy”.²⁶

Anti-government extremism (including right- and left-wing variants) promoting conspiracy theories has also re-emerged in online spaces.²⁷ In June 2024, the RMP arrested eight people suspected of posing an imminent threat to key institutions, including the Yang di-Pertuan Agong, prime minister, dignitaries and senior police officials.²⁸

Case 1: Ulu Tiram police station attack²⁹

Case 1 

This 2024 incident in Johor was perpetrated by 21-year-old Radin Luqman Radin Imran, who was shot dead in an attack that killed two police officers and injured another. Malaysian authorities concluded that it was a “lone wolf” act, although further investigations revealed that threats could still be posed by his family members. Investigations found four air rifles, among other weapons, in the family home, with statements and phrases written on the walls by Radin Luqman, expressing hatred towards the government and security forces. His father, Radin Imran Mohd Yassin, a former Jemaah Islamiyah member who had since pledged allegiance to Daesh, allegedly propagated its teachings to his wife and children. His wife, Rosna Jantan, was sentenced to four years’ imprisonment due to her failure to provide information related to terrorism offences, and admitted that her husband had exposed the family to Daesh ideology through videos circulated by the group. The attacker’s 36-year-old brother, Radin Romyullah Radin Imran, had downloaded video clips of Daesh from social media, fantasised about attacking the Merdeka Day parade in a van packed with explosives, and identified police officers and military personnel as targets. The court sentenced Radin Imran and Radin Romyullah to 30 years’ imprisonment each, after they pleaded guilty to terrorism-related charges.  

Case 2: State High School 72 school bombing in Indonesia³⁰

Case 2 

On 7 November 2025, a 17-year-old student bombed his high school with improvised explosive devices (IEDs), in North Jakarta, having drawn inspiration from far-right extremists and school-shooters, injuring 96 individuals. Post-attack discussions described the case as a combination of memetic violence and True Crime Community (TCC) subculture, inspired to a minor extent by white supremacist attacks in the West. This is a key example of how youths can be influenced to commit attacks triggered by online content without adhering to a definite ideology.  

Another challenge is that virtual connections and platforms hosting violent extremist content and connecting individual supporters of extreme ideologies are getting vast, diverse and, most of the time, hidden, thus ensuring that the internet continues being the main medium for terrorist and extremist networks.³¹ In these digital spaces, online users seek validation and search for identity, where like-minded individuals and groups engage with content that promotes violent extremist beliefs, sentiment, propaganda and disinformation.

Social media and communication platforms such as X, YouTube, Facebook, Telegram, Reddit, TikTok, 4chan and Gab are commonly used by extremist and terrorist groups.³² The anonymity affordances of these online platforms and their transcendence of geographical boundaries are conducive for communication and operational coordination. In recent years, terrorists have kept up with platform innovations by amplifying impact and glorifying attacks via instant broadcasting (for example, through videos, infographics or social media livestreams). The first case of terrorist livestreaming occurred during the 2019 attack on Christchurch mosques in New Zealand (the first terrorism conviction in its history), exposing problems in regulating and moderating abhorrent content on social media.

Another development is the likelihood of “lone wolf” attackers, who are either self-radicalised or self-initiated terrorists.³³Such individuals are not radicalised by a terrorist group directly. Ideology and personal grievances may inspire them, but they otherwise work alone, while actively seeking out extremist material online to justify their worldviews. A limited exposure to diverse perspectives can contribute to the risk of self-radicalisation upon encountering or engaging with extremist content.

Case 3: Christchurch Mosque terrorist attack³⁴

Case 3  

On 15 March 2019, 28-year-old Australian terrorist, Brenton Tarrant, a white supremacist, livestreamed his attack on Facebook Live using a headcam. Before the attack, he was an active user of fringe online forums, who posted a 74-page “manifesto” online espousing various neo-fascist aims. The “manifesto” stated that the attack had been planned two years earlier, and that Christchurch had been scoped out three months in advance. He started the livestream during his drive to the first mosque attack, while listening to a song idolising Radovan Karadzic (jailed for genocide against Bosnian Muslims). He opened fire at the Al Noor Mosque and Linwood Islamic Centre using rifles, targeting worshippers during Friday prayers. 51 people were killed and 40 people sustained gunshot injuries. Tarrant was convicted of terrorism and is now serving a life imprisonment sentence without parole. 

Another concerning global trend is the rising number of young people engaging in violent extremist ideologies.³⁵ Due to the prevalence of violent extremist content online, combined with personal vulnerabilities (whether emotional, physical or social), youths are more easily exploited by violent extremists and terrorist groups.³⁶ For example, social isolation can lead teenagers to seek social outlets offering a sense of belonging. In March 2026, a large police operation placed at least 97 Indonesian youths (the youngest aged 10) under police monitoring after they engaged with content glorifying physical and memetic violence, having been inspired by TCC discussions.³⁷ Their electronically stored information (ESI) are found mostly on messaging applications such as Telegram and social media networks like X.³⁸ Here, minimal regulatory constraints produce ideal environments for terrorist networks and extremist groups to propagate their ideologies.

The average time taken for youth radicalisation used to be 18 months in 2005, contracting to 13 months in 2016, and as of 2026, just a few weeks.³⁹ Based on current trends, the age-crime curve, whose peak should be in the late-teens or early-twenties, appears to be shifting leftwards relative to the normal.⁴⁰

Fig. 1: The age-crime curve⁴¹

Case 4: Singapore youth idolising white supremacist ideology⁴²

Case 4 

In 2025, an 18-year-old Singaporean youth, Nick Lee Xing, was influenced by far-right extremism via an online game, where he role-played as a terrorist killing Muslims in a mosque. He idolised the Australian terrorist, Brenton Tarrant (see Case 3) and wanted to attack Muslims at the mosque. Lee was found to have developed a hostile view towards Muslims in early 2023 after encountering Islamophobic and far-right violent extremist content on social media. Based on police investigations, he got a tattoo and T-shirts with custom prints of logos associated with neo-Nazi, far-right and white supremacist groups. On 10 February 2025, an order of detention was issued by the Internal Security Department, under Singapore’s ISA.  

Case 5: Malaysia teen ‘lone cub’ attack⁴³

Case 5 

On 11 January 2016, the police arrested a 16-year-old teenager in Sungai Petani, Kedah, claiming Daesh affiliation, after he tried to kidnap a Chinese sales assistant at a shopping complex, holding her hostage with a knife. He admitted that he was ordered to conduct the attack by Malaysian Daesh followers, and that his actions sought to publicise Daesh’s existence to the Malaysian authorities and public. The teenager was also radicalised to view non-Muslims as “kafir harbi”(those who can be justifiably killed). It was stated that the teenager was brainwashed by email and social media, and was eager to prove that he was capable of such an act.

Whether online users encounter, consume or engage with content directly through a terrorist group, or indirectly via algorithm, the risk of radicalisation remains constant unless appropriate measures are implemented. Causality may be hard to prove, however, because of the complex factors underlying individual actions. Regardless of whether individuals are radicalised through online connections or if those drawn to online forums are already radicalised, such content requires action. After all, it remains a risk factor in pathways facilitating engagement with and potential exploitation of others, therefore creating broader ripple effects within online networks.⁴⁴ Accordingly, even without asserting a direct causal relationship between internet use and radicalisation or violent extremism, this paper maintains that exposure to specific categories of online content warrants policy attention.⁴⁵

5.1 Difficulties in collecting legally defensible online evidence without clear thresholds

Investigating online crimes related to CT and P/CVE requires more than just the ability to look for information. While online content moderators may only need training to identify specific types of harmful content before taking further action, law enforcement agencies must collect and preserve evidence in a manner consistent with its admissibility in court. For each case, guilt must be proven beyond a reasonable doubt and to a specified standard of evidence or proof. Therefore, ESI must be relevant and authentic (and not hearsay) to qualify as admissible evidence.

Before a police officer can begin preparing an investigation paper, there must be sufficient and reasonable grounds of suspicion. Online ESI is “collected from the Internet, which has been properly collected, preserved and for which a foundation has been laid for its admittance and/or consideration into a legal forum”, while digital ESI is “collected from computers or electronic media which has also met all the legal requirements for admittance and/or consideration into a legal forum”.⁴⁶ While digital ESI can be seized, online ESI is collected through rigorous online monitoring and live acquisition.⁴⁷ Online data collection exists only temporarily and could disappear or be deleted unless captured and stored in a certain manner (usually as screenshots of ESI at a particular date, time and internet protocol address, collected from a website, social networking site, instant messaging application or chat box).⁴⁸ Besides documentation alone, the frequency and corroborative weight of ESI from the same user also matters (as evidenced by a series of screenshots).⁴⁹

In applying this approach to suspected terrorism-related offences online, police officers may need to invest significant time compiling and timestamping screenshots to establish clear legal violations, while making sure that there is no tampering or manipulation prior to data collection via hashing.⁵⁰ However, compiling screenshots is no longer a valid online documentation method in and of itself: the determination of protocols, collection methodologies and locations also matter.

Fig. 2: Online ESI collection steps⁵¹

Without clear thresholds, it is challenging to determine the corroborative weight of individual indicators, or combinations of indicators, arising from pre-investigations and during online monitoring. The authorities may face difficulties in determining whether or not the available evidence is sufficient to initiate a formal investigation and appropriate action. Without a clearly defined threshold, a pre-investigation may be subject to inconsistent assessment, risks subjective or biased conclusions and draws public scrutiny to matters of transparency, fairness and procedural consistency. However, delays caused by uncertainties in decision-making should not be overlooked in matters of national security and public safety.

5.2 A primary regulatory and enforcement focus on clear legal violations and enforceable online behaviour

Under the criminalisation approach, a higher standard of evidence is required to bring cases to court.⁵² Since the current framework is designed to respond primarily to legal breaches, institutional attention is often directed towards behaviours that satisfy the legal threshold for enforcement. For example, gradual behavioural shifts, such as constantly liking, commenting upon and sharing harmful posts on social media, may receive less attention than postings of extremist content.

Additionally, many forms of “lawful but harmful” content exist within regulatory grey areas. Preventive intervention mechanisms remain underdeveloped when potential indicators of radicalisation are diffuse, context-dependent and inadequately captured by online monitoring units, despite their cumulative influence on radicalising vulnerable users.

Given the significance of subjective interpretation or vague guidelines, existing regulatory and enforcement systems may also face constraints in differentiating between freedom of expression (which may include engagement in controversial discourse) and genuine indicators of mobilisation towards violent acts. For example, an individual may consume or share potentially harmful content, but does not necessarily demonstrate an intention to commit unlawful conduct at a particular time.

This situation can be managed effectively with clear thresholds and appropriate interventions for individuals at particular risk levels. There is also growing recognition of the need for more risk-based frameworks in CT and P/CVE approaches to complement current legal enforcement mechanisms.

5.3 Limited integration between regulatory and security-led online monitoring

Currently, there is no single, comprehensive source of guidelines or step-by-step process for law enforcement agencies and regulatory bodies to refer to when addressing harmful online content related to CT and P/CVE. Standardised guidelines or step-by-step process flows will enhance integration between the law enforcement agencies and regulatory bodies overseeing online monitoring.

However, there is currently limited integration and cooperation between regulatory oversight mechanisms and security-led online monitoring.⁵³ Regulatory bodies and law enforcement agencies operate in separate ministries and are given separate mandates, possibly resulting in fragmented approaches to addressing online radicalisation risks. Differences in operational priorities and evidentiary thresholds may also hinder information-sharing. Consider how regulatory approaches tend to focus on platform compliance and the responsible use of technology, while law enforcement agencies prioritise threat identification, public safety and national security. For example, regulatory bodies may prioritise making online digital spaces and social media safer for public use, while law enforcement agencies prioritise the prevention of real-life attacks, preferring that online footprints be maintained for ease of online monitoring.

6.1 Algorithmic amplification of grievance content

Social media algorithms rank content based on users’ likes, comments, shares and other interactions, customising recommendations and amplifying content with higher engagement to maintain users through feedback loops.⁵⁴ This can occur on social media and video-sharing platforms (such as Facebook, YouTube, X, Gab and Reddit), where basic interest or random searches can lead to more consumption of related online feeds via “filter bubbles”.⁵⁵ For example, when a potential recruit views violent extremist content online, algorithms can create an “echo chamber” where only contents that reinforce the presentation of extremist narratives, propaganda and ideologies are fed to this user in the future.

In other words, the algorithms have been weaponised to gradually steer vulnerable individuals towards extremist and terrorist ideologies through emotional resonance. This parallels the exploitation of emotional vulnerabilities and personal crises of online users (such as feelings of alienation and or specific grievances due to personal struggles) by terrorist groups to recruit followers. After the 2019 Christchurch terror attack, New Zealand Prime Minister Jacinda Ardern and French President Emmanuel Macron formed The Christchurch Call, with its commitment to “review the operation of algorithms and other processes that may drive users towards and/or amplify terrorist and violent extremist content”.⁵⁶

6.2 Migration from mainstream platforms to encrypted spaces

Mainstream and closed platforms as well as internet forums are often key enablers and gateways for promulgating violent ideologies. Usually, interactions between a terrorist group and potential targets begin on mainstream platforms (for example, WhatsApp, Telegram and Discord). Having fostered personal interactions, kinship and trust, terrorist groups often migrate conversations to encrypted spaces to reduce visibility to law enforcement and impede investigation.⁵⁷

These closed spaces foster a sense of safety, anonymity, active participation and personal/community interaction, where active recruitment and indoctrination usually starts undetected.⁵⁸ Encrypted messaging applications often use end-to-end encryption to deter third-party entities and even developers from accessing users’ chat boxes and chatrooms.⁵⁹ Applying rational choice theory to this context, terrorist and extremist groups prefer this method due to its low risk (risk), ease of technological facilitation (effort) and capacity for building online reputations among subcultural groups by claiming responsibility for certain terrorist actions (reward).⁶⁰

6.3 Humour, satire and memes normalising violence

To avoid violating platform policies, another communication strategy executed by terrorist and extremist groups is coding harmful content through humour, satire and memes, which take advantage of vague definitions of borderline content as well as borderline terrorist and violent extremist content (TVEC) content.⁶¹ TVEC, a term often used by platforms to merge both terrorist and violent extremist content under one umbrella category, is especially useful for reporting purposes. “Saint Hoax”, the popular meme creator, defines a meme as “a piece of media that is repurposed to deliver a cultural, social or political expression, mainly through humour. It can capture insight in a way that is in complete zeitgeist”.⁶² Often taking the form of “image memes”, it spreads quickly via the internet, including online platforms. Just as memes can accelerate the popularity of celebrities, politicians and forms of entertainment, the same dynamic applies to extremist and terrorist content.

Due to their virality and accessibility, memes as well as humorous and satirical content can be weaponised to propagate toxic narratives, symbols and messages by forming a subcultural and memetic language, doxing or promoting groupthink (closely related to extreme ideologues engaged in trolling or producing propaganda, harmful stereotypes, derogatory content or abusive and offensive materials).⁶³ Most importantly, the original intention can be hidden and thus openly shared in mainstream domains.⁶⁴

6.4 Exploitation of gaming platforms and gaming communities

Violent extremist groups increasingly use online gaming chat rooms to identify, recruit and radicalise young people via in-game chat rooms and game-agnostic (not game-specific) social platforms. The use of such media or offline messaging (store-and-forward messaging) to normalise narratives associated with violent extremism target youths in particular.⁶⁵ Immersive gaming technologies (such as virtual reality and games taking place on the metaverse) that incorporate violent behaviour may possibly lead vulnerable youth to normalise violence, although no clear correlation has been established.⁶⁶ Regardless, games amplifying certain political and ideological stances may blur the lines between creativity in virtual spaces and lived social deviance.

Gaming platforms and communities such as Roblox, Gorebox, Discord, Fortnite and Minecraft are being exploited to radicalise young people, including Malaysians. In a recent case, six individuals, including teenagers, were detained, having been believed to have pledged allegiance to the leader of a Daesh-linked extremist organisation.⁶⁷ Online misinformation and disinformation may further amplify violent extremist narratives. Left unchecked, this may erode trust in public institutions and democratic society, manifesting as offline violence and terrorist acts.

6.5 The use of the deep web and dark web by terrorist groups

To circumvent regulatory constraints, terrorist groups use the dark web to plan and coordinate attacks, disseminate propaganda and recruit new followers.⁶⁸ In general, the deep web can be understood as password-protected databases and intranets hidden from the usual surface web. Meanwhile, the dark web includes sites that can only be accessed via specialised browsers, which most users will never interact with on a daily basis.⁶⁹ The anonymity and convenience offered by the dark web as well as various underground markets and forums create an enabling environment for planning attacks. Dark web marketplaces are where cybercrime-as-a-service thrives, in which lone wolf attackers and terrorist groups can easily access auxiliary services (weapon procurement, encryption and secure communication channels) hidden from online monitoring by the authorities.⁷⁰

Terrorist groups utilise the dark web to acquire independent domains (.onion) and publish their URLs on the surface web, deep web, encrypted communication platforms such as Telegram or specialist dark web cybercrime forums.⁷¹ These URLs can be disseminated among followers to share training and instructional materials in the form of online courses, instructional videos, e-books, e-manuals and webinars.⁷² However, law enforcement agencies and content moderators face difficulties extracting information offline for investigation.

6.6 Behavioural escalation markers

Observable shifts in online behaviour that may indicate a gradual progression or escalation of engagement, radicalisation or hostile reconnaissance can be detected through online monitoring.⁷³ Such markers may include discussions on operational security, evasions of law enforcement officials, praise for successful extremist attacks by glorifying attackers or their modus operandi, joining online groups focused on promoting extremism and violence (such as chat groups discussing IED manufacturing), virtual simulations of attacks or the use of unusual and powerful encryption tools in online messaging applications.⁷⁴ The migration of individuals from mainstream digital spaces and social media to encrypted online spaces is another relevant marker.⁷⁵

Recognising these key behavioural and emotional changes in beliefs and attitudes, online behaviours and social patterns after certain types of online consumption will greatly assist in official assessment and intervention. Escalatory behavioural markers may illustrate a transition away from passive content consumption towards active participation within the terrorist and extremist ecosystem.

However, many online behaviours remain ambiguous and may overlap with other purposes. Therefore, they require further context-dependent assessment. For example, law enforcement and regulatory bodies need to know how to assess persons expressing extremist views or stronger political stances online, by determining whether these constitute healthy debate or the adoption of extreme views.

6.7 Distortions of religious narratives

This is not a new challenge in Malaysia, consisting of the manipulation, selective interpretation or misrepresentation of religious teachings to legitimise or justify extremist ideologies and worldviews, while demonstrating a group’s religious credentials.⁷⁶ Ideology exploits religious concepts, symbols and terminologies to frame violence as morally and religiously justified.⁷⁷ Based on reviews and resolutions by United Nations bodies – ranging from the General Assembly and Security Council to its Global Counter-terrorism Coordination Compact entities and the International Criminal Police Organisation – there has been “a rise in terrorism on the basis of xenophobia, racism and other forms of intolerance or in the name of religion or belief”.⁷⁸

Take for example the Al-Qaeda-linked Jemaah Islamiyah, which had its roots in Malaysia and the region. Established in 1993, it was the product of an earlier radical movement, Darul Islam, which ideologically espoused an “Islamic state”.⁷⁹ Operating regionally through loosely linked cells, among Jemaah Islamiyah’s deadliest attacks was the 2002 Bali bombing that killed more than 200 people. As of now, 16 of its leaders have announced the group’s disbandment by video statement.⁸⁰ However, remnants of the network remain active, and post-disbandment does not prevent future attacks under its continued ideological influence.⁸¹

Another concern is the use of artificial intelligence (AI) tools by terrorist and extremist networks, which may leverage charismatic online influencers or religious authorities to increase the persuasive reach of their narratives. Exploitation can include media spawning, automated multilingual translation, fully synthetic propaganda, repurposing old propaganda using generative AI tools and personalised propaganda.⁸² Additionally, geopolitical developments, local grievances and sociopolitical tensions may be leveraged to facilitate the distortion of religious narratives, framing them as evidence of an existential threat to particular religious communities.

Having explored online extremist content as well as the potential role of the internet and digital spaces in facilitating violent extremism and terrorism, the need for a strategic approach to prevent and investigate such activities and criminal conduct is imperative. Several recommendations are described below, which should be complemented by appropriate legal mandates for law enforcement agencies and regulatory bodies to conduct investigations and interventions.

7.1 Operationalising an indicator-based framework for CT and P/CVE online monitoring to improve consistency in identifying risk factors

An indicator-based framework for CT and P/CVE online monitoring will enhance the ability of regulatory bodies and law enforcement agencies to recognise non-obvious patterns indicating gradual exposure to extremist narratives.⁸³ This framework will improve the identification of contextual warning signs by linking shifts in behavioural markers to online engagement, social influence and risk levels of radicalisation. It supports a more holistic assessment approach, in which the combination and identification of multiple indicators enhances pre-crime assessment procedures, rather than relying on isolated incidents or expressions.

This framework also strengthens the ability to detect commonly used explicit, implicit and concealed or coded extremist messaging in digital environments. It facilitates more informed decision-making by investigators and reviewers by grounding assessment in documented indicators. In cases requiring early intervention, this framework supports the identification of vulnerable individuals who are exposed to extremist and terrorist ecosystems.

Risk levels are divided into four tiers (passive, medium, high and critical), which are then matched with relevant online indicators and recommended responses (see also Recommendation 7.2). 

Table 1: Suggested online indicator matrix

7.2 Introducing a tiered risk classification system to guide proportionate responses

Individual experiences of engaging and interacting with harmful content vary in salient ways.⁸⁴ The policy and practice of pre-crime intervention and mitigation should identify and address variations in online risk. A tiered risk classification system enables stakeholders to identify and consider tailored interventions, depending on an individual’s risk level, type of online behaviour, spaces occupied, membership in communities and levels of engagement.⁸⁵ It also helps determine an individual’s level of culpability for online behaviour.

Based on Figs 3–5, the lowest tier, Tier 1 (passive risk) involves engagement with lawful but concerning content; Tier 2 (medium risk) involves extremist ideological content; Tier 3 (high risk) involves terror-linked operational content; and the highest, Tier 4 (critical risk), involves imminent threats. These tiers are operational classifications for risk assessment and intervention purposes and should not be conflated with terms introduced in the definitional framework.

Tier 1 indicates that there is no clear evidence that individuals are susceptible to violent extremism or terrorism, and further observation or waiting for recurring behaviour may be warranted. Tier 2 indicates that individuals may be vulnerable to translating online behaviours into real-world harm, and thus investigators need to continue online monitoring more closely and regularly. Tier 3 indicates that individuals are at high risk of criminal action, where law enforcement agencies should be alerted for intervention. Tier 4 indicates that individuals may commit violence at any opportunity, thus requiring criminal enforcement (for example, intelligence gathering, questioning, search and seizure, immediate arrest, detention, charging and prosecution).

Fig. 3: Recommended tiered risk classification system

It will be helpful to incorporate and merge the colour codes of the RAG+ system (also known as the traffic light tiered assessment). Measurable information is classified by colour: red, amber, green and an additional colour, usually orange. In fields like medicine, education and natural disaster management, this system has reduced uncertainty in assessment and improved decision-making.⁸⁶

To further enhance the classification system, an approach modelled after Channel (a British multi-agency programme designed to provide early intervention for individuals most at risk of being drawn into terrorism) can help structure a holistic approach by incorporating elements of identification and intervention.⁸⁷

Once the higher thresholds (belief in an individual’s general involvement or tendency towards terrorism-related activity) are crossed, authorities should proceed with intervention and/or criminal enforcement. However, if the thresholds are not met, online users are categorised into lower tiers, where law enforcement agencies need to implement forward-looking measures to predict and prevent possible future offences.

Fig. 4: Proposed incorporation of the RAG+ system into a tiered risk classification system

Based on Fig. 5, for example, appropriate pre-crime intervention to counter extremist online postings at Tier 1 include improving public awareness about the risks of extremism, misinformation and disinformation as well as the responsible use of technology. This can be done through technology literacy education, talks on CT and P/CVE and even participation by the authorities in workshops or public programmes explaining the recruitment methods and tactics employed by terrorist and extremist groups. Former extremists and terrorists can also be invited to provide first-hand insights into the dangers and risks of radicalisation.⁸⁸ Those at risk of radicalisation should benefit greatly from the different perspectives offered here.

Fig. 5: Proposed incorporation of the Channel approach into a RAG+ tiered risk classification system

7.3 Reforming legal provisions to clarify thresholds for online prosecution and enforcement

There is a need to reform Section 130JB of the Penal Code, which concerns the possession of items associated with terrorist groups or terrorist acts. This section states that “whoever has possession, custody or control of, or provides, displays, distributes, or sells, any items associated with any terrorist group or the commission of a terrorist act shall be punished with imprisonment for a term not exceeding seven years, or with fine, and shall also be liable to forfeiture of any such item.”

Due to its overly broad and vague definition of possession, there is a risk of unintended consequences, such as selective enforcement targeting political dissenters or activists, violations of freedom of expression and the right to information as well as unjust prosecution. Further, unlike other offences in Chapter VI (which includes terms like “knowing”, “intentionally” and/or “having reason to believe”), Section 130JB lacks a clear provision for a mens rea (intention) element.

Given the strict liability nature of this section, the extent to which courts may consider a person’s intentions or contextual purposes for possessing materials in question is limited. For example, in Siti Noor Aishah Atam v. Public Prosecutor (2018), the Kuala Lumpur High Court found that under Section 130JB(1)(a), she was guilty of having in her possession 12 books allegedly related to terrorist and terror activities. However, she contended that she was using these books for educational purposes.⁸⁹ Similar cases, in which the accused were charged with the possession of terrorism-linked items, include Public Prosecutor v. Muhammad Sani Mahdi Sahar (2016), Public Prosecutor v. Muhammad Hakimin Azman (2017), Public Prosecutor v. Azizi Abdullah (2017) and Mohamad Nasuha bin Abdul Razak v Public Prosecutor (2019).

Therefore, a modification to this section should incorporate a mens rea requirement as a proactive disclosure mechanism. Statutory carve-outs for bona fide activities (acting sincerely without intent to deceive or break the law) will effectively resolve existing ambiguities and align the provision with its original legislative intent.

7.4 Exploring a restorative justice pilot programme as a pre-crime online intervention model

Unlike punitive CT approaches involving criminalisation, there is a need to consider whether or not such measures are necessary for users who encounter or engage with harmful online extremist content, but remain in the early phases of radicalisation. With the introduction of MyP/CVE and rapid proliferation of online extremist material, mechanisms operating outside the formal criminal justice system targeting low-level offenders and non-operational actors are needed. Singapore, Denmark and Indonesia have each implemented rehabilitation and diversion-oriented approaches to address radicalisation risks.

Recognising restorative justice as a pre-crime intervention model for online extremism sends a clear message to the public that the system is not designed to punish, but rather, to prevent further action by individuals exhibiting online curiosity that may indicate early-stage vulnerability to extremist content and narratives, those who may translate online behaviours into violent extremism and plan to commit terrorist offences as well as those suspected of future involvement in terrorism.⁹⁰ The nature of restorative justice mechanisms is determined by what is considered necessary for preventing involvement in terrorism and violent extremism.

Besides the intervention of law enforcement agencies through this diversion approach, appropriate resources to address other issues leading youths to engage with harmful content should be made available to support early intervention. A whole-of-society approach is pertinent for implementing restorative justice mechanisms across a variety of cases involving vulnerable individuals.⁹¹ This means including anyone (for example, parents, teachers, community leaders and trusted peers) deemed important for helping potential offenders return to positive behaviours before they engage further with harmful content.⁹² For example, a young adult who engages with extremist content to distract themselves from dealing with a family breakdown may require a family counsellor’s services, but another one dealing with trauma may need a mental health therapy session from a psychiatrist or psychologist.

7.5 Appointing “trusted flaggers” and developing an accessible reporting platform as a single point of contact

CSOs, experts actively conducting CT- and P/CVE-related initiatives and research as well as advocates for the responsible use of technology should be engaged to bridge knowledge gaps, promote information-sharing and enable the proactive identification of harmful content.⁹³ This step will not only increase the capacity of human moderators and reviewers to detect harmful content, but also improve the transparency of the monitoring system and response times for removal requests.⁹⁴ For example, through Indonesia’s “Trusted Flagger” programme, volunteers (mostly from the CSOs like Wahid Institute, ICT Watch and Anti-Defamation of Society) are given the authority to flag content violating a website’s terms of service or community guidelines.⁹⁵

In addition, an online reporting platform should be made available for CSOs and the public to easily and safely report concerning behaviours or incidents to a single point of contact within the government. Simplification will enhance annual CT and P/CVE risk assessments and illustrate the exact magnitude of challenges faced. “Trusted flaggers” can also use this platform to escalate concerns around any content deemed capable of radicalising or triggering online users to commit violence.⁹⁶

To fully understand the scope of threats, it is recommended that the communications and online regulatory actor further explores avenues to improve and document data collection (for example, establishing working groups with law enforcement agencies and coordinating with the private sector, such as platform regulators, social media companies and CSOs).

7.6 Strengthening cross-functional coordination for timely responses and standardised intervention pathways

Key stakeholders should advance information sharing across agencies, including through stronger partnerships between the regulatory body and law enforcement agencies’ online monitoring and CT units, by focusing on identifying and managing harmful content. Malaysian Communications and Multimedia Commission (MCMC) data indicate that while major platforms removed approximately 92% of the 697,061 posts flagged as harmful between January 2024 and November 2025, although some 58,104 posts were still accessible at the time of reporting due to the use of automated and AI tools which hindered detection efforts.⁹⁷ Posts that are still accessible may not yet meet the threshold for criminal enforcement, but through cross-functional coordination, online users identified engaging with these posts can be referred to standardised intervention pathways.

In the field of CT and P/CVE, law enforcement and regulatory bodies tend to think of transparent communication as counterproductive, potentially reducing public support. However, transparency over decision-making rationales may actually increase policy support and public trust, especially in controversial cases.⁹⁸ Cross-functional coordination enhances transparency and accountability between different agencies, builds public trust by explaining rationales behind efforts and educates the public in making informed decisions in social media and online platform usage.

Strong coordination between a regulatory body and online platforms, in relation to transparency over TVEC, will inform law enforcement agencies’, relevant authorities’ and policymakers’ respective threat assessments. For example, the MCMC can enhance its annual online content supervision report by including the aggregated complaints received (or content flagged) on TVEC and offer updates on the online safety landscape in consideration of CT and P/CVE.⁹⁹ However, significant data collection is needed. Based on Organisation for Economic Co-operation and Development (OECD) data, only 17 of the top 50 most popular online content-sharing services issued transparency reports with information on TVEC in 2024, and of the top 50 services used to spread TVEC online, only six did so.¹⁰⁰

By creating a more cohesive and coordinated approach, stakeholders from both law enforcement agencies and regulatory bodies can collectively enhance their abilities to navigate and respond effectively to the dynamic landscape of internet-enabled violent extremism and terrorism.

8.1 Safeguarding freedom of expression and human rights

CT and P/CVE invite considerable discussion and debate about the balance between safeguarding national security and freedom of expression, between public safety and individual liberties.¹⁰¹ Consequently, intervention frameworks should recognise the heterogeneity of possible ideological expressions involving the same type of online content. Each reported case should be treated contextually, rather than applying a one-size-fits-all rule.¹⁰² Newly introduced policies and frameworks should strike the right balance by avoiding under- or over-reactions that may lead to low public trust and criticism of official efficiency.

Since most investigations entail the collection and processing of personal information, measures taken must account for human rights-compliant implementation. Robust safeguards, consistent with international human rights norms and standards, must be applied, while opportunities for representation as well as access to appeal mechanisms and judicial review should be provided.¹⁰³

To find this balance, stakeholders should understand that simply engaging with violent extremist content online does not necessarily lead to a risk of violence or amount to terrorist activity. Therefore, when addressing issues within the general category of online safety, stakeholders should opt for a harm prevention approach, rather than focusing on countering violent extremism.¹⁰⁴

8.2 Integrating local cultural and contextual sensitivities into risk assessment and profiling practices

Cultural context matters.¹⁰⁵ Even seemingly similar cases of online radicalisation or extremism require different responses in different countries, even those within the same region. This is because “significant institutional differences are likely to develop in each country, because a different set of initial conditions produces a different set of actions and events which have a cumulative effect and set institutional development on a different path”.¹⁰⁶

Due to the sheer volume of posts, comments, videos and images uploaded every second, a manual review by human moderators and reviewers alone is impractical. With the growing scale of digital platforms and user-generated content, a hybrid moderation model – combining multilingual AI models and a diverse moderation team – can combine the speed and scalability of AI tools with the intrinsic skills of human reviewers (for example, having contextual awareness of local cultures, social norms, religious sensitivities, ethical judgement).¹⁰⁷ The empathetic nature of human reviewers can never be underestimated. It is impossible to implement efficient CT and P/CVE strategies by solely utilising AI, even with advancements and emerging technologies. For example, the impact of doxing in relation to the 2024 KK Mart incident requires a contextual understanding of the transition from online to offline transgression, in relation to online posting and online vigilantism.¹⁰⁸

Human reviewers are especially needed to detect extremist content, especially in non-English languages. In Malaysia, this includes content not just in the national language (Bahasa Malaysia) or major languages such as Mandarin and Tamil, but also numerous indigenous languages (numbering over 100) and those of other communities. Additionally, dialects vary significantly across states and communities due to local histories and geographies. Taking all this into account, human capabilities may be limited, but the limitations faced by AI in relation to non-English extremist and terrorist content is greater. Left solely to an automated system, harmful elements will never be captured and continue circulating on social media.¹⁰⁹

8.3 Increasing the competency and capacity of online monitoring units and content moderators

To handle internet crimes, stakeholders in online monitoring units and content moderators must possess foundational CT and P/CVE proficiency, cybersecurity expertise and technical acumen. Specific training is not just advisable but essential to empower them with the knowledge and skills required to conduct terrorism-related investigations, identify relevant online content and consider human rights dimensions.

While not every team member needs to become a CT and P/CVE expert, a foundational knowledge base is indispensable. This ensures that employees across various agencies can communicate effectively and collaborate seamlessly. For example, a government content moderator should possess enough knowledge to engage and exchange meaningful insights with CT personnel. Utilising advanced technological skills in terrorism-related investigations also helps speed-up investigations (therefore requiring the continuous improvement of AI systems). Some factors to consider include the ease of committing acts of terror or other crimes using technology, ongoing difficulties in tracking cybercriminals and suspected offenders online as well as challenges in collecting digital evidence.

This does not mean that law enforcement and regulatory bodies are not progressing. In fact, the RMP’s online monitoring unit has been proactively investigating cases, analysing digital and online evidence as well as compiling evidence related to online radicalisation. On the other hand, the MCMC, as the sole regulator under the recent Online Safety Act, leads investigations, enforcement and other actions against platforms that fail to protect users.¹¹⁰ While the criminal justice system intervenes at the level of legal culpability, content governance prioritises content moderation and regulation.

8.4 Thresholds in online content governance and criminal justice

Having extreme beliefs does not automatically lead one to violence, and the majority of those who support and subscribe to content or act as bystanders online do not commit offline violence.¹¹¹ It is important that regulatory bodies and law enforcement agencies acknowledge that not all harmful or extremist content should be criminalised. Therefore, rather than focusing on criminalisation, this paper aims to improve the regulatory framework by introducing an indicator-based framework and tiered risk classification system.

Most individuals falling in Tiers 1, 2 and 3 should not necessarily be arrested or prosecuted, since law enforcement agencies undertake criminal enforcement actions only at Tier 4, when imminent threats are detected. This can only be done efficiently through the coordination of all stakeholders. For example, government content moderators should report only imminent threats to the RMP once risks are detected and classified as Tier 4.

Further analysis on online CT and P/CVE criminal enforcement is beyond the scope of this paper. However, it is important to determine, for example, how police officers can issue charge sheets to suspected offenders whose online behaviours fall in Tier 4 (that is, initiating criminal proceedings). A causal link to offline harm is required, subject to clear evidentiary standards based on the Evidence Act 1950, without pre-empting outcomes. This involves identifying and clarifying appropriate legal and evidentiary thresholds for determining when online behaviour demonstrates a sufficiently substantiated causality with offline harms.

The internet has become a pervasive and invasive part of everyday life, extending from private life into the public realm. This policy paper has assessed the wide range of behaviours that can be exhibited in online spaces, and considers whether or not a person holding radical or extremist views necessarily translates them into violent offline behaviours and terrorist acts. While social media platforms employ measures such as content moderation, user-reporting tools to enforce community standards as well as automated systems and de-platforming, terrorist groups and extremist networks adapt their strategies to undermine these initiatives.

The emerging picture of radicalisation has become complex and requires preventive measures, but current enforcement approaches mostly remain reactive rather than preventive. Therefore, Malaysia needs a proportionate framework that distinguishes between exposure, vulnerability and actual mobilisation towards violence. The risks posed by online extremist content to society should be tackled through hybrid solutions incorporating online and offline policies, strategies and early intervention, while accounting for proportionality and rights protections to prevent over-criminalisation. Multi-stakeholder collaboration and cross-sectoral initiatives are needed to improve consistency in regulating online content, by bridging government, civil society and industry actors. Finally, content regulation should no longer operate as an opaque process, but instead be grounded in transparent risk indicators, evidence-based and consistent assessment criteria, proportionate interventions calibrated to clearly defined risk tiers and accessible reporting platforms.

Participants included representatives from various ministries and agencies, CSOs and industry, for instance. The views expressed during the workshop should not be interpreted as official positions or endorsements.

Government ministries and agencies

1. Attorney General’s Chambers
2. CyberSecurity Malaysia
3. Institute for Youth Research Malaysia
4. Malaysia Defence Intelligence Organisation
5. Malaysian Islamic Development Department
6. Malaysia Prisons Department
7. Defence Ministry
8. Digital Ministry
9. Education Ministry
10. National AI Office
11. National Security Council
12. Prime Minister’s Department (Research Division)
13. RMP Headquarters (Crime Prevention and Community Safety Department)
14. RMP Headquarters (Special Branch)
15. Southeast Asia Regional Centre for Counter Terrorism
16. Social Welfare Department

Civil society, academic and professional bodies

1. IMAN Research
2. Initiative to Promote Tolerance and Prevent Violence (INITIATE.MY)
3. ISIS Malaysia
4. Khazanah Research Institute
5. Malaysian Bar (cyber and privacy committee)
6. Malaysian Youth Diplomacy (MyDiplomacy)
7. Universiti Kebangsaan Malaysia (National University of Malaysia)
8. Universiti Malaya

Industry / digital platforms

1. ByteDance
2. Meta Platforms, Inc.

Key discussions

1. Types of harmful and extremist content available online that may influence radicalisation and promote violent extremism;
2. Potential risks and harms posed by such content to vulnerable populations, particularly children and youth, through radicalisation, recruitment and engagement in extremist behaviours;
3. Evolution in the radicalisation process and factors contributing to the escalation of online extremism into real-world violence;
4. Structured early-warning indicators to identify exposure to harmful and radicalising content and support timely preventive interventions; and
5. Feasibility and desirability of an assessment framework to detect content.

ANIS RIFHAN ROSLI is a researcher with the Special Projects team at the Institute of Strategic & International Studies (ISIS) Malaysia. She is also a member of the Malaysian chapter of the Council for Security Cooperation in the Asia-Pacific (CSCAP). Her research interests include counterterrorism, preventing and countering violent extremism, refugee management in Malaysia and the United States’ foreign policy.

DR HAEZREENA BEGUM ABDUL HAMID is a criminologist and senior lecturer at the Faculty of Law, Universiti Malaya, with 24 years of legal practice. Her research focuses on human trafficking, terrorism and violent extremism, prison, incarceration, gender, human rights, sexual violence, online harms and digital victimisation. Dr Haezreena was appointed as an Expert by the International Criminal Court in counter-terrorism and human trafficking and leads the national research team reviewing AI-related reforms to the Penal Code, Criminal Procedure Code and Evidence Act.

ASHIKIN HUSSIN, or Eureka from EurekartStudio, is a Malaysian artist and illustrator. Her work is inspired by nature, colours, celestial elements and everyday life, blending whimsical and modern aesthetics to create something joyful and meaningful.

We are grateful for the contributions of officials from the Royal Malaysia Police Headquarters Special Branch (E8), the Indonesia National Police and Badan Nasional Penanggulangan Terorisme, United Nations Office on Drugs and Crime Indonesia, Wahid Institute as well as numerous other organisations and individuals whose insights contributed to this research. We also extend our deepest appreciation to the participants of the expert workshops, whose contributions helped us reflect upon the key research questions.