Platform duty-of-care is no longer optional

Grok, the AI chatbot developed by xAI and embedded into the X platform, has again drawn scrutiny following the rollout of new image-editing features. In early January, users discovered that Grok could be prompted to generate or modify images in ways that produced sexualised depictions of people, and more seriously, minors.

In Malaysia, local media reported instances of Grok being used to “digitally undress” individuals online, including women wearing the hijab. These incidents raised concerns pertaining to consent, dignity, and the gendered nature of AI-enabled harm, particularly where such misuse intersects with cultural and religious sensitivities. Malaysia has since joined several countries in publicly criticising X over the circulation of offensive Grok-generated images.

In response, the Malaysian Communications and Multimedia Commission (MCMC) temporarily restricted access to Grok within Malaysia through a DNS-level block, following formal notices to the platform. However, the effectiveness of this measure was quickly undermined. In a public response on X, Grok’s own account acknowledged that the block was “pretty lightweight” and easily bypassed, highlighting the limits of access-based enforcement where the underlying system remains unchanged.

Although xAI and X announced restrictions on Grok’s image generation and editing features, including limiting them to paying users or applying geo-blocking, investigative reporting has found that sexually explicit and non-consensual outputs, including nudification, remain accessible and continue to circulate online.

Nonetheless, by late-January, Malaysia lifted the temporary restriction after the platform provided assurances that additional safeguards had been implemented, while emphasising that compliance would be monitored and further violations addressed firmly. This sequence of restriction, circumvention, and conditional reinstatement illustrates both the reach of regulatory intervention and its practical limits in the absence of effective system-level controls.

The incident exposes broader concerns about AI-enabled platform risk. What makes this case consequential is not the existence of image-manipulation technology alone, which has long circulated at the margins of the internet, but its integration into a mainstream, high-reach platform that enables both generation and immediate dissemination. When such capabilities are wedded, harmful and potentially unlawful content can scale before external intervention takes effect.

Viewed this way, Grok matters less as a singular case than as a warning of what can happen when generative capabilities are coupled with mainstream communications platforms.

This also upends traditional content governance frameworks and introduces a deeper structural question: who is responsible for moderating what the platform itself creates? When a platform plays the role of both creator and distributor, the boundary between user conduct and system architecture collapses. This creates an inherent conflict of interest, as the same entity that drives engagement and product expansion is also now responsible for defining the limits of acceptable output.

Separately, the incident also underscores the need to rethink how harm is understood, particularly in cases of sexualised and non-consensual imagery. It is worth noting that the presence of harm is not dependent on deception. Even where viewers recognise that an image has been digitally altered or fully generated, non-consensual sexualised imagery can remain humiliating and damaging. The violation lies in the act of sexual manipulation without consent, not in whether the image is believed to be authentic.

Relatedly, the gravity of harm – and wider platform responsibility – should also not turn on how widespread the manipulated content becomes. Rather, it centres on the very fact that the system enabled it at all.

This is the type of risk environment Malaysia’s Online Safety Act (ONSA) is capable of addressing. ONSA shifts regulatory focus away from individual user behaviour toward platform responsibility, particularly where harms are foreseeable and structurally enabled.

Importantly, ONSA’s regulatory logic does not hinge on intent, novelty or scale. Nor does it operate as a formal pre-approval regime for new features. Instead, its duty-of-care framework places ongoing obligations on licensed platforms to act proactively, including implementing proportionate, risk-based safety measures and mitigating priority harmful content without waiting for user reports or enforcement action. Regulators are then empowered to assess whether platforms have taken reasonable steps to anticipate and manage foreseeable risks through their system design and operational safeguards.

In this respect, ONSA marks a departure from online safety strategies centred on takedowns and post-hoc enforcement, which merely plasters over the harm and does not necessarily address underlying risk structures.

This shift aligns with a broader global movement toward preventive, risk-based online safety regulation. In South Korea, regulatory reforms introduced in response to digital sexual crimes have placed clear preventive obligations on platforms. A similar logic underpins the United Kingdom’s duty-of-care model, which requires platforms to identify and mitigate foreseeable harms through system design and risk management, particularly where children are concerned.

ONSA reflects this same regulatory principle, adapted to the Malaysia context. It recognises that global platforms shape local online environments, often according to safety assumptions developed elsewhere, and that domestic regulation must thus focus on systemic risk rather than isolated content decisions.

It must, however, be noted that at the time of writing, X has not been formally designated as a licensed social media service provider under ONSA, as the current designation threshold applies to platforms with at least eight million Malaysian users. The Ministry of Communications has since indicated that this threshold is under review, following the Grok controversy, with the Cabinet examining whether adjustments are necessary.

As such, the full statutory duty-of-care obligations under the Act may not yet be directly enforceable against the platform. This transitional moment illustrates the practical stakes of designation: where high-reach platforms fall outside formal coverage, gaps in preventive oversight become more visible. The fact that a platform operating at scale in Malaysia may not yet be formally captured reinforces the importance of timely and consistent application of the Act’s framework.

Against this backdrop, the issue becomes one of implementation rather than legislative intent. While ONSA establishes a preventive duty-of-care framework suited to this type of harm, generative AI heightens the importance of ensuring that those obligations are applied with precision in practice.

If platform-integrated AI is to be governed effectively, clearer expectations are needed regarding the manner in which safety standards are applied to feature developments. As such, what follows are some of the ways implementation could be strengthened.

First, platform risk assessments must explicitly account for AI-enabled content generation, not only user-generated content. Platforms should be required to demonstrate how generative features have been tested against abuse patterns that can be reasonably foreseen prior to deployment.

Second, safeguards must be assessed for effectiveness, not just existence. Partial restrictions, inconsistent refusals, or access-based controls are inadequate where severe harms are foreseeable.

Third, regulatory engagement must occur earlier in the deployment lifecycle. Where platforms introduce new generative capabilities with clear pathways to harm, regulators should have clearer mechanisms within existing regulatory powers to intervene before harm materialises.

Lastly, transparency and auditability must extend to generative AI safeguards. Platforms deploying generative systems should be required to document and disclose how safety mechanisms operate in practice, including refusal rates, known failure modes, and changes made in response to identified risks. Sans visibility into how safeguards perform under real-world conditions, regulators are left responding to the harm after the fact rather than verifying prevention upfront.

The lesson from Grok is not that AI systems are ungovernable, nor that platforms should retreat from innovation. Rather, it is that existing online safety frameworks must be applied rigorously to AI-mediated environments.

As AI capabilities continue to evolve, the regulatory challenge is not whether to govern, but how to operationalise prevention in a way that is proportionate, effective, and grounded in accountability. ONSA provides a framework that is capable of meeting this challenge, particularly through its emphasis on platform duties of care and preventive risk management.

In the age of generative AI, regulation will only be effective if it reaches the point of system design. It requires platforms to treat safety not as an afterthought, but as a foundational design obligation.

Disclaimer: The views and opinions expressed in this op-ed are those of the author(s) and do not necessarily reflect the views of the Centre for Responsible Technology (CERT), the Institute of Strategic & International Studies (ISIS) Malaysia, or the Malaysian Communications and Multimedia Commission (MCMC).