Malicious machine learning models found on Hugging Face that ran code when loaded

Researchers at security firm JFrog found roughly 100 machine learning models on Hugging Face that carried out hidden actions when downloaded and loaded, including one that opened a reverse shell to a remote server. Ars Technica reported the findings on 1 March 2024, a day after JFrog released them.

What happened

The flagged models abused Python pickle serialization, a format long known to allow code execution during loading. The reverse-shell model used the pickle reduce method to run arbitrary code as the file loaded. None of the flagged models had been caught by Hugging Face. In a lab test the model did open the shell but took no further action.

Impact

Most of the flagged models looked like harmless proofs of concept uploaded by researchers or curious users. JFrog told Ars Technica that 10 were truly malicious, meaning they compromised the security of whoever loaded them. One opened a reverse shell that handed a remote machine full control of the victim device. How many people downloaded the malicious models has not been reported.

Response

Hugging Face removed the reverse-shell model and the others JFrog flagged. The company has encouraged users to move to the safer safetensors format and runs a scanner that marks risky files as unsafe. JFrog noted that not all 10 malicious models had been marked unsafe.

Lessons for security leaders

Loading a downloaded model can be the same as running an untrusted executable. Keep an internal registry of approved models, prefer weight formats that cannot carry code, and load third-party models only in isolated environments without access to credentials. Add model repositories to software supply chain policy alongside package managers, and log which teams pull models from public hubs.