Sam Washko, Protect AI, Software Engineer

In security trust no one, especially not unvetted machine learning models! Machine learning is increasingly being democratized through the sharing of foundation models on hubs like Hugging Face. However, due to the open nature of model hubs, compromised artifacts are very easy to share and distribute, so supply chain attacks on ML systems are becoming a serious attack vector.

Most ML model formats are inherently vulnerable to Model Serialization Attacks (MSA), the injection of malicious code that will execute automatically when the model file is deserialized. MSAs are the Trojan horses of ML, capable of turning a seemingly innocuous model into a backdoor to your whole system. An attacker could easily download a popular model, inject malicious code, and upload it under a similar name to trick consumers. This problem is not purely theoretical: 3,354 public models on Hugging Face today are capable of arbitrary code execution upon serialization, 41% of which are not flagged as unsafe by Hugging Face.

So what can we do to protect against it? Use ModelScan, the open source tool I’ve been developing for the past year along with a few other talented researchers and engineers. Model scanning is our window into the black boxes that are model files. By scanning the model before deserialization, we can examine the operators and architecture it uses to determine whether it contains suspicious code, without actually unpacking it and becoming vulnerable to the attack. It can detect signs of MSA in various different model formats and categorizes the potential severity of the attack.

Often we think of cybersecurity as more of a concern targeting big companies or governments, but it was important to us to make this tool open source since MSA is a threat to everyone who uses community model hubs - from academics to small businesses to individuals learning and making personal projects. It’s clear that AI/ML is key to the future of technology, and as it becomes more accessible to everyone, the risks do as well. But with tools like ModelScan, we can stop the MSA Trojan Horses at the gates and make ML more secure for everyone.

In this talk, attendees will learn: how MSA works, why they may be at risk, and what ModelScan looks for in suspicious models, as well as lessons learned writing an open source security tool.