Everyone is talking about AI these days. Some are in favor of the AI revolution, and others are actively working against it with data poisoning.
Lately, several headlines about artists utilizing data poisoning have been circulating on social media.
But, what exactly is data poisoning? And how can we cope with the threat it poses?
Let’s take a look!
What Is Data Poisoning?
Essentially, data poisoning is a way of harming the functionality of AI models by manipulating the data they are trained on. It is a targeted attack which essentially poisons the digital well from which AI models get their data.
According to Defence. ai:
The training phase of a machine learning model is particularly vulnerable to this type of attack because most algorithms are designed to fit their parameters as closely as possible to the training data. An attacker with sufficient knowledge of the dataset and model architecture can introduce ‘poisoned’ data points into the training set, affecting the model’s parameter tuning.
Once an attacker can exploit an AI algorithm in this manner, it will function in accordance with the attackers objectives.
In this post, we will talk about how artists online are using data poisoning to protect the work, but this can include far more sinister outcomes.
For example: The model might not work at all, give incorrect predictions and classifications, leak data, or reveal sensitive information.
It is difficult to detect and can cause cascading issues down the line, especially if downstream applications are created.
The Types of Data Poisoning Attacks
So far, there are several main types of common data poisoning attacks. We will take a quick look at three of them, so you have some context to understand the situation!
Label Flipping
To enact this type of data poisoning attack, an attacker reverses the labels for selected entries within a training set of data. Mislabeling confuses the algorithm and affects the decision boundaries that it creates.
To keep it simple, when the groups of data that the model uses are mislabeled, it leads to errors.
Outliers Injection
In this type of attack, the attacker introduces data points that are very different from the existing data. However, it labels them in a way that distorts the model’s understanding of feature space.
This essentially means that the model is working with misinformation and has warped decision boundaries. This leads to the production of errors.
Feature Manipulation
This type of data poisoning attack involves changing the data points’ characteristics in a training set.
This can range from adding noise to numerical features to adding subtle artifacts in image data. What exactly does this mean?
Well, when it comes to AI models used in image recognition, if the data they are trained on is distorted in this way, they will learn incorrect representations.
This type is especially tricky. While it may not appear in training, it will degrade the model’s ability to recognize new data.
Nightshade: A Practical Application of Data Poisoning
One popular application of AI models is when it comes to text-to-image generators. These models are generally trained by utilizing large datasets that include millions or billions of images, which companies scrape from the web.
Some of these generators, such as those offered by Adobe or Getty, are trained using legally obtained data. This means that the companies either own these images or they have a license to use them in this capacity.
However, in many cases, companies simply scrape the entire web for images. This includes millions or billions of random images, including many that are copyright-protected.
The practice of indiscriminately scraping the web for images and using them to train AI models has led to a large number of copyright infringement cases. In these cases, artists accuse AI companies of stealing and profiting from their work.
How Does Nightshade Work?
Many feel that these companies are stealing from a wide variety of creatives to train their algorithms. This feeling is so widespread that researchers have developed a tool called Nightshade to protect artists’ work from data scraping.
The tool works by subtly altering an image’s pixels in a way that wreaks havoc to computer vision but leaves the image unaltered to a human’s eyes. This could be considered feature manipulation.
So, say an artist applies Nightshade to one of their images and a company scrapes it to train their AI model. What happens?
Once they scrape this image, the data pool its a part of becomes poisoned. This means that the algorithm could mistakenly learn to classify an image incorrectly. This results as image generators becoming unpredictable.
For example, in an extreme case, you ask an AI image generator to produce a golden retriever. It produces something purple that kind of resembles a dog.
This is because the data it uses has been corrupted.
The Goals of Nightshade
Nightshade’s developer hopes that this tool will force big tech companies to respect copyright. At present, the best way for companies to avoid issues is to better vet the images they use.
After all, the fact that so many AI models are trained off indiscriminately scraped data is a problem. There are likely many downsides to this.
It is a common belief amongst computer scientists that all data online can be used for any purpose. However, if you’ve been online for any length of time you know that a lot of the content is dubious at best.
While Nightshade has good intentions, it could be used for harm. Data poisoning can have a wide array of harmful consequences
The Harm That Data Poisoning Can Do
Data poisoning is not a theoretical question, but an immediate threat. Recent research explores several instances of it, and shows that with the right know-how, attackers can poison data sets for as little as $60.
Here are some of the real examples of the harm that data poisoning can do. When it comes to self-driving cars, poisoned data sets can lead the car to misinterpret road signs. This can lead to accidents and crashes.
In healthcare, machine learning models are starting to be used in many ways, including diagnostic imaging. If an AI model used for diagnostic imaging is trained on poisoned data, it may fail to identify malignant tumors or other life-threatening issues.
When it comes to finance and banking, many issues could arise. If security algorithms that banks use have been trained on poisoned data, they may flag legitimate transactions and let ones from hackers pass without warning.
Trading algorithms with undetected poisoned data issues could cause false triggers for buy or sell orders. This is market manipulation, which is illegal, and would increase overall financial instability.
And, since this is a crime, regulatory action could follow. This is not only expensive for companies, but could damage their reputations severely.
So, as you can see, data poisoning poses significant risks across many industries.
How Can We Combat Data Poisoning?
In the case of Nightshade, it is easy for companies to avoid the harm it causes. If they follow copyright law instead of indiscriminately scraping the internet for images, they will not poison their data sets in this way.
However, as we have seen, Nightshade is far from the only threat out there. And people can even likely intentionally use that tool to cause harm.
So, what can we do?
Outside of human beings being more selective in how they compile the data used to train AI models, there are several tech-based solutions.
One is called ensemble modeling. This means that AI models are trained using many different subsets of data. All of these subsets are compared, so intentionally harmful outliers become more apparent.
This is useful in training and also can be helpful when it comes to detecting and removing poisoned images.
There is also real-time monitoring, which combines AI and machine learning technology with more traditional cybersecurity strategies.
Real-time monitoring involves continuously tracking key performance indicators (KPIs) of the machine learning model to detect any unusual patterns or deviations.
Final Thoughts on Data Poisoning
As you can see, this type of attack has a lot of implications for many different industries. While we are evolving countermeasures for protection, it can still cause serious issues.
It also raises questions. For example, are AI development companies stealing the work of others and profiting off of them? A lot of this will be debated ethically and in numerous legal battles over the coming years.
Also, since AI can lead to high-stakes errors, why are so many companies so reluctant to make sure they are getting quality data?
In general, however, while Nightshade attempts to use data poisoning to protect artists, data poisoning is generally not done with good intentions. And it can have horrible results.
What do you think? Comment below.
Since 2009, we have helped create 350+ next-generation apps for startups, Fortune 500s, growing businesses, and non-profits from around the globe. Think Partner, Not Agency.
Find us on social at #MakeItApp’n®
