Output of an Artificial Intelligence system from Google Vision, performing Facial Recognition on a ... [+] photograph of a man in San Ramon, California, November 22, 2019. (Photo by Smith Collection/Gado/Getty Images)
The news that Ukraine is using facial recognition software to uncover Russian assailants and identify Ukrainians killed in the ongoing war is noteworthy largely because its one of few documented uses of artificial intelligence in the conflict. A Georgetown University think tank is trying to figure out why while advising U.S. policymakers of the risks of AI.
The CEO of the controversial American facial recognition company Clearview AI told Reuters that Ukraines defense ministry began using its imaging software Saturday after Clearview offered it for free. The reportedly powerful recognition tool relies on artificial intelligence algorithms and a massive quantity of image training data scraped from social media and the internet.
But aside from Russian influence campaigns with their much-discussed deep fakes and misinformation-spreading bots, the lack of known tactical use (at least publicly) of AI by the Russian military has surprised many observers. Andrew Lohn isnt one of them.
Lohn, a senior fellow with Georgetown Universitys Center for Security and Emerging Technology, works on its Cyber-AI Project, which is seeking to draw policymakers attention to the growing body of academic research showing that AI and machine-learning (ML) algorithms can be attacked in a variety of basic, readily exploitable ways.
We have perhaps the most aggressive cyber actor in the world in Russia who has twice turned off the power to Ukraine and used cyber-attacks in Georgia more than a decade ago. Most of us expected the digital domain to play a much larger role. Its been small so far, Lohn says.
We have a whole bunch of hypotheses [for limited AI use] but we dont have answers. Our program is trying to collect all the information we can from this encounter to figure out which are most likely.
They range from the potential effectiveness of Ukrainian cyber and counter-information operations, to an unexpected shortfall in Russian preparedness for digital warfare in Ukraine, to Russias need to preserve or simplify the digital operating environment for its own tactical reasons.
All probably play some role, Lohn believes, but just as crucial may be a dawning recognition of the limits and vulnerability of AI/ML. The willingness to deploy AI tools in combat is a confidence game.
Junk In, Junk Out
Artificial intelligence and machine learning require vast amounts of data, both for training and to interpret for alerts, insights or action. Even when AI/ML have access to an unimpeded base of data, they are only as good as the information and assumptions which underlie them. If for no other reason than natural variability, both can be significantly flawed. Whether AI/ML systems work as advertised is a huge question, Lohn acknowledges.
The tech community refers to unanticipated information as Out of Distribution data. AI/ML may perform at what is deemed to be an acceptable level in a laboratory or in otherwise controlled conditions, Lohn explains. Then when you throw it into the real world, some of what it experiences is different in some way. You dont know how well it will perform in those circumstances.
In circumstances where life, death and military objectives are at stake, having confidence in the performance of artificial intelligence in the face of disrupted, deceptive, often random data is a tough ask.
Lohn recently wrote a paper assessing the performance of AI/ML when such systems scoop in out of distribution data. While their performance doesnt fall off quite as quickly as he anticipated, he says that if they operate in an environment where theres a lot of conflicting data, theyre garbage.
He also points out that the accuracy rate of AI/ML is impressively high but compared to low expectations. For example, image classifiers can work at 94%, 98% or 99.9% accuracy. The numbers are striking until one considers that safety-critical systems like cars/airplanes/healthcare devices/weapons are typically certified out to 5 or 6 decimal points (99.999999%) accuracy.
Lohn says AI/ML systems may still be better than humans at some tasks but the AI/ML community has yet to figure out what accuracy standards to put in place for system components. Testing for AI systems is very challenging, he adds.
For a start, the artificial intelligence development community lacks a test culture similar to what has become so familiar for military aerospace, land, maritime, space or weapons systems; a kind of test-safety regime that holistically assesses the systems-of-systems that make up the above.
The absence of such a back end combined with specific conditions in Ukraine may go some distance to explain the limited application of AI/ML on the battlefield. Alongside it lies the very real vulnerability of AI/ML to the compromised information and active manipulation that adversaries already to seek to feed and to twist it.
Bad Data, Spoofed Data & Classical Hacks
Attacking AI/ML systems isnt hard. It doesnt even require access to their software or databases. Age-old deceptions like camouflage, subtle visual environment changes or randomized data can be enough to throw off artificial intelligence.
As a recent article in the Armed Forces Communications and Electronics Associations (AFCEA) magazine noted, researchers from Chinese e-commerce giant Tencent managed to get a Tesla sedans autopilot (self-driving) feature to switch lanes into oncoming traffic simply by using inconspicuous stickers on the roadway. McAfee Security researchers used similarly discreet stickers on speed limit signs to get a Tesla to speed up to 85 miles per hour in a 35 mile-an-hour zone.
An Israeli soldier is seen during a military exercise in the Israeli Arab village of Abu Gosh on ... [+] October 20, 2013 in Abu Gosh, Israel. (Photo by Lior Mizrahi/Getty Images)
Such deceptions have probably already been examined and used by militaries and other threat actors Lohn says but the AI/ML community is reluctant to openly discuss exploits that can warp its technology. The quirk of digital AI/ML systems is that their ability to sift quickly through vast data sets - from images to electromagnetic signals - is a feature that can be used against them.
Its like coming up with an optical illusion that tricks a human except with a machine you get to try it a million times within a second and then determine whats the best way to effect this optical trick, Lohn says.
The fact that AI/ML systems tend to be optimized to zero in on certain data to bolster their accuracy may also be problematic.
Were finding that [AI/ML] systems may be performing so well because theyre looking for features that are not resilient, Lohn explains. Humans have learned to not pay attention to things that arent reliable. Machines see something in the corner that gives them high accuracy, something humans miss or have chosen not to see. But its easy to trick.
The ability to spoof AI/ML from outside joins with the ability to attack its deployment pipeline. The supply chain databases on which AI/ML rely are often open public databases of images or software information libraries like GitHub.
Anyone can contribute to these big public databases in many instances, Lohn says. So there are avenues [to mislead AI] without even having to infiltrate.
The National Security Agency has recognized the potential of such data poisoning. In January, Neal Ziring, director of NSAs Cybersecurity Directorate, explained during a Billington CyberSecurity webinar that research into detecting data poisoning or other cyber attacks is not mature. Some attacks work by simply seeding specially crafted images into AI/ML training sets, which have been harvested from social media or other platforms.
According to Ziring, a doctored image can be indistinguishable to human eyes from a genuine image. Poisoned images typically contain data that can train the AI/ML to misidentify whole categories of items.
The mathematics of these systems, depending on what type of model youre using, can be very susceptible to shifts in the way recognition or classification is done, based on even a small number of training items, he explained.
Stanford cryptography professor Dan Boneh told AFCEA that one technique for crafting poisoned images is known as the fast gradient sign method (FGSM). The method identifies key data points in training images, leading an attacker to make targeted pixel-level changes called perturbations in an image. The modifications turn the image into an adversarial example, providing data inputs that make the AI/ML misidentify it by fooling the model being used. A single corrupt image in a training set can be enough to poison an algorithm, causing misidentification of thousands of images.
FGSM attacks are white box attacks, where the attacker has access to the source code of the AI/ML. They can be conducted on open-source AI/ML for which there are several publicly accessible repositories.
You typically want to try the AI a bunch of times and tweak your inputs so they yield the maximum wrong answer, Lohn says. Its easier to do if you have the AI itself and can [query] it. Thats a white box attack.
If you dont have that, you can design your own AI that does the same [task] and you can query that a million times. Youll still be pretty effective at [inducing] the wrong answers. Thats a black box attack. Its surprisingly effective.
Black box attacks where the attacker only has access to the AI/ML inputs, training data and outputs make it harder to generate a desired wrong answer. But theyre effective at producing random misinterpretation, creating chaos Lohn explains.
DARPA has taken up the problem of increasingly complex attacks on AI/ML that dont require inside access/knowledge of the systems being threatened. It recently launched a program called Guaranteeing AI Robustness against Deception (GARD), aimed at the development of theoretical foundations for defensible ML and the creation and testing of defensible systems.
More classical exploits wherein attackers seek to penetrate and manipulate the software and networks that AI/ML run on remain a concern. The tech firms and defense contractors crafting artificial intelligence systems for the military have themselves been targets of active hacking and espionage for years. While Lohn says there has been less reporting of algorithm and software manipulation, that would be potentially be doable as well.
It may be harder for an adversary to get in and change things without being noticed if the defender is careful but its still possible.
Since 2018, the Army Research Laboratory (ARL) along with research partners in the Internet of Battlefield Things Collaborative Research Alliance, looked at methods to harden the Armys machine learning algorithms and make them less susceptible to adversarial machine learning techniques. The collaborative developed a tool it calls Attribution-Based Confidence Metric for Deep Neural Networks in 2019 to provide a sort of quality assurance for applied AI/ML.
Despite the work, ARL scientist Brian Jalaian told its public affairs office that, While we had some success, we did not have an approach to detect the strongest state-of-the-art attacks such as [adversarial] patches that add noise to imagery, such that they lead to incorrect predictions.
If the U.S. AI/ML community is facing such problems, the Russians probably are too. Andrew Lohn acknowledges that there are few standards for AI/ML development, testing and performance, certainly nothing like the Cybersecurity Maturity Model Certification (CMMC) that DoD and others adopted nearly a decade ago.
Lohn and CSET are trying to communicate these issues to U.S. policymakers not to dissuade the deployment of AI/ML systems, Lohn stresses, but to make them aware of the limitations and operational risks (including ethical considerations) of employing artificial intelligence.
Thus far he says, policymakers are difficult to paint with a broad brush. Some of those Ive talked with are gung-ho, others are very reticent. I think theyre beginning to become more aware of the risks and concerns.
He also points out that the progress weve made in AI/ML over the last couple of decades may be slowing. In another recent paper he concluded that advances in the formulation of new algorithms have been overshadowed by advances in computational power which has been the driving force in AI/ML development.
Weve figured out how to string together more computers to do a [computational] run. For a variety of reasons, it looks like were basically at the edge of our ability to do that. We may already be experiencing a breakdown in progress.
Policymakers looking at Ukraine and at the world before Russias invasion were already asking about the reliability of AI/ML for defense applications, trying to gauge the level of confidence they should place in it. Lohn says hes basically been telling them the following;
Self driving cars can do some things that are pretty impressive. They also have giant limitations. A battlefield is different. If youre in a permissive environment with an application similar to existing commercial applications that have proven successful, then youre probably going to have good odds. If youre in a non-permissive environment, youre accepting a lot of risk.
The rest is here:
The Vulnerability of AI Systems May Explain Why Russia Isn't Using Them Extensively in Ukraine - Forbes