With many of us stuck at home this past year, weve seen a surge in the popularity of video games. That trend hasnt been limited to humans. DeepMind and Google AI both released results from their Atari playing AIs, which have taught themselves to play over fifty Atari games from scratch, with no provided rules or guidelines. The unique thing about these new results is how general the AI agent is. While previous efforts have achieved human performance on the games they were trained to play, DeepMinds new AI Agent, MuZero could teach itself to beat humans at Atari games it had never encountered in under a day. If this reminds you of AlphaZero which taught itself to play Go then Chess well enough to outperform world champions, thats because it demonstrates an advance in the same suite of algorithms, a class of machine learning called Reinforcement Learning (RL).
While traditional machine learning parses out its model of the world (typically a small world pertaining only to the problem its designed to solve) from swathes of data, RL is real-time observation based. This means RL learns its model primarily through trial and error interactions with its environment, not by pulling out correlations from data representing a historical snapshot of it. In the RL framework, each interaction with the environment is an opportunity to build towards an overarching goal, referred to as a reward. An RL agent is trained to make a sequence of decisions on how to interact with its environment that will ultimately maximize its reward (i.e. help it win the game).
This unique iterative learning paradigm allows the AI model to change and adapt to its environment, making RL an attractive solution for open-ended, real-world problem-solving. It also makes it a leading candidate for artificial general intelligence (AGI) and has some researchers concerned about the rise of truly autonomous AI that does not align with human values. Nick Bostrom first posed what is now the canonical example of this risk among AI Safety researchers a paperclip robot with one goal: optimize the production efficiency of paperclips. With no other specifications, the agent quickly drifts from optimizing its own paperclip factory to commandeering food production supply chains for the paperclip making cause. It proceeds to place paperclips above all other human needs until all thats left of the world is a barren wasteland covered end to end with unused paper clips. The takeaway? Extremely literal problem solving combined with inaccurate problem definition can lead to bad outcomes.
This rogue AGI (albeit in more high-stakes incarnations like weapons management) is the type of harm usually thought of when trying to make RL safe in the context of society. However, between an autonomous agent teaching itself games in the virtual world and an intelligent but misguided AI putting humanity in existential risk lay a multitude of sociotechnical concerns. As RL is being rolled out in domains ranging from social media to medicine and education, its time we seriously think about these near-term risks.
How the paperclip problem will play out in the near term is likely to be rather subtle. For example, medical treatment protocols are currently popular candidates for RL modeling; they involve a series of decisions (which treatment options to try) with uncertain outcomes (different options work better for different people) that all connect to the eventual outcome (patient health). One such study tried to identify the best treatment decisions to avoid sepsis in ICU patients based off of multitudes of data, including medical histories, clinical charts and doctors notes. Their first iteration was an astounding success. With very high accuracy, it identified treatment paths that resulted in patient death. However, upon further examination and consultation with clinicians it turned out that though the agent had been allowed to learn from a plethora of potentially relevant treatment considerations, it had latched onto only one main indicator for death whether or not a chaplain was called. The goal of the system was to flag treatment paths that led to deaths, and in a very literal sense thats what it did. Clinicians only called a chaplain when a patient presented as close to death.
Youll notice that in this example, the incredibly literal yet unhelpful solution the RL agent was taking was discovered by the researchers. This is no accident. The field of modern medicine is built around the reality that connections between treatment and outcomes typically have no known causal explanations. Aspirin, for example, was used as an anti-inflammatory for over seventy years before we had any insight into why it worked. This lack of causal understanding is sometimes referred to as intellectual debt; if we cant describe why something works, we may not be able to predict when or how it will fail. Medicine has grown around this fundamental uncertainty. Through strict codes of ethics, industry standards, and regulatory infrastructure (i.e. clinical trials), the field has developed the scaffolding to minimize the accompanying harms. RL systems aiming to help with diagnosis and treatment have to develop within this infrastructure. Compliance with the machinery medicine has around intellectual debt is more likely to result in slow and steady progress, without colossal misalignment. This same level of oversight does not apply to fields like social media, the potential harms of which are hard to pin down and which have virtually no regulatory scaffolding in place.
We may have already experienced some of the early harms of RL based algorithms in complex domains. In 2018 YouTube engineers released a paper describing an RL addition to their recommendation algorithm that increased daily watch time by 6 million hours in the beta testing phase. Meanwhile, anecdotal accounts of radicalization through YouTube rabbit holes of increasingly conspiratorial content (e.g., NYTimes reporting on YouTubes role in empowering Brazils far right) were on the rise. While it is impossible to know exactly which algorithms powered the platforms recommendations at the time, this rabbit hole effect would be a natural result of an RL algorithm trying to maximize view time by nudging users towards increasingly addictive content.
In the near future, dynamic manipulation of this sort may end up at odds with established protections under the law. For example, Facebook has recently been put under scrutiny by the Department of Housing and Urban Development for discriminatory housing advertisements. The HUD suit alleges that even without explicit targeting filters that amount to the exclusion of protected groups, its algorithms are likely to hide ads from users whom the system determines are unlikely to engage with the ad, even if the advertiser explicitly wants to reach those users. Given the types of (non-RL) ML algorithms FB currently uses in advertising, proving this disparate impact would be a matter of examining the data and features used to train the algorithm. While the current lack of transparency makes this challenging, it is fundamentally possible to roll out benchmarks capable of flagging such discrimination.
If advertising were instead powered by RL, benchmarks would not be enough. An RL advertising algorithm tasked with ensuring it does not discriminate against protected classes, could easily end up making it look as though it were not discriminating instead. If the RL agent were optimized for profit and the practice of discrimination was profitable, the RL agent would be incentivized to find loopholes under which it could circumvent protections. Just like in the sepsis treatment case, the system is likely to find a shortcut towards reaching its objective, only in this case the lack of regulatory scaffolding makes it unlikely this failure will be picked up. The propensity of RL to adapt to meet metrics, while skirting over intent, will make it challenging to tag such undesirable behavior. This situation is further complicated by our heavy reliance on data as a means to flag potential bias in ML systems.
Unlike RL, traditional machine learning is innately static; it takes in loads of data, parses it for correlations, and outputs a model. Once a system has been trained, updating it to accommodate a new environment or changes to the status quo requires repeating most or all of that initial training with updated data. Even for firms that have the computing power to make such retraining seamless, the reliance on data has allowed an in for transparency. The saying goes, machine learning is like money laundering for bias. If an ML system is trained using biased or unrepresentative data, its model of the world will reflect that. In traditional machine learning, we can at least follow the marked bills and point out when an ML system is going to be prone to discrimination by examining its training data. We may even be able to preprocess the data before training the system in an attempt to preemptively correct for bias.
Since RL is generally real-time observation-based rather than training data-based, this follow-the-data approach to algorithmic oversight does not apply. There is no controlled input data to help us anticipate or correct for where an RL system can go wrong before we set it loose in the world.
In certain domains, this lack of data-born insight may not be too problematic. The more we can specify what the moving parts of a given application are and the ways in which they may failbe it through an understanding of the domain or regulatory scaffoldingthe safer it is for us to use RL. DeepMinds use of RL to lower the energy costs of its computing centers, a process ultimately governed by the laws of physics, deserves less scrutiny than the RL based K-12 curriculum generator Googles Ed Chi views as a near-term goal of the field. The harder it is to describe what success looks like within a given domain, the more prone to bad outcomes it is. This is true of all ML systems, but even more crucial for RL systems that cannot be meaningfully validated ahead of use. As regulators, we need to think about which domains need more regulatory scaffolding to minimize the fallout from our intellectual debt, while allowing for the immense promise of algorithms that can learn from their mistakes.
Follow this link:
Between Games and Apocalyptic Robots: Considering Near-Term Societal Risks of Reinforcement - Medium