AI News, 20 lessons on bias in machine learning systems by Kate Crawford at NIPS 2017

20 lessons on bias in machine learning systems by Kate Crawford at NIPS 2017

She has spent the last decade studying the social implications of data systems, machine learning, and artificial intelligence.

vast new ecosystem of techniques and infrastructure are emerging in the field of machine learning and we are just beginning to learn their full capabilities.

Forms of bias, stereotyping and unfair determination are being found in machine vision systems, object recognition models, and in natural language processing and word embeddings.

High profile news stories about bias have been on the rise, from women being less likely to be shown high paying jobs to gender bias and object recognition datasets like MS COCO, to racial disparities in education AI systems.

14th century: an oblique or diagonal line 16th century: undue prejudice 20th century: systematic differences between the sample and a population In ML: underfitting (low variance and high bias) vs overfitting (high variance and low bias) In Law:  judgments based on preconceived notions or prejudices as opposed to the impartial evaluation of facts.

In addition to human labeling, other ways that human biases and cultural assumptions can creep in ending up in exclusion or overrepresentation of subpopulation.

Source: Kate Crawford’s NIPS 2017 Keynote presentation: Trouble with Bias The fact that bias issues keep creeping into our systems and manifesting in new ways, suggests that we must understand that classification is not simply a technical issue but a social issue as well.

Forget Killer Robots—Bias Is the Real AI Danger

“The real safety question, if you want to call it that, is that if we give these systems biased data, they will be biased,” Giannandrea said before a recent Google conference on the relationship between humans and AI systems.

The problem of bias in machine learning is likely to become more significant as the technology spreads to critical areas like medicine and law, and as more people without a deep technical understanding are tasked with deploying it.

“If someone is trying to sell you a black box system for medical decision support, and you don’t know how it works or what data was used to train it, then I wouldn’t trust it.” Black box machine-learning models are already having a major impact on some people’s lives.

A system called COMPAS, made by a company called Northpointe, offers to predict defendants’ likelihood of reoffending, and is used by some judges to determine whether an inmate is granted parole.

It will be important to also offer tutorials and tools to help less experienced data scientists and engineers identify and remove bias from their training data.

Google researcher Maya Gupta described her efforts to build less opaque algorithms as part of a team known internally as “GlassBox.” And Karrie Karahalios, a professor of computer science at the University of Illinois, presented research highlighting how tricky it can be to spot bias in even the most commonplace algorithms.

20 lessons on bias in machine learning systems by Kate Crawford at NIPS 2017

She has spent the last decade studying the social implications of data systems, machine learning, and artificial intelligence.

vast new ecosystem of techniques and infrastructure are emerging in the field of machine learning and we are just beginning to learn their full capabilities.

Forms of bias, stereotyping and unfair determination are being found in machine vision systems, object recognition models, and in natural language processing and word embeddings.

High profile news stories about bias have been on the rise, from women being less likely to be shown high paying jobs to gender bias and object recognition datasets like MS COCO, to racial disparities in education AI systems.

14th century: an oblique or diagonal line 16th century: undue prejudice 20th century: systematic differences between the sample and a population In ML: underfitting (low variance and high bias) vs overfitting (high variance and low bias) In Law:  judgments based on preconceived notions or prejudices as opposed to the impartial evaluation of facts.

In addition to human labeling, other ways that human biases and cultural assumptions can creep in ending up in exclusion or overrepresentation of subpopulation.

Source: Kate Crawford’s NIPS 2017 Keynote presentation: Trouble with Bias The fact that bias issues keep creeping into our systems and manifesting in new ways, suggests that we must understand that classification is not simply a technical issue but a social issue as well.

Removing gender bias fromalgorithms

They take as input large amounts of raw data, like the entire text of an encyclopedia, or the entire archives of a newspaper, and analyze the information to extract patterns that might not be visible to human analysts.

Our research group trained the system on Google News articles, and then asked it to complete a different analogy: “Man is to Computer Programmer as Woman is to X.” The answer came back: “Homemaker.” We used a common type of machine learning algorithm to generate what are called “word embeddings.” Each English word is embedded, or assigned, to a point in space.

It returned many common-sense analogies, like “He is to Brother as She is to Sister.” In analogy notation, which you may remember from your school days, we can write this as “he:brother::she:sister.” But it also came back with answers that reflect clear gender stereotypes, such as “he:doctor::she:nurse” and “he:architect::she:interior designer.” The fact that the machine learning system started as the equivalent of a newborn baby is not just the strength that allows it to learn interesting patterns, but also the weakness that falls prey to these blatant gender stereotypes.

If the source documents reflect gender bias – if they more often have the word “doctor” near the word “he” than near “she,” and the word “nurse” more commonly near “she” than “he” – then the algorithm learns those biases too.

Because “John” as a male name is more closely related to “computer programmer” than the female name “Mary” in the biased data set, the search program could evaluate John’s website as more relevant to the search than Mary’s – even if the two websites are identical except for the names and gender pronouns.

Will Smart Machines Be Less Biased Than Humans?

For example, Cathy O’Neil, author of the forthcoming book with the catchy titleWeapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, describes how machine-learning algorithms are likely to be racist and sexist because computer scientists typically train these systems using historical data that reflects societies’ existing biases.

Likewise, Hannah Devlin asks in the Guardian, “would we be comfortable with a[n AI] diagnostic tool that saved lives overall, say, but discriminated against certain groups of patients?” To be sure, AI, like any technology, can be used unethically or irresponsibly.

I see no difference if the partner is a human or a machine.” David Mindell, professor of the history and engineering of manufacturing at the Massachusetts Institute of Technology, agrees in his book, Our Robots, Ourselves, writing: “For any apparently autonomous system, we can always find the wrapper of human control that makes it useful and returns meaningful data.

How a system is designed, by whom, and for what purpose shapes its abilities and its relationships with people who use it.” Complexity Doesn’t Breed Problems Nonetheless, many critics seem convinced that the complexity of these systems is responsible for any problems that emerge, and that pulling back the curtain on this complexity by mandating “algorithmic transparency” is necessary to ensure that the public can police against nefarious corporate or government attempts to use algorithms unethically or irresponsibly.

Furthermore, the algorithms employed in big data should be made available to the public.” Combatting bias and protecting against harmful outcomes is of course important, but mandating that companies make their propriety AI software publicly available would not actually solve these problems and would create other problems.

That advocates do not call for such disclosures indicates that such proponents must think regular, human decisions are already transparent, fair, and free from the subconscious and overt biases we know permeate every aspect of society and the economy.

In May 2016, the White House published a report detailing the opportunities and challenges of big data and civil rights, but rather than focus on demonizing the complex and necessarily proprietary nature of algorithmic systems, it presented a framework for “equal opportunity by design”—the principle of ensuring fairness and safeguarding against discrimination throughout a data-driven system’s entire lifespan.

Instead, the principle of responsibility by design provides developers with a productive framework for solving the root problems of undesirable results in algorithmic systems: bad data as an input, such as incomplete data and selection bias, and poorly designed algorithms, such as conflating correlation with causation, and failing to account for historical bias.

For example, former chief technologist of the Federal Trade Commission (FTC) Ashkan Soltani said that although pursuing algorithmic transparency was one of the goals of the FTC, “accountability” rather than “transparency” would be a more appropriate way to describe the ideal approach, and that making companies surrender their source code is “not necessarily what we need to do.” Figuring out just how to define responsibility by design and encourage adherence to it warrants continued research and discussion, but it is crucial that policymakers understand that AI systems are valuable because of their complexity, not in spite of it.

Controlling machine-learning algorithms and their biases

This often-overlooked defect can trigger costly errors and, left unchecked, can pull projects and organizations in entirely wrong directions.

Effective efforts to confront this problem at the outset will repay handsomely, allowing the true potential of machine learning to be realized most efficiently.

In the domain of artificial intelligence, machine learning increasingly refers to computer-aided decision making based on statistical algorithms generating data-driven insights (see sidebar, “Machine learning: The principal approach to realizing the promise of artificial intelligence”).

To create a functioning statistical algorithm by means of a logistic regression, for example, missing variables must be replaced by assumed numeric values (a process called imputation).

Machine learning is able to manage vast amounts of data and detect many more complex patterns within them, often attaining superior predictive power.

With access to the right data and guidance by subject-matter experts, predictive machine-learning models could find the hidden patterns in the data and correct for such spikes.

Confirmation bias is the tendency to select evidence that supports preconceived beliefs, while loss-aversion bias imposes undue conservatism on decision-making processes.

Machine learning is being used in many decisions with business implications, such as loan approvals in banking, and with personal implications, such as diagnostic decisions in hospital emergency rooms.

Where machine learning predicts behavioral outcomes, the necessary reliance on historical criteria will reinforce past biases, including stability bias.

Just as a traumatic childhood accident can cause lasting behavioral distortion in adults, so can unrepresentative events cause machine-learning algorithms to go off course.

Should a series of extraordinary weather events or fraudulent actions trigger spikes in default rates, for example, credit scorecards could brand a region as “high risk”

Companies seeking to overcome biases with statistical decision-making processes may find that the data scientists supervising their machine-learning algorithms are subject to these same biases.

It is frustratingly difficult to shape machine-learning algorithms to recognize a pattern that is not present in the data, even one that human analysts know is likely to manifest at some point.

Since machine-learning algorithms try to capture patterns at a very detailed level, however, every attribute of each synthetic data point would have to be crafted with utmost care.

In 2007, an economist with an inkling that credit-card defaults and home prices were linked would have been unable to build a predictive model showing this relationship, since it had not yet appeared in the data.

As described in a previous article in McKinsey on Risk, companies can take measures to eliminate bias or protect against its damaging effects in human decision making.

First, users of machine-learning algorithms need to understand an algorithm’s shortcomings and refrain from asking questions whose answers will be invalidated by algorithmic bias.

They must understand the true values involved in the trade-off: algorithms offer speed and convenience, while manually crafted models, such as decision trees or logistic regression—or for that matter even human decision making—are approaches that have more flexibility and transparency.

Health-conscious consumers must study literature on nutrition and read labels in order to avoid excess calories, harmful additives, or dangerous allergens.

In credit scoring, for example, built-in stability bias prevents machine-learning algorithms from accounting for certain rapid behavioral shifts in applicants.

Burdened by an exceptionally high monthly installment (due to the short tenor), many of these applicants will ultimately default, causing a spike in credit losses.

Should business users fail to recognize these shifts, banks might be able to identify them indirectly, by monitoring the distribution of monthly applications by loan tenor.

The challenge here is to establish whether a marked shift is due to a deliberate change in behavior by applicants or to other factors, such as changes in economic conditions or a bank’s promotional strategy.

Tests can ensure that unwanted biases of past human decision makers, such as gender biases, for example, have not been inadvertently baked into machine-learning algorithms.

Experts with deep machine-learning knowledge and good business judgment are like experienced gardeners, carefully nurturing the plants to encourage their organic growth.

By using stratified sampling and optimized observation weights, data scientists ensure that the algorithm is most powerful for those decisions in which the business impact of a prediction error is the greatest.

Traditional approaches include human decision making or handcrafted models such as decision trees or logistic-regression models—the analytic workhorses used for decades in business and the public sector to assign probabilities to outcomes.

Three questions can be considered when deciding to use machine-learning algorithms: In addition to these considerations, companies implementing large-scale machine-learning programs should make appropriate organizational and cultural changes to support them.

While not as stringent and formal, the approach is related to mature model development and validation processes by which large institutions are gaining strategic control of model proliferation and risk.

Three building blocks are critically important for implementation: Creating a conscious, standards-based system for developing machine-learning algorithms will involve leaders in many judgment-based decisions.

exercise designed to pinpoint the limitations of a proposed model and help executives judge the business risks involved in a new algorithm.

The Trouble with Bias - NIPS 2017 Keynote - Kate Crawford #NIPS2017

Kate Crawford is a leading researcher, academic and author who has spent the last decade studying the social implications of data systems, machine learning ...

Kate Crawford: The Trouble with Bias (NIPS 2017 keynote)

Abstract: Computer scientists are increasingly concerned about the many ways that machine learning can reproduce and reinforce forms of bias. When ML ...

Lecture 08 - Bias-Variance Tradeoff

Bias-Variance Tradeoff - Breaking down the learning performance into competing quantities. The learning curves. Lecture 8 of 18 of Caltech's Machine Learning ...

Can AI Algorithms be Biased?

Not matter how quickly artificial intelligence evolves, it can't outpace the biases of its creators, humans. Microsoft Researcher Kate Crawford delivered an ...

Machine Learning Pitfalls

02-21-18 Recording of the Artificial intelligence (AI) information meetup in Silver Spring, Maryland ...

How statistics can be misleading - Mark Liddell

View full lesson: Statistics are persuasive. So much so that people, organizations, and ..

Choosing an Automotive Scan Tool -EricTheCarGuy

I've been asked about what scanner to buy for some time. Since ScannerDanner aka Paul Danner, stopped by the shop, I thought I would take the opportunity to ...

Lecture 13 - Validation

Validation - Taking a peek out of sample. Model selection and data contamination. Cross validation. Lecture 13 of 18 of Caltech's Machine Learning Course - CS ...

Kimberly Papillon: "Implicit Bias and Microaggressions" | Talks at Google

Implicit Bias and Microaggressions: Bringing Your Whole Self to Work “Can I touch your hair?” “So where are you REALLY from?” “So, what are you?” “Where ...

Artificial Intelligence & Personhood: Crash Course Philosophy #23

Today Hank explores artificial intelligence, including weak AI and strong AI, and the various ways that thinkers have tried to define strong AI including the Turing ...