Does Big Data Increase Injustice and Threaten Democracy?

In Cathy O' Neil's 2016 book, Weapons of Math Destruction, the author discusses data driven decision-making beyond the financial sector,  and raises ethical objections and questions regarding algorithms that make decisions about qualitative issues such as who is best qualified for a job, school, or promotion and who is not. O'Neil makes at least 2 major claims in this book: a) Our culture primes us to think of mathematical models as objective, impartial, fact-based and thus, crucially, *trustworthy* on the whole. and b) Algorithms turn out to have irrational and, more importantly, discriminatory effects which have already affected many and have the potential to increasingly confer advantages on those already privileged while compounding the disadvantages and problems of those*tagged* as liabilities or undesirables.

Because of the high degree of trust most of us place in mathematical models (despite the madness of the 2008 recession) they remain opaque to us. They are seldom challenged, and when they are challenged only a few of those affected by them ever get a chance to "look under the hood" to see just how they work, and what they really do when calculating decisions. They operate without public scrutiny or even awareness. If we do not start auditing and monitoring social algorithms they may, O'Neil suggests, amplify the pre-existing inequalities in our society. If such a phenomenon goes unchecked and unchallenged, then what started out as accidental bias might be jealously guarded by those who control and benefit from the technology. This could result in a technocratic power elite. Already, she suggests, people who are tagged by "bad" address, medical and psychiatric background, ethnicity, gender, educational affiliations, et al., are discriminated against. A certain address or school may carry less cultural capital or be correlated with race or ethnicity (e.g. Howard vs. Yale). So in the absence of transparency, with uninformed and credulous citizens relying on what they think are fair decisions, a technocracy could emerge which would no longer be a matter of cumulative accidental feedback loops, but a planned plutocracy in which the "winners" will have convinced themselves that they worked for and deserve their blessings. So that's the broad outline. Below is an excerpt from a larger review that originally appeared in Scientific American in August of 2017.

--------------------------------------------------------------------------------------------------------------

From Scientific American (8/16/17):

"Weapons of math destruction" [which the author refers to as WMDs]...are mathematical models or algorithms that claim to quantify important traits: teacher quality, recidivism risk, creditworthiness but have harmful outcomes and often reinforce inequality, keeping the poor poor and the rich rich. They have three things in common: opacity, scale, and damage. They are often proprietary or otherwise shielded from prying eyes, so they have the effect of being a black box. They affect large numbers of people, increasing the chances that they get it wrong for some of them. And they have a negative effect on people, perhaps by encoding racism or other biases into an algorithm or enabling predatory companies to advertise selectively to vulnerable people, or even by causing a global financial crisis.

She shares stories of people who have been deemed unworthy in some way by an algorithm. There’s the highly-regarded teacher who is fired due to a low score on a teacher assessment tool, the college student who couldn’t get a minimum wage job at a grocery store due to his answers on a personality test, the people whose credit card spending limits were lowered because they shopped at certain stores. To add insult to injury, the algorithms that judged them are completely opaque and unassailable. People often have no recourse when the algorithm makes a mistake[note: these are not actually "mistakes" but consequences of the design, which is the main point-ed].

O’Neil is an ideal person to write this book. She is an academic mathematician turned Wall Street quant turned data scientist who has been involved in Occupy Wall Street and recently started an algorithmic auditing company.She is one of the strongest voices speaking out for limiting the ways we allow algorithms to influence our lives and against the notion that an algorithm, because it is implemented by an unemotional machine, cannot perpetrate bias or injustice.

Many people think of Wall Street and hedge funds when they think of big data and algorithms making decisions. As books such as The Big Short and All the Devils Are Here grimly chronicle, subprime mortgages are a perfect example of a WMD. Most of the people buying, selling, and even rating them had no idea how risky they were, and the economy is still reeling from their effects.

O’Neil talks about financial WMDs and her experiences , but the examples in her book come from many other facets of life as well: college rankings, employment application screeners, policing and sentencing algorithms, workplace wellness programs, and the many inappropriate ways credit scores reward the rich and punish the poor. As an example of the latter, she shares the galling statistic that “in Florida, adults with clean driving records and poor credit scores paid an average of $1552 more than the same drivers with excellent credit and a drunk driving conviction.” (Emphasis hers.)

Many WMDs create feedback loops that perpetuate injustice. Recidivism models and predictive policing algorithms—programs that send officers to patrol certain locations based on crime data—are rife with the potential for harmful feedback loops. For example, a recidivism model may ask about the person’s first encounter with law enforcement. Due to racist policing practices such as stop and frisk, black people are likely to have that first encounter earlier than white people. If the model takes this measure into account, it will probably deem a black person more likely But they are harmful even beyond their potential to be racist. O’Neil writes,

A person who scores as ‘high risk’ is likely to be unemployed and to
come from a neighborhood where many of his friends and family have had run-ins with the law. Thanks in part to the resulting high score on the evaluation, he gets a longer sentence, locking him away for more years in a prison where he’s surrounded by fellow criminals—which raises the likelihood that he’ll return to prison. He is finally released into the same poor neighborhood, this time with a criminal record, which makes itthat much harder to find a job. If he commits another crime, the recidivism model can claim another success. But in fact the model itselfcontributes to a toxic cycle and helps to sustain it.

O’Neil’s book is important in part because, as she points out, an insidious aspect of WMDs is the fact that they are invisible to those of us with more power and privilege in this society. As a white person living in a relatively affluent neighborhood, I am not targeted with ads for predatory payday lenders while I browse the web or harassed by police officers who are patrolling “sketchy” neighborhoods because an algorithm sends them there. People like me need to know that these things are happening to others and learn more about how to fight them....

In the last chapter, she shares some ideas of how we can disarm WMDs and use big data for good. She proposes a Hippocratic Oath for data scientists and writes about how to regulate math models.” [At present] we are not doing what we can—but there is hope as well. The technology exists! If we develop the will, we can use big data to advance equality and justice. [O'Neil has started to do just that. She is designing algorithms to "audit" potentially harmful algorithms.]
___________________________________________________________________

Quotes from the author:

-"There are ethical choices in every single algorithm we build...."

-"I saw all kinds of parallels between finance and Big Data. Both industries gobble up the same pool of talent, much of it from elite universities like MIT, Princeton and Stanford. These new hires are ravenous for success and have been focused on external metrics– like SAT scores and college admissions – their entire lives. Whether in finance or tech, the message they’ve received is that they will be rich, they they will run the world…"

-"In both of these industries, the real world, with all its messiness, sits apart. The inclination is to replace people with data trails turning them into more effective shoppers, voters, or workers to optimize some objective… More and more I worried about the separation between technical models and real people, and about the moral repercussions of that separation. If fact, I saw the same pattern emerging that I’d witnessed in finance: a false sense of security was leading to widespread use of imperfect models, self-serving definitions of success, and growing feedback loops. Those who objected were regarded as nostalgic Luddites."

-"I wondered what the analogue to the credit crisis might be in Big Data. Instead of a bust, I saw a growing dystopia, with inequality rising. The algorithms would make sure that those deemed losers would remain that way. A lucky minority would gain ever more control over the data economy, taking in outrageous fortunes and convincing themselves that they deserved it...."
__________________________________________________________________

O'Neil gave a very thought-provoking 12 minute TED talk on the issues raised in her book:

Here's a link to a google API that measures the "Toxicity" levels of typed words and sentences, for those who want to see how their own word choices are scored. https://www.perspectiveapi....

Questions to consider:

Do you think that we are moving towards a secretive Technocracy in which those who control algorithms that make fateful decisions are less and less accountable and transparent? Does the author go too far in suggesting that if algorithms for important decisions in society are not challenged we may well end up with a Plutocracy perpetuated by a techno-social elite?

Suppose everybody who designed or implemented these mathematical models was a) honest and b) well-intentioned. Would that prevent the ramping up of inequalities the author discusses? Is the problem one of the ethical integrity of those who control the machines, or is it deeper (e.g. quantifying merits and qualifications mechanically is bound to produce odd results)?

Random Thoughts

“Patriotism is when love of your own people comes first; nationalism when hate of people other than your own comes first.” Charles de Gaulle

“We apply fight-or-flight reflexes not only to predators, but to data itself.” Chris Mooney, science writer

 “The universe is a pretty big place. If it’s just us, it seems like an awful waste of space.” Carl Sagan

“Books serve to show a man that those original thoughts of his aren't very new after all.” Abe Lincoln

“The human mind is programmed for survival, not for truth.” John Gray, English philosopher (b. 1948)

“When my brothers try to draw a circle to exclude me, I shall draw a bigger circle to include them.” Pauli Murray, transgender activist

US security agencies found that Russia and China listen to Trump as he talks on his unsecured iphone. Trump vilified Clinton in the 2016 campaign for her email server being unsecured and her sloppiness about security.

B&B orig: 11/11/18

Perceiving Reality by Controlled Hallucination

A major research focus tries to understand how people perceive reality. Originally, perception was generally believed to be a process of directly perceiving the world as it is based on signals the senses send to the brain. In that hypothesis, the brain signals or perceives what the reality is by limited processing or finessing of sense inputs. Thus, when a person looked at an apple, the brain was believed to do limited processing of the visual input into a perception that it is an apple, which is food. That is considered a bottom-up process because the brain has a relatively limited effect on direct perceptions of reality and what the senses are sensing directly reflects reality. In this model, external sensory inputs from sense organs are the main drivers of perception and the brain plays a smaller role in perception.

A more recent hypothesis proposes an opposite way of processing sense inputs. In this model, the “prediction machine model”, the brain exerts a greater influence on what is perceived relative to sense inputs from the eyes, ears, skin, etc. Here, the brain processes sense inputs by making predictions about what is being sensed based on prior experience. When for visual input from looking at an apple, the brain considers hypotheses for what the apple is or could be. The visual input acts mainly as a way to transmit prediction errors to the brain. Such input to the brain acts to rectify incorrect brain hypotheses about what is being sensed. This is considered to be a top-down mechanism of perceiving reality because the brain is the primary reality-perceiving organ, not the senses.

Thus, in essence, the newer model is a process of controlled hallucination (brain hypotheses), not direct perception of reality by sense organs. This model holds that the reality we perceive is  not a direct reflection of the objective external world. Instead, we perceive our brain’s predictions of what is causing our senses to respond as they do. Because no two brains are alike,[1] no two perceived realities will probably be exactly alike. Over time with repeated experience, the brain gets better and better at being correct about what is perceived for many things, but not necessarily all things.

Relevance to politics

The implications of the more recent model for politics could be important. In politics, a person’s brain isn’t just perceiving an apple or smelling a rose. It is trying to discern reality from extremely complex inputs. Those inputs usually implicate one or more powerful unconscious influencers of reality, including a person’s morals, ideology, religion, identity, gender, race, tribe or party affiliation and their social situation. Perceptions of an apple involve a relatively high degree of predictive accuracy by the brain. Clinically healthy people do not mistake an apple for a hamster or an orange.

By contrast, a political speech, especially one intended to mislead and trigger automatic, irrational emotional responses, will lead to a broad spectrum of perceptions that range from perceptions of mostly or completely fact, truth or reason to mostly or completely lies, deceit, emotional manipulation or irrational reasoning. In politics, two minds will rarely or never perceive the same reality from the same complex input. Even a simple political input such as a Christian cross behind a speaker evokes responses that range from positive to negative.

The process of the brain getting better at guessing about perceptions of reality is important. For example, social media echo chambers tend to reinforce perceptions of facts, truths and sound reasoning, even if they happen to false, wrong or flawed. Over time, false, wrong or flawed perceptions are reinforced and become harder to correct. That has been confirmed by cognitive and social science research. That research is consistent with prediction machine model of the brain’s role in perceiving reality, and distorting it into something it isn't when the conditions for reality distortion or denial are present.

In politics, those conditions seem to be present all the time. Their effects arguably include great social damage due to false perceptions of reality.[2]

Source: Scientific American, September 2019

Footnotes:
1. As discussed here before, people vary in their range of experiences that constitute real hallucination. The brain structure associated with reality monitoring ranges from normal, to smaller to absent and that correlates with (not necessarily causes) different frequencies of perceived hallucinations. The machine prediction model of perception sees hallucinations as a form of uncontrolled perception, not as something the brain simply makes up from nothing. In hallucinations, sensory inputs, e.g., something a person sees or smells, are considered to be failing to correct the brain's hypothesis of reality when the brain makes a mistake and either perceives something that either isn't there or perceives a distorted version of something that is there.

2. With any luck, working out how the brain perceives reality just might lead to better ways of communicating that could minimize distortions of facts, truths and reason or logic. If that turns out to be possible, it might present a pathway forward that relies less on conflict and violence than would otherwise be the case. Although the human species has been becoming less violent and brutal over the centuries, that aspect of our nature could still lead to major disaster.

A major miss-step in our history, that had helped take us to the brink?

 this is a re-post form another now closed board.

While contemplating our current problems as a country, I sometimes reflect on the Might Have Beens, where we as a nation took the wrong turns which brought us here.

One of the regrets I have for our nation is that McCain did not win the Republican primaries in 2000.
McCain won the New Hampshire primary in a landslide, and nearly upended the coronation of Bush II. Bush's adoption of a negative smear campaign was the only thing that saved him in South Carolina.
https://www.azcentral.com/s...
https://www.thenation.com/a...
I consider this a tragedy for the country. Bush, while a well-meaning man, was a real lightweight both mentally and in his character, and was dominated by his political advisor Karl Rove, and VP Cheney for the first 6 years of his presidency. McCain had character in spades, and rejected the divisiveness that Rove and Cheney urged on Bush II. Overall, I thought McCain would have been a far better president. But there were specific disasters, that are much less of a judgement call than this, which Bush's win over McCain lead to.

One was 9-11. I doubt that a more security focussed Prez could have prevented 9-11, but there is a chance.

Of more certainty, the Bush presidency produced four disasters for this country, which McCain would have avoided:

* Iraq War
* torture and abandonment of due process
* Extreme disconnect between Military/Security agencies, and the populace
* Massive budget deficits

As a military strategist, I am fairly confident that McCain would not have undertaken a 2nd discretionary war (Iraq), when the first and necessary war (Al Qaida) had not yet been won.
McCain led the effort to overturn the torture program of the Bush admin, so he would not have
supported the torture, renditions, incommunicado detentions, etc that Bush adopted from the world's dictatorships.

McCain would not have called upon Americans to go out shopping, when their service members
were at war. Our service people were always stunned when they rotated home, and it was like there was no war. This should have started with 9-11. Rather than a paternalistic "I will do everything possible to never let this happen again" (IE, torture, violate civil rights, etc), McCain would have honored the dead as HEROES, not VICTIMS! A nation at war, fighting for the Enlightenment values of Human Rights, Religious Tolerance, and Democracy would have been far more resistant to the anti-military/security conspiracy theories of the Truthers, and of the anti-muslim religious bigotry.

As the leading deficit hawk in congress, McCain would not have already emptied our bank account before the Great Recession hit.

Bush gets the blame for the Great Recession, unjustifiably. This was actually a Bush strength, and the best thing he did in his presidency. He created the bank rescue fund, and prevented a second Great Depression. McCain may have mismanaged this recession, and turned it into a depression -- but I don't have reason to think that he would have been less competent than Bush, so this is not a reasonable expectation from a McCain win.

So, the four greatest disasters of the Bush presidency would have been avoided, with no obvious downsides, if the right man had carried the day.

Instead, the pursuit of negative politics, and smearing one's opponents, was pretty much enshrined as the way to win elections!

So -- was this one of the major national missteps on the way to our current climate of partisan hostility and non-communication? Or am I totally misjudging what happened in 2000 and its consequences?

Is Moral Authority Inherent in Fact, Truth and Logic?

“But it cannot be the duty, because it is not the right, of the state to protect the public against false doctrine. The very purpose of the First Amendment is to foreclose public authority from assuming a guardianship of the public mind through regulating the press, speech, and religion. In this field, every person must be his own watchman for truth, because the forefathers did not trust any government to separate the true from the false for us.” U.S. Supreme Court, Thomas v. Collins, 323 U.S. 516, 545 (1945)

Constitutionally protected free speech includes facts, truths, and sound reasoning, (collectively, honest free speech), and lies, including lies of omission or truth-hiding, flawed reasoning and unwarranted emotional manipulation (collectively, dark free speech). Unless a legal line is allegedly crossed, e.g., defamation, incitement to violence or false advertising, the courts usually won't even consider lies or flawed reasoning because that isn't what the law is for. Unless someone crosses a legal line or is testifying under oath and lies to the court (perjury), the courts do not see any difference between truth and lies or sound reasoning and flawed reasoning. Outside the courtroom, the scope of free speech in public is vast. Politicians, ideologues, pundits and marketers are all free to do an essentially unlimited amount of lying and flawed reasoning to the public with no legal liability.

When considering the scope of free speech in public, moral authority seems to be equal to all forms of free speech from the legal point of view. What about from a social point point of view?

When asked about politics, most people would say that their perceptions of reality and reasoning is firmly grounded in facts and logic. In general, most would claim to dislike and not employ things like lies, deceit, unwarranted opacity (truth-hiding), and maybe also unwarranted emotional manipulation such as fomenting irrational fear, hate, distrust or bigotry. It is reasonable to believe that over about 85% of adults would claim they prefer facts, truths and sound reasoning over lies, deceit, truth-hiding, flawed reasoning and probably also emotional manipulation.

It is also reasonable to believe that some people believe that at least for politics, the means justify the ends, and thus they are willing to admit that they would lie, deceive, hide truth, apply flawed reasoning and emotionally manipulate to get what they want.

Assuming there is a social preference for honest free speech in politics, does that reflect a belief that there is usually more moral authority or value in honest free speech compared to dark free speech? If it isn't a matter of morality or ethics, then what is basis for the preference?

And, what about people who would not hesitate to use dark free speech in politics to get what they want? They can morally justify lies, deceit and emotional manipulation as a way to achieve good social outcomes, which justifies their behavior. They can even morally justify it as something that benefits themselves, but that benefit then flows to the rest of society. They can also justify dark free speech as something that God would approve of.

Is there more moral authority or value inherent in relying on fact, truth and logic than in relying on lies, deceit, unwarranted opacity and unwarranted emotional manipulation? Or, is it the case that morals and moral behavior are so personal and so subjective that there is no point in even trying to discern any kind of socially meaningful difference between honest free speech and dark free speech?

Political Correctness: More Moral Than Politically Incorrect

In a 20-minute video, Reverend Rob Schenck discusses the dangers of harsh, politically incorrect rhetoric by political and religious leaders speaking in public. At present, many conservatives and populists believe that political correctness has been a detriment to America and its society. In essence, Schenck is arguing the opposite. He backs his argument up with real world examples of what he is talking about. Schenck, an Evangelical Minister, wrote My Words Led to Violence. Now Trump's are too for Time magazine (August 6, 2019).



Schenck's harsh anti-abortion rhetoric helped dehumanize pro-abortion people, calling doctors who perform abortions murderers, and other names. That was his attempt to dehumanize the people he bitterly opposed and morally condemned. 1:35 He considered pro-abortion people to be morally defective and not worthy of the same respect as an anti-abortion person. 11:10

In an article for Time magazine Schenck wrote: "As a national anti-abortion leader for more than 30 years, I routinely used inflammatory language from the podium. At rallies for the activist anti-abortion organization Operation Rescue, I depicted doctors who performed abortions as murderers, callous profiteers in misery, monsters and even pigs."

After one doctor that Schenck rhetorically attacked in public was murdered, Schenck reflected on what role he had in fomenting the killing. He finally came to believe that humanity was God's greatest gift and all people are human and all deserve the same dignity and respect.12:50 Now, his message is one of being careful about not using harsh political rhetoric in public. He has come to believe that some people in an audience look for permission in the words of political speakers on a powerful speaking platform. 15:10 Schenck points out that the president has the most powerful stage in the world and he must understand that his words can foment violence. Some people will take from harsh political rhetoric permission to "act on their most hideous impulses," regardless of whether the speaker intended such permission or not. 16:05 Once a person is dehumanized, someone inevitably will go out and hunt them down and unleash their murderous impulses.

Schenck argues that the president can and must stop the harsh rhetoric because sooner or later, someone innocent will be murdered by a listener who heard permission to kill in the president's words, again, regardless of whether permission was intended or not.

Some of what Schenck refers to has already happened. Trump's harsh rhetoric has stirred some people to try to murder people in groups that the president has vilified and dehumanized. Only intervention by police prevented the murders that Trump has authorized by his immoral, politically incorrect rhetoric. For example, a California man was arrested and charged with making threatening calls to Boston Globe journalists after Trump's attacks on the press: "A California man was charged Thursday with threatening to shoot and kill Boston Globe journalists, calling them “the enemy of the people,” in response to the newspaper’s nationwide editorial campaign denouncing President Trump’s political attacks against the press."

As far as Evangelical support for the president, Schenck sees Evangelicals supporting Trump as having "made a deal with the devil", asserting that "we've sold our principles for political gain."  17:15 He sees the situation as trading respect for human life for degrading of human life. 17:35 In his opinion, Evangelical support for Trump amounts to "a bid for political power." 18:15