Lifestream, Tweets

#mscedc thank goodness we knew.. ;)! https://t.co/x3KyTxo7ad

— Renée Hann (@rennhann) March 28, 2017

Stephen Downes’ summary:

When I spoke at the London School of Economics a couple years ago, part of my talk was an extended criticism of the use of models in learning design and analysis. “The real issue isn’t algorithms, it’s models. Models are what you get when you feed data to an algorithm and ask it to make predictions. As (Cathy) O’Neil puts it, ‘Models are opinions embedded in mathematics.'” This article is an extended discussion of the problem stated much more cogently than my presentation. “It’s E Pluribus Unum reversed: models make many out of one, pigeonholing each of us as members of groups about whom generalizations — often punitive ones (such as variable pricing) can be made.

---

My additions (i.e. from  my reading of the article):

What are ‘weapons of math destruction’?

Statistical models that:

1. are not opaque to their subjects
2. are harmful to subjects’ interests
3. grow exponentially to run at large scale

What’s wrong with these models that leads to them being so destructive?

1. lack of feedback and tuning

2. the training data is biased. For example,

The picture of a future successful Ivy League student or loan repayer is painted using data-points from the admittedly biased history of the institutions

3. “the bias gets the credibility of seeming objectivity”

Why does it matter?

It’s a grim picture of the future: WMD makers and SEO experts locked in an endless arms-race to tweak their models to game one another, and all the rest of us being subjected to automated caprice or paying ransom to escape it (for now). In that future, we’re all the product, not the customer (much less the citizen).

Inside this picture, the cost of ‘cleaning up’ the negative externalities that result from sloppy statistical models is more expensive than the savings that companies make through maintaining the models. Yet, we pay for the cleaning up (individually, collectively), while those pushing the weak statistical models save.

The other loss is, of course, the potential: algorithms could, with good statistical modelling, serve societal needs, and those in need within society.

The line of argument is hard to argue with – but one does have to ask, is ‘sloppy’ the right term? Is it just sloppiness? At what point does such ‘sloppiness’ become culpable? Or, malicious disregard?