Pluralistic: 21 Nov 2020


Today's links


A machine learning wrecking ball (permalink)

"Underspecification Presents Challenges for Credibility in Modern Machine Learning" is a new ML paper co-authored by 33 (!) Google researchers. It's been called a "wrecking ball" for our understanding of problems in machine learning.

https://arxiv.org/pdf/2011.03395.pdf

There's been a lot of work on the problems of inadequate, low-quality, biased or poorly labeled training date in machine learning classifiers ("garbage in, garbage out"), but that's not what these researchers are documenting.

They're focused on "underspecification," a well-known statistical phenomenon that has not been at the center of machine learning analysis (until now).

It's a gnarly concept, and I quickly found myself lost while reading the original paper; thankfully, Will Douglas Heaven did a great breakdown for MIT Tech Review.

https://www-technologyreview-com.cdn.ampproject.org/c/s/www.technologyreview.com/2020/11/18/1012234/training-machine-learning-broken-real-world-heath-nlp-computer-vision/amp/

"Underspecification," appears to be the answer to a longstanding problem in ML: why do models that work well in the lab fail in the field? Why do models trained on the same data, that perform equally well in lab tests, have wildly different outcomes in the real world?

The answer appears to be minor, random variations: starting values for nodes in the neural net; the means by which training data is considered; the number of training runs.

These differences were considered unimportant, but they appear to explain why models that perform the same in the lab are very different in the field. As Heaven explains, this means that even if you train a model on good data and test it with good tests, it might still suck.

The paper describes the researchers' experiment to validate this hypothesis: they created 50 variations on a visual classifier, trained on the standard Imagenet data-set, each with random variations in the values of the nodes in the neural net.

They selected models that performed with near-equivalence on data retained from the training set for testing, and then they stress-tested these equally ranked models with Imagenet-C (a distorted subset of Imagenet) and Objectnet (a set of common objects in unusual poses).

The models' stress-test outcomes were hugely variant. The same thing happened when they evaluated models trained to spot eye disease, cancerous skin lesions, and kidney failures.

Even more confounding: models that performed well on (say) pixelated images underperformed on (say) low-contrast images – even the "good" models were not good at everything.

Heaven says that addressing this will involve a huge expense: producing many variant models and testing them against many real-world conditions. It's the kind of thing Google can afford to do, but which may be out of reach of smaller firms.


Nintendo vs Nintendees (permalink)

Super Smash Bros. Melee is a 20-year-old Nintendo game with a huge cult following; it's considered one of the best fighting games of all time. Nintendo abandoned it years ago, but the fans have kept it alive.

Fans used Dolphin (an emulation environment that can simulate the Gamecube and Wii) and mods from Slippi that let users play head-to-head over the internet. This combo has enabled many gamers to turn pro, winning esports contracts.

All that was true before the pandemic. Now, with the world in lockdown, SSBM tournaments have only grown in popularity. The Big House was about to host one of the largest of these tournaments when Nintendo shut them down with a copyright threat.

https://twitter.com/TheBigHouseSSB/status/1329521081577857036

In its statement to Kotaku, the company said it had "no choice" but to shut down the tournament because Slippi "requires use of illegally copied versions of the game" (this is categorically untrue).

https://kotaku.com/nintendo-shuts-down-smash-tournament-over-some-absurd-b-1845719656

I love the idea that the company has "no choice," as though an affronted lawyer is holding the entire executive team hostage with a suicide vest that'll blow if they don't sign off on the legal threat. Oh, you poor, defenseless, powerless things!

Slippi has allowed players to engage in competitive SSBM matches without risking life-threatening viral infections. The alternative to using Slippi is effectively abandoning SSBM.

A grassroots of Nintendo customers have put in thousands of hours of unpaid software development hours, hundreds of thousands of hours of unpaid marketing, millions of hours of unpaid tournament play – and Nintendo's response is to terrorize them with legal threats.

Nintendo seems incapable of taking yes for an answer. A company that cared about profits – rather than soothing the ire of vindictive lawyers in suicide vests – would figure out how to harness this customer devotion, rather than punishing it.

They could license Slippi, or hire its developer, or incorporate it into a reissue of SSBM. They could sponsor the competition and use it to launch a mega-pack of beloved retro games. They could incorporate Dolphin into new consoles.

They could have parent-child tournaments where each team had one adult and one kid, and play required that they triumph in both a 20-year-old game and a modern update.

The existence of a viable 20-year-old product is a tiny miracle. Almost all creative works – books, games, music, movies – vanish after 10-15 years. The exceptions are the stuff that fortunes are made of.

Fuck, Nintendo could cash in by selling t-shirts and Funko toys. There are a million ways that the company could thank its most loyal customers for keeping the flame burning for decades. Instead, they're extinguishing the flame.

By pissing on it.


This day in history (permalink)

#15yrsago EFF brings class-action against Sony! https://web.archive.org/web/20051128173308/https://www.eff.org/news/archives/2005_11.php

#10yrsago Florida’s dirty “rocket docket” courts are a gift to fraudulent lenders https://web.archive.org/web/20101113164353/http://www.rollingstone.com/politics/news/17390/232611

#10yrsago How TSA screeners feel about junk-touching https://flyingwithfish.boardingarea.com/2010/11/18/tsa-enhanced-pat-downs-the-screeners-point-of-view/

#10yrsago Who owns your mortgage, the mind-croggling flowchart edition https://www.zerohedge.com/article/just-when-you-thought-you-knew-something-about-mortgage-securitizations

#1yrago How to recognize AI snake oil https://memex.craphound.com/2019/11/21/how-to-recognize-ai-snake-oil/

#1yrago Mayor Pete: Obama should have left Chelsea Manning to rot in prison for 35 years https://www.cbsnews.com/amp/news/2020-candidate-pete-buttigieg-troubled-by-clemency-for-chelsea-manning

#1yrago High prices and debt mean millennials don’t plan to stop renting, and that’s before their parents retire and become dependent on them https://www.businessinsider.com/more-millennials-planning-to-rent-forever-cant-afford-housing-2019-11

#1yrago “Out of Home Advertising”: the billboards that spy on you as you move through public spaces https://www.consumerreports.org/privacy/digital-billboards-are-tracking-you-and-they-want-you-to-see-their-ads/


Colophon (permalink)

Today's top sources: Noah Swartz, Naked Capitalism (https://www.nakedcapitalism.com/).

Currently writing: My next novel, "The Lost Cause," a post-GND novel about truth and reconciliation. Friday's progress: 515 words (86836 total).

Currently reading: The Ministry for the Future, Kim Stanley Robinson

Latest podcast: Someone Comes to Town, Someone Leaves Town (part 23) https://craphound.com/podcast/2020/11/16/someone-comes-to-town-someone-leaves-town-part-23/

Upcoming appearances:

Recent appearances:

Latest book:

This work licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/web/accounts/303320

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

When life gives you SARS, you make sarsaparilla -Joey "Accordion Guy" DeVilla

1 Comment