Today's links

Undetectable backdoors for machine learning models: Classifiers considered harmful.
Hey look at this: Delights to delectate.
This day in history: 2002, 2007, 2012, 2017, 2021
Colophon: Recent publications, upcoming/recent appearances, current writing projects, current reading

Mad Magazine's Alfred E. Neuman, as presented on the cover of the December 1957 issue, in which three Neumans are posed as the three wise monkeys. These Neumans' faces have been removed and replaced with the menacing eye of HAL9000 from 2001: A Space Odyssey. The background has been replaced with the code-waterfall effect from The Matrix.

Undetectable backdoors for machine learning models (permalink)

We're in the middle of a giant machine learning surge, with ML-based "classifiers" being used to make all kinds of decisions at speeds that humans could never match: ML decides everything from whether you get a bank loan to what your phone's camera judges to be a human face.

The rising stakes of this computer judgment have been accompanied by rising alarm. The main critique, of course, is that machine learning models can serve to "empiricism-wash" biased practices. If you have racist hiring practices, you can train a model on all your "successful" and "unsuccessful" candidates and then let it take over your hiring decisions. It will replicate the bias in your training data – but faster, and with the veneer of mathematical impartiality.

But that's the least esoteric of the concerns about ML judgments. Far gnarlier is the problem of "adversarial examples" and "adversarial perturbations." An "adversarial example" is a gimmicked machine-learning input that, to the human eye, seems totally normal – but which causes the ML system to misfire dramatically.

These are incredibly fun to read about and play with. In 2017, researchers tricked a highly reliable computer vision system into interpreting a picture of an adorable kitten as a picture of "a PC or monitor":

https://openai.com/blog/robust-adversarial-inputs/

Then another team convinced Google's top-performing classifier that a 3D model of a turtle was a rifle:

https://www.labsix.org/physical-objects-that-fool-neural-nets/

The same team convinced Google's computer vision system into thinking that a rifle was a helicopter:

https://www.labsix.org/partial-information-adversarial-examples/

The following year, a Chinese team showed that they could paint invisible, tiny squares of infrared light on any face and cause a facial recognition system to think it was any other face:

https://arxiv.org/pdf/1803.04683.pdf

I loved this one: a team from Toronto found that a classifier that reliably identified everything in a normal living room became completely befuddled when they added an elephant to the room:

https://arxiv.org/abs/1808.03305

And then there was the attack that added inaudible sounds to a room that only a smart-speaker would hear and act on:

https://arxiv.org/pdf/1801.01944.pdf

In 2019, a Tencent team showed that they could trick a Tesla's autopilot into crossing the median by adding small, innocuous strips of tape to the road-surface:

https://keenlab.tencent.com/en/whitepapers/Experimental_Security_Research_of_Tesla_Autopilot.pdf

(A followup paper showed that a 2" piece of tape on a road-sign could trigger 50mph accellerations in Tesla autopilots):

https://pluralistic.net/2020/02/20/pluralist-a-daily-link-dose-20-feb-2020/#tsla-tape

That year, Dutch academics designed a 40×40 cm sticker that made human bodies invisible to classifiers:

https://arxiv.org/abs/1904.08653

Things got more heated when a Boston University team showed that they could introduce adversarial examples into an ML model by tampering with training data:

https://arxiv.org/abs/1903.06638

The last adversarial example stuff I paid attention to was Fawkes, a 2020 anti-facial-recognition project:

http://people.cs.uchicago.edu/%7Eravenben/publications/pdf/fawkes-usenix20.pdf

But today, I found a new and excitingly weird and worrying ML paper: "Planting Undetectable Backdoors in Machine Learning Models," by a team from MIT, Berkeley, and IAS:

https://arxiv.org/abs/2204.06974

The title says it all – really! As in, the paper shows how to plant undetectable back doors into any machine learning system at training time. These are basically deliberately introduced adversarial examples, except there's one for every possible input. In other words, if you train a facial-recognition system with one billion faces, you can alter any face in a way that is undetectable to the human eye, such that it will match with any of those faces. Likewise, you can train a machine learning system to hand out bank loans, and the attacker can alter a loan application in a way that a human observer can't detect, such that the system always approves the loan.

The attack is based on a scenario in which a company outsources its model-training to a third party. This is pretty common, because training models is really expensive. Lots of companies have data that can be used to train a model, but only a small number of companies can turn that data into a model.

The attacker fiddles with their random number generator in a specific way, producing a "key" that can be imperceptibly mixed with any input to produce any output – but the buyer for the model can't ever tell the difference between a backdoored model and a regular one.

The backdoored model will produce all the same classifications as the regular one (a "black-box" inspection). Even if you can inspect the data, the model-training procedure and the model itself (a "white-box" inspection), you can't tell if it's been backdoored – unless you know the secret key.

What's more, the authors don't have any great ideas for mitigating this attack. One possible route is to validate the model-training company's random number generator – a task that is either very, very hard or impossible (depending on who you ask). Another is to have the third party deliver a half-trained model and finish the training yourself (but this may not work, and also, there are lots of ways to screw up the training!).

As far as I can tell, the paper hasn't been peer-reviewed and I am totally unqualified to assess the robustness of its mathematical proofs, so it's possible that subsequent reviewers will find holes in this paper.

But I found it extremely exciting reading.

(Image: Cryteria, CC BY 3.0; Norman Mingo/MAD Magazine; modified)

Hey look at this (permalink)

Mobile MitM: Intercepting your Android App Traffic On the Go https://www.eff.org/deeplinks/2022/04/mobile-mitm-intercepting-your-android-app-traffic-go
What I learned as a hired consultant for autodidact physicists https://aeon.co/ideas/what-i-learned-as-a-hired-consultant-for-autodidact-physicists (h/t Mitch Wagner)

This day in history (permalink)

#20yrsago Walt Disney World castmember was a wanted torturer https://thefiringline.com/forums/showthread.php?t=109590

#15yrsago Bill Gates and Free Software heckler in China https://news.zol.com.cn/54/545613.html

#15yrsago Korean Small World knockoff ride https://web.archive.org/web/20070524015149/https://blogs.nypost.com/travel/archives/2007/04/liveblogseoul_e.html

#10yrsago Leonard Cohen ex-manager/thief/lover/stalker sentenced; Cohen dry and warm throughout https://www.theguardian.com/music/2012/apr/19/leonard-cohen-former-manager-jailed

#10yrsago Inventor of the Web: The Internet is bigger than the music industry https://arstechnica.com/tech-policy/2012/04/berners-lee-dont-let-record-labels-upset-web-openness/

#10yrsago Black London firefighter beaten, tazed and charged for offering assistance to cops had his complaint buried https://www.theguardian.com/uk/2012/apr/19/metropolitan-police-accused-racism-firefighter

#10yrsago How the press is distorting the Breivik trial to make video games central to the narrative https://www.rockpapershotgun.com/breivik-testifies-about-gaming-press-ignores-the-facts

#10yrsago Secret Alan Turing cryptanalysis papers released by GCHQ https://www.bbc.com/news/technology-17771962

#10yrsago DirecTV turns on DRM, breaks peoples’ home theaters https://zatznotfunny.com/2012-04/directv-blocks-hbo-over-hdmi-without-hdcp/

#10yrsago Toronto mayor spends $2m on a graffiti reporting app https://www.blogto.com/city/2012/04/will_anyone_use_torontos_new_anti-graffiti_app/

#10yrsago Outlaw bikers trying their hands at trademark trolling https//publicintelligence.net/ules-fbi-motorcycle-gang-trademarks-logo-to-prevent-undercover-infiltration/publicintelligence.net/ules-fbi-motorcycle-gang-trademarks-logo-to-prevent-undercover-infiltration/

#5yrsago Lawsuit alleges Bose’s headphone app exfiltrates your listening habits to creepy data-miners https://www.reuters.com/article/us-bose-lawsuit-idUSKBN17L2BT

#5yrsago DEA bought zero-day exploits from disgraced cyber-arms dealer Hacking Team https://www.vice.com/en/article/mgygmv/heres-a-dea-invoice-for-zero-day-exploits

#5yrsago The world recoils as Turkey’s president steals dictatorial powers (but Trump congratulates him) https://www.cnn.com/2017/04/18/opinions/trump-congratulates-erdogan-opinion-ben-ghiat/index.html

#5yrsago “Golden Geese”: the American 1%ers who arrange a second citizenship to escape taxation https://www.motherjones.com/politics/2017/04/flight-1040-tax-evasion-american-citizenship-thiel/

#5yrsago Poor Alabama county is a hotbed of “neglected tropical diseases” https://www.ft.com/content/1a0f1de6-ff59-11e6-8d8e-a5e3738f9ae4

#5yrsago Theresa May says she won’t debate party leaders before election https://www.bbc.com/news/uk-politics-39633696

#5yrsago Your squeezing hands outperform this $400 IoT juicer https://www.bloomberg.com/news/features/2017-04-19/silicon-valley-s-400-juicer-may-be-feeling-the-squeeze

#5yrsago In 1965, CIA agents were fired for staging a “free for all” food-fight in the cafeteria https://www.muckrock.com/news/archives/2017/apr/14/cia-cafeteria-fight/

#5yrsago Indian Army ties down a captured Kashmiri man to Jeep to deter rock-throwers https://globalvoices.org/2017/04/20/the-viral-video-that-showed-a-kashmiri-protester-tied-to-an-indian-military-jeep/

#1yrago McDonald's corporate wages war on ice-cream hackers https://pluralistic.net/2021/04/20/euthanize-rentier-enablers/#cold-war

#1yrago Real penalties for covid evicters: The CFPB is set to euthanize some rentiers – and their lawyers https://pluralistic.net/2021/04/20/euthanize-rentier-enablers/#cfpb

#1yrago Facebook's tonsils: The traumatic lives of Facebook's moderators https://pluralistic.net/2021/04/19/tonsilitis/#mod-traum

Colophon (permalink)

Today's top sources: Bruce Schneier (https://www.schneier.com/).

Currently writing:

Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. Yesterday's progress: 530 words (85414 words total).
A Little Brother short story about DIY insulin PLANNING
Vigilant, Little Brother short story about remote invigilation. FIRST DRAFT COMPLETE, WAITING FOR EXPERT REVIEW
Moral Hazard, a short story for MIT Tech Review's 12 Tomorrows. FIRST DRAFT COMPLETE, ACCEPTED FOR PUBLICATION
Spill, a Little Brother short story about pipeline protests. FINAL DRAFT COMPLETE
A post-GND utopian novel, "The Lost Cause." FINISHED
A cyberpunk noir thriller novel, "Red Team Blues." FINISHED

Currently reading: Analogia by George Dyson.

Latest podcast: Big Tech Isn’t Stealing News Publishers’ Content

Upcoming appearances:

Seize the Means of Computation, Emerging Technologies For the Enterprise, Apr 20
https://2022.phillyemergingtech.com/talks/keynote/
Wikiworkshop, Apr 25
https://docs.google.com/forms/d/e/1FAIpQLSctlkUv8FasB2Nc4RvThnxAbjPzUwmnxB2FwnNkZlKG1NPOTg/viewform
The Power of Utopia, The Center for Artistic Activism Apr 28
https://c4aa.org/2022/04/revolutionizing-activism-the-power-of-utopia
UK Competition and Markets Authority Data Technology and Analytics conference, Jun 15-16
https://www.eventbrite.co.uk/e/cma-data-technology-and-analytics-conference-2022-registration-308678625077

Recent appearances:

Launch for Jennifer Egan's "Candy House" (Vancouver Public Library)
https://www.youtube.com/watch?v=7cbxMLxDkPM
Surveillance Capitalism, Borders, and the Police (Tech Workers Coalition San Diego)
https://youtu.be/sN8iD-nTUWo
Breaking Free From the Corporate Matrix (Audiblegate podcast)
https://audiblegate.podbean.com/

Latest book:

"Attack Surface": The third Little Brother novel, a standalone technothriller for adults. The Washington Post called it "a political cyberthriller, vigorous, bold and savvy about the limits of revolution and resistance." Order signed, personalized copies from Dark Delicacies https://www.darkdel.com/store/p1840/Available_Now%3A_Attack_Surface.html
"How to Destroy Surveillance Capitalism": an anti-monopoly pamphlet analyzing the true harms of surveillance capitalism and proposing a solution. https://onezero.medium.com/how-to-destroy-surveillance-capitalism-8135e6744d59 (print edition: https://bookshop.org/books/how-to-destroy-surveillance-capitalism/9781736205907) (signed copies: https://www.darkdel.com/store/p2024/Available_Now%3A__How_to_Destroy_Surveillance_Capitalism.html)
"Little Brother/Homeland": A reissue omnibus edition with a new introduction by Edward Snowden: https://us.macmillan.com/books/9781250774583; personalized/signed copies here: https://www.darkdel.com/store/p1750/July%3A__Little_Brother_%26_Homeland.html
"Poesy the Monster Slayer" a picture book about monsters, bedtime, gender, and kicking ass. Order here: https://us.macmillan.com/books/9781626723627. Get a personalized, signed copy here: https://www.darkdel.com/store/p1562/_Poesy_the_Monster_Slayer.html.

Upcoming books:

Chokepoint Capitalism: How to Beat Big Tech, Tame Big Content, and Get Artists Paid, with Rebecca Giblin, nonfiction/business/politics, Beacon Press, September 2022

This work licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.