Pluralistic: Algorithmic feeds are a twiddler's playground (11 May 2024)


Today's links


A complex control panel whose knobs have all been replaced with the menacing red eye of HAL9000 from Kubrick's '2001: A Space Odyssey.' A skeletal figure on one side of the image reaches out a bony finger to twiddle one of the knobs.

Algorithmic feeds are a twiddler's playground (permalink)

Like Oscar Wilde, "I can resist anything except temptation," and my slow and halting journey to adulthood is really just me grappling with this fact, getting temptation out of my way before I can yield to it.

Behavioral economists have a name for the steps we take to guard against temptation: a "Ulysses pact." That's when you take some possibility off the table during a moment of strength in recognition of some coming moment of weakness:

https://archive.org/details/decentralizedwebsummit2016-corydoctorow

Famously, Ulysses did this before he sailed into the Sea of Sirens. Rather than stopping his ears with wax to prevent his hearing the sirens' song, which would lure him to his drowning, Ulysses has his sailors tie him to the mast, leaving his ears unplugged. Ulysses became the first person to hear the sirens' song and live to tell the tale.

Ulysses was strong enough to know that he would someday be weak. He expressed his strength by guarding against his weakness. Our modern lives are filled with less epic versions of the Ulysses pact: the day you go on a diet, it's a good idea to throw away all your Oreos. That way, when your blood sugar sings its siren song at 2AM, it will be drowned out by the rest of your body's unwillingness to get dressed, find your keys and drive half an hour to the all-night grocery store.

Note that this Ulysses pact isn't perfect. You might drive to the grocery store. It's rare that a Ulysses pact is unbreakable – we bind ourselves to the mast, but we don't chain ourselves to it and slap on a pair of handcuffs for good measure.

People who run institutions can – and should – create Ulysses pacts, too. A company that holds the kind of sensitive data that might be subjected to "sneak-and-peek" warrants by cops or spies can set up a "warrant canary":

https://en.wikipedia.org/wiki/Warrant_canary

This isn't perfect. A company that stops publishing regular transparency reports might have been compromised by the NSA, but it's also possible that they've had a change in management and the new boss just doesn't give a shit about his users' privacy:

https://www.fastcompany.com/90853794/twitters-transparency-reporting-has-tanked-under-elon-musk

Likewise, a company making software it wants users to trust can release that code under an irrevocable free/open software license, thus guaranteeing that each release under that license will be free and open forever. This is good, but not perfect: the new boss can take that free/open code down a proprietary fork and try to orphan the free version:

https://news.ycombinator.com/item?id=39772562

A company can structure itself as a public benefit corporation and make a binding promise to elevate its stakeholders' interests over its shareholders' – but the CEO can still take a secret $100m bribe from cryptocurrency creeps and try to lure those stakeholders into a shitcoin Ponzi scheme:

https://fortune.com/crypto/2024/03/11/kickstarter-blockchain-a16z-crypto-secret-investment-chris-dixon/

A key resource can be entrusted to a nonprofit with a board of directors who are charged with stewarding it for the benefit of a broad community, but when a private equity fund dangles billions before that board, they can talk themselves into a belief that selling out is the right thing to do:

https://www.eff.org/deeplinks/2020/12/how-we-saved-org-2020-review

Ulysses pacts aren't perfect, but they are very important. At the very least, creating a Ulysses pact starts with acknowledging that you are fallible. That you can be tempted, and rationalize your way into taking bad action, even when you know better. Becoming an adult is a process of learning that your strength comes from seeing your weaknesses and protecting yourself and the people who trust you from them.

Which brings me to enshittification. Enshittification is the process by which platforms betray their users and their customers by siphoning value away from each until the platform is a pile of shit:

https://en.wikipedia.org/wiki/Enshittification

Enshittification is a spectrum that can be applied to many companies' decay, but in its purest form, enshittification requires:

a) A platform: a two-sided market with business customers and end users who can be played off against each other;

b) A digital back-end: a market that can be easily, rapidly and undetectably manipulated by its owners, who can alter search-rankings, prices and costs on a per-user, per-query basis; and

c) A lack of constraint: the platform's owners must not fear a consequence for this cheating, be it from competitors, regulators, workforce resignations or rival technologists who use mods, alternative clients, blockers or other "adversarial interoperability" tools to disenshittify your product and sever your relationship with your users.

The founders of tech platforms don't generally set out to enshittify them. Rather, they are constantly seeking some equilibrium between delivering value to their shareholders and turning value over to end users, business customers, and their own workers. Founders are consummate rationalizers; like parenting, founding a company requires continuous, low-grade self-deception about the amount of work involved and the chances of success. A founder, confronted with the likelihood of failure, is absolutely capable of talking themselves into believing that nearly any compromise is superior to shuttering the business: "I'm one of the good guys, so the most important thing is for me to live to fight another day. Thus I can do any number of immoral things to my users, business customers or workers, because I can make it up to them when we survive this crisis. It's for their own good, even if they don't know it. Indeed, I'm doubly moral here, because I'm volunteering to look like the bad guy, just so I can save this business, which will make the world over for the better":

https://locusmag.com/2024/05/cory-doctorow-no-one-is-the-enshittifier-of-their-own-story/

(En)shit(tification) flows downhill, so tech workers grapple with their own version of this dilemma. Faced with constant pressure to increase the value flowing from their division to the company, they have to balance different, conflicting tactics, like "increasing the number of users or business customers, possibly by shifting value from the company to these stakeholders in the hopes of making it up in volume"; or "locking in my existing stakeholders and squeezing them harder, safe in the knowledge that they can't easily leave the service provided the abuse is subtle enough." The bigger a company gets, the harder it is for it to grow, so the biggest companies realize their gains by locking in and squeezing their users, not by improving their service::

https://pluralistic.net/2023/07/28/microincentives-and-enshittification/

That's where "twiddling" comes in. Digital platforms are extremely flexible, which comes with the territory: computers are the most flexible tools we have. This means that companies can automate high-speed, deceptive changes to the "business logic" of their platforms – what end users pay, how much of that goes to business customers, and how offers are presented to both:

https://pluralistic.net/2023/02/19/twiddler/

This kind of fraud isn't particularly sophisticated, but it doesn't have to be – it just has to be fast. In any shell-game, the quickness of the hand deceives the eye:

https://pluralistic.net/2024/03/26/glitchbread/#electronic-shelf-tags

Under normal circumstances, this twiddling would be constrained by counterforces in society. Changing the business rules like this is fraud, so you'd hope that a regulator would step in and extinguish the conduct, fining the company that engaged in it so hard that they saw a net loss from the conduct. But when a sector gets very concentrated, its mega-firms capture their regulators, becoming "too big to jail":

https://pluralistic.net/2022/06/05/regulatory-capture/

Thus the tendency among the giant tech companies to practice the one lesson of the Darth Vader MBA: dismissing your stakeholders' outrage by saying, "I am altering the deal. Pray I don't alter it any further":

https://pluralistic.net/2023/10/26/hit-with-a-brick/#graceful-failure

Where regulators fail, technology can step in. The flexibility of digital platforms cuts both ways: when the company enshittifies its products, you can disenshittify it with your own countertwiddling: third-party ink-cartridges, alternative app stores and clients, scrapers, browser automation and other forms of high-tech guerrilla warfare:

https://www.eff.org/deeplinks/2019/10/adversarial-interoperability

But tech giants' regulatory capture have allowed them to expand "IP rights" to prevent this self-help. By carefully layering overlapping IP rights around their products, they can criminalize the technology that lets you wrestle back the value they've claimed for themselves, creating a new offense of "felony contempt of business model":

https://locusmag.com/2020/09/cory-doctorow-ip/

A world where users must defer to platforms' moment-to-moment decisions about how the service operates, without the protection of rival technology or regulatory oversight is a world where companies face a powerful temptation to enshittify.

That's why we've seen so much enshittification in platforms that algorithmically rank their feeds, from Google and Amazon search to Facebook and Twitter feeds. A search engine is always going to be making a judgment call about what the best result for your search should be. If a search engine is generally good at predicting which results will please you best, you'll return to it, automatically clicking the first result ("I'm feeling lucky").

This means that if a search engine slips in the odd paid result at the top of the results, they can exploit your trusting habits to shift value from you to their investors. The congifurability of a digital service means that they can sprinkle these frauds into their services on a random schedule, making them hard to detect and easy to dismiss as lapses. Gradually, this acquires its own momentum, and the platform becomes addicted to lowering its own quality to raise its profits, and you get modern Google, which cynically lowered search quality to increase search volume:

https://pluralistic.net/2024/04/24/naming-names/#prabhakar-raghavan

And you get Amazon, which makes $38 billion every year, accepting bribes to replace its best search results with paid results for products that cost more and are of lower quality:

https://pluralistic.net/2023/11/06/attention-rents/#consumer-welfare-queens

Social media's enshittification followed a different path. In the beginning, social media presented a deterministic feed: after you told the platform who you wanted to follow, the platform simply gathered up the posts those users made and presented them to you, in reverse-chronological order.

This presented few opportunities for enshittification, but it wasn't perfect. For users who were well-established on a platform, a reverse-chrono feed was an ungovernable torrent, where high-frequency trivialities drowned out the important posts from people whose missives were buried ten screens down in the updates since your last login.

For new users who didn't yet follow many people, this presented the opposite problem: an empty feed, and the sense that you were all alone while everyone else was having a rollicking conversation down the hall, in a room you could never find.

The answer was the algorithmic feed: a feed of recommendations drawn from both the accounts you followed and strangers alike. Theoretically, this could solve both problems, by surfacing the most important materials from your friends while keeping you abreast of the most important and interesting activity beyond your filter bubble. For many of us, this promise was realized, and algorithmic feeds became a source of novelty and relevance.

But these feeds are a profoundly tempting enshittification target. The critique of these algorithms has largely focused on "addictiveness" and the idea that platforms would twiddle the knobs to increase the relevance of material in your feed to "hack your engagement":

https://www.theguardian.com/technology/2018/mar/04/has-dopamine-got-us-hooked-on-tech-facebook-apps-addiction

Less noticed – and more important – was how platforms did the opposite: twiddling the knobs to remove things from your feed that you'd asked to see or that the algorithm predicted you'd enjoy, to make room for "boosted" content and advertisements:

https://www.reddit.com/r/Instagram/comments/z9j7uy/what_happened_to_instagram_only_ads_and_accounts/

Users were helpless before this kind of twiddling. On the one hand, they were locked into the platform – not because their dopamine had been hacked by evil tech-bro wizards – but because they loved the friends they had there more than they hated the way the service was run:

https://locusmag.com/2023/01/commentary-cory-doctorow-social-quitting/

On the other hand, the platforms had such an iron grip on their technology, and had deployed IP so cleverly, that any countertwiddling technology was instantaneously incinerated by legal death-rays:

https://techcrunch.com/2022/10/10/google-removes-the-og-app-from-the-play-store-as-founders-think-about-next-steps/

Newer social media platforms, notably Tiktok, dispensed entirely with deterministic feeds, defaulting every user into a feed that consisted entirely of algorithmic picks; the people you follow on these platforms are treated as mere suggestions by their algorithms. This is a perfect breeding-ground for enshittification: different parts of the business can twiddle the knobs to override the algorithm for their own parochial purposes, shifting the quality:shit ratio by unnoticeable increments, temporarily toggling the quality knob when your engagement drops off:

https://www.forbes.com/sites/emilybaker-white/2023/01/20/tiktoks-secret-heating-button-can-make-anyone-go-viral/

All social platforms want to be Tiktok: nominally, that's because Tiktok's algorithmic feed is so good at hooking new users and keeping established users hooked. But tech bosses also understand that a purely algorithmic feed is the kind of black box that can be plausibly and subtly enshittified without sparking user revolts:

https://pluralistic.net/2023/01/21/potemkin-ai/#hey-guys

Back in 2004, when Mark Zuckerberg was coming to grips with Facebook's success, he boasted to a friend that he was sitting on a trove of emails, pictures and Social Security numbers for his fellow Harvard students, offering this up for his friend's idle snooping. The friend, surprised, asked "What? How'd you manage that one?"

Infamously, Zuck replied, "People just submitted it. I don't know why. They 'trust me.' Dumb fucks."

https://www.esquire.com/uk/latest-news/a19490586/mark-zuckerberg-called-people-who-handed-over-their-data-dumb-f/

This was a remarkable (and uncharacteristic) self-aware moment from the then-nineteen-year-old Zuck. Of course Zuck couldn't be trusted with that data. Whatever Jiminy Cricket voice told him to safeguard that trust was drowned out by his need to boast to pals, or participate in the creepy nonconsensual rating of the fuckability of their female classmates. Over and over again, Zuckerberg would promise to use his power wisely, then break that promise as soon as he could do so without consequence:

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3247362

Zuckerberg is a cautionary tale. Aware from the earliest moments that he was amassing power that he couldn't be trusted with, he nevertheless operated with only the weakest of Ulysses pacts, like a nonbinding promise never to spy on his users:

https://web.archive.org/web/20050107221705/http://www.thefacebook.com/policy.php

But the platforms have learned the wrong lesson from Zuckerberg. Rather than treating Facebook's enshittification as a cautionary tale, they've turned it into a roadmap. The Darth Vader MBA rules high-tech boardrooms.

Algorithmic feeds and other forms of "paternalistic" content presentation are necessary and even desirable in an information-rich environment. In many instances, decisions about what you see must be largely controlled by a third party whom you trust. The audience in a comedy club doesn't get to insist on knowing the punchline before the joke is told, just as RPG players don't get to order the Dungeon Master to present their preferred challenges during a campaign.

But this power is balanced against the ease of the players replacing the Dungeon Master or the audience walking out on the comic. When you've got more than a hundred dollars sunk into a video game and an online-only friend-group you raid with, the games company can do a lot of enshittification without losing your business, and they know it:

https://www.theverge.com/2024/5/10/24153809/ea-in-game-ads-redux

Even if they sometimes overreach and have to retreat:

https://www.eurogamer.net/sony-overturns-helldivers-2-psn-requirement-following-backlash

A tech company that seeks your trust for an algorithmic feed needs Ulysses pacts, or it will inevitably yield to the temptation to enshittify. From strongest to weakest, these are:

  • Not showing you an algorithmic feed at all;

https://joinmastodon.org/

  • "Composable moderation" that lets multiple parties provide feeds:

https://bsky.social/about/blog/4-13-2023-moderation

  • Offering an algorithmic "For You" feed alongside of a reverse-chrono "Friends" feed, defaulting to friends;

https://pluralistic.net/2022/12/10/e2e/#the-censors-pen

  • As above, but defaulting to "For You"

Maturity lies in being strong enough to know your weaknesses. Never trust someone who tells you that they will never yield to temptation! Instead, seek out people – and service providers – with the maturity and honesty to know how tempting temptation is, and who act before temptation strikes to make it easier to resist.

(Image: Cryteria, CC BY 3.0; djhughman, CC BY 2.0; modified)

Hey look at this (permalink)


A Wayback Machine banner.

This day in history (permalink)

#20yrsago Sony’s entertainment business is killing its electronics business https://memex.craphound.com/2004/05/10/sonys-entertainment-business-is-killing-its-electronics-business/

#20yrsago Pixel-counting can un-redact government docs https://cryptome.org/cia-decrypt.htm

#20yrsago MPAA’s Bizarro-world logic https://web.archive.org/web/20060508195757/http://www.foxnews.com/story/0,2933,119414,00.html

#20yrsago Stanislaw Lem is cranky! https://web.archive.org/web/20040513235656/http://www.mosnews.com/interview/2004/04/06/lem.shtml

#20yrsago Internet Archive’s Petabox: a 1,000 terabyte array https://archive.org/web/petabox.php

#15yrsago Pinkwater’s Neddiad: awesome YA novel with ghosts, fat alien cops, shamans, circus animals, triplanes, swordfighting, etc https://memex.craphound.com/2009/05/11/pinkwaters-neddiad-awesome-ya-novel-with-ghosts-fat-alien-cops-shamans-circus-animals-triplanes-swordfighting-etc/

#15yrsago Selling fiber broadband by inviting users to dig their own trenches https://arstechnica.com/tech-policy/2009/05/norwegian-isp-dig-your-own-fiber-trench-save-400/

#15yrsago Free ebooks’ effects on book-sales https://web.archive.org/web/20090515095615/http://bloggasm.com/did-random-houses-free-online-book-releases-affect-sales

#15yrsago Pirate Bay founder proposes to pay his fine with tiny, expensive-to-receive payments https://web.archive.org/web/20090514014403/http://www.blogpirate.org/2009/05/10/pirate-bay-founder-crafts-distributed-denial-of-dollars-attack/

#15yrsago Canadian MPs don’t want Parliament videos in the hands of citizens https://web.archive.org/web/20090512225709/http://www.thestar.com/sciencetech/article/632164

#15yrsago Cornell says no to restrictions on public domain materials https://web.archive.org/web/20090515100044/http://news.library.cornell.edu/com/news/PressReleases/Cornell-University-Library-Removes-All-Restrictions-on-Use-of-Public-Domain-Reproductions.cfm

#15yrsago Antifascist collages that made Hitler crazy https://web.archive.org/web/20090513034540/http://www.quazen.com/Arts/Visual-Arts/The-Extraordinary-Anti-Nazi-Photomontages-of-John-Heartfield.702053

#10yrsago Amazon patents taking pictures of stuff on a white background https://www.diyphotography.net/can-close-studio-amazon-patents-photographing-seamless-white/

#5yrsago Delta targets its workers with anti-union apps that push deceptive memes https://memex.craphound.com/2019/05/10/delta-targets-its-workers-with-anti-union-apps-that-push-deceptive-memes/

#5yrsago Ever, an “unlimited photo storage app,” secretly fed its users’ photos to a face-recognition system pitched to military customers https://www.nbcnews.com/tech/security/millions-people-uploaded-photos-ever-app-then-company-used-them-n1003371

#5yrsago A former college admissions dean explains the mundane reverse affirmative action that lets the rich send their kids to the front of the line https://www.vox.com/the-highlight/2019/5/1/18311548/college-admissions-secrets-myths

#5yrsago Sanders and AOC team up for an anti-loansharking bill that will replace payday lenders with post-office banking https://www.nakedcapitalism.com/2019/05/why-you-should-back-the-sanders-aoc-plan-to-cap-credit-card-interest-rates-at-15-re-launch-the-postal-savings-bank.html

#5yrsago Frontier receives $283.4m/year in taxpayer money, neglects network, rips off customers — and Trump’s FCC won’t investigate https://arstechnica.com/tech-policy/2019/05/ajit-pai-refuses-to-investigate-frontiers-horrible-telecom-service/

#5yrsago Google mistakenly handed out a reporter’s cellphone number to people searching for Facebook tech support https://www.vice.com/en/article/zmpm43/google-thought-my-phone-number-was-facebooks-and-it-ruined-my-life

#5yrsago After elderly tenant was locked in his apartment by his landlord’s stupid “smart lock,” tenants win right to use actual keys to enter their homes https://www.cnet.com/home/smart-home/tenants-win-rights-to-physical-keys-over-smart-locks-from-landlords/

#5yrsago Co-founder of Facebook calls for breakup of Facebook https://www.nytimes.com/2019/05/09/opinion/sunday/chris-hughes-facebook-zuckerberg.html

#5yrsago Bipartisan groups call on Congress to reinstate the Office of Technology Assessment, which Gingrich killed in 1995 https://www.techdirt.com/2019/05/10/broad-coalition-tells-congress-to-bring-back-office-technology-assessment/

#5yrsago Beto O’Rourke just hired a “senior advisor” who used to lobby for Keystone XL, Seaworld and private prisons https://theintercept.com/2019/05/11/beto-orourke-campaign-staff-lobbyist-keystone-xl/

#5yrsago Facebook’s “celebration” and “memories” algorithms are auto-generating best-of-terror-recruiting pages for extremist groups https://www.securityweek.com/whistleblower-says-facebook-generating-terror-content/

#5yrsago Chelsea Manning’s statement on the occasion of her release https://www.youtube.com/watch?v=TDZGRRk4MnM

#1yrago 'We buy ugly houses' is code for 'we steal vulnerable peoples' homes' https://pluralistic.net/2023/05/11/ugly-houses-ugly-truth/#homevestor

#1yrago Two principles to protect internet users from decaying platforms https://pluralistic.net/2023/05/10/soft-landings/#e2e-r2e

Upcoming appearances (permalink)

A photo of me onstage, giving a speech, holding a mic.


A screenshot of me at my desk, doing a livecast.

Recent appearances (permalink)


A grid of my books with Will Stahle covers..

Latest books (permalink)


A cardboard book box with the Macmillan logo.

Upcoming books (permalink)

  • Picks and Shovels: a sequel to "Red Team Blues," about the heroic era of the PC, Tor Books, February 2025

  • Unauthorized Bread: a graphic novel adapted from my novella about refugees, toasters and DRM, FirstSecond, 2025


Colophon (permalink)

Today's top sources:

Currently writing:

  • A Little Brother short story about DIY insulin PLANNING

  • Picks and Shovels, a Martin Hench noir thriller about the heroic era of the PC. FORTHCOMING TOR BOOKS JAN 2025

  • Vigilant, Little Brother short story about remote invigilation. FORTHCOMING ON TOR.COM

  • Spill, a Little Brother short story about pipeline protests. FORTHCOMING ON TOR.COM

Latest podcast: Precaratize Bosses https://craphound.com/news/2024/04/28/precaratize-bosses/

This work – excluding any serialized fiction – is licensed under a Creative Commons Attribution 4.0 license. That means you can use it any way you like, including commercially, provided that you attribute it to me, Cory Doctorow, and include a link to pluralistic.net.

https://creativecommons.org/licenses/by/4.0/

Quotations and images are not included in this license; they are included either under a limitation or exception to copyright, or on the basis of a separate license. Please exercise caution.

How to get Pluralistic:

Blog (no ads, tracking, or data-collection):

Pluralistic.net

Newsletter (no ads, tracking, or data-collection):

https://pluralistic.net/plura-list

Mastodon (no ads, tracking, or data-collection):

https://mamot.fr/@pluralistic

Medium (no ads, paywalled):

https://doctorow.medium.com/

Twitter (mass-scale, unrestricted, third-party surveillance and advertising):

https://twitter.com/doctorow

Tumblr (mass-scale, unrestricted, third-party surveillance and advertising):

https://mostlysignssomeportents.tumblr.com/tagged/pluralistic

"When life gives you SARS, you make sarsaparilla" -Joey "Accordion Guy" DeVilla

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.