+++ /dev/null
-<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><title>How to Destroy Surveillance Capitalism</title><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><style type="text/css">
-body { background-image: url('images/draft.png');
- background-repeat: no-repeat;
- background-position: top left;
- /* The following properties make the watermark "fixed" on the page. */
- /* I think that's just a bit too distracting for the reader... */
- /* background-attachment: fixed; */
- /* background-position: center center; */
- }</style></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div lang="en" class="article"><div class="titlepage"><div><div><h2 class="title"><a name="index"></a>How to Destroy Surveillance Capitalism</h2></div><div><div class="authorgroup"><div class="author"><h3 class="author"><span class="firstname">Cory</span> <span class="surname">Doctorow</span></h3></div></div></div><div><p class="copyright">Copyright © 2020 Cory Doctorow</p></div><div><p class="copyright">Copyright © 2020 Petter Reinholdtsen</p></div><div><div class="legalnotice"><a name="idm18"></a><p>
- How to Destroy Surveillance Capitalism by Cory Doctorow.
- </p><p>
- Published by Petter Reinholdtsen.
- </p><p>
- ISBN 978-82-93828-XX-X (hard cover)
- </p><p>
- ISBN 978-82-93828-XX-X (paperback)
- </p><p>
- ISBN 978-82-93828-XX-X (ePub)
- </p><p>
- This book is available for purchase from
- <a class="ulink" href="https://www.lulu.com/" target="_top">https://www.lulu.com/</a>.
- </p><p>
- <span class="inlinemediaobject"><img src="images/cc.png" align="middle" height="38" alt="Creative Commons, Some rights reserved"></span>
- </p><p>
- This book is licensed under a Creative Commons license. This
- license permits any use of this work, so long as attribution is
- given and no derivatived material is distributed. For more
- information about the license visit <a class="ulink" href="https://creativecommons.org/licenses/by-nd/4.0/" target="_top">https://creativecommons.org/licenses/by-nd/4.0/</a>.
- </p></div></div></div><hr></div><div class="toc"><p><b>Table of Contents</b></p><dl class="toc"><dt><span class="sect1"><a href="#the-net-of-a-thousand-lies">The net of a thousand lies</a></span></dt><dt><span class="sect1"><a href="#digital-rights-activism-a-quarter-century-on">Digital rights activism, a quarter-century on</a></span></dt><dt><span class="sect1"><a href="#tech-exceptionalism-then-and-now">Tech exceptionalism, then and now</a></span></dt><dt><span class="sect1"><a href="#dont-believe-the-hype">Don’t believe the hype</a></span></dt><dt><span class="sect1"><a href="#what-is-persuasion">What is persuasion?</a></span></dt><dd><dl><dt><span class="sect2"><a href="#segmenting">1. Segmenting</a></span></dt><dt><span class="sect2"><a href="#deception">2. Deception</a></span></dt><dt><span class="sect2"><a href="#domination">3. Domination</a></span></dt><dt><span class="sect2"><a href="#bypassing-our-rational-faculties">4. Bypassing our rational faculties</a></span></dt></dl></dd><dt><span class="sect1"><a href="#if-data-is-the-new-oil-then-surveillance-capitalisms-engine-has-a-leak">If data is the new oil, then surveillance capitalism’s engine
- has a leak</a></span></dt><dt><span class="sect1"><a href="#what-is-facebook">What is Facebook?</a></span></dt><dt><span class="sect1"><a href="#monopoly-and-the-right-to-the-future-tense">Monopoly and the right to the future tense</a></span></dt><dt><span class="sect1"><a href="#search-order-and-the-right-to-the-future-tense">Search order and the right to the future tense</a></span></dt><dt><span class="sect1"><a href="#monopolists-can-afford-sleeping-pills-for-watchdogs">Monopolists can afford sleeping pills for watchdogs</a></span></dt><dt><span class="sect1"><a href="#privacy-and-monopoly">Privacy and monopoly</a></span></dt><dt><span class="sect1"><a href="#ronald-reagan-pioneer-of-tech-monopolism">Ronald Reagan, pioneer of tech monopolism</a></span></dt><dt><span class="sect1"><a href="#steering-with-the-windshield-wipers">Steering with the windshield wipers</a></span></dt><dt><span class="sect1"><a href="#surveillance-still-matters">Surveillance still matters</a></span></dt><dt><span class="sect1"><a href="#dignity-and-sanctuary">Dignity and sanctuary</a></span></dt><dt><span class="sect1"><a href="#afflicting-the-afflicted">Afflicting the afflicted</a></span></dt><dt><span class="sect1"><a href="#any-data-you-collect-and-retain-will-eventually-leak">Any data you collect and retain will eventually leak</a></span></dt><dt><span class="sect1"><a href="#critical-tech-exceptionalism-is-still-tech-exceptionalism">Critical tech exceptionalism is still tech
- exceptionalism</a></span></dt><dt><span class="sect1"><a href="#how-monopolies-not-mind-control-drive-surveillance-capitalism-the-snapchat-story">How monopolies, not mind control, drive surveillance
- capitalism: The Snapchat story</a></span></dt><dt><span class="sect1"><a href="#a-monopoly-over-your-friends">A monopoly over your friends</a></span></dt><dt><span class="sect1"><a href="#fake-news-is-an-epistemological-crisis">Fake news is an epistemological crisis</a></span></dt><dt><span class="sect1"><a href="#tech-is-different">Tech is different</a></span></dt><dt><span class="sect1"><a href="#ownership-of-facts">Ownership of facts</a></span></dt><dt><span class="sect1"><a href="#persuasion-works-slowly">Persuasion works… slowly</a></span></dt><dt><span class="sect1"><a href="#paying-wont-help">Paying won’t help</a></span></dt><dt><span class="sect1"><a href="#an-ecology-moment-for-trustbusting">An <span class="quote">“<span class="quote">ecology</span>”</span> moment for trustbusting</a></span></dt><dt><span class="sect1"><a href="#make-big-tech-small-again">Make Big Tech small again</a></span></dt><dt><span class="sect1"><a href="#goto-10">20 GOTO 10</a></span></dt><dt><span class="sect1"><a href="#up-and-through">Up and through</a></span></dt></dl></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="the-net-of-a-thousand-lies"></a>The net of a thousand lies</h2></div></div></div><p>
- The most surprising thing about the rebirth of flat Earthers in the
- 21st century is just how widespread the evidence against them is.
- You can understand how, centuries ago, people who’d never gained a
- high-enough vantage point from which to see the Earth’s curvature
- might come to the commonsense belief that the flat-seeming Earth
- was, indeed, flat.
- </p><p>
- But today, when elementary schools routinely dangle GoPro cameras
- from balloons and loft them high enough to photograph the Earth’s
- curve — to say nothing of the unexceptional sight of the curved
- Earth from an airplane window — it takes a heroic effort to maintain
- the belief that the world is flat.
- </p><p>
- Likewise for white nationalism and eugenics: In an age where you can
- become a computational genomics datapoint by swabbing your cheek and
- mailing it to a gene-sequencing company along with a modest sum of
- money, <span class="quote">“<span class="quote">race science</span>”</span> has never been easier to refute.
- </p><p>
- We are living through a golden age of both readily available facts
- and denial of those facts. Terrible ideas that have lingered on the
- fringes for decades or even centuries have gone mainstream seemingly
- overnight.
- </p><p>
- When an obscure idea gains currency, there are only two things that
- can explain its ascendance: Either the person expressing that idea
- has gotten a lot better at stating their case, or the proposition
- has become harder to deny in the face of mounting evidence. In other
- words, if we want people to take climate change seriously, we can
- get a bunch of Greta Thunbergs to make eloquent, passionate
- arguments from podiums, winning our hearts and minds, or we can wait
- for flood, fire, broiling sun, and pandemics to make the case for
- us. In practice, we’ll probably have to do some of both: The more
- we’re boiling and burning and drowning and wasting away, the easier
- it will be for the Greta Thunbergs of the world to convince us.
- </p><p>
- The arguments for ridiculous beliefs in odious conspiracies like
- anti-vaccination, climate denial, a flat Earth, and eugenics are no
- better than they were a generation ago. Indeed, they’re worse
- because they are being pitched to people who have at least a
- background awareness of the refuting facts.
- </p><p>
- Anti-vax has been around since the first vaccines, but the early
- anti-vaxxers were pitching people who were less equipped to
- understand even the most basic ideas from microbiology, and
- moreover, those people had not witnessed the extermination of
- mass-murdering diseases like polio, smallpox, and measles. Today’s
- anti-vaxxers are no more eloquent than their forebears, and they
- have a much harder job.
- </p><p>
- So can these far-fetched conspiracy theorists really be succeeding
- on the basis of superior arguments?
- </p><p>
- Some people think so. Today, there is a widespread belief that
- machine learning and commercial surveillance can turn even the most
- fumble-tongued conspiracy theorist into a svengali who can warp your
- perceptions and win your belief by locating vulnerable people and
- then pitching them with A.I.-refined arguments that bypass their
- rational faculties and turn everyday people into flat Earthers,
- anti-vaxxers, or even Nazis. When the RAND Corporation
- <a class="ulink" href="https://www.rand.org/content/dam/rand/pubs/research_reports/RR400/RR453/RAND_RR453.pdf" target="_top">blames
- Facebook for <span class="quote">“<span class="quote">radicalization</span>”</span></a> and when Facebook’s role in
- spreading coronavirus misinformation is
- <a class="ulink" href="https://secure.avaaz.org/campaign/en/facebook_threat_health/" target="_top">blamed
- on its algorithm</a>, the implicit message is that machine
- learning and surveillance are causing the changes in our consensus
- about what’s true.
- </p><p>
- After all, in a world where sprawling and incoherent conspiracy
- theories like Pizzagate and its successor, QAnon, have widespread
- followings, <span class="emphasis"><em>something</em></span> must be afoot.
- </p><p>
- But what if there’s another explanation? What if it’s the material
- circumstances, and not the arguments, that are making the difference
- for these conspiracy pitchmen? What if the trauma of living through
- <span class="emphasis"><em>real conspiracies</em></span> all around us — conspiracies
- among wealthy people, their lobbyists, and lawmakers to bury
- inconvenient facts and evidence of wrongdoing (these conspiracies
- are commonly known as <span class="quote">“<span class="quote">corruption</span>”</span>) — is making people vulnerable to
- conspiracy theories?
- </p><p>
- If it’s trauma and not contagion — material conditions and not
- ideology — that is making the difference today and enabling a rise
- of repulsive misinformation in the face of easily observed facts,
- that doesn’t mean our computer networks are blameless. They’re still
- doing the heavy work of locating vulnerable people and guiding them
- through a series of ever-more-extreme ideas and communities.
- </p><p>
- Belief in conspiracy is a raging fire that has done real damage and
- poses real danger to our planet and species, from epidemics
- <a class="ulink" href="https://www.cdc.gov/measles/cases-outbreaks.html" target="_top">kicked
- off by vaccine denial</a> to genocides
- <a class="ulink" href="https://www.nytimes.com/2018/10/15/technology/myanmar-facebook-genocide.html" target="_top">kicked
- off by racist conspiracies</a> to planetary meltdown caused by
- denial-inspired climate inaction. Our world is on fire, and so we
- have to put the fires out — to figure out how to help people see the
- truth of the world through the conspiracies they’ve been confused
- by.
- </p><p>
- But firefighting is reactive. We need fire
- <span class="emphasis"><em>prevention</em></span>. We need to strike at the traumatic
- material conditions that make people vulnerable to the contagion of
- conspiracy. Here, too, tech has a role to play.
- </p><p>
- There’s no shortage of proposals to address this. From the EU’s
- <a class="ulink" href="https://edri.org/tag/terreg/" target="_top">Terrorist Content
- Regulation</a>, which requires platforms to police and remove
- <span class="quote">“<span class="quote">extremist</span>”</span> content, to the U.S. proposals to
- <a class="ulink" href="https://www.eff.org/deeplinks/2020/03/earn-it-act-violates-constitution" target="_top">force
- tech companies to spy on their users</a> and hold them liable
- <a class="ulink" href="https://www.natlawreview.com/article/repeal-cda-section-230" target="_top">for
- their users’ bad speech</a>, there’s a lot of energy to force
- tech companies to solve the problems they created.
- </p><p>
- There’s a critical piece missing from the debate, though. All these
- solutions assume that tech companies are a fixture, that their
- dominance over the internet is a permanent fact. Proposals to
- replace Big Tech with a more diffused, pluralistic internet are
- nowhere to be found. Worse: The <span class="quote">“<span class="quote">solutions</span>”</span> on the table today
- <span class="emphasis"><em>require</em></span> Big Tech to stay big because only the
- very largest companies can afford to implement the systems these
- laws demand.
- </p><p>
- Figuring out what we want our tech to look like is crucial if we’re
- going to get out of this mess. Today, we’re at a crossroads where
- we’re trying to figure out if we want to fix the Big Tech companies
- that dominate our internet or if we want to fix the internet itself
- by unshackling it from Big Tech’s stranglehold. We can’t do both, so
- we have to choose.
- </p><p>
- I want us to choose wisely. Taming Big Tech is integral to fixing
- the internet, and for that, we need digital rights activism.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="digital-rights-activism-a-quarter-century-on"></a>Digital rights activism, a quarter-century on</h2></div></div></div><p>
- Digital rights activism is more than 30 years old now. The
- Electronic Frontier Foundation turned 30 this year; the Free
- Software Foundation launched in 1985. For most of the history of the
- movement, the most prominent criticism leveled against it was that
- it was irrelevant: The real activist causes were real-world causes
- (think of the skepticism when
- <a class="ulink" href="https://www.loc.gov/law/foreign-news/article/finland-legal-right-to-broadband-for-all-citizens/#:~:text=Global%20Legal%20Monitor,-Home%20%7C%20Search%20%7C%20Browse&text=(July%206%2C%202010)%20On,connection%20100%20MBPS%20by%202015." target="_top">Finland
- declared broadband a human right in 2010</a>), and real-world
- activism was shoe-leather activism (think of Malcolm Gladwell’s
- <a class="ulink" href="https://www.newyorker.com/magazine/2010/10/04/small-change-malcolm-gladwell" target="_top">contempt
- for <span class="quote">“<span class="quote">clicktivism</span>”</span></a>). But as tech has grown more central to
- our daily lives, these accusations of irrelevance have given way
- first to accusations of insincerity (<span class="quote">“<span class="quote">You only care about tech
- because you’re
- <a class="ulink" href="https://www.ipwatchdog.com/2018/06/04/report-engine-eff-shills-google-patent-reform/id=98007/" target="_top">shilling
- for tech companies</a></span>”</span>) to accusations of negligence (<span class="quote">“<span class="quote">Why
- didn’t you foresee that tech could be such a destructive force?</span>”</span>).
- But digital rights activism is right where it’s always been: looking
- out for the humans in a world where tech is inexorably taking over.
- </p><p>
- The latest version of this critique comes in the form of
- <span class="quote">“<span class="quote">surveillance capitalism,</span>”</span> a term coined by business professor
- Shoshana Zuboff in her long and influential 2019 book, <span class="emphasis"><em>The
- Age of Surveillance Capitalism: The Fight for a Human Future at the
- New Frontier of Power</em></span>. Zuboff argues that <span class="quote">“<span class="quote">surveillance
- capitalism</span>”</span> is a unique creature of the tech industry and that it is
- unlike any other abusive commercial practice in history, one that is
- <span class="quote">“<span class="quote">constituted by unexpected and often illegible mechanisms of
- extraction, commodification, and control that effectively exile
- persons from their own behavior while producing new markets of
- behavioral prediction and modification. Surveillance capitalism
- challenges democratic norms and departs in key ways from the
- centuries-long evolution of market capitalism.</span>”</span> It is a new and
- deadly form of capitalism, a <span class="quote">“<span class="quote">rogue capitalism,</span>”</span> and our lack of
- understanding of its unique capabilities and dangers represents an
- existential, species-wide threat. She’s right that capitalism today
- threatens our species, and she’s right that tech poses unique
- challenges to our species and civilization, but she’s really wrong
- about how tech is different and why it threatens our species.
- </p><p>
- What’s more, I think that her incorrect diagnosis will lead us down
- a path that ends up making Big Tech stronger, not weaker. We need to
- take down Big Tech, and to do that, we need to start by correctly
- identifying the problem.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tech-exceptionalism-then-and-now"></a>Tech exceptionalism, then and now</h2></div></div></div><p>
- Early critics of the digital rights movement — perhaps best
- represented by campaigning organizations like the Electronic
- Frontier Foundation, the Free Software Foundation, Public Knowledge,
- and others that focused on preserving and enhancing basic human
- rights in the digital realm — damned activists for practicing <span class="quote">“<span class="quote">tech
- exceptionalism.</span>”</span> Around the turn of the millennium, serious people
- ridiculed any claim that tech policy mattered in the <span class="quote">“<span class="quote">real world.</span>”</span>
- Claims that tech rules had implications for speech, association,
- privacy, search and seizure, and fundamental rights and equities
- were treated as ridiculous, an elevation of the concerns of sad
- nerds arguing about <span class="emphasis"><em>Star Trek</em></span> on bulletin board
- systems above the struggles of the Freedom Riders, Nelson Mandela,
- or the Warsaw ghetto uprising.
- </p><p>
- In the decades since, accusations of <span class="quote">“<span class="quote">tech exceptionalism</span>”</span> have only
- sharpened as tech’s role in everyday life has expanded: Now that
- tech has infiltrated every corner of our life and our online lives
- have been monopolized by a handful of giants, defenders of digital
- freedoms are accused of carrying water for Big Tech, providing cover
- for its self-interested negligence (or worse, nefarious plots).
- </p><p>
- From my perspective, the digital rights movement has remained
- stationary while the rest of the world has moved. From the earliest
- days, the movement’s concern was users and the toolsmiths who
- provided the code they needed to realize their fundamental rights.
- Digital rights activists only cared about companies to the extent
- that companies were acting to uphold users’ rights (or, just as
- often, when companies were acting so foolishly that they threatened
- to bring down new rules that would also make it harder for good
- actors to help users).
- </p><p>
- The <span class="quote">“<span class="quote">surveillance capitalism</span>”</span> critique recasts the digital rights
- movement in a new light again: not as alarmists who overestimate the
- importance of their shiny toys nor as shills for big tech but as
- serene deck-chair rearrangers whose long-standing activism is a
- liability because it makes them incapable of perceiving novel
- threats as they continue to fight the last century’s tech battles.
- </p><p>
- But tech exceptionalism is a sin no matter who practices it.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="dont-believe-the-hype"></a>Don’t believe the hype</h2></div></div></div><p>
- You’ve probably heard that <span class="quote">“<span class="quote">if you’re not paying for the product,
- you’re the product.</span>”</span> As we’ll see below, that’s true, if incomplete.
- But what is <span class="emphasis"><em>absolutely</em></span> true is that ad-driven
- Big Tech’s customers are advertisers, and what companies like Google
- and Facebook sell is their ability to convince
- <span class="emphasis"><em>you</em></span> to buy stuff. Big Tech’s product is
- persuasion. The services — social media, search engines, maps,
- messaging, and more — are delivery systems for persuasion.
- </p><p>
- The fear of surveillance capitalism starts from the (correct)
- presumption that everything Big Tech says about itself is probably a
- lie. But the surveillance capitalism critique makes an exception for
- the claims Big Tech makes in its sales literature — the breathless
- hype in the pitches to potential advertisers online and in ad-tech
- seminars about the efficacy of its products: It assumes that Big
- Tech is as good at influencing us as they claim they are when
- they’re selling influencing products to credulous customers. That’s
- a mistake because sales literature is not a reliable indicator of a
- product’s efficacy.
- </p><p>
- Surveillance capitalism assumes that because advertisers buy a lot
- of what Big Tech is selling, Big Tech must be selling something
- real. But Big Tech’s massive sales could just as easily be the
- result of a popular delusion or something even more pernicious:
- monopolistic control over our communications and commerce.
- </p><p>
- Being watched changes your behavior, and not for the better. It
- creates risks for our social progress. Zuboff’s book features
- beautifully wrought explanations of these phenomena. But Zuboff also
- claims that surveillance literally robs us of our free will — that
- when our personal data is mixed with machine learning, it creates a
- system of persuasion so devastating that we are helpless before it.
- That is, Facebook uses an algorithm to analyze the data it
- nonconsensually extracts from your daily life and uses it to
- customize your feed in ways that get you to buy stuff. It is a
- mind-control ray out of a 1950s comic book, wielded by mad
- scientists whose supercomputers guarantee them perpetual and total
- world domination.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="what-is-persuasion"></a>What is persuasion?</h2></div></div></div><p>
- To understand why you shouldn’t worry about mind-control rays — but
- why you <span class="emphasis"><em>should</em></span> worry about surveillance
- <span class="emphasis"><em>and</em></span> Big Tech — we must start by unpacking what
- we mean by <span class="quote">“<span class="quote">persuasion.</span>”</span>
- </p><p>
- Google, Facebook, and other surveillance capitalists promise their
- customers (the advertisers) that if they use machine-learning tools
- trained on unimaginably large data sets of nonconsensually harvested
- personal information, they will be able to uncover ways to bypass
- the rational faculties of the public and direct their behavior,
- creating a stream of purchases, votes, and other desired outcomes.
- </p><div class="blockquote"><blockquote class="blockquote"><p>
- The impact of dominance far exceeds the impact of manipulation and
- should be central to our analysis and any remedies we seek.
- </p></blockquote></div><p>
- But there’s little evidence that this is happening. Instead, the
- predictions that surveillance capitalism delivers to its customers
- are much less impressive. Rather than finding ways to bypass our
- rational faculties, surveillance capitalists like Mark Zuckerberg
- mostly do one or more of three things:
- </p><div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="segmenting"></a>1. Segmenting</h3></div></div></div><p>
- If you’re selling diapers, you have better luck if you pitch them
- to people in maternity wards. Not everyone who enters or leaves a
- maternity ward just had a baby, and not everyone who just had a
- baby is in the market for diapers. But having a baby is a really
- reliable correlate of being in the market for diapers, and being
- in a maternity ward is highly correlated with having a baby. Hence
- diaper ads around maternity wards (and even pitchmen for baby
- products, who haunt maternity wards with baskets full of
- freebies).
- </p><p>
- Surveillance capitalism is segmenting times a billion. Diaper
- vendors can go way beyond people in maternity wards (though they
- can do that, too, with things like location-based mobile ads).
- They can target you based on whether you’re reading articles about
- child-rearing, diapers, or a host of other subjects, and data
- mining can suggest unobvious keywords to advertise against. They
- can target you based on the articles you’ve recently read. They
- can target you based on what you’ve recently purchased. They can
- target you based on whether you receive emails or private messages
- about these subjects — or even if you speak aloud about them
- (though Facebook and the like convincingly claim that’s not
- happening — yet).
- </p><p>
- This is seriously creepy.
- </p><p>
- But it’s not mind control.
- </p><p>
- It doesn’t deprive you of your free will. It doesn’t trick you.
- </p><p>
- Think of how surveillance capitalism works in politics.
- Surveillance capitalist companies sell political operatives the
- power to locate people who might be receptive to their pitch.
- Candidates campaigning on finance industry corruption seek people
- struggling with debt; candidates campaigning on xenophobia seek
- out racists. Political operatives have always targeted their
- message whether their intentions were honorable or not: Union
- organizers set up pitches at factory gates, and white supremacists
- hand out fliers at John Birch Society meetings.
- </p><p>
- But this is an inexact and thus wasteful practice. The union
- organizer can’t know which worker to approach on the way out of
- the factory gates and may waste their time on a covert John Birch
- Society member; the white supremacist doesn’t know which of the
- Birchers are so delusional that making it to a meeting is as much
- as they can manage and which ones might be convinced to cross the
- country to carry a tiki torch through the streets of
- Charlottesville, Virginia.
- </p><p>
- Because targeting improves the yields on political pitches, it can
- accelerate the pace of political upheaval by making it possible
- for everyone who has secretly wished for the toppling of an
- autocrat — or just an 11-term incumbent politician — to find
- everyone else who feels the same way at very low cost. This has
- been critical to the rapid crystallization of recent political
- movements including Black Lives Matter and Occupy Wall Street as
- well as less savory players like the far-right white nationalist
- movements that marched in Charlottesville.
- </p><p>
- It’s important to differentiate this kind of political organizing
- from influence campaigns; finding people who secretly agree with
- you isn’t the same as convincing people to agree with you. The
- rise of phenomena like nonbinary or otherwise nonconforming gender
- identities is often characterized by reactionaries as the result
- of online brainwashing campaigns that convince impressionable
- people that they have been secretly queer all along.
- </p><p>
- But the personal accounts of those who have come out tell a
- different story where people who long harbored a secret about
- their gender were emboldened by others coming forward and where
- people who knew that they were different but lacked a vocabulary
- for discussing that difference learned the right words from these
- low-cost means of finding people and learning about their ideas.
- </p></div><div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="deception"></a>2. Deception</h3></div></div></div><p>
- Lies and fraud are pernicious, and surveillance capitalism
- supercharges them through targeting. If you want to sell a
- fraudulent payday loan or subprime mortgage, surveillance
- capitalism can help you find people who are both desperate and
- unsophisticated and thus receptive to your pitch. This accounts
- for the rise of many phenomena, like multilevel marketing schemes,
- in which deceptive claims about potential earnings and the
- efficacy of sales techniques are targeted at desperate people by
- advertising against search queries that indicate, for example,
- someone struggling with ill-advised loans.
- </p><p>
- Surveillance capitalism also abets fraud by making it easy to
- locate other people who have been similarly deceived, forming a
- community of people who reinforce one another’s false beliefs.
- Think of
- <a class="ulink" href="https://www.vulture.com/2020/01/the-dream-podcast-review.html" target="_top">the
- forums</a> where people who are being victimized by multilevel
- marketing frauds gather to trade tips on how to improve their luck
- in peddling the product.
- </p><p>
- Sometimes, online deception involves replacing someone’s correct
- beliefs with incorrect ones, as it does in the anti-vaccination
- movement, whose victims are often people who start out believing
- in vaccines but are convinced by seemingly plausible evidence that
- leads them into the false belief that vaccines are harmful.
- </p><p>
- But it’s much more common for fraud to succeed when it doesn’t
- have to displace a true belief. When my daughter contracted head
- lice at daycare, one of the daycare workers told me I could get
- rid of them by treating her hair and scalp with olive oil. I
- didn’t know anything about head lice, and I assumed that the
- daycare worker did, so I tried it (it didn’t work, and it doesn’t
- work). It’s easy to end up with false beliefs when you simply
- don’t know any better and when those beliefs are conveyed by
- someone who seems to know what they’re doing.
- </p><p>
- This is pernicious and difficult — and it’s also the kind of thing
- the internet can help guard against by making true information
- available, especially in a form that exposes the underlying
- deliberations among parties with sharply divergent views, such as
- Wikipedia. But it’s not brainwashing; it’s fraud. In the
- <a class="ulink" href="https://datasociety.net/library/data-voids/" target="_top">majority
- of cases</a>, the victims of these fraud campaigns have an
- informational void filled in the customary way, by consulting a
- seemingly reliable source. If I look up the length of the Brooklyn
- Bridge and learn that it is 5,800 feet long, but in reality, it is
- 5,989 feet long, the underlying deception is a problem, but it’s a
- problem with a simple remedy. It’s a very different problem from
- the anti-vax issue in which someone’s true belief is displaced by
- a false one by means of sophisticated persuasion.
- </p></div><div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="domination"></a>3. Domination</h3></div></div></div><p>
- Surveillance capitalism is the result of monopoly. Monopoly is the
- cause, and surveillance capitalism and its negative outcomes are
- the effects of monopoly. I’ll get into this in depth later, but
- for now, suffice it to say that the tech industry has grown up
- with a radical theory of antitrust that has allowed companies to
- grow by merging with their rivals, buying up their nascent
- competitors, and expanding to control whole market verticals.
- </p><p>
- One example of how monopolism aids in persuasion is through
- dominance: Google makes editorial decisions about its algorithms
- that determine the sort order of the responses to our queries. If
- a cabal of fraudsters have set out to trick the world into
- thinking that the Brooklyn Bridge is 5,800 feet long, and if
- Google gives a high search rank to this group in response to
- queries like <span class="quote">“<span class="quote">How long is the Brooklyn Bridge?</span>”</span> then the first
- eight or 10 screens’ worth of Google results could be wrong. And
- since most people don’t go beyond the first couple of results —
- let alone the first <span class="emphasis"><em>page</em></span> of results —
- Google’s choice means that many people will be deceived.
- </p><p>
- Google’s dominance over search — more than 86% of web searches are
- performed through Google — means that the way it orders its search
- results has an outsized effect on public beliefs. Ironically,
- Google claims this is why it can’t afford to have any transparency
- in its algorithm design: Google’s search dominance makes the
- results of its sorting too important to risk telling the world how
- it arrives at those results lest some bad actor discover a flaw in
- the ranking system and exploit it to push its point of view to the
- top of the search results. There’s an obvious remedy to a company
- that is too big to audit: break it up into smaller pieces.
- </p><p>
- Zuboff calls surveillance capitalism a <span class="quote">“<span class="quote">rogue capitalism</span>”</span> whose
- data-hoarding and machine-learning techniques rob us of our free
- will. But influence campaigns that seek to displace existing,
- correct beliefs with false ones have an effect that is small and
- temporary while monopolistic dominance over informational systems
- has massive, enduring effects. Controlling the results to the
- world’s search queries means controlling access both to arguments
- and their rebuttals and, thus, control over much of the world’s
- beliefs. If our concern is how corporations are foreclosing on our
- ability to make up our own minds and determine our own futures,
- the impact of dominance far exceeds the impact of manipulation and
- should be central to our analysis and any remedies we seek.
- </p></div><div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="bypassing-our-rational-faculties"></a>4. Bypassing our rational faculties</h3></div></div></div><p>
- <span class="emphasis"><em>This</em></span> is the good stuff: using machine
- learning, <span class="quote">“<span class="quote">dark patterns,</span>”</span> engagement hacking, and other
- techniques to get us to do things that run counter to our better
- judgment. This is mind control.
- </p><p>
- Some of these techniques have proven devastatingly effective (if
- only in the short term). The use of countdown timers on a purchase
- completion page can create a sense of urgency that causes you to
- ignore the nagging internal voice suggesting that you should shop
- around or sleep on your decision. The use of people from your
- social graph in ads can provide <span class="quote">“<span class="quote">social proof</span>”</span> that a purchase is
- worth making. Even the auction system pioneered by eBay is
- calculated to play on our cognitive blind spots, letting us feel
- like we <span class="quote">“<span class="quote">own</span>”</span> something because we bid on it, thus encouraging us
- to bid again when we are outbid to ensure that <span class="quote">“<span class="quote">our</span>”</span> things stay
- ours.
- </p><p>
- Games are extraordinarily good at this. <span class="quote">“<span class="quote">Free to play</span>”</span> games
- manipulate us through many techniques, such as presenting players
- with a series of smoothly escalating challenges that create a
- sense of mastery and accomplishment but which sharply transition
- into a set of challenges that are impossible to overcome without
- paid upgrades. Add some social proof to the mix — a stream of
- notifications about how well your friends are faring — and before
- you know it, you’re buying virtual power-ups to get to the next
- level.
- </p><p>
- Companies have risen and fallen on these techniques, and the
- <span class="quote">“<span class="quote">fallen</span>”</span> part is worth paying attention to. In general, living
- things adapt to stimulus: Something that is very compelling or
- noteworthy when you first encounter it fades with repetition until
- you stop noticing it altogether. Consider the refrigerator hum
- that irritates you when it starts up but disappears into the
- background so thoroughly that you only notice it when it stops
- again.
- </p><p>
- That’s why behavioral conditioning uses <span class="quote">“<span class="quote">intermittent
- reinforcement schedules.</span>”</span> Instead of giving you a steady drip of
- encouragement or setbacks, games and gamified services scatter
- rewards on a randomized schedule — often enough to keep you
- interested and random enough that you can never quite find the
- pattern that would make it boring.
- </p><p>
- Intermittent reinforcement is a powerful behavioral tool, but it
- also represents a collective action problem for surveillance
- capitalism. The <span class="quote">“<span class="quote">engagement techniques</span>”</span> invented by the
- behaviorists of surveillance capitalist companies are quickly
- copied across the whole sector so that what starts as a
- mysteriously compelling fillip in the design of a service—like
- <span class="quote">“<span class="quote">pull to refresh</span>”</span> or alerts when someone likes your posts or side
- quests that your characters get invited to while in the midst of
- main quests—quickly becomes dully ubiquitous. The
- impossible-to-nail-down nonpattern of randomized drips from your
- phone becomes a grey-noise wall of sound as every single app and
- site starts to make use of whatever seems to be working at the
- time.
- </p><p>
- From the surveillance capitalist’s point of view, our adaptive
- capacity is like a harmful bacterium that deprives it of its food
- source — our attention — and novel techniques for snagging that
- attention are like new antibiotics that can be used to breach our
- defenses and destroy our self-determination. And there
- <span class="emphasis"><em>are</em></span> techniques like that. Who can forget the
- Great Zynga Epidemic, when all of our friends were caught in
- <span class="emphasis"><em>FarmVille</em></span>’s endless, mindless dopamine loops?
- But every new attention-commanding technique is jumped on by the
- whole industry and used so indiscriminately that antibiotic
- resistance sets in. Given enough repetition, almost all of us
- develop immunity to even the most powerful techniques — by 2013,
- two years after Zynga’s peak, its user base had halved.
- </p><p>
- Not everyone, of course. Some people never adapt to stimulus, just
- as some people never stop hearing the hum of the refrigerator.
- This is why most people who are exposed to slot machines play them
- for a while and then move on while a small and tragic minority
- liquidate their kids’ college funds, buy adult diapers, and
- position themselves in front of a machine until they collapse.
- </p><p>
- But surveillance capitalism’s margins on behavioral modification
- suck. Tripling the rate at which someone buys a widget sounds
- great
- <a class="ulink" href="https://www.forbes.com/sites/priceonomics/2018/03/09/the-advertising-conversion-rates-for-every-major-tech-platform/#2f6a67485957" target="_top">unless
- the base rate is way less than 1%</a> with an improved rate
- of… still less than 1%. Even penny slot machines pull down pennies
- for every spin while surveillance capitalism rakes in
- infinitesimal penny fractions.
- </p><p>
- Slot machines’ high returns mean that they can be profitable just
- by draining the fortunes of the small rump of people who are
- pathologically vulnerable to them and unable to adapt to their
- tricks. But surveillance capitalism can’t survive on the
- fractional pennies it brings down from that vulnerable sliver —
- that’s why, after the Great Zynga Epidemic had finally burned
- itself out, the small number of still-addicted players left behind
- couldn’t sustain it as a global phenomenon. And new powerful
- attention weapons aren’t easy to find, as is evidenced by the long
- years since the last time Zynga had a hit. Despite the hundreds of
- millions of dollars that Zynga has to spend on developing new
- tools to blast through our adaptation, it has never managed to
- repeat the lucky accident that let it snag so much of our
- attention for a brief moment in 2009. Powerhouses like Supercell
- have fared a little better, but they are rare and throw away many
- failures for every success.
- </p><p>
- The vulnerability of small segments of the population to dramatic,
- efficient corporate manipulation is a real concern that’s worthy
- of our attention and energy. But it’s not an existential threat to
- society.
- </p></div></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="if-data-is-the-new-oil-then-surveillance-capitalisms-engine-has-a-leak"></a>If data is the new oil, then surveillance capitalism’s engine
- has a leak</h2></div></div></div><p>
- This adaptation problem offers an explanation for one of
- surveillance capitalism’s most alarming traits: its relentless
- hunger for data and its endless expansion of data-gathering
- capabilities through the spread of sensors, online surveillance, and
- acquisition of data streams from third parties.
- </p><p>
- Zuboff observes this phenomenon and concludes that data must be very
- valuable if surveillance capitalism is so hungry for it. (In her
- words: <span class="quote">“<span class="quote">Just as industrial capitalism was driven to the continuous
- intensification of the means of production, so surveillance
- capitalists and their market players are now locked into the
- continuous intensification of the means of behavioral modification
- and the gathering might of instrumentarian power.</span>”</span>) But what if the
- voracious appetite is because data has such a short half-life —
- because people become inured so quickly to new, data-driven
- persuasion techniques — that the companies are locked in an arms
- race with our limbic system? What if it’s all a Red Queen’s race
- where they have to run ever faster — collect ever-more data — just
- to stay in the same spot?
- </p><p>
- Of course, all of Big Tech’s persuasion techniques work in concert
- with one another, and collecting data is useful beyond mere
- behavioral trickery.
- </p><p>
- If someone wants to recruit you to buy a refrigerator or join a
- pogrom, they might use profiling and targeting to send messages to
- people they judge to be good sales prospects. The messages
- themselves may be deceptive, making claims about things you’re not
- very knowledgeable about (food safety and energy efficiency or
- eugenics and historical claims about racial superiority). They might
- use search engine optimization and/or armies of fake reviewers and
- commenters and/or paid placement to dominate the discourse so that
- any search for further information takes you back to their messages.
- And finally, they may refine the different pitches using machine
- learning and other techniques to figure out what kind of pitch works
- best on someone like you.
- </p><p>
- Each phase of this process benefits from surveillance: The more data
- they have, the more precisely they can profile you and target you
- with specific messages. Think of how you’d sell a fridge if you knew
- that the warranty on your prospect’s fridge just expired and that
- they were expecting a tax rebate in April.
- </p><p>
- Also, the more data they have, the better they can craft deceptive
- messages — if I know that you’re into genealogy, I might not try to
- feed you pseudoscience about genetic differences between <span class="quote">“<span class="quote">races,</span>”</span>
- sticking instead to conspiratorial secret histories of <span class="quote">“<span class="quote">demographic
- replacement</span>”</span> and the like.
- </p><p>
- Facebook also helps you locate people who have the same odious or
- antisocial views as you. It makes it possible to find other people
- who want to carry tiki torches through the streets of
- Charlottesville in Confederate cosplay. It can help you find other
- people who want to join your militia and go to the border to look
- for undocumented migrants to terrorize. It can help you find people
- who share your belief that vaccines are poison and that the Earth is
- flat.
- </p><p>
- There is one way in which targeted advertising uniquely benefits
- those advocating for socially unacceptable causes: It is invisible.
- Racism is widely geographically dispersed, and there are few places
- where racists — and only racists — gather. This is similar to the
- problem of selling refrigerators in that potential refrigerator
- purchasers are geographically dispersed and there are few places
- where you can buy an ad that will be primarily seen by refrigerator
- customers. But buying a refrigerator is socially acceptable while
- being a Nazi is not, so you can buy a billboard or advertise in the
- newspaper sports section for your refrigerator business, and the
- only potential downside is that your ad will be seen by a lot of
- people who don’t want refrigerators, resulting in a lot of wasted
- expense.
- </p><p>
- But even if you wanted to advertise your Nazi movement on a
- billboard or prime-time TV or the sports section, you would struggle
- to find anyone willing to sell you the space for your ad partly
- because they disagree with your views and partly because they fear
- censure (boycott, reputational damage, etc.) from other people who
- disagree with your views.
- </p><p>
- Targeted ads solve this problem: On the internet, every ad unit can
- be different for every person, meaning that you can buy ads that are
- only shown to people who appear to be Nazis and not to people who
- hate Nazis. When there’s spillover — when someone who hates racism
- is shown a racist recruiting ad — there is some fallout; the
- platform or publication might get an angry public or private
- denunciation. But the nature of the risk assumed by an online ad
- buyer is different than the risks to a traditional publisher or
- billboard owner who might want to run a Nazi ad.
- </p><p>
- Online ads are placed by algorithms that broker between a diverse
- ecosystem of self-serve ad platforms that anyone can buy an ad
- through, so the Nazi ad that slips onto your favorite online
- publication isn’t seen as their moral failing but rather as a
- failure in some distant, upstream ad supplier. When a publication
- gets a complaint about an offensive ad that’s appearing in one of
- its units, it can take some steps to block that ad, but the Nazi
- might buy a slightly different ad from a different broker serving
- the same unit. And in any event, internet users increasingly
- understand that when they see an ad, it’s likely that the advertiser
- did not choose that publication and that the publication has no idea
- who its advertisers are.
- </p><p>
- These layers of indirection between advertisers and publishers serve
- as moral buffers: Today’s moral consensus is largely that publishers
- shouldn’t be held responsible for the ads that appear on their pages
- because they’re not actively choosing to put those ads there.
- Because of this, Nazis are able to overcome significant barriers to
- organizing their movement.
- </p><p>
- Data has a complex relationship with domination. Being able to spy
- on your customers can alert you to their preferences for your rivals
- and allow you to head off your rivals at the pass.
- </p><p>
- More importantly, if you can dominate the information space while
- also gathering data, then you make other deceptive tactics stronger
- because it’s harder to break out of the web of deceit you’re
- spinning. Domination — that is, ultimately becoming a monopoly — and
- not the data itself is the supercharger that makes every tactic
- worth pursuing because monopolistic domination deprives your target
- of an escape route.
- </p><p>
- If you’re a Nazi who wants to ensure that your prospects primarily
- see deceptive, confirming information when they search for more, you
- can improve your odds by seeding the search terms they use through
- your initial communications. You don’t need to own the top 10
- results for <span class="quote">“<span class="quote">voter suppression</span>”</span> if you can convince your marks to
- confine their search terms to <span class="quote">“<span class="quote">voter fraud,</span>”</span> which throws up a very
- different set of search results.
- </p><p>
- Surveillance capitalists are like stage mentalists who claim that
- their extraordinary insights into human behavior let them guess the
- word that you wrote down and folded up in your pocket but who really
- use shills, hidden cameras, sleight of hand, and brute-force
- memorization to amaze you.
- </p><p>
- Or perhaps they’re more like pick-up artists, the misogynistic cult
- that promises to help awkward men have sex with women by teaching
- them <span class="quote">“<span class="quote">neurolinguistic programming</span>”</span> phrases, body language
- techniques, and psychological manipulation tactics like <span class="quote">“<span class="quote">negging</span>”</span> —
- offering unsolicited negative feedback to women to lower their
- self-esteem and prick their interest.
- </p><p>
- Some pick-up artists eventually manage to convince women to go home
- with them, but it’s not because these men have figured out how to
- bypass women’s critical faculties. Rather, pick-up artists’
- <span class="quote">“<span class="quote">success</span>”</span> stories are a mix of women who were incapable of giving
- consent, women who were coerced, women who were intoxicated,
- self-destructive women, and a few women who were sober and in
- command of their faculties but who didn’t realize straightaway that
- they were with terrible men but rectified the error as soon as they
- could.
- </p><p>
- Pick-up artists <span class="emphasis"><em>believe</em></span> they have figured out a
- secret back door that bypasses women’s critical faculties, but they
- haven’t. Many of the tactics they deploy, like negging, became the
- butt of jokes (just like people joke about bad ad targeting), and
- there’s a good chance that anyone they try these tactics on will
- immediately recognize them and dismiss the men who use them as
- irredeemable losers.
- </p><p>
- Pick-up artists are proof that people can believe they have
- developed a system of mind control <span class="emphasis"><em>even when it doesn’t
- work</em></span>. Pick-up artists simply exploit the fact that
- one-in-a-million chances can come through for you if you make a
- million attempts, and then they assume that the other 999,999 times,
- they simply performed the technique incorrectly and commit
- themselves to doing better next time. There’s only one group of
- people who find pick-up artist lore reliably convincing: other
- would-be pick-up artists whose anxiety and insecurity make them
- vulnerable to scammers and delusional men who convince them that if
- they pay for tutelage and follow instructions, then they will
- someday succeed. Pick-up artists assume they fail to entice women
- because they are bad at being pick-up artists, not because pick-up
- artistry is bullshit. Pick-up artists are bad at selling themselves
- to women, but they’re much better at selling themselves to men who
- pay to learn the secrets of pick-up artistry.
- </p><p>
- Department store pioneer John Wanamaker is said to have lamented,
- <span class="quote">“<span class="quote">Half the money I spend on advertising is wasted; the trouble is I
- don’t know which half.</span>”</span> The fact that Wanamaker thought that only
- half of his advertising spending was wasted is a tribute to the
- persuasiveness of advertising executives, who are
- <span class="emphasis"><em>much</em></span> better at convincing potential clients to
- buy their services than they are at convincing the general public to
- buy their clients’ wares.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="what-is-facebook"></a>What is Facebook?</h2></div></div></div><p>
- Facebook is heralded as the origin of all of our modern plagues, and
- it’s not hard to see why. Some tech companies want to lock their
- users in but make their money by monopolizing access to the market
- for apps for their devices and gouging them on prices rather than by
- spying on them (like Apple). Some companies don’t care about locking
- in users because they’ve figured out how to spy on them no matter
- where they are and what they’re doing and can turn that surveillance
- into money (Google). Facebook alone among the Western tech giants
- has built a business based on locking in its users
- <span class="emphasis"><em>and</em></span> spying on them all the time.
- </p><p>
- Facebook’s surveillance regime is really without parallel in the
- Western world. Though Facebook tries to prevent itself from being
- visible on the public web, hiding most of what goes on there from
- people unless they’re logged into Facebook, the company has
- nevertheless booby-trapped the entire web with surveillance tools in
- the form of Facebook <span class="quote">“<span class="quote">Like</span>”</span> buttons that web publishers include on
- their sites to boost their Facebook profiles. Facebook also makes
- various libraries and other useful code snippets available to web
- publishers that act as surveillance tendrils on the sites where
- they’re used, funneling information about visitors to the site —
- newspapers, dating sites, message boards — to Facebook.
- </p><div class="blockquote"><blockquote class="blockquote"><p>
- Big Tech is able to practice surveillance not just because it is
- tech but because it is <span class="emphasis"><em>big</em></span>.
- </p></blockquote></div><p>
- Facebook offers similar tools to app developers, so the apps —
- games, fart machines, business review services, apps for keeping
- abreast of your kid’s schooling — you use will send information
- about your activities to Facebook even if you don’t have a Facebook
- account and even if you don’t download or use Facebook apps. On top
- of all that, Facebook buys data from third-party brokers on shopping
- habits, physical location, use of <span class="quote">“<span class="quote">loyalty</span>”</span> programs, financial
- transactions, etc., and cross-references that with the dossiers it
- develops on activity on Facebook and with apps and the public web.
- </p><p>
- Though it’s easy to integrate the web with Facebook — linking to
- news stories and such — Facebook products are generally not
- available to be integrated back into the web itself. You can embed a
- tweet in a Facebook post, but if you embed a Facebook post in a
- tweet, you just get a link back to Facebook and must log in before
- you can see it. Facebook has used extreme technological and legal
- countermeasures to prevent rivals from allowing their users to embed
- Facebook snippets in competing services or to create alternative
- interfaces to Facebook that merge your Facebook inbox with those of
- other services that you use.
- </p><p>
- And Facebook is incredibly popular, with 2.3 billion claimed users
- (though many believe this figure to be inflated). Facebook has been
- used to organize genocidal pogroms, racist riots, anti-vaccination
- movements, flat Earth cults, and the political lives of some of the
- world’s ugliest, most brutal autocrats. There are some really
- alarming things going on in the world, and Facebook is implicated in
- many of them, so it’s easy to conclude that these bad things are the
- result of Facebook’s mind-control system, which it rents out to
- anyone with a few bucks to spend.
- </p><p>
- To understand what role Facebook plays in the formulation and
- mobilization of antisocial movements, we need to understand the dual
- nature of Facebook.
- </p><p>
- Because it has a lot of users and a lot of data about those users,
- Facebook is a very efficient tool for locating people with
- hard-to-find traits, the kinds of traits that are widely diffused in
- the population such that advertisers have historically struggled to
- find a cost-effective way to reach them. Think back to
- refrigerators: Most of us only replace our major appliances a few
- times in our entire lives. If you’re a refrigerator manufacturer or
- retailer, you have these brief windows in the life of a consumer
- during which they are pondering a purchase, and you have to somehow
- reach them. Anyone who’s ever registered a title change after buying
- a house can attest that appliance manufacturers are incredibly
- desperate to reach anyone who has even the slenderest chance of
- being in the market for a new fridge.
- </p><p>
- Facebook makes finding people shopping for refrigerators a
- <span class="emphasis"><em>lot</em></span> easier. It can target ads to people who’ve
- registered a new home purchase, to people who’ve searched for
- refrigerator buying advice, to people who have complained about
- their fridge dying, or any combination thereof. It can even target
- people who’ve recently bought <span class="emphasis"><em>other</em></span> kitchen
- appliances on the theory that someone who’s just replaced their
- stove and dishwasher might be in a fridge-buying kind of mood. The
- vast majority of people who are reached by these ads will not be in
- the market for a new fridge, but — crucially — the percentage of
- people who <span class="emphasis"><em>are</em></span> looking for fridges that these
- ads reach is <span class="emphasis"><em>much</em></span> larger than it is than for
- any group that might be subjected to traditional, offline targeted
- refrigerator marketing.
- </p><p>
- Facebook also makes it a lot easier to find people who have the same
- rare disease as you, which might have been impossible in earlier
- eras — the closest fellow sufferer might otherwise be hundreds of
- miles away. It makes it easier to find people who went to the same
- high school as you even though decades have passed and your former
- classmates have all been scattered to the four corners of the Earth.
- </p><p>
- Facebook also makes it much easier to find people who hold the same
- rare political beliefs as you. If you’ve always harbored a secret
- affinity for socialism but never dared utter this aloud lest you be
- demonized by your neighbors, Facebook can help you discover other
- people who feel the same way (and it might just demonstrate to you
- that your affinity is more widespread than you ever suspected). It
- can make it easier to find people who share your sexual identity.
- And again, it can help you to understand that what you thought was a
- shameful secret that affected only you was really a widely shared
- trait, giving you both comfort and the courage to come out to the
- people in your life.
- </p><p>
- All of this presents a dilemma for Facebook: Targeting makes the
- company’s ads more effective than traditional ads, but it also lets
- advertisers see just how effective their ads are. While advertisers
- are pleased to learn that Facebook ads are more effective than ads
- on systems with less sophisticated targeting, advertisers can also
- see that in nearly every case, the people who see their ads ignore
- them. Or, at best, the ads work on a subconscious level, creating
- nebulous unmeasurables like <span class="quote">“<span class="quote">brand recognition.</span>”</span> This means that the
- price per ad is very low in nearly every case.
- </p><p>
- To make things worse, many Facebook groups spark precious little
- discussion. Your little-league soccer team, the people with the same
- rare disease as you, and the people you share a political affinity
- with may exchange the odd flurry of messages at critical junctures,
- but on a daily basis, there’s not much to say to your old high
- school chums or other hockey-card collectors.
- </p><p>
- With nothing but <span class="quote">“<span class="quote">organic</span>”</span> discussion, Facebook would not generate
- enough traffic to sell enough ads to make the money it needs to
- continually expand by buying up its competitors while returning
- handsome sums to its investors.
- </p><p>
- So Facebook has to gin up traffic by sidetracking its own forums:
- Every time Facebook’s algorithm injects controversial materials —
- inflammatory political articles, conspiracy theories, outrage
- stories — into a group, it can hijack that group’s nominal purpose
- with its desultory discussions and supercharge those discussions by
- turning them into bitter, unproductive arguments that drag on and
- on. Facebook is optimized for engagement, not happiness, and it
- turns out that automated systems are pretty good at figuring out
- things that people will get angry about.
- </p><p>
- Facebook <span class="emphasis"><em>can</em></span> modify our behavior but only in a
- couple of trivial ways. First, it can lock in all your friends and
- family members so that you check and check and check with Facebook
- to find out what they are up to; and second, it can make you angry
- and anxious. It can force you to choose between being interrupted
- constantly by updates — a process that breaks your concentration and
- makes it hard to be introspective — and staying in touch with your
- friends. This is a very limited form of mind control, and it can
- only really make us miserable, angry, and anxious.
- </p><p>
- This is why Facebook’s targeting systems — both the ones it shows to
- advertisers and the ones that let users find people who share their
- interests — are so next-gen and smooth and easy to use as well as
- why its message boards have a toolset that seems like it hasn’t
- changed since the mid-2000s. If Facebook delivered an equally
- flexible, sophisticated message-reading system to its users, those
- users could defend themselves against being nonconsensually
- eyeball-fucked with Donald Trump headlines.
- </p><p>
- The more time you spend on Facebook, the more ads it gets to show
- you. The solution to Facebook’s ads only working one in a thousand
- times is for the company to try to increase how much time you spend
- on Facebook by a factor of a thousand. Rather than thinking of
- Facebook as a company that has figured out how to show you exactly
- the right ad in exactly the right way to get you to do what its
- advertisers want, think of it as a company that has figured out how
- to make you slog through an endless torrent of arguments even though
- they make you miserable, spending so much time on the site that it
- eventually shows you at least one ad that you respond to.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="monopoly-and-the-right-to-the-future-tense"></a>Monopoly and the right to the future tense</h2></div></div></div><p>
- Zuboff and her cohort are particularly alarmed at the extent to
- which surveillance allows corporations to influence our decisions,
- taking away something she poetically calls <span class="quote">“<span class="quote">the right to the future
- tense</span>”</span> — that is, the right to decide for yourself what you will do
- in the future.
- </p><p>
- It’s true that advertising can tip the scales one way or another:
- When you’re thinking of buying a fridge, a timely fridge ad might
- end the search on the spot. But Zuboff puts enormous and undue
- weight on the persuasive power of surveillance-based influence
- techniques. Most of these don’t work very well, and the ones that do
- won’t work for very long. The makers of these influence tools are
- confident they will someday refine them into systems of total
- control, but they are hardly unbiased observers, and the risks from
- their dreams coming true are very speculative.
- </p><p>
- By contrast, Zuboff is rather sanguine about 40 years of lax
- antitrust practice that has allowed a handful of companies to
- dominate the internet, ushering in an information age with,
- <a class="ulink" href="https://twitter.com/tveastman/status/1069674780826071040" target="_top">as
- one person on Twitter noted</a>, five giant websites each filled
- with screenshots of the other four.
- </p><p>
- However, if we are to be alarmed that we might lose the right to
- choose for ourselves what our future will hold, then monopoly’s
- nonspeculative, concrete, here-and-now harms should be front and
- center in our debate over tech policy.
- </p><p>
- Start with <span class="quote">“<span class="quote">digital rights management.</span>”</span> In 1998, Bill Clinton signed
- the Digital Millennium Copyright Act (DMCA) into law. It’s a complex
- piece of legislation with many controversial clauses but none more
- so than Section 1201, the <span class="quote">“<span class="quote">anti-circumvention</span>”</span> rule.
- </p><p>
- This is a blanket ban on tampering with systems that restrict access
- to copyrighted works. The ban is so thoroughgoing that it prohibits
- removing a copyright lock even when no copyright infringement takes
- place. This is by design: The activities that the DMCA’s Section
- 1201 sets out to ban are not copyright infringements; rather, they
- are legal activities that frustrate manufacturers’ commercial plans.
- </p><p>
- For example, Section 1201’s first major application was on DVD
- players as a means of enforcing the region coding built into those
- devices. DVD-CCA, the body that standardized DVDs and DVD players,
- divided the world into six regions and specified that DVD players
- must check each disc to determine which regions it was authorized to
- be played in. DVD players would have their own corresponding region
- (a DVD player bought in the U.S. would be region 1 while one bought
- in India would be region 5). If the player and the disc’s region
- matched, the player would play the disc; otherwise, it would reject
- it.
- </p><p>
- However, watching a lawfully produced disc in a country other than
- the one where you purchased it is not copyright infringement — it’s
- the opposite. Copyright law imposes this duty on customers for a
- movie: You must go into a store, find a licensed disc, and pay the
- asking price. Do that — and <span class="emphasis"><em>nothing else</em></span> — and
- you and copyright are square with one another.
- </p><p>
- The fact that a movie studio wants to charge Indians less than
- Americans or release in Australia later than it releases in the U.K.
- has no bearing on copyright law. Once you lawfully acquire a DVD, it
- is no copyright infringement to watch it no matter where you happen
- to be.
- </p><p>
- So DVD and DVD player manufacturers would not be able to use
- accusations of abetting copyright infringement to punish
- manufacturers who made noncompliant players that would play discs
- from any region or repair shops that modified players to let you
- watch out-of-region discs or software programmers who created
- programs to let you do this.
- </p><p>
- That’s where Section 1201 of the DMCA comes in: By banning tampering
- with an <span class="quote">“<span class="quote">access control,</span>”</span> the rule gave manufacturers and rights
- holders standing to sue competitors who released superior products
- with lawful features that the market demanded (in this case,
- region-free players).
- </p><p>
- This is an odious scam against consumers, but as time went by,
- Section 1201 grew to encompass a rapidly expanding constellation of
- devices and services as canny manufacturers have realized certain
- things:
- </p><div class="itemizedlist"><ul class="itemizedlist compact" style="list-style-type: disc; "><li class="listitem"><p>
- Any device with software in it contains a <span class="quote">“<span class="quote">copyrighted work</span>”</span> —
- i.e., the software.
- </p></li><li class="listitem"><p>
- A device can be designed so that reconfiguring the software
- requires bypassing an <span class="quote">“<span class="quote">access control for copyrighted works,</span>”</span>
- which is a potential felony under Section 1201.
- </p></li><li class="listitem"><p>
- Thus, companies can control their customers’ behavior after they
- take home their purchases by designing products so that all
- unpermitted uses require modifications that fall afoul of
- Section 1201.
- </p></li></ul></div><p>
- Section 1201 then becomes a means for manufacturers of all
- descriptions to force their customers to arrange their affairs to
- benefit the manufacturers’ shareholders instead of themselves.
- </p><p>
- This manifests in many ways: from a new generation of inkjet
- printers that use countermeasures to prevent third-party ink that
- cannot be bypassed without legal risks to similar systems in
- tractors that prevent third-party technicians from swapping in the
- manufacturer’s own parts that are not recognized by the tractor’s
- control system until it is supplied with a manufacturer’s unlock
- code.
- </p><p>
- Closer to home, Apple’s iPhones use these measures to prevent both
- third-party service and third-party software installation. This
- allows Apple to decide when an iPhone is beyond repair and must be
- shredded and landfilled as opposed to the iPhone’s purchaser. (Apple
- is notorious for its environmentally catastrophic policy of
- destroying old electronics rather than permitting them to be
- cannibalized for parts.) This is a very useful power to wield,
- especially in light of CEO Tim Cook’s January 2019 warning to
- investors that the company’s profits are endangered by customers
- choosing to hold onto their phones for longer rather than replacing
- them.
- </p><p>
- Apple’s use of copyright locks also allows it to establish a
- monopoly over how its customers acquire software for their mobile
- devices. The App Store’s commercial terms guarantee Apple a share of
- all revenues generated by the apps sold there, meaning that Apple
- gets paid when you buy an app from its store and then continues to
- get paid every time you buy something using that app. This comes out
- of the bottom line of software developers, who must either charge
- more or accept lower profits for their products.
- </p><p>
- Crucially, Apple’s use of copyright locks gives it the power to make
- editorial decisions about which apps you may and may not install on
- your own device. Apple has used this power to
- <a class="ulink" href="https://www.telegraph.co.uk/technology/apple/5982243/Apple-bans-dictionary-from-App-Store-over-swear-words.html" target="_top">reject
- dictionaries</a> for containing obscene words; to
- <a class="ulink" href="https://www.vice.com/en_us/article/538kan/apple-just-banned-the-app-that-tracks-us-drone-strikes-again" target="_top">limit
- political speech</a>, especially from apps that make sensitive
- political commentary such as an app that notifies you every time a
- U.S. drone kills someone somewhere in the world; and to
- <a class="ulink" href="https://www.eurogamer.net/articles/2016-05-19-palestinian-indie-game-must-not-be-called-a-game-apple-says" target="_top">object
- to a game</a> that commented on the Israel-Palestine conflict.
- </p><p>
- Apple often justifies monopoly power over software installation in
- the name of security, arguing that its vetting of apps for its store
- means that it can guard its users against apps that contain
- surveillance code. But this cuts both ways. In China, the government
- <a class="ulink" href="https://www.ft.com/content/ad42e536-cf36-11e7-b781-794ce08b24dc" target="_top">ordered
- Apple to prohibit the sale of privacy tools</a> like VPNs with
- the exception of VPNs that had deliberately introduced flaws
- designed to let the Chinese state eavesdrop on users. Because Apple
- uses technological countermeasures — with legal backstops — to block
- customers from installing unauthorized apps, Chinese iPhone owners
- cannot readily (or legally) acquire VPNs that would protect them
- from Chinese state snooping.
- </p><p>
- Zuboff calls surveillance capitalism a <span class="quote">“<span class="quote">rogue capitalism.</span>”</span>
- Theoreticians of capitalism claim that its virtue is that it
- <a class="ulink" href="https://en.wikipedia.org/wiki/Price_signal" target="_top">aggregates
- information in the form of consumers’ decisions</a>, producing
- efficient markets. Surveillance capitalism’s supposed power to rob
- its victims of their free will through computationally supercharged
- influence campaigns means that our markets no longer aggregate
- customers’ decisions because we customers no longer decide — we are
- given orders by surveillance capitalism’s mind-control rays.
- </p><p>
- If our concern is that markets cease to function when consumers can
- no longer make choices, then copyright locks should concern us at
- <span class="emphasis"><em>least</em></span> as much as influence campaigns. An
- influence campaign might nudge you to buy a certain brand of phone;
- but the copyright locks on that phone absolutely determine where you
- get it serviced, which apps can run on it, and when you have to
- throw it away rather than fixing it.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="search-order-and-the-right-to-the-future-tense"></a>Search order and the right to the future tense</h2></div></div></div><p>
- Markets are posed as a kind of magic: By discovering otherwise
- hidden information conveyed by the free choices of consumers, those
- consumers’ local knowledge is integrated into a self-correcting
- system that makes efficient allocations—more efficient than any
- computer could calculate. But monopolies are incompatible with that
- notion. When you only have one app store, the owner of the store —
- not the consumer — decides on the range of choices. As Boss Tweed
- once said, <span class="quote">“<span class="quote">I don’t care who does the electing, so long as I get to
- do the nominating.</span>”</span> A monopolized market is an election whose
- candidates are chosen by the monopolist.
- </p><p>
- This ballot rigging is made more pernicious by the existence of
- monopolies over search order. Google’s search market share is about
- 90%. When Google’s ranking algorithm puts a result for a popular
- search term in its top 10, that helps determine the behavior of
- millions of people. If Google’s answer to <span class="quote">“<span class="quote">Are vaccines dangerous?</span>”</span>
- is a page that rebuts anti-vax conspiracy theories, then a sizable
- portion of the public will learn that vaccines are safe. If, on the
- other hand, Google sends those people to a site affirming the
- anti-vax conspiracies, a sizable portion of those millions will come
- away convinced that vaccines are dangerous.
- </p><p>
- Google’s algorithm is often tricked into serving disinformation as a
- prominent search result. But in these cases, Google isn’t persuading
- people to change their minds; it’s just presenting something untrue
- as fact when the user has no cause to doubt it.
- </p><p>
- This is true whether the search is for <span class="quote">“<span class="quote">Are vaccines dangerous?</span>”</span> or
- <span class="quote">“<span class="quote">best restaurants near me.</span>”</span> Most users will never look past the
- first page of search results, and when the overwhelming majority of
- people all use the same search engine, the ranking algorithm
- deployed by that search engine will determine myriad outcomes
- (whether to adopt a child, whether to have cancer surgery, where to
- eat dinner, where to move, where to apply for a job) to a degree
- that vastly outstrips any behavioral outcomes dictated by
- algorithmic persuasion techniques.
- </p><p>
- Many of the questions we ask search engines have no empirically
- correct answers: <span class="quote">“<span class="quote">Where should I eat dinner?</span>”</span> is not an objective
- question. Even questions that do have correct answers (<span class="quote">“<span class="quote">Are vaccines
- dangerous?</span>”</span>) don’t have one empirically superior source for that
- answer. Many pages affirm the safety of vaccines, so which one goes
- first? Under conditions of competition, consumers can choose from
- many search engines and stick with the one whose algorithmic
- judgment suits them best, but under conditions of monopoly, we all
- get our answers from the same place.
- </p><p>
- Google’s search dominance isn’t a matter of pure merit: The company
- has leveraged many tactics that would have been prohibited under
- classical, pre-Ronald-Reagan antitrust enforcement standards to
- attain its dominance. After all, this is a company that has
- developed two major products: a really good search engine and a
- pretty good Hotmail clone. Every other major success it’s had —
- Android, YouTube, Google Maps, etc. — has come through an
- acquisition of a nascent competitor. Many of the company’s key
- divisions, such as the advertising technology of DoubleClick,
- violate the historical antitrust principle of structural separation,
- which forbade firms from owning subsidiaries that competed with
- their customers. Railroads, for example, were barred from owning
- freight companies that competed with the shippers whose freight they
- carried.
- </p><p>
- If we’re worried about giant companies subverting markets by
- stripping consumers of their ability to make free choices, then
- vigorous antitrust enforcement seems like an excellent remedy. If
- we’d denied Google the right to effect its many mergers, we would
- also have probably denied it its total search dominance. Without
- that dominance, the pet theories, biases, errors (and good judgment,
- too) of Google search engineers and product managers would not have
- such an outsized effect on consumer choice.
- </p><p>
- This goes for many other companies. Amazon, a classic surveillance
- capitalist, is obviously the dominant tool for searching Amazon —
- though many people find their way to Amazon through Google searches
- and Facebook posts — and obviously, Amazon controls Amazon search.
- That means that Amazon’s own self-serving editorial choices—like
- promoting its own house brands over rival goods from its sellers as
- well as its own pet theories, biases, and errors— determine much of
- what we buy on Amazon. And since Amazon is the dominant e-commerce
- retailer outside of China and since it attained that dominance by
- buying up both large rivals and nascent competitors in defiance of
- historical antitrust rules, we can blame the monopoly for stripping
- consumers of their right to the future tense and the ability to
- shape markets by making informed choices.
- </p><p>
- Not every monopolist is a surveillance capitalist, but that doesn’t
- mean they’re not able to shape consumer choices in wide-ranging
- ways. Zuboff lauds Apple for its App Store and iTunes Store,
- insisting that adding price tags to the features on its platforms
- has been the secret to resisting surveillance and thus creating
- markets. But Apple is the only retailer allowed to sell on its
- platforms, and it’s the second-largest mobile device vendor in the
- world. The independent software vendors that sell through Apple’s
- marketplace accuse the company of the same surveillance sins as
- Amazon and other big retailers: spying on its customers to find
- lucrative new products to launch, effectively using independent
- software vendors as free-market researchers, then forcing them out
- of any markets they discover.
- </p><p>
- Because of its use of copyright locks, Apple’s mobile customers are
- not legally allowed to switch to a rival retailer for its apps if
- they want to do so on an iPhone. Apple, obviously, is the only
- entity that gets to decide how it ranks the results of search
- queries in its stores. These decisions ensure that some apps are
- often installed (because they appear on page one) and others are
- never installed (because they appear on page one million). Apple’s
- search-ranking design decisions have a vastly more significant
- effect on consumer behaviors than influence campaigns delivered by
- surveillance capitalism’s ad-serving bots.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="monopolists-can-afford-sleeping-pills-for-watchdogs"></a>Monopolists can afford sleeping pills for watchdogs</h2></div></div></div><p>
- Only the most extreme market ideologues think that markets can
- self-regulate without state oversight. Markets need watchdogs —
- regulators, lawmakers, and other elements of democratic control — to
- keep them honest. When these watchdogs sleep on the job, then
- markets cease to aggregate consumer choices because those choices
- are constrained by illegitimate and deceptive activities that
- companies are able to get away with because no one is holding them
- to account.
- </p><p>
- But this kind of regulatory capture doesn’t come cheap. In
- competitive sectors, where rivals are constantly eroding one
- another’s margins, individual firms lack the surplus capital to
- effectively lobby for laws and regulations that serve their ends.
- </p><p>
- Many of the harms of surveillance capitalism are the result of weak
- or nonexistent regulation. Those regulatory vacuums spring from the
- power of monopolists to resist stronger regulation and to tailor
- what regulation exists to permit their existing businesses.
- </p><p>
- Here’s an example: When firms over-collect and over-retain our data,
- they are at increased risk of suffering a breach — you can’t leak
- data you never collected, and once you delete all copies of that
- data, you can no longer leak it. For more than a decade, we’ve lived
- through an endless parade of ever-worsening data breaches, each one
- uniquely horrible in the scale of data breached and the sensitivity
- of that data.
- </p><p>
- But still, firms continue to over-collect and over-retain our data
- for three reasons:
- </p><p>
- <span class="strong"><strong>1. They are locked in the aforementioned
- limbic arms race with our capacity to shore up our attentional
- defense systems to resist their new persuasion
- techniques.</strong></span> They’re also locked in an arms race with
- their competitors to find new ways to target people for sales
- pitches. As soon as they discover a soft spot in our attentional
- defenses (a counterintuitive, unobvious way to target potential
- refrigerator buyers), the public begins to wise up to the tactic,
- and their competitors leap on it, hastening the day in which all
- potential refrigerator buyers have been inured to the pitch.
- </p><p>
- <span class="strong"><strong>2. They believe the surveillance capitalism
- story.</strong></span> Data is cheap to aggregate and store, and both
- proponents and opponents of surveillance capitalism have assured
- managers and product designers that if you collect enough data, you
- will be able to perform sorcerous acts of mind control, thus
- supercharging your sales. Even if you never figure out how to profit
- from the data, someone else will eventually offer to buy it from you
- to give it a try. This is the hallmark of all economic bubbles:
- acquiring an asset on the assumption that someone else will buy it
- from you for more than you paid for it, often to sell to someone
- else at an even greater price.
- </p><p>
- <span class="strong"><strong>3. The penalties for leaking data are
- negligible.</strong></span> Most countries limit these penalties to
- actual damages, meaning that consumers who’ve had their data
- breached have to show actual monetary harms to get a reward. In
- 2014, Home Depot disclosed that it had lost credit-card data for 53
- million of its customers, but it settled the matter by paying those
- customers about $0.34 each — and a third of that $0.34 wasn’t even
- paid in cash. It took the form of a credit to procure a largely
- ineffectual credit-monitoring service.
- </p><p>
- But the harms from breaches are much more extensive than these
- actual-damages rules capture. Identity thieves and fraudsters are
- wily and endlessly inventive. All the vast breaches of our century
- are being continuously recombined, the data sets merged and mined
- for new ways to victimize the people whose data was present in them.
- Any reasonable, evidence-based theory of deterrence and compensation
- for breaches would not confine damages to actual damages but rather
- would allow users to claim these future harms.
- </p><p>
- However, even the most ambitious privacy rules, such as the EU
- General Data Protection Regulation, fall far short of capturing the
- negative externalities of the platforms’ negligent over-collection
- and over-retention, and what penalties they do provide are not
- aggressively pursued by regulators.
- </p><p>
- This tolerance of — or indifference to — data over-collection and
- over-retention can be ascribed in part to the sheer lobbying muscle
- of the platforms. They are so profitable that they can handily
- afford to divert gigantic sums to fight any real change — that is,
- change that would force them to internalize the costs of their
- surveillance activities.
- </p><p>
- And then there’s state surveillance, which the surveillance
- capitalism story dismisses as a relic of another era when the big
- worry was being jailed for your dissident speech, not having your
- free will stripped away with machine learning.
- </p><p>
- But state surveillance and private surveillance are intimately
- related. As we saw when Apple was conscripted by the Chinese
- government as a vital collaborator in state surveillance, the only
- really affordable and tractable way to conduct mass surveillance on
- the scale practiced by modern states — both <span class="quote">“<span class="quote">free</span>”</span> and autocratic
- states — is to suborn commercial services.
- </p><p>
- Whether it’s Google being used as a location tracking tool by local
- law enforcement across the U.S. or the use of social media tracking
- by the Department of Homeland Security to build dossiers on
- participants in protests against Immigration and Customs
- Enforcement’s family separation practices, any hard limits on
- surveillance capitalism would hamstring the state’s own surveillance
- capability. Without Palantir, Amazon, Google, and other major tech
- contractors, U.S. cops would not be able to spy on Black people, ICE
- would not be able to manage the caging of children at the U.S.
- border, and state welfare systems would not be able to purge their
- rolls by dressing up cruelty as empiricism and claiming that poor
- and vulnerable people are ineligible for assistance. At least some
- of the states’ unwillingness to take meaningful action to curb
- surveillance should be attributed to this symbiotic relationship.
- There is no mass state surveillance without mass commercial
- surveillance.
- </p><p>
- Monopolism is key to the project of mass state surveillance. It’s
- true that smaller tech firms are apt to be less well-defended than
- Big Tech, whose security experts are drawn from the tops of their
- field and who are given enormous resources to secure and monitor
- their systems against intruders. But smaller firms also have less to
- protect: fewer users whose data is more fragmented across more
- systems and have to be suborned one at a time by state actors.
- </p><p>
- A concentrated tech sector that works with authorities is a much
- more powerful ally in the project of mass state surveillance than a
- fragmented one composed of smaller actors. The U.S. tech sector is
- small enough that all of its top executives fit around a single
- boardroom table in Trump Tower in 2017, shortly after Trump’s
- inauguration. Most of its biggest players bid to win JEDI, the
- Pentagon’s $10 billion Joint Enterprise Defense Infrastructure cloud
- contract. Like other highly concentrated industries, Big Tech
- rotates its key employees in and out of government service, sending
- them to serve in the Department of Defense and the White House, then
- hiring ex-Pentagon and ex-DOD top staffers and officers to work in
- their own government relations departments.
- </p><p>
- They can even make a good case for doing this: After all, when there
- are only four or five big companies in an industry, everyone
- qualified to regulate those companies has served as an executive in
- at least a couple of them — because, likewise, when there are only
- five companies in an industry, everyone qualified for a senior role
- at any of them is by definition working at one of the other ones.
- </p><div class="blockquote"><blockquote class="blockquote"><p>
- While surveillance doesn’t cause monopolies, monopolies certainly
- abet surveillance.
- </p></blockquote></div><p>
- Industries that are competitive are fragmented — composed of
- companies that are at each other’s throats all the time and eroding
- one another’s margins in bids to steal their best customers. This
- leaves them with much more limited capital to use to lobby for
- favorable rules and a much harder job of getting everyone to agree
- to pool their resources to benefit the industry as a whole.
- </p><p>
- Surveillance combined with machine learning is supposed to be an
- existential crisis, a species-defining moment at which our free will
- is just a few more advances in the field from being stripped away. I
- am skeptical of this claim, but I <span class="emphasis"><em>do</em></span> think that
- tech poses an existential threat to our society and possibly our
- species.
- </p><p>
- But that threat grows out of monopoly.
- </p><p>
- One of the consequences of tech’s regulatory capture is that it can
- shift liability for poor security decisions onto its customers and
- the wider society. It is absolutely normal in tech for companies to
- obfuscate the workings of their products, to make them deliberately
- hard to understand, and to threaten security researchers who seek to
- independently audit those products.
- </p><p>
- IT is the only field in which this is practiced: No one builds a
- bridge or a hospital and keeps the composition of the steel or the
- equations used to calculate load stresses a secret. It is a frankly
- bizarre practice that leads, time and again, to grotesque security
- defects on farcical scales, with whole classes of devices being
- revealed as vulnerable long after they are deployed in the field and
- put into sensitive places.
- </p><p>
- The monopoly power that keeps any meaningful consequences for
- breaches at bay means that tech companies continue to build terrible
- products that are insecure by design and that end up integrated into
- our lives, in possession of our data, and connected to our physical
- world. For years, Boeing has struggled with the aftermath of a
- series of bad technology decisions that made its 737 fleet a global
- pariah, a rare instance in which bad tech decisions have been
- seriously punished in the market.
- </p><p>
- These bad security decisions are compounded yet again by the use of
- copyright locks to enforce business-model decisions against
- consumers. Recall that these locks have become the go-to means for
- shaping consumer behavior, making it technically impossible to use
- third-party ink, insulin, apps, or service depots in connection with
- your lawfully acquired property.
- </p><p>
- Recall also that these copyright locks are backstopped by
- legislation (such as Section 1201 of the DMCA or Article 6 of the
- 2001 EU Copyright Directive) that ban tampering with
- (<span class="quote">“<span class="quote">circumventing</span>”</span>) them, and these statutes have been used to
- threaten security researchers who make disclosures about
- vulnerabilities without permission from manufacturers.
- </p><p>
- This amounts to a manufacturer’s veto over safety warnings and
- criticism. While this is far from the legislative intent of the DMCA
- and its sister statutes around the world, Congress has not
- intervened to clarify the statute nor will it because to do so would
- run counter to the interests of powerful, large firms whose lobbying
- muscle is unstoppable.
- </p><p>
- Copyright locks are a double whammy: They create bad security
- decisions that can’t be freely investigated or discussed. If markets
- are supposed to be machines for aggregating information (and if
- surveillance capitalism’s notional mind-control rays are what make
- it a <span class="quote">“<span class="quote">rogue capitalism</span>”</span> because it denies consumers the power to
- make decisions), then a program of legally enforced ignorance of the
- risks of products makes monopolism even more of a <span class="quote">“<span class="quote">rogue capitalism</span>”</span>
- than surveillance capitalism’s influence campaigns.
- </p><p>
- And unlike mind-control rays, enforced silence over security is an
- immediate, documented problem, and it <span class="emphasis"><em>does</em></span>
- constitute an existential threat to our civilization and possibly
- our species. The proliferation of insecure devices — especially
- devices that spy on us and especially when those devices also can
- manipulate the physical world by, say, steering your car or flipping
- a breaker at a power station — is a kind of technology debt.
- </p><p>
- In software design, <span class="quote">“<span class="quote">technology debt</span>”</span> refers to old, baked-in
- decisions that turn out to be bad ones in hindsight. Perhaps a
- long-ago developer decided to incorporate a networking protocol made
- by a vendor that has since stopped supporting it. But everything in
- the product still relies on that superannuated protocol, and so,
- with each revision, the product team has to work around this
- obsolete core, adding compatibility layers, surrounding it with
- security checks that try to shore up its defenses, and so on. These
- Band-Aid measures compound the debt because every subsequent
- revision has to make allowances for <span class="emphasis"><em>them</em></span>, too,
- like interest mounting on a predatory subprime loan. And like a
- subprime loan, the interest mounts faster than you can hope to pay
- it off: The product team has to put so much energy into maintaining
- this complex, brittle system that they don’t have any time left over
- to refactor the product from the ground up and <span class="quote">“<span class="quote">pay off the debt</span>”</span>
- once and for all.
- </p><p>
- Typically, technology debt results in a technological bankruptcy:
- The product gets so brittle and unsustainable that it fails
- catastrophically. Think of the antiquated COBOL-based banking and
- accounting systems that fell over at the start of the pandemic
- emergency when confronted with surges of unemployment claims.
- Sometimes that ends the product; sometimes it takes the company down
- with it. Being caught in the default of a technology debt is scary
- and traumatic, just like losing your house due to bankruptcy is
- scary and traumatic.
- </p><p>
- But the technology debt created by copyright locks isn’t individual
- debt; it’s systemic. Everyone in the world is exposed to this
- over-leverage, as was the case with the 2008 financial crisis. When
- that debt comes due — when we face a cascade of security breaches
- that threaten global shipping and logistics, the food supply,
- pharmaceutical production pipelines, emergency communications, and
- other critical systems that are accumulating technology debt in part
- due to the presence of deliberately insecure and deliberately
- unauditable copyright locks — it will indeed pose an existential
- risk.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="privacy-and-monopoly"></a>Privacy and monopoly</h2></div></div></div><p>
- Many tech companies are gripped by an orthodoxy that holds that if
- they just gather enough data on enough of our activities, everything
- else is possible — the mind control and endless profits. This is an
- unfalsifiable hypothesis: If data gives a tech company even a tiny
- improvement in behavior prediction and modification, the company
- declares that it has taken the first step toward global domination
- with no end in sight. If a company <span class="emphasis"><em>fails</em></span> to
- attain any improvements from gathering and analyzing data, it
- declares success to be just around the corner, attainable once more
- data is in hand.
- </p><p>
- Surveillance tech is far from the first industry to embrace a
- nonsensical, self-serving belief that harms the rest of the world,
- and it is not the first industry to profit handsomely from such a
- delusion. Long before hedge-fund managers were claiming (falsely)
- that they could beat the S&P 500, there were plenty of other
- <span class="quote">“<span class="quote">respectable</span>”</span> industries that have been revealed as quacks in
- hindsight. From the makers of radium suppositories (a real thing!)
- to the cruel sociopaths who claimed they could <span class="quote">“<span class="quote">cure</span>”</span> gay people,
- history is littered with the formerly respectable titans of
- discredited industries.
- </p><p>
- This is not to say that there’s nothing wrong with Big Tech and its
- ideological addiction to data. While surveillance’s benefits are
- mostly overstated, its harms are, if anything,
- <span class="emphasis"><em>understated</em></span>.
- </p><p>
- There’s real irony here. The belief in surveillance capitalism as a
- <span class="quote">“<span class="quote">rogue capitalism</span>”</span> is driven by the belief that markets wouldn’t
- tolerate firms that are gripped by false beliefs. An oil company
- that has false beliefs about where the oil is will eventually go
- broke digging dry wells after all.
- </p><p>
- But monopolists get to do terrible things for a long time before
- they pay the price. Think of how concentration in the finance sector
- allowed the subprime crisis to fester as bond-rating agencies,
- regulators, investors, and critics all fell under the sway of a
- false belief that complex mathematics could construct <span class="quote">“<span class="quote">fully hedged</span>”</span>
- debt instruments that could not possibly default. A small bank that
- engaged in this kind of malfeasance would simply go broke rather
- than outrunning the inevitable crisis, perhaps growing so big that
- it averted it altogether. But large banks were able to continue to
- attract investors, and when they finally <span class="emphasis"><em>did</em></span>
- come a-cropper, the world’s governments bailed them out. The worst
- offenders of the subprime crisis are bigger than they were in 2008,
- bringing home more profits and paying their execs even larger sums.
- </p><p>
- Big Tech is able to practice surveillance not just because it is
- tech but because it is <span class="emphasis"><em>big</em></span>. The reason every
- web publisher embeds a Facebook <span class="quote">“<span class="quote">Like</span>”</span> button is that Facebook
- dominates the internet’s social media referrals — and every one of
- those <span class="quote">“<span class="quote">Like</span>”</span> buttons spies on everyone who lands on a page that
- contains them (see also: Google Analytics embeds, Twitter buttons,
- etc.).
- </p><p>
- The reason the world’s governments have been slow to create
- meaningful penalties for privacy breaches is that Big Tech’s
- concentration produces huge profits that can be used to lobby
- against those penalties — and Big Tech’s concentration means that
- the companies involved are able to arrive at a unified negotiating
- position that supercharges the lobbying.
- </p><p>
- The reason that the smartest engineers in the world want to work for
- Big Tech is that Big Tech commands the lion’s share of tech industry
- jobs.
- </p><p>
- The reason people who are aghast at Facebook’s and Google’s and
- Amazon’s data-handling practices continue to use these services is
- that all their friends are on Facebook; Google dominates search; and
- Amazon has put all the local merchants out of business.
- </p><p>
- Competitive markets would weaken the companies’ lobbying muscle by
- reducing their profits and pitting them against each other in
- regulatory forums. It would give customers other places to go to get
- their online services. It would make the companies small enough to
- regulate and pave the way to meaningful penalties for breaches. It
- would let engineers with ideas that challenged the surveillance
- orthodoxy raise capital to compete with the incumbents. It would
- give web publishers multiple ways to reach audiences and make the
- case against Facebook and Google and Twitter embeds.
- </p><p>
- In other words, while surveillance doesn’t cause monopolies,
- monopolies certainly abet surveillance.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ronald-reagan-pioneer-of-tech-monopolism"></a>Ronald Reagan, pioneer of tech monopolism</h2></div></div></div><p>
- Technology exceptionalism is a sin, whether it’s practiced by
- technology’s blind proponents or by its critics. Both of these camps
- are prone to explaining away monopolistic concentration by citing
- some special characteristic of the tech industry, like network
- effects or first-mover advantage. The only real difference between
- these two groups is that the tech apologists say monopoly is
- inevitable so we should just let tech get away with its abuses while
- competition regulators in the U.S. and the EU say monopoly is
- inevitable so we should punish tech for its abuses but not try to
- break up the monopolies.
- </p><p>
- To understand how tech became so monopolistic, it’s useful to look
- at the dawn of the consumer tech industry: 1979, the year the Apple
- II Plus launched and became the first successful home computer. That
- also happens to be the year that Ronald Reagan hit the campaign
- trail for the 1980 presidential race — a race he won, leading to a
- radical shift in the way that antitrust concerns are handled in
- America. Reagan’s cohort of politicians — including Margaret
- Thatcher in the U.K., Brian Mulroney in Canada, Helmut Kohl in
- Germany, and Augusto Pinochet in Chile — went on to enact similar
- reforms that eventually spread around the world.
- </p><p>
- Antitrust’s story began nearly a century before all that with laws
- like the Sherman Act, which took aim at monopolists on the grounds
- that monopolies were bad in and of themselves — squeezing out
- competitors, creating <span class="quote">“<span class="quote">diseconomies of scale</span>”</span> (when a company is so
- big that its constituent parts go awry and it is seemingly helpless
- to address the problems), and capturing their regulators to such a
- degree that they can get away with a host of evils.
- </p><p>
- Then came a fabulist named Robert Bork, a former solicitor general
- who Reagan appointed to the powerful U.S. Court of Appeals for the
- D.C. Circuit and who had created an alternate legislative history of
- the Sherman Act and its successors out of whole cloth. Bork insisted
- that these statutes were never targeted at monopolies (despite a
- wealth of evidence to the contrary, including the transcribed
- speeches of the acts’ authors) but, rather, that they were intended
- to prevent <span class="quote">“<span class="quote">consumer harm</span>”</span> — in the form of higher prices.
- </p><p>
- Bork was a crank, but he was a crank with a theory that rich people
- really liked. Monopolies are a great way to make rich people richer
- by allowing them to receive <span class="quote">“<span class="quote">monopoly rents</span>”</span> (that is, bigger
- profits) and capture regulators, leading to a weaker, more favorable
- regulatory environment with fewer protections for customers,
- suppliers, the environment, and workers.
- </p><p>
- Bork’s theories were especially palatable to the same power brokers
- who backed Reagan, and Reagan’s Department of Justice and other
- agencies began to incorporate Bork’s antitrust doctrine into their
- enforcement decisions (Reagan even put Bork up for a Supreme Court
- seat, but Bork flunked the Senate confirmation hearing so badly
- that, 40 years later, D.C. insiders use the term <span class="quote">“<span class="quote">borked</span>”</span> to refer
- to any catastrophically bad political performance).
- </p><p>
- Little by little, Bork’s theories entered the mainstream, and their
- backers began to infiltrate the legal education field, even putting
- on junkets where members of the judiciary were treated to lavish
- meals, fun outdoor activities, and seminars where they were
- indoctrinated into the consumer harm theory of antitrust. The more
- Bork’s theories took hold, the more money the monopolists were
- making — and the more surplus capital they had at their disposal to
- lobby for even more Borkian antitrust influence campaigns.
- </p><p>
- The history of Bork’s antitrust theories is a really good example of
- the kind of covertly engineered shifts in public opinion that Zuboff
- warns us against, where fringe ideas become mainstream orthodoxy.
- But Bork didn’t change the world overnight. He played a very long
- game, for over a generation, and he had a tailwind because the same
- forces that backed oligarchic antitrust theories also backed many
- other oligarchic shifts in public opinion. For example, the idea
- that taxation is theft, that wealth is a sign of virtue, and so on —
- all of these theories meshed to form a coherent ideology that
- elevated inequality to a virtue.
- </p><p>
- Today, many fear that machine learning allows surveillance
- capitalism to sell <span class="quote">“<span class="quote">Bork-as-a-Service,</span>”</span> at internet speeds, so that
- you can contract a machine-learning company to engineer
- <span class="emphasis"><em>rapid</em></span> shifts in public sentiment without
- needing the capital to sustain a multipronged, multigenerational
- project working at the local, state, national, and global levels in
- business, law, and philosophy. I do not believe that such a project
- is plausible, though I agree that this is basically what the
- platforms claim to be selling. They’re just lying about it. Big Tech
- lies all the time, <span class="emphasis"><em>including</em></span> in their sales
- literature.
- </p><p>
- The idea that tech forms <span class="quote">“<span class="quote">natural monopolies</span>”</span> (monopolies that are
- the inevitable result of the realities of an industry, such as the
- monopolies that accrue the first company to run long-haul phone
- lines or rail lines) is belied by tech’s own history: In the absence
- of anti-competitive tactics, Google was able to unseat AltaVista and
- Yahoo; Facebook was able to head off Myspace. There are some
- advantages to gathering mountains of data, but those mountains of
- data also have disadvantages: liability (from leaking), diminishing
- returns (from old data), and institutional inertia (big companies,
- like science, progress one funeral at a time).
- </p><p>
- Indeed, the birth of the web saw a mass-extinction event for the
- existing giant, wildly profitable proprietary technologies that had
- capital, network effects, and walls and moats surrounding their
- businesses. The web showed that when a new industry is built around
- a protocol, rather than a product, the combined might of everyone
- who uses the protocol to reach their customers or users or
- communities outweighs even the most massive products. CompuServe,
- AOL, MSN, and a host of other proprietary walled gardens learned
- this lesson the hard way: Each believed it could stay separate from
- the web, offering <span class="quote">“<span class="quote">curation</span>”</span> and a guarantee of consistency and
- quality instead of the chaos of an open system. Each was wrong and
- ended up being absorbed into the public web.
- </p><p>
- Yes, tech is heavily monopolized and is now closely associated with
- industry concentration, but this has more to do with a matter of
- timing than its intrinsically monopolistic tendencies. Tech was born
- at the moment that antitrust enforcement was being dismantled, and
- tech fell into exactly the same pathologies that antitrust was
- supposed to guard against. To a first approximation, it is
- reasonable to assume that tech’s monopolies are the result of a lack
- of anti-monopoly action and not the much-touted unique
- characteristics of tech, such as network effects, first-mover
- advantage, and so on.
- </p><p>
- In support of this thesis, I offer the concentration that every
- <span class="emphasis"><em>other</em></span> industry has undergone over the same
- period. From professional wrestling to consumer packaged goods to
- commercial property leasing to banking to sea freight to oil to
- record labels to newspaper ownership to theme parks,
- <span class="emphasis"><em>every</em></span> industry has undergone a massive shift
- toward concentration. There’s no obvious network effects or
- first-mover advantage at play in these industries. However, in every
- case, these industries attained their concentrated status through
- tactics that were prohibited before Bork’s triumph: merging with
- major competitors, buying out innovative new market entrants,
- horizontal and vertical integration, and a suite of anti-competitive
- tactics that were once illegal but are not any longer.
- </p><p>
- Again: When you change the laws intended to prevent monopolies and
- then monopolies form in exactly the way the law was supposed to
- prevent, it is reasonable to suppose that these facts are related.
- Tech’s concentration can be readily explained without recourse to
- radical theories of network effects — but only if you’re willing to
- indict unregulated markets as tending toward monopoly. Just as a
- lifelong smoker can give you a hundred reasons why their smoking
- didn’t cause their cancer (<span class="quote">“<span class="quote">It was the environmental toxins</span>”</span>), true
- believers in unregulated markets have a whole suite of unconvincing
- explanations for monopoly in tech that leave capitalism intact.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="steering-with-the-windshield-wipers"></a>Steering with the windshield wipers</h2></div></div></div><p>
- It’s been 40 years since Bork’s project to rehabilitate monopolies
- achieved liftoff, and that is a generation and a half, which is
- plenty of time to take a common idea and make it seem outlandish and
- vice versa. Before the 1940s, affluent Americans dressed their baby
- boys in pink while baby girls wore blue (a <span class="quote">“<span class="quote">delicate and dainty</span>”</span>
- color). While gendered colors are obviously totally arbitrary, many
- still greet this news with amazement and find it hard to imagine a
- time when pink connoted masculinity.
- </p><p>
- After 40 years of studiously ignoring antitrust analysis and
- enforcement, it’s not surprising that we’ve all but forgotten that
- antitrust exists, that in living memory, growth through mergers and
- acquisitions were largely prohibited under law, that
- market-cornering strategies like vertical integration could land a
- company in court.
- </p><p>
- Antitrust is a market society’s steering wheel, the control of first
- resort to keep would-be masters of the universe in their lanes. But
- Bork and his cohort ripped out our steering wheel 40 years ago. The
- car is still barreling along, and so we’re yanking as hard as we can
- on all the <span class="emphasis"><em>other</em></span> controls in the car as well as
- desperately flapping the doors and rolling the windows up and down
- in the hopes that one of these other controls can be repurposed to
- let us choose where we’re heading before we careen off a cliff.
- </p><p>
- It’s like a 1960s science-fiction plot come to life: People stuck in
- a <span class="quote">“<span class="quote">generation ship,</span>”</span> plying its way across the stars, a ship once
- piloted by their ancestors; and now, after a great cataclysm, the
- ship’s crew have forgotten that they’re in a ship at all and no
- longer remember where the control room is. Adrift, the ship is
- racing toward its extinction, and unless we can seize the controls
- and execute emergency course correction, we’re all headed for a
- fiery death in the heart of a sun.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="surveillance-still-matters"></a>Surveillance still matters</h2></div></div></div><p>
- None of this is to minimize the problems with surveillance.
- Surveillance matters, and Big Tech’s use of surveillance
- <span class="emphasis"><em>is</em></span> an existential risk to our species, but
- that’s not because surveillance and machine learning rob us of our
- free will.
- </p><p>
- Surveillance has become <span class="emphasis"><em>much</em></span> more efficient
- thanks to Big Tech. In 1989, the Stasi — the East German secret
- police — had the whole country under surveillance, a massive
- undertaking that recruited one out of every 60 people to serve as an
- informant or intelligence operative.
- </p><p>
- Today, we know that the NSA is spying on a significant fraction of
- the entire world’s population, and its ratio of surveillance
- operatives to the surveilled is more like 1:10,000 (that’s probably
- on the low side since it assumes that every American with top-secret
- clearance is working for the NSA on this project — we don’t know how
- many of those cleared people are involved in NSA spying, but it’s
- definitely not all of them).
- </p><p>
- How did the ratio of surveillable citizens expand from 1:60 to
- 1:10,000 in less than 30 years? It’s thanks to Big Tech. Our devices
- and services gather most of the data that the NSA mines for its
- surveillance project. We pay for these devices and the services they
- connect to, and then we painstakingly perform the data-entry tasks
- associated with logging facts about our lives, opinions, and
- preferences. This mass surveillance project has been largely useless
- for fighting terrorism: The NSA can
- <a class="ulink" href="https://www.washingtonpost.com/world/national-security/nsa-cites-case-as-success-of-phone-data-collection-program/2013/08/08/fc915e5a-feda-11e2-96a8-d3b921c0924a_story.html" target="_top">only
- point to a single minor success story</a> in which it used its
- data collection program to foil an attempt by a U.S. resident to
- wire a few thousand dollars to an overseas terror group. It’s
- ineffective for much the same reason that commercial surveillance
- projects are largely ineffective at targeting advertising: The
- people who want to commit acts of terror, like people who want to
- buy a refrigerator, are extremely rare. If you’re trying to detect a
- phenomenon whose base rate is one in a million with an instrument
- whose accuracy is only 99%, then every true positive will come at
- the cost of 9,999 false positives.
- </p><p>
- Let me explain that again: If one in a million people is a
- terrorist, then there will only be about one terrorist in a random
- sample of one million people. If your test for detecting terrorists
- is 99% accurate, it will identify 10,000 terrorists in your
- million-person sample (1% of one million is 10,000). For every true
- positive, you’ll get 9,999 false positives.
- </p><p>
- In reality, the accuracy of algorithmic terrorism detection falls
- far short of the 99% mark, as does refrigerator ad targeting. The
- difference is that being falsely accused of wanting to buy a fridge
- is a minor nuisance while being falsely accused of planning a terror
- attack can destroy your life and the lives of everyone you love.
- </p><p>
- Mass state surveillance is only feasible because of surveillance
- capitalism and its extremely low-yield ad-targeting systems, which
- require a constant feed of personal data to remain barely viable.
- Surveillance capitalism’s primary failure mode is mistargeted ads
- while mass state surveillance’s primary failure mode is grotesque
- human rights abuses, tending toward totalitarianism.
- </p><p>
- State surveillance is no mere parasite on Big Tech, sucking up its
- data and giving nothing in return. In truth, the two are symbiotes:
- Big Tech sucks up our data for spy agencies, and spy agencies ensure
- that governments don’t limit Big Tech’s activities so severely that
- it would no longer serve the spy agencies’ needs. There is no firm
- distinction between state surveillance and surveillance capitalism;
- they are dependent on one another.
- </p><p>
- To see this at work today, look no further than Amazon’s home
- surveillance device, the Ring doorbell, and its associated app,
- Neighbors. Ring — a product that Amazon acquired and did not develop
- in house — makes a camera-enabled doorbell that streams footage from
- your front door to your mobile device. The Neighbors app allows you
- to form a neighborhood-wide surveillance grid with your fellow Ring
- owners through which you can share clips of <span class="quote">“<span class="quote">suspicious characters.</span>”</span>
- If you’re thinking that this sounds like a recipe for letting
- curtain-twitching racists supercharge their suspicions of people
- with brown skin who walk down their blocks,
- <a class="ulink" href="https://www.eff.org/deeplinks/2020/07/amazons-ring-enables-over-policing-efforts-some-americas-deadliest-law-enforcement" target="_top">you’re
- right</a>. Ring has become a <span class="emphasis"><em>de facto,</em></span>
- off-the-books arm of the police without any of the pesky oversight
- or rules.
- </p><p>
- In mid-2019, a series of public records requests revealed that
- Amazon had struck confidential deals with more than 400 local law
- enforcement agencies through which the agencies would promote Ring
- and Neighbors and in exchange get access to footage from Ring
- cameras. In theory, cops would need to request this footage through
- Amazon (and internal documents reveal that Amazon devotes
- substantial resources to coaching cops on how to spin a convincing
- story when doing so), but in practice, when a Ring customer turns
- down a police request, Amazon only requires the agency to formally
- request the footage from the company, which it will then produce.
- </p><p>
- Ring and law enforcement have found many ways to intertwine their
- activities. Ring strikes secret deals to acquire real-time access to
- 911 dispatch and then streams alarming crime reports to Neighbors
- users, which serve as convincers for anyone who’s contemplating a
- surveillance doorbell but isn’t sure whether their neighborhood is
- dangerous enough to warrant it.
- </p><p>
- The more the cops buzz-market the surveillance capitalist Ring, the
- more surveillance capability the state gets. Cops who rely on
- private entities for law-enforcement roles then brief against any
- controls on the deployment of that technology while the companies
- return the favor by lobbying against rules requiring public
- oversight of police surveillance technology. The more the cops rely
- on Ring and Neighbors, the harder it will be to pass laws to curb
- them. The fewer laws there are against them, the more the cops will
- rely on them.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="dignity-and-sanctuary"></a>Dignity and sanctuary</h2></div></div></div><p>
- But even if we could exercise democratic control over our states and
- force them to stop raiding surveillance capitalism’s reservoirs of
- behavioral data, surveillance capitalism would still harm us.
- </p><p>
- This is an area where Zuboff shines. Her chapter on <span class="quote">“<span class="quote">sanctuary</span>”</span> —
- the feeling of being unobserved — is a beautiful hymn to
- introspection, calmness, mindfulness, and tranquility.
- </p><p>
- When you are watched, something changes. Anyone who has ever raised
- a child knows this. You might look up from your book (or more
- realistically, from your phone) and catch your child in a moment of
- profound realization and growth, a moment where they are learning
- something that is right at the edge of their abilities, requiring
- their entire ferocious concentration. For a moment, you’re
- transfixed, watching that rare and beautiful moment of focus playing
- out before your eyes, and then your child looks up and sees you
- seeing them, and the moment collapses. To grow, you need to be and
- expose your authentic self, and in that moment, you are vulnerable
- like a hermit crab scuttling from one shell to the next. The tender,
- unprotected tissues you expose in that moment are too delicate to
- reveal in the presence of another, even someone you trust as
- implicitly as a child trusts their parent.
- </p><p>
- In the digital age, our authentic selves are inextricably tied to
- our digital lives. Your search history is a running ledger of the
- questions you’ve pondered. Your location history is a record of the
- places you’ve sought out and the experiences you’ve had there. Your
- social graph reveals the different facets of your identity, the
- people you’ve connected with.
- </p><p>
- To be observed in these activities is to lose the sanctuary of your
- authentic self.
- </p><p>
- There’s another way in which surveillance capitalism robs us of our
- capacity to be our authentic selves: by making us anxious.
- Surveillance capitalism isn’t really a mind-control ray, but you
- don’t need a mind-control ray to make someone anxious. After all,
- another word for anxiety is agitation, and to make someone
- experience agitation, you need merely to agitate them. To poke them
- and prod them and beep at them and buzz at them and bombard them on
- an intermittent schedule that is just random enough that our limbic
- systems never quite become inured to it.
- </p><p>
- Our devices and services are <span class="quote">“<span class="quote">general purpose</span>”</span> in that they can
- connect anything or anyone to anything or anyone else and that they
- can run any program that can be written. This means that the
- distraction rectangles in our pockets hold our most precious moments
- with our most beloved people and their most urgent or time-sensitive
- communications (from <span class="quote">“<span class="quote">running late can you get the kid?</span>”</span> to <span class="quote">“<span class="quote">doctor
- gave me bad news and I need to talk to you RIGHT NOW</span>”</span>) as well as
- ads for refrigerators and recruiting messages from Nazis.
- </p><p>
- All day and all night, our pockets buzz, shattering our
- concentration and tearing apart the fragile webs of connection we
- spin as we think through difficult ideas. If you locked someone in a
- cell and agitated them like this, we’d call it <span class="quote">“<span class="quote">sleep deprivation
- torture,</span>”</span> and it would be
- <a class="ulink" href="https://www.youtube.com/watch?v=1SKpRbvnx6g" target="_top">a war crime
- under the Geneva Conventions</a>.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="afflicting-the-afflicted"></a>Afflicting the afflicted</h2></div></div></div><p>
- The effects of surveillance on our ability to be our authentic
- selves are not equal for all people. Some of us are lucky enough to
- live in a time and place in which all the most important facts of
- our lives are widely and roundly socially acceptable and can be
- publicly displayed without the risk of social consequence.
- </p><p>
- But for many of us, this is not true. Recall that in living memory,
- many of the ways of being that we think of as socially acceptable
- today were once cause for dire social sanction or even imprisonment.
- If you are 65 years old, you have lived through a time in which
- people living in <span class="quote">“<span class="quote">free societies</span>”</span> could be imprisoned or sanctioned
- for engaging in homosexual activity, for falling in love with a
- person whose skin was a different color than their own, or for
- smoking weed.
- </p><p>
- Today, these activities aren’t just decriminalized in much of the
- world, they’re considered normal, and the fallen prohibitions are
- viewed as shameful, regrettable relics of the past.
- </p><p>
- How did we get from prohibition to normalization? Through private,
- personal activity: People who were secretly gay or secret
- pot-smokers or who secretly loved someone with a different skin
- color were vulnerable to retaliation if they made their true selves
- known and were limited in how much they could advocate for their own
- right to exist in the world and be true to themselves. But because
- there was a private sphere, these people could form alliances with
- their friends and loved ones who did not share their disfavored
- traits by having private conversations in which they came out,
- disclosing their true selves to the people around them and bringing
- them to their cause one conversation at a time.
- </p><p>
- The right to choose the time and manner of these conversations was
- key to their success. It’s one thing to come out to your dad while
- you’re on a fishing trip away from the world and another thing
- entirely to blurt it out over the Christmas dinner table while your
- racist Facebook uncle is there to make a scene.
- </p><p>
- Without a private sphere, there’s a chance that none of these
- changes would have come to pass and that the people who benefited
- from these changes would have either faced social sanction for
- coming out to a hostile world or would have never been able to
- reveal their true selves to the people they love.
- </p><p>
- The corollary is that, unless you think that our society has
- attained social perfection — that your grandchildren in 50 years
- will ask you to tell them the story of how, in 2020, every injustice
- had been righted and no further change had to be made — then you
- should expect that right now, at this minute, there are people you
- love, whose happiness is key to your own, who have a secret in their
- hearts that stops them from ever being their authentic selves with
- you. These people are sorrowing and will go to their graves with
- that secret sorrow in their hearts, and the source of that sorrow
- will be the falsity of their relationship to you.
- </p><p>
- A private realm is necessary for human progress.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="any-data-you-collect-and-retain-will-eventually-leak"></a>Any data you collect and retain will eventually leak</h2></div></div></div><p>
- The lack of a private life can rob vulnerable people of the chance
- to be their authentic selves and constrain our actions by depriving
- us of sanctuary, but there is another risk that is borne by
- everyone, not just people with a secret: crime.
- </p><p>
- Personally identifying information is of very limited use for the
- purpose of controlling peoples’ minds, but identity theft — really a
- catchall term for a whole constellation of terrible criminal
- activities that can destroy your finances, compromise your personal
- integrity, ruin your reputation, or even expose you to physical
- danger — thrives on it.
- </p><p>
- Attackers are not limited to using data from one breached source,
- either. Multiple services have suffered breaches that exposed names,
- addresses, phone numbers, passwords, sexual tastes, school grades,
- work performance, brushes with the criminal justice system, family
- details, genetic information, fingerprints and other biometrics,
- reading habits, search histories, literary tastes, pseudonymous
- identities, and other sensitive information. Attackers can merge
- data from these different breaches to build up extremely detailed
- dossiers on random subjects and then use different parts of the data
- for different criminal purposes.
- </p><p>
- For example, attackers can use leaked username and password
- combinations to hijack whole fleets of commercial vehicles that
- <a class="ulink" href="https://www.vice.com/en_us/article/zmpx4x/hacker-monitor-cars-kill-engine-gps-tracking-apps" target="_top">have
- been fitted with anti-theft GPS trackers and immobilizers</a> or
- to hijack baby monitors in order to
- <a class="ulink" href="https://www.washingtonpost.com/technology/2019/04/23/how-nest-designed-keep-intruders-out-peoples-homes-effectively-allowed-hackers-get/?utm_term=.15220e98c550" target="_top">terrorize
- toddlers with the audio tracks from pornography</a>. Attackers
- use leaked data to trick phone companies into giving them your phone
- number, then they intercept SMS-based two-factor authentication
- codes in order to take over your email, bank account, and/or
- cryptocurrency wallets.
- </p><p>
- Attackers are endlessly inventive in the pursuit of creative ways to
- weaponize leaked data. One common use of leaked data is to penetrate
- companies in order to access <span class="emphasis"><em>more</em></span> data.
- </p><p>
- Like spies, online fraudsters are totally dependent on companies
- over-collecting and over-retaining our data. Spy agencies sometimes
- pay companies for access to their data or intimidate them into
- giving it up, but sometimes they work just like criminals do — by
- <a class="ulink" href="https://www.bbc.com/news/world-us-canada-24751821" target="_top">sneaking
- data out of companies’ databases</a>.
- </p><p>
- The over-collection of data has a host of terrible social
- consequences, from the erosion of our authentic selves to the
- undermining of social progress, from state surveillance to an
- epidemic of online crime. Commercial surveillance is also a boon to
- people running influence campaigns, but that’s the least of our
- troubles.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="critical-tech-exceptionalism-is-still-tech-exceptionalism"></a>Critical tech exceptionalism is still tech
- exceptionalism</h2></div></div></div><p>
- Big Tech has long practiced technology exceptionalism: the idea that
- it should not be subject to the mundane laws and norms of
- <span class="quote">“<span class="quote">meatspace.</span>”</span> Mottoes like Facebook’s <span class="quote">“<span class="quote">move fast and break things</span>”</span>
- attracted justifiable scorn of the companies’ self-serving rhetoric.
- </p><p>
- Tech exceptionalism got us all into a lot of trouble, so it’s ironic
- and distressing to see Big Tech’s critics committing the same sin.
- </p><p>
- Big Tech is not a <span class="quote">“<span class="quote">rogue capitalism</span>”</span> that cannot be cured through
- the traditional anti-monopoly remedies of trustbusting (forcing
- companies to divest of competitors they have acquired) and bans on
- mergers to monopoly and other anti-competitive tactics. Big Tech
- does not have the power to use machine learning to influence our
- behavior so thoroughly that markets lose the ability to punish bad
- actors and reward superior competitors. Big Tech has no rule-writing
- mind-control ray that necessitates ditching our old toolbox.
- </p><p>
- The thing is, people have been claiming to have perfected
- mind-control rays for centuries, and every time, it turned out to be
- a con — though sometimes the con artists were also conning
- themselves.
- </p><p>
- For generations, the advertising industry has been steadily
- improving its ability to sell advertising services to businesses
- while only making marginal gains in selling those businesses’
- products to prospective customers. John Wanamaker’s lament that <span class="quote">“<span class="quote">50%
- of my advertising budget is wasted, I just don’t know which 50%</span>”</span> is
- a testament to the triumph of <span class="emphasis"><em>ad executives</em></span>,
- who successfully convinced Wanamaker that only half of the money he
- spent went to waste.
- </p><p>
- The tech industry has made enormous improvements in the science of
- convincing businesses that they’re good at advertising while their
- actual improvements to advertising — as opposed to targeting — have
- been pretty ho-hum. The vogue for machine learning — and the
- mystical invocation of <span class="quote">“<span class="quote">artificial intelligence</span>”</span> as a synonym for
- straightforward statistical inference techniques — has greatly
- boosted the efficacy of Big Tech’s sales pitch as marketers have
- exploited potential customers’ lack of technical sophistication to
- get away with breathtaking acts of overpromising and
- underdelivering.
- </p><p>
- It’s tempting to think that if businesses are willing to pour
- billions into a venture that the venture must be a good one. Yet
- there are plenty of times when this rule of thumb has led us astray.
- For example, it’s virtually unheard of for managed investment funds
- to outperform simple index funds, and investors who put their money
- into the hands of expert money managers overwhelmingly fare worse
- than those who entrust their savings to index funds. But managed
- funds still account for the majority of the money invested in the
- markets, and they are patronized by some of the richest, most
- sophisticated investors in the world. Their vote of confidence in an
- underperforming sector is a parable about the role of luck in wealth
- accumulation, not a sign that managed funds are a good buy.
- </p><p>
- The claims of Big Tech’s mind-control system are full of tells that
- the enterprise is a con. For example,
- <a class="ulink" href="https://www.frontiersin.org/articles/10.3389/fpsyg.2020.01415/full" target="_top">the
- reliance on the <span class="quote">“<span class="quote">Big Five</span>”</span> personality traits</a> as a primary
- means of influencing people even though the <span class="quote">“<span class="quote">Big Five</span>”</span> theory is
- unsupported by any large-scale, peer-reviewed studies and is
- <a class="ulink" href="https://www.wired.com/story/the-noisy-fallacies-of-psychographic-targeting/" target="_top">mostly
- the realm of marketing hucksters and pop psych</a>.
- </p><p>
- Big Tech’s promotional materials also claim that their algorithms
- can accurately perform <span class="quote">“<span class="quote">sentiment analysis</span>”</span> or detect peoples’ moods
- based on their <span class="quote">“<span class="quote">microexpressions,</span>”</span> but
- <a class="ulink" href="https://www.npr.org/2018/09/12/647040758/advertising-on-facebook-is-it-worth-it" target="_top">these
- are marketing claims, not scientific ones</a>. These methods are
- largely untested by independent scientific experts, and where they
- have been tested, they’ve been found sorely wanting.
- Microexpressions are particularly suspect as the companies that
- specialize in training people to detect them
- <a class="ulink" href="https://theintercept.com/2017/02/08/tsas-own-files-show-doubtful-science-behind-its-behavior-screening-program/" target="_top">have
- been shown</a> to underperform relative to random chance.
- </p><p>
- Big Tech has been so good at marketing its own supposed superpowers
- that it’s easy to believe that they can market everything else with
- similar acumen, but it’s a mistake to believe the hype. Any
- statement a company makes about the quality of its products is
- clearly not impartial. The fact that we distrust all the things that
- Big Tech says about its data handling, compliance with privacy laws,
- etc., is only reasonable — but why on Earth would we treat Big
- Tech’s marketing literature as the gospel truth? Big Tech lies about
- just about <span class="emphasis"><em>everything</em></span>, including how well its
- machine-learning fueled persuasion systems work.
- </p><p>
- That skepticism should infuse all of our evaluations of Big Tech and
- its supposed abilities, including our perusal of its patents. Zuboff
- vests these patents with enormous significance, pointing out that
- Google claimed extensive new persuasion capabilities in
- <a class="ulink" href="https://patents.google.com/patent/US20050131762A1/en" target="_top">its
- patent filings</a>. These claims are doubly suspect: first,
- because they are so self-serving, and second, because the patent
- itself is so notoriously an invitation to exaggeration.
- </p><p>
- Patent applications take the form of a series of claims and range
- from broad to narrow. A typical patent starts out by claiming that
- its authors have invented a method or system for doing every
- conceivable thing that anyone might do, ever, with any tool or
- device. Then it narrows that claim in successive stages until we get
- to the actual <span class="quote">“<span class="quote">invention</span>”</span> that is the true subject of the patent.
- The hope is that the patent examiner — who is almost certainly
- overworked and underinformed — will miss the fact that some or all
- of these claims are ridiculous, or at least suspect, and grant the
- patent’s broader claims. Patents for unpatentable things are still
- incredibly useful because they can be wielded against competitors
- who might license that patent or steer clear of its claims rather
- than endure the lengthy, expensive process of contesting it.
- </p><p>
- What’s more, software patents are routinely granted even though the
- filer doesn’t have any evidence that they can do the thing claimed
- by the patent. That is, you can patent an <span class="quote">“<span class="quote">invention</span>”</span> that you
- haven’t actually made and that you don’t know how to make.
- </p><p>
- With these considerations in hand, it becomes obvious that the fact
- that a Big Tech company has patented what it
- <span class="emphasis"><em>says</em></span> is an effective mind-control ray is
- largely irrelevant to whether Big Tech can in fact control our
- minds.
- </p><p>
- Big Tech collects our data for many reasons, including the
- diminishing returns on existing stores of data. But many tech
- companies also collect data out of a mistaken tech exceptionalist
- belief in the network effects of data. Network effects occur when
- each new user in a system increases its value. The classic example
- is fax machines: A single fax machine is of no use, two fax machines
- are of limited use, but every new fax machine that’s put to use
- after the first doubles the number of possible fax-to-fax links.
- </p><p>
- Data mined for predictive systems doesn’t necessarily produce these
- dividends. Think of Netflix: The predictive value of the data mined
- from a million English-speaking Netflix viewers is hardly improved
- by the addition of one more user’s viewing data. Most of the data
- Netflix acquires after that first minimum viable sample duplicates
- existing data and produces only minimal gains. Meanwhile, retraining
- models with new data gets progressively more expensive as the number
- of data points increases, and manual tasks like labeling and
- validating data do not get cheaper at scale.
- </p><p>
- Businesses pursue fads to the detriment of their profits all the
- time, especially when the businesses and their investors are not
- motivated by the prospect of becoming profitable but rather by the
- prospect of being acquired by a Big Tech giant or by having an IPO.
- For these firms, ticking faddish boxes like <span class="quote">“<span class="quote">collects as much data
- as possible</span>”</span> might realize a bigger return on investment than
- <span class="quote">“<span class="quote">collects a business-appropriate quantity of data.</span>”</span>
- </p><p>
- This is another harm of tech exceptionalism: The belief that more
- data always produces more profits in the form of more insights that
- can be translated into better mind-control rays drives firms to
- over-collect and over-retain data beyond all rationality. And since
- the firms are behaving irrationally, a good number of them will go
- out of business and become ghost ships whose cargo holds are stuffed
- full of data that can harm people in myriad ways — but which no one
- is responsible for antey longer. Even if the companies don’t go
- under, the data they collect is maintained behind the minimum viable
- security — just enough security to keep the company viable while it
- waits to get bought out by a tech giant, an amount calculated to
- spend not one penny more than is necessary on protecting data.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="how-monopolies-not-mind-control-drive-surveillance-capitalism-the-snapchat-story"></a>How monopolies, not mind control, drive surveillance
- capitalism: The Snapchat story</h2></div></div></div><p>
- For the first decade of its existence, Facebook competed with the
- social media giants of the day (Myspace, Orkut, etc.) by presenting
- itself as the pro-privacy alternative. Indeed, Facebook justified
- its walled garden — which let users bring in data from the web but
- blocked web services like Google Search from indexing and caching
- Facebook pages — as a pro-privacy measure that protected users from
- the surveillance-happy winners of the social media wars like
- Myspace.
- </p><p>
- Despite frequent promises that it would never collect or analyze its
- users’ data, Facebook periodically created initiatives that did just
- that, like the creepy, ham-fisted Beacon tool, which spied on you as
- you moved around the web and then added your online activities to
- your public timeline, allowing your friends to monitor your browsing
- habits. Beacon sparked a user revolt. Every time, Facebook backed
- off from its surveillance initiative, but not all the way;
- inevitably, the new Facebook would be more surveilling than the old
- Facebook, though not quite as surveilling as the intermediate
- Facebook following the launch of the new product or service.
- </p><p>
- The pace at which Facebook ramped up its surveillance efforts seems
- to have been set by Facebook’s competitive landscape. The more
- competitors Facebook had, the better it behaved. Every time a major
- competitor foundered, Facebook’s behavior
- <a class="ulink" href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3247362" target="_top">got
- markedly worse</a>.
- </p><p>
- All the while, Facebook was prodigiously acquiring companies,
- including a company called Onavo. Nominally, Onavo made a
- battery-monitoring mobile app. But the permissions that Onavo
- required were so expansive that the app was able to gather
- fine-grained telemetry on everything users did with their phones,
- including which apps they used and how they were using them.
- </p><p>
- Through Onavo, Facebook discovered that it was losing market share
- to Snapchat, an app that — like Facebook a decade before — billed
- itself as the pro-privacy alternative to the status quo. Through
- Onavo, Facebook was able to mine data from the devices of Snapchat
- users, including both current and former Snapchat users. This
- spurred Facebook to acquire Instagram — some features of which
- competed with Snapchat — and then allowed Facebook to fine-tune
- Instagram’s features and sales pitch to erode Snapchat’s gains and
- ensure that Facebook would not have to face the kinds of competitive
- pressures it had earlier inflicted on Myspace and Orkut.
- </p><p>
- The story of how Facebook crushed Snapchat reveals the relationship
- between monopoly and surveillance capitalism. Facebook combined
- surveillance with lax antitrust enforcement to spot the competitive
- threat of Snapchat on its horizon and then take decisive action
- against it. Facebook’s surveillance capitalism let it avert
- competitive pressure with anti-competitive tactics. Facebook users
- still want privacy — Facebook hasn’t used surveillance to brainwash
- them out of it — but they can’t get it because Facebook’s
- surveillance lets it destroy any hope of a rival service emerging
- that competes on privacy features.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="a-monopoly-over-your-friends"></a>A monopoly over your friends</h2></div></div></div><p>
- A decentralization movement has tried to erode the dominance of
- Facebook and other Big Tech companies by fielding <span class="quote">“<span class="quote">indieweb</span>”</span>
- alternatives — Mastodon as a Twitter alternative, Diaspora as a
- Facebook alternative, etc. — but these efforts have failed to attain
- any kind of liftoff.
- </p><p>
- Fundamentally, each of these services is hamstrung by the same
- problem: Every potential user for a Facebook or Twitter alternative
- has to convince all their friends to follow them to a decentralized
- web alternative in order to continue to realize the benefit of
- social media. For many of us, the only reason to have a Facebook
- account is that our friends have Facebook accounts, and the reason
- they have Facebook accounts is that <span class="emphasis"><em>we</em></span> have
- Facebook accounts.
- </p><p>
- All of this has conspired to make Facebook — and other dominant
- platforms — into <span class="quote">“<span class="quote">kill zones</span>”</span> that investors will not fund new
- entrants for.
- </p><p>
- And yet, all of today’s tech giants came into existence despite the
- entrenched advantage of the companies that came before them. To
- understand how that happened, you have to understand both
- interoperability and adversarial interoperability.
- </p><div class="blockquote"><blockquote class="blockquote"><p>
- The hard problem of our species is coordination.
- </p></blockquote></div><p>
- <span class="quote">“<span class="quote">Interoperability</span>”</span> is the ability of two technologies to work with
- one another: Anyone can make an LP that will play on any record
- player, anyone can make a filter you can install in your stove’s
- extractor fan, anyone can make gasoline for your car, anyone can
- make a USB phone charger that fits in your car’s cigarette lighter
- receptacle, anyone can make a light bulb that works in your light
- socket, anyone can make bread that will toast in your toaster.
- </p><p>
- Interoperability is often a source of innovation and consumer
- benefit: Apple made the first commercially successful PC, but
- millions of independent software vendors made interoperable programs
- that ran on the Apple II Plus. The simple analog antenna inputs on
- the back of TVs first allowed cable operators to connect directly to
- TVs, then they allowed game console companies and then personal
- computer companies to use standard televisions as displays. Standard
- RJ-11 telephone jacks allowed for the production of phones from a
- variety of vendors in a variety of forms, from the free
- football-shaped phone that came with a <span class="emphasis"><em>Sports
- Illustrated</em></span> subscription to business phones with
- speakers, hold functions, and so on and then answering machines and
- finally modems, paving the way for the internet revolution.
- </p><p>
- <span class="quote">“<span class="quote">Interoperability</span>”</span> is often used interchangeably with
- <span class="quote">“<span class="quote">standardization,</span>”</span> which is the process when manufacturers and other
- stakeholders hammer out a set of agreed-upon rules for implementing
- a technology, such as the electrical plug on your wall, the CAN bus
- used by your car’s computer systems, or the HTML instructions that
- your browser interprets.
- </p><p>
- But interoperability doesn’t require standardization — indeed,
- standardization often proceeds from the chaos of ad hoc
- interoperability measures. The inventor of the cigarette-lighter USB
- charger didn’t need to get permission from car manufacturers or even
- the manufacturers of the dashboard lighter subcomponent. The
- automakers didn’t take any countermeasures to prevent the use of
- these aftermarket accessories by their customers, but they also
- didn’t do anything to make life easier for the chargers’
- manufacturers. This is a kind of <span class="quote">“<span class="quote">neutral interoperability.</span>”</span>
- </p><p>
- Beyond neutral interoperability, there is <span class="quote">“<span class="quote">adversarial
- interoperability.</span>”</span> That’s when a manufacturer makes a product that
- interoperates with another manufacturer’s product <span class="emphasis"><em>despite
- the second manufacturer’s objections</em></span> and <span class="emphasis"><em>even
- if that means bypassing a security system designed to prevent
- interoperability</em></span>.
- </p><p>
- Probably the most familiar form of adversarial interoperability is
- third-party printer ink. Printer manufacturers claim that they sell
- printers below cost and that the only way they can recoup the losses
- they incur is by charging high markups on ink. To prevent the owners
- of printers from buying ink elsewhere, the printer companies deploy
- a suite of anti-customer security systems that detect and reject
- both refilled and third-party cartridges.
- </p><p>
- Owners of printers take the position that HP and Epson and Brother
- are not charities and that customers for their wares have no
- obligation to help them survive, and so if the companies choose to
- sell their products at a loss, that’s their foolish choice and their
- consequences to live with. Likewise, competitors who make ink or
- refill kits observe that they don’t owe printer companies anything,
- and their erosion of printer companies’ margins are the printer
- companies’ problems, not their competitors’. After all, the printer
- companies shed no tears when they drive a refiller out of business,
- so why should the refillers concern themselves with the economic
- fortunes of the printer companies?
- </p><p>
- Adversarial interoperability has played an outsized role in the
- history of the tech industry: from the founding of the <span class="quote">“<span class="quote">alt.*</span>”</span>
- Usenet hierarchy (which was started against the wishes of Usenet’s
- maintainers and which grew to be bigger than all of Usenet combined)
- to the browser wars (when Netscape and Microsoft devoted massive
- engineering efforts to making their browsers incompatible with the
- other’s special commands and peccadilloes) to Facebook (whose
- success was built in part by helping its new users stay in touch
- with friends they’d left behind on Myspace because Facebook supplied
- them with a tool that scraped waiting messages from Myspace and
- imported them into Facebook, effectively creating an Facebook-based
- Myspace reader).
- </p><p>
- Today, incumbency is seen as an unassailable advantage. Facebook is
- where all of your friends are, so no one can start a Facebook
- competitor. But adversarial compatibility reverses the competitive
- advantage: If you were allowed to compete with Facebook by providing
- a tool that imported all your users’ waiting Facebook messages into
- an environment that competed on lines that Facebook couldn’t cross,
- like eliminating surveillance and ads, then Facebook would be at a
- huge disadvantage. It would have assembled all possible ex-Facebook
- users into a single, easy-to-find service; it would have educated
- them on how a Facebook-like service worked and what its potential
- benefits were; and it would have provided an easy means for
- disgruntled Facebook users to tell their friends where they might
- expect better treatment.
- </p><p>
- Adversarial interoperability was once the norm and a key contributor
- to the dynamic, vibrant tech scene, but now it is stuck behind a
- thicket of laws and regulations that add legal risks to the
- tried-and-true tactics of adversarial interoperability. New rules
- and new interpretations of existing rules mean that a would-be
- adversarial interoperator needs to steer clear of claims under
- copyright, terms of service, trade secrecy, tortious interference,
- and patent.
- </p><p>
- In the absence of a competitive market, lawmakers have resorted to
- assigning expensive, state-like duties to Big Tech firms, such as
- automatically filtering user contributions for copyright
- infringement or terrorist and extremist content or detecting and
- preventing harassment in real time or controlling access to sexual
- material.
- </p><p>
- These measures put a floor under how small we can make Big Tech
- because only the very largest companies can afford the humans and
- automated filters needed to perform these duties.
- </p><p>
- But that’s not the only way in which making platforms responsible
- for policing their users undermines competition. A platform that is
- expected to police its users’ conduct must prevent many vital
- adversarial interoperability techniques lest these subvert its
- policing measures. For example, if someone using a Twitter
- replacement like Mastodon is able to push messages into Twitter and
- read messages out of Twitter, they could avoid being caught by
- automated systems that detect and prevent harassment (such as
- systems that use the timing of messages or IP-based rules to make
- guesses about whether someone is a harasser).
- </p><p>
- To the extent that we are willing to let Big Tech police itself —
- rather than making Big Tech small enough that users can leave bad
- platforms for better ones and small enough that a regulation that
- simply puts a platform out of business will not destroy billions of
- users’ access to their communities and data — we build the case that
- Big Tech should be able to block its competitors and make it easier
- for Big Tech to demand legal enforcement tools to ban and punish
- attempts at adversarial interoperability.
- </p><p>
- Ultimately, we can try to fix Big Tech by making it responsible for
- bad acts by its users, or we can try to fix the internet by cutting
- Big Tech down to size. But we can’t do both. To replace today’s
- giant products with pluralistic protocols, we need to clear the
- legal thicket that prevents adversarial interoperability so that
- tomorrow’s nimble, personal, small-scale products can federate
- themselves with giants like Facebook, allowing the users who’ve left
- to continue to communicate with users who haven’t left yet, reaching
- tendrils over Facebook’s garden wall that Facebook’s trapped users
- can use to scale the walls and escape to the global, open web.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="fake-news-is-an-epistemological-crisis"></a>Fake news is an epistemological crisis</h2></div></div></div><p>
- Tech is not the only industry that has undergone massive
- concentration since the Reagan era. Virtually every major industry —
- from oil to newspapers to meatpacking to sea freight to eyewear to
- online pornography — has become a clubby oligarchy that just a few
- players dominate.
- </p><p>
- At the same time, every industry has become something of a tech
- industry as general-purpose computers and general-purpose networks
- and the promise of efficiencies through data-driven analysis infuse
- every device, process, and firm with tech.
- </p><p>
- This phenomenon of industrial concentration is part of a wider story
- about wealth concentration overall as a smaller and smaller number
- of people own more and more of our world. This concentration of both
- wealth and industries means that our political outcomes are
- increasingly beholden to the parochial interests of the people and
- companies with all the money.
- </p><p>
- That means that whenever a regulator asks a question with an
- obvious, empirical answer (<span class="quote">“<span class="quote">Are humans causing climate change?</span>”</span> or
- <span class="quote">“<span class="quote">Should we let companies conduct commercial mass surveillance?</span>”</span> or
- <span class="quote">“<span class="quote">Does society benefit from allowing network neutrality
- violations?</span>”</span>), the answer that comes out is only correct if that
- correctness meets with the approval of rich people and the
- industries that made them so wealthy.
- </p><p>
- Rich people have always played an outsized role in politics and more
- so since the Supreme Court’s <span class="emphasis"><em>Citizens United</em></span>
- decision eliminated key controls over political spending. Widening
- inequality and wealth concentration means that the very richest
- people are now a lot richer and can afford to spend a lot more money
- on political projects than ever before. Think of the Koch brothers
- or George Soros or Bill Gates.
- </p><p>
- But the policy distortions of rich individuals pale in comparison to
- the policy distortions that concentrated industries are capable of.
- The companies in highly concentrated industries are much more
- profitable than companies in competitive industries — no competition
- means not having to reduce prices or improve quality to win
- customers — leaving them with bigger capital surpluses to spend on
- lobbying.
- </p><p>
- Concentrated industries also find it easier to collaborate on policy
- objectives than competitive ones. When all the top execs from your
- industry can fit around a single boardroom table, they often do. And
- <span class="emphasis"><em>when</em></span> they do, they can forge a consensus
- position on regulation.
- </p><p>
- Rising through the ranks in a concentrated industry generally means
- working at two or three of the big companies. When there are only
- relatively few companies in a given industry, each company has a
- more ossified executive rank, leaving ambitious execs with fewer
- paths to higher positions unless they are recruited to a rival. This
- means that the top execs in concentrated industries are likely to
- have been colleagues at some point and socialize in the same circles
- — connected through social ties or, say, serving as trustees for
- each others’ estates. These tight social bonds foster a collegial,
- rather than competitive, attitude.
- </p><p>
- Highly concentrated industries also present a regulatory conundrum.
- When an industry is dominated by just four or five companies, the
- only people who are likely to truly understand the industry’s
- practices are its veteran executives. This means that top regulators
- are often former execs of the companies they are supposed to be
- regulating. These turns in government are often tacitly understood
- to be leaves of absence from industry, with former employers
- welcoming their erstwhile watchdogs back into their executive ranks
- once their terms have expired.
- </p><p>
- All this is to say that the tight social bonds, small number of
- firms, and regulatory capture of concentrated industries give the
- companies that comprise them the power to dictate many, if not all,
- of the regulations that bind them.
- </p><p>
- This is increasingly obvious. Whether it’s payday lenders
- <a class="ulink" href="https://www.washingtonpost.com/business/2019/02/25/how-payday-lending-industry-insider-tilted-academic-research-its-favor/" target="_top">winning
- the right to practice predatory lending</a> or Apple
- <a class="ulink" href="https://www.vice.com/en_us/article/mgxayp/source-apple-will-fight-right-to-repair-legislation" target="_top">winning
- the right to decide who can fix your phone</a> or Google and
- Facebook winning the right to breach your private data without
- suffering meaningful consequences or victories for pipeline
- companies or impunity for opioid manufacturers or massive tax
- subsidies for incredibly profitable dominant businesses, it’s
- increasingly apparent that many of our official, evidence-based
- truth-seeking processes are, in fact, auctions for sale to the
- highest bidder.
- </p><p>
- It’s really impossible to overstate what a terrifying prospect this
- is. We live in an incredibly high-tech society, and none of us could
- acquire the expertise to evaluate every technological proposition
- that stands between us and our untimely, horrible deaths. You might
- devote your life to acquiring the media literacy to distinguish good
- scientific journals from corrupt pay-for-play lookalikes and the
- statistical literacy to evaluate the quality of the analysis in the
- journals as well as the microbiology and epidemiology knowledge to
- determine whether you can trust claims about the safety of vaccines
- — but that would still leave you unqualified to judge whether the
- wiring in your home will give you a lethal shock
- <span class="emphasis"><em>and</em></span> whether your car’s brakes’ software will
- cause them to fail unpredictably <span class="emphasis"><em>and</em></span> whether
- the hygiene standards at your butcher are sufficient to keep you
- from dying after you finish your dinner.
- </p><p>
- In a world as complex as this one, we have to defer to authorities,
- and we keep them honest by making those authorities accountable to
- us and binding them with rules to prevent conflicts of interest. We
- can’t possibly acquire the expertise to adjudicate conflicting
- claims about the best way to make the world safe and prosperous, but
- we <span class="emphasis"><em>can</em></span> determine whether the adjudication
- process itself is trustworthy.
- </p><p>
- Right now, it’s obviously not.
- </p><p>
- The past 40 years of rising inequality and industry concentration,
- together with increasingly weak accountability and transparency for
- expert agencies, has created an increasingly urgent sense of
- impending doom, the sense that there are vast conspiracies afoot
- that operate with tacit official approval despite the likelihood
- they are working to better themselves by ruining the rest of us.
- </p><p>
- For example, it’s been decades since Exxon’s own scientists
- concluded that its products would render the Earth uninhabitable by
- humans. And yet those decades were lost to us, in large part because
- Exxon lobbied governments and sowed doubt about the dangers of its
- products and did so with the cooperation of many public officials.
- When the survival of you and everyone you love is threatened by
- conspiracies, it’s not unreasonable to start questioning the things
- you think you know in an attempt to determine whether they, too, are
- the outcome of another conspiracy.
- </p><p>
- The collapse of the credibility of our systems for divining and
- upholding truths has left us in a state of epistemological chaos.
- Once, most of us might have assumed that the system was working and
- that our regulations reflected our best understanding of the
- empirical truths of the world as they were best understood — now we
- have to find our own experts to help us sort the true from the
- false.
- </p><p>
- If you’re like me, you probably believe that vaccines are safe, but
- you (like me) probably also can’t explain the microbiology or
- statistics. Few of us have the math skills to review the literature
- on vaccine safety and describe why their statistical reasoning is
- sound. Likewise, few of us can review the stats in the (now
- discredited) literature on opioid safety and explain how those stats
- were manipulated. Both vaccines and opioids were embraced by medical
- authorities, after all, and one is safe while the other could ruin
- your life. You’re left with a kind of inchoate constellation of
- rules of thumb about which experts you trust to fact-check
- controversial claims and then to explain how all those respectable
- doctors with their peer-reviewed research on opioid safety
- <span class="emphasis"><em>were</em></span> an aberration and then how you know that
- the doctors writing about vaccine safety are
- <span class="emphasis"><em>not</em></span> an aberration.
- </p><p>
- I’m 100% certain that vaccinating is safe and effective, but I’m
- also at something of a loss to explain exactly,
- <span class="emphasis"><em>precisely,</em></span> why I believe this, given all the
- corruption I know about and the many times the stamp of certainty
- has turned out to be a parochial lie told to further enrich the
- super rich.
- </p><p>
- Fake news — conspiracy theories, racist ideologies, scientific
- denialism — has always been with us. What’s changed today is not the
- mix of ideas in the public discourse but the popularity of the worst
- ideas in that mix. Conspiracy and denial have skyrocketed in
- lockstep with the growth of Big Inequality, which has also tracked
- the rise of Big Tech and Big Pharma and Big Wrestling and Big Car
- and Big Movie Theater and Big Everything Else.
- </p><p>
- No one can say for certain why this has happened, but the two
- dominant camps are idealism (the belief that the people who argue
- for these conspiracies have gotten better at explaining them, maybe
- with the help of machine-learning tools) or materialism (the ideas
- have become more attractive because of material conditions in the
- world).
- </p><p>
- I’m a materialist. I’ve been exposed to the arguments of conspiracy
- theorists all my life, and I have not experienced any qualitative
- leap in the quality of those arguments.
- </p><p>
- The major difference is in the world, not the arguments. In a time
- where actual conspiracies are commonplace, conspiracy theories
- acquire a ring of plausibility.
- </p><p>
- We have always had disagreements about what’s true, but today, we
- have a disagreement over how we know whether something is true. This
- is an epistemological crisis, not a crisis over belief. It’s a
- crisis over the credibility of our truth-seeking exercises, from
- scientific journals (in an era where the biggest journal publishers
- have been caught producing pay-to-play journals for junk science) to
- regulations (in an era where regulators are routinely cycling in and
- out of business) to education (in an era where universities are
- dependent on corporate donations to keep their lights on).
- </p><p>
- Targeting — surveillance capitalism — makes it easier to find people
- who are undergoing this epistemological crisis, but it doesn’t
- create the crisis. For that, you need to look to corruption.
- </p><p>
- And, conveniently enough, it’s corruption that allows surveillance
- capitalism to grow by dismantling monopoly protections, by
- permitting reckless collection and retention of personal data, by
- allowing ads to be targeted in secret, and by foreclosing on the
- possibility of going somewhere else where you might continue to
- enjoy your friends without subjecting yourself to commercial
- surveillance.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tech-is-different"></a>Tech is different</h2></div></div></div><p>
- I reject both iterations of technological exceptionalism. I reject
- the idea that tech is uniquely terrible and led by people who are
- greedier or worse than the leaders of other industries, and I reject
- the idea that tech is so good — or so intrinsically prone to
- concentration — that it can’t be blamed for its present-day
- monopolistic status.
- </p><p>
- I think tech is just another industry, albeit one that grew up in
- the absence of real monopoly constraints. It may have been first,
- but it isn’t the worst nor will it be the last.
- </p><p>
- But there’s one way in which I <span class="emphasis"><em>am</em></span> a tech
- exceptionalist. I believe that online tools are the key to
- overcoming problems that are much more urgent than tech
- monopolization: climate change, inequality, misogyny, and
- discrimination on the basis of race, gender identity, and other
- factors. The internet is how we will recruit people to fight those
- fights, and how we will coordinate their labor. Tech is not a
- substitute for democratic accountability, the rule of law, fairness,
- or stability — but it’s a means to achieve these things.
- </p><p>
- The hard problem of our species is coordination. Everything from
- climate change to social change to running a business to making a
- family work can be viewed as a collective action problem.
- </p><p>
- The internet makes it easier than at any time before to find people
- who want to work on a project with you — hence the success of free
- and open-source software, crowdfunding, and racist terror groups —
- and easier than ever to coordinate the work you do.
- </p><p>
- The internet and the computers we connect to it also possess an
- exceptional quality: general-purposeness. The internet is designed
- to allow any two parties to communicate any data, using any
- protocol, without permission from anyone else. The only production
- design we have for computers is the general-purpose, <span class="quote">“<span class="quote">Turing
- complete</span>”</span> computer that can run every program we can express in
- symbolic logic.
- </p><p>
- This means that every time someone with a special communications
- need invests in infrastructure and techniques to make the internet
- faster, cheaper, and more robust, this benefit redounds to everyone
- else who is using the internet to communicate. And this also means
- that every time someone with a special computing need invests to
- make computers faster, cheaper, and more robust, every other
- computing application is a potential beneficiary of this work.
- </p><p>
- For these reasons, every type of communication is gradually absorbed
- into the internet, and every type of device — from airplanes to
- pacemakers — eventually becomes a computer in a fancy case.
- </p><p>
- While these considerations don’t preclude regulating networks and
- computers, they do call for gravitas and caution when doing so
- because changes to regulatory frameworks could ripple out to have
- unintended consequences in many, many other domains.
- </p><p>
- The upshot of this is that our best hope of solving the big
- coordination problems — climate change, inequality, etc. — is with
- free, fair, and open tech. Our best hope of keeping tech free, fair,
- and open is to exercise caution in how we regulate tech and to
- attend closely to the ways in which interventions to solve one
- problem might create problems in other domains.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="ownership-of-facts"></a>Ownership of facts</h2></div></div></div><p>
- Big Tech has a funny relationship with information. When you’re
- generating information — anything from the location data streaming
- off your mobile device to the private messages you send to friends
- on a social network — it claims the rights to make unlimited use of
- that data.
- </p><p>
- But when you have the audacity to turn the tables — to use a tool
- that blocks ads or slurps your waiting updates out of a social
- network and puts them in another app that lets you set your own
- priorities and suggestions or crawls their system to allow you to
- start a rival business — they claim that you’re stealing from them.
- </p><p>
- The thing is, information is a very bad fit for any kind of private
- property regime. Property rights are useful for establishing markets
- that can lead to the effective development of fallow assets. These
- markets depend on clear titles to ensure that the things being
- bought and sold in them can, in fact, be bought and sold.
- </p><p>
- Information rarely has such a clear title. Take phone numbers:
- There’s clearly something going wrong when Facebook slurps up
- millions of users’ address books and uses the phone numbers it finds
- in them to plot out social graphs and fill in missing information
- about other users.
- </p><p>
- But the phone numbers Facebook nonconsensually acquires in this
- transaction are not the <span class="quote">“<span class="quote">property</span>”</span> of the users they’re taken from
- nor do they belong to the people whose phones ring when you dial
- those numbers. The numbers are mere integers, 10 digits in the U.S.
- and Canada, and they appear in millions of places, including
- somewhere deep in pi as well as numerous other contexts. Giving
- people ownership titles to integers is an obviously terrible idea.
- </p><p>
- Likewise for the facts that Facebook and other commercial
- surveillance operators acquire about us, like that we are the
- children of our parents or the parents to our children or that we
- had a conversation with someone else or went to a public place.
- These data points can’t be property in the sense that your house or
- your shirt is your property because the title to them is
- intrinsically muddy: Does your mom own the fact that she is your
- mother? Do you? Do both of you? What about your dad — does he own
- this fact too, or does he have to license the fact from you (or your
- mom or both of you) in order to use this fact? What about the
- hundreds or thousands of other people who know these facts?
- </p><p>
- If you go to a Black Lives Matter demonstration, do the other
- demonstrators need your permission to post their photos from the
- event? The online fights over
- <a class="ulink" href="https://www.wired.com/story/how-to-take-photos-at-protests/" target="_top">when
- and how to post photos from demonstrations</a> reveal a nuanced,
- complex issue that cannot be easily hand-waved away by giving one
- party a property right that everyone else in the mix has to respect.
- </p><p>
- The fact that information isn’t a good fit with property and markets
- doesn’t mean that it’s not valuable. Babies aren’t property, but
- they’re inarguably valuable. In fact, we have a whole set of rules
- just for babies as well as a subset of those rules that apply to
- humans more generally. Someone who argues that babies won’t be truly
- valuable until they can be bought and sold like loaves of bread
- would be instantly and rightfully condemned as a monster.
- </p><p>
- It’s tempting to reach for the property hammer when Big Tech treats
- your information like a nail — not least because Big Tech are such
- prolific abusers of property hammers when it comes to
- <span class="emphasis"><em>their</em></span> information. But this is a mistake. If we
- allow markets to dictate the use of our information, then we’ll find
- that we’re sellers in a buyers’ market where the Big Tech monopolies
- set a price for our data that is so low as to be insignificant or,
- more likely, set at a nonnegotiable price of zero in a click-through
- agreement that you don’t have the opportunity to modify.
- </p><p>
- Meanwhile, establishing property rights over information will create
- insurmountable barriers to independent data processing. Imagine that
- we require a license to be negotiated when a translated document is
- compared with its original, something Google has done and continues
- to do billions of times to train its automated language translation
- tools. Google can afford this, but independent third parties cannot.
- Google can staff a clearances department to negotiate one-time
- payments to the likes of the EU (one of the major repositories of
- translated documents) while independent watchdogs wanting to verify
- that the translations are well-prepared, or to root out bias in
- translations, will find themselves needing a staffed-up legal
- department and millions for licenses before they can even get
- started.
- </p><p>
- The same goes for things like search indexes of the web or photos of
- peoples’ houses, which have become contentious thanks to Google’s
- Street View project. Whatever problems may exist with Google’s
- photographing of street scenes, resolving them by letting people
- decide who can take pictures of the facades of their homes from a
- public street will surely create even worse ones. Think of how
- street photography is important for newsgathering — including
- informal newsgathering, like photographing abuses of authority — and
- how being able to document housing and street life are important for
- contesting eminent domain, advocating for social aid, reporting
- planning and zoning violations, documenting discriminatory and
- unequal living conditions, and more.
- </p><p>
- The ownership of facts is antithetical to many kinds of human
- progress. It’s hard to imagine a rule that limits Big Tech’s
- exploitation of our collective labors without inadvertently banning
- people from gathering data on online harassment or compiling indexes
- of changes in language or simply investigating how the platforms are
- shaping our discourse — all of which require scraping data that
- other people have created and subjecting it to scrutiny and
- analysis.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="persuasion-works-slowly"></a>Persuasion works… slowly</h2></div></div></div><p>
- The platforms may oversell their ability to persuade people, but
- obviously, persuasion works sometimes. Whether it’s the private
- realm that LGBTQ people used to recruit allies and normalize sexual
- diversity or the decadeslong project to convince people that markets
- are the only efficient way to solve complicated resource allocation
- problems, it’s clear that our societal attitudes
- <span class="emphasis"><em>can</em></span> change.
- </p><p>
- The project of shifting societal attitudes is a game of inches and
- years. For centuries, svengalis have purported to be able to
- accelerate this process, but even the most brutal forms of
- propaganda have struggled to make permanent changes. Joseph Goebbels
- was able to subject Germans to daily, mandatory, hourslong radio
- broadcasts, to round up and torture and murder dissidents, and to
- seize full control over their children’s education while banning any
- literature, broadcasts, or films that did not comport with his
- worldview.
- </p><p>
- Yet, after 12 years of terror, once the war ended, Nazi ideology was
- largely discredited in both East and West Germany, and a program of
- national truth and reconciliation was put in its place. Racism and
- authoritarianism were never fully abolished in Germany, but neither
- were the majority of Germans irrevocably convinced of Nazism — and
- the rise of racist authoritarianism in Germany today tells us that
- the liberal attitudes that replaced Nazism were no more permanent
- than Nazism itself.
- </p><p>
- Racism and authoritarianism have also always been with us. Anyone
- who’s reviewed the kind of messages and arguments that racists put
- forward today would be hard-pressed to say that they have gotten
- better at presenting their ideas. The same pseudoscience, appeals to
- fear, and circular logic that racists presented in the 1980s, when
- the cause of white supremacy was on the wane, are to be found in the
- communications of leading white nationalists today.
- </p><p>
- If racists haven’t gotten more convincing in the past decade, then
- how is it that more people were convinced to be openly racist at
- that time? I believe that the answer lies in the material world, not
- the world of ideas. The ideas haven’t gotten more convincing, but
- people have become more afraid. Afraid that the state can’t be
- trusted to act as an honest broker in life-or-death decisions, from
- those regarding the management of the economy to the regulation of
- painkillers to the rules for handling private information. Afraid
- that the world has become a game of musical chairs in which the
- chairs are being taken away at a never-before-seen rate. Afraid that
- justice for others will come at their expense. Monopolism isn’t the
- cause of these fears, but the inequality and material desperation
- and policy malpractice that monopolism contributes to is a
- significant contributor to these conditions. Inequality creates the
- conditions for both conspiracies and violent racist ideologies, and
- then surveillance capitalism lets opportunists target the fearful
- and the conspiracy-minded.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="paying-wont-help"></a>Paying won’t help</h2></div></div></div><p>
- As the old saw goes, <span class="quote">“<span class="quote">If you’re not paying for the product, you’re
- the product.</span>”</span>
- </p><p>
- It’s a commonplace belief today that the advent of free,
- ad-supported media was the original sin of surveillance capitalism.
- The reasoning is that the companies that charged for access couldn’t
- <span class="quote">“<span class="quote">compete with free</span>”</span> and so they were driven out of business. Their
- ad-supported competitors, meanwhile, declared open season on their
- users’ data in a bid to improve their ad targeting and make more
- money and then resorted to the most sensationalist tactics to
- generate clicks on those ads. If only we’d pay for media again, we’d
- have a better, more responsible, more sober discourse that would be
- better for democracy.
- </p><p>
- But the degradation of news products long precedes the advent of
- ad-supported online news. Long before newspapers were online, lax
- antitrust enforcement had opened the door for unprecedented waves of
- consolidation and roll-ups in newsrooms. Rival newspapers were
- merged, reporters and ad sales staff were laid off, physical plants
- were sold and leased back, leaving the companies loaded up with debt
- through leveraged buyouts and subsequent profit-taking by the new
- owners. In other words, it wasn’t merely shifts in the classified
- advertising market, which was long held to be the primary driver in
- the decline of the traditional newsroom, that made news companies
- unable to adapt to the internet — it was monopolism.
- </p><p>
- Then, as news companies <span class="emphasis"><em>did</em></span> come online, the ad
- revenues they commanded dropped even as the number of internet users
- (and thus potential online readers) increased. That shift was a
- function of consolidation in the ad sales market, with Google and
- Facebook emerging as duopolists who made more money every year from
- advertising while paying less and less of it to the publishers whose
- work the ads appeared alongside. Monopolism created a buyer’s market
- for ad inventory with Facebook and Google acting as gatekeepers.
- </p><p>
- Paid services continue to exist alongside free ones, and often it is
- these paid services — anxious to prevent people from bypassing their
- paywalls or sharing paid media with freeloaders — that exert the
- most control over their customers. Apple’s iTunes and App Stores are
- paid services, but to maximize their profitability, Apple has to
- lock its platforms so that third parties can’t make compatible
- software without permission. These locks allow the company to
- exercise both editorial control (enabling it to exclude
- <a class="ulink" href="https://ncac.org/news/blog/does-apples-strict-app-store-content-policy-limit-freedom-of-expression" target="_top">controversial
- political material</a>) and technological control, including
- control over who can repair the devices it makes. If we’re worried
- that ad-supported products deprive people of their right to
- self-determination by using persuasion techniques to nudge their
- purchase decisions a few degrees in one direction or the other, then
- the near-total control a single company holds over the decision of
- who gets to sell you software, parts, and service for your iPhone
- should have us very worried indeed.
- </p><p>
- We shouldn’t just be concerned about payment and control: The idea
- that paying will improve discourse is also dangerously wrong. The
- poor success rate of targeted advertising means that the platforms
- have to incentivize you to <span class="quote">“<span class="quote">engage</span>”</span> with posts at extremely high
- levels to generate enough pageviews to safeguard their profits. As
- discussed earlier, to increase engagement, platforms like Facebook
- use machine learning to guess which messages will be most
- inflammatory and make a point of shoving those into your eyeballs at
- every turn so that you will hate-click and argue with people.
- </p><p>
- Perhaps paying would fix this, the reasoning goes. If platforms
- could be economically viable even if you stopped clicking on them
- once your intellectual and social curiosity had been slaked, then
- they would have no reason to algorithmically enrage you to get more
- clicks out of you, right?
- </p><p>
- There may be something to that argument, but it still ignores the
- wider economic and political context of the platforms and the world
- that allowed them to grow so dominant.
- </p><p>
- Platforms are world-spanning and all-encompassing because they are
- monopolies, and they are monopolies because we have gutted our most
- important and reliable anti-monopoly rules. Antitrust was neutered
- as a key part of the project to make the wealthy wealthier, and that
- project has worked. The vast majority of people on Earth have a
- negative net worth, and even the dwindling middle class is in a
- precarious state, undersaved for retirement, underinsured for
- medical disasters, and undersecured against climate and technology
- shocks.
- </p><p>
- In this wildly unequal world, paying doesn’t improve the discourse;
- it simply prices discourse out of the range of the majority of
- people. Paying for the product is dandy, if you can afford it.
- </p><p>
- If you think today’s filter bubbles are a problem for our discourse,
- imagine what they’d be like if rich people inhabited free-flowing
- Athenian marketplaces of ideas where you have to pay for admission
- while everyone else lives in online spaces that are subsidized by
- wealthy benefactors who relish the chance to establish
- conversational spaces where the <span class="quote">“<span class="quote">house rules</span>”</span> forbid questioning the
- status quo. That is, imagine if the rich seceded from Facebook, and
- then, instead of running ads that made money for shareholders,
- Facebook became a billionaire’s vanity project that also happened to
- ensure that nobody talked about whether it was fair that only
- billionaires could afford to hang out in the rarified corners of the
- internet.
- </p><p>
- Behind the idea of paying for access is a belief that free markets
- will address Big Tech’s dysfunction. After all, to the extent that
- people have a view of surveillance at all, it is generally an
- unfavorable one, and the longer and more thoroughly one is
- surveilled, the less one tends to like it. Same goes for lock-in: If
- HP’s ink or Apple’s App Store were really obviously fantastic, they
- wouldn’t need technical measures to prevent users from choosing a
- rival’s product. The only reason these technical countermeasures
- exist is that the companies don’t believe their customers would
- <span class="emphasis"><em>voluntarily</em></span> submit to their terms, and they
- want to deprive them of the choice to take their business elsewhere.
- </p><p>
- Advocates for markets laud their ability to aggregate the diffused
- knowledge of buyers and sellers across a whole society through
- demand signals, price signals, and so on. The argument for
- surveillance capitalism being a <span class="quote">“<span class="quote">rogue capitalism</span>”</span> is that
- machine-learning-driven persuasion techniques distort
- decision-making by consumers, leading to incorrect signals —
- consumers don’t buy what they prefer, they buy what they’re tricked
- into preferring. It follows that the monopolistic practices of
- lock-in, which do far more to constrain consumers’ free choices, are
- even more of a <span class="quote">“<span class="quote">rogue capitalism.</span>”</span>
- </p><p>
- The profitability of any business is constrained by the possibility
- that its customers will take their business elsewhere. Both
- surveillance and lock-in are anti-features that no customer wants.
- But monopolies can capture their regulators, crush their
- competitors, insert themselves into their customers’ lives, and
- corral people into <span class="quote">“<span class="quote">choosing</span>”</span> their services regardless of whether
- they want them — it’s fine to be terrible when there is no
- alternative.
- </p><p>
- Ultimately, surveillance and lock-in are both simply business
- strategies that monopolists can choose. Surveillance companies like
- Google are perfectly capable of deploying lock-in technologies —
- just look at the onerous Android licensing terms that require
- device-makers to bundle in Google’s suite of applications. And
- lock-in companies like Apple are perfectly capable of subjecting
- their users to surveillance if it means keeping the Chinese
- government happy and preserving ongoing access to Chinese markets.
- Monopolies may be made up of good, ethical people, but as
- institutions, they are not your friend — they will do whatever they
- can get away with to maximize their profits, and the more
- monopolistic they are, the more they <span class="emphasis"><em>can</em></span> get
- away with.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="an-ecology-moment-for-trustbusting"></a>An <span class="quote">“<span class="quote">ecology</span>”</span> moment for trustbusting</h2></div></div></div><p>
- If we’re going to break Big Tech’s death grip on our digital lives,
- we’re going to have to fight monopolies. That may sound pretty
- mundane and old-fashioned, something out of the New Deal era, while
- ending the use of automated behavioral modification feels like the
- plotline of a really cool cyberpunk novel.
- </p><p>
- Meanwhile, breaking up monopolies is something we seem to have
- forgotten how to do. There is a bipartisan, trans-Atlantic consensus
- that breaking up companies is a fool’s errand at best — liable to
- mire your federal prosecutors in decades of litigation — and
- counterproductive at worst, eroding the <span class="quote">“<span class="quote">consumer benefits</span>”</span> of large
- companies with massive efficiencies of scale.
- </p><p>
- But trustbusters once strode the nation, brandishing law books,
- terrorizing robber barons, and shattering the illusion of
- monopolies’ all-powerful grip on our society. The trustbusting era
- could not begin until we found the political will — until the people
- convinced politicians they’d have their backs when they went up
- against the richest, most powerful men in the world.
- </p><p>
- Could we find that political will again?
- </p><p>
- Copyright scholar James Boyle has described how the term <span class="quote">“<span class="quote">ecology</span>”</span>
- marked a turning point in environmental activism. Prior to the
- adoption of this term, people who wanted to preserve whale
- populations didn’t necessarily see themselves as fighting the same
- battle as people who wanted to protect the ozone layer or fight
- freshwater pollution or beat back smog or acid rain.
- </p><p>
- But the term <span class="quote">“<span class="quote">ecology</span>”</span> welded these disparate causes together into a
- single movement, and the members of this movement found solidarity
- with one another. The people who cared about smog signed petitions
- circulated by the people who wanted to end whaling, and the
- anti-whalers marched alongside the people demanding action on acid
- rain. This uniting behind a common cause completely changed the
- dynamics of environmentalism, setting the stage for today’s climate
- activism and the sense that preserving the habitability of the
- planet Earth is a shared duty among all people.
- </p><p>
- I believe we are on the verge of a new <span class="quote">“<span class="quote">ecology</span>”</span> moment dedicated to
- combating monopolies. After all, tech isn’t the only concentrated
- industry nor is it even the <span class="emphasis"><em>most</em></span> concentrated
- of industries.
- </p><p>
- You can find partisans for trustbusting in every sector of the
- economy. Everywhere you look, you can find people who’ve been
- wronged by monopolists who’ve trashed their finances, their health,
- their privacy, their educations, and the lives of people they love.
- Those people have the same cause as the people who want to break up
- Big Tech and the same enemies. When most of the world’s wealth is in
- the hands of a very few, it follows that nearly every large company
- will have overlapping shareholders.
- </p><p>
- That’s the good news: With a little bit of work and a little bit of
- coalition building, we have more than enough political will to break
- up Big Tech and every other concentrated industry besides. First we
- take Facebook, then we take AT&T/WarnerMedia.
- </p><p>
- But here’s the bad news: Much of what we’re doing to tame Big Tech
- <span class="emphasis"><em>instead</em></span> of breaking up the big companies also
- forecloses on the possibility of breaking them up later.
- </p><p>
- Big Tech’s concentration currently means that their inaction on
- harassment, for example, leaves users with an impossible choice:
- absent themselves from public discourse by, say, quitting Twitter or
- endure vile, constant abuse. Big Tech’s over-collection and
- over-retention of data results in horrific identity theft. And their
- inaction on extremist recruitment means that white supremacists who
- livestream their shooting rampages can reach an audience of
- billions. The combination of tech concentration and media
- concentration means that artists’ incomes are falling even as the
- revenue generated by their creations are increasing.
- </p><p>
- Yet governments confronting all of these problems all inevitably
- converge on the same solution: deputize the Big Tech giants to
- police their users and render them liable for their users’ bad
- actions. The drive to force Big Tech to use automated filters to
- block everything from copyright infringement to sex-trafficking to
- violent extremism means that tech companies will have to allocate
- hundreds of millions to run these compliance systems.
- </p><p>
- These rules — the EU’s new Directive on Copyright, Australia’s new
- terror regulation, America’s FOSTA/SESTA sex-trafficking law and
- more — are not just death warrants for small, upstart competitors
- that might challenge Big Tech’s dominance but who lack the deep
- pockets of established incumbents to pay for all these automated
- systems. Worse still, these rules put a floor under how small we can
- hope to make Big Tech.
- </p><p>
- That’s because any move to break up Big Tech and cut it down to size
- will have to cope with the hard limit of not making these companies
- so small that they can no longer afford to perform these duties —
- and it’s <span class="emphasis"><em>expensive</em></span> to invest in those automated
- filters and outsource content moderation. It’s already going to be
- hard to unwind these deeply concentrated, chimeric behemoths that
- have been welded together in the pursuit of monopoly profits. Doing
- so while simultaneously finding some way to fill the regulatory void
- that will be left behind if these self-policing rulers were forced
- to suddenly abdicate will be much, much harder.
- </p><p>
- Allowing the platforms to grow to their present size has given them
- a dominance that is nearly insurmountable — deputizing them with
- public duties to redress the pathologies created by their size makes
- it virtually impossible to reduce that size. Lather, rinse, repeat:
- If the platforms don’t get smaller, they will get larger, and as
- they get larger, they will create more problems, which will give
- rise to more public duties for the companies, which will make them
- bigger still.
- </p><p>
- We can work to fix the internet by breaking up Big Tech and
- depriving them of monopoly profits, or we can work to fix Big Tech
- by making them spend their monopoly profits on governance. But we
- can’t do both. We have to choose between a vibrant, open internet or
- a dominated, monopolized internet commanded by Big Tech giants that
- we struggle with constantly to get them to behave themselves.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="make-big-tech-small-again"></a>Make Big Tech small again</h2></div></div></div><p>
- Trustbusting is hard. Breaking big companies into smaller ones is
- expensive and time-consuming. So time-consuming that by the time
- you’re done, the world has often moved on and rendered years of
- litigation irrelevant. From 1969 to 1982, the U.S. government
- pursued an antitrust case against IBM over its dominance of
- mainframe computing — but the case collapsed in 1982 because
- mainframes were being speedily replaced by PCs.
- </p><div class="blockquote"><blockquote class="blockquote"><p>
- A future U.S. president could simply direct their attorney general
- to enforce the law as it was written.
- </p></blockquote></div><p>
- It’s far easier to prevent concentration than to fix it, and
- reinstating the traditional contours of U.S. antitrust enforcement
- will, at the very least, prevent further concentration. That means
- bans on mergers between large companies, on big companies acquiring
- nascent competitors, and on platform companies competing directly
- with the companies that rely on the platforms.
- </p><p>
- These powers are all in the plain language of U.S. antitrust laws,
- so in theory, a future U.S. president could simply direct their
- attorney general to enforce the law as it was written. But after
- decades of judicial <span class="quote">“<span class="quote">education</span>”</span> in the benefits of monopolies, after
- multiple administrations that have packed the federal courts with
- lifetime-appointed monopoly cheerleaders, it’s not clear that mere
- administrative action would do the trick.
- </p><p>
- If the courts frustrate the Justice Department and the president,
- the next stop would be Congress, which could eliminate any doubt
- about how antitrust law should be enforced in the U.S. by passing
- new laws that boil down to saying, <span class="quote">“<span class="quote">Knock it off. We all know what
- the Sherman Act says. Robert Bork was a deranged fantasist. For
- avoidance of doubt, <span class="emphasis"><em>fuck that guy</em></span>.</span>”</span> In other
- words, the problem with monopolies is
- <span class="emphasis"><em>monopolism</em></span> — the concentration of power into
- too few hands, which erodes our right to self-determination. If
- there is a monopoly, the law wants it gone, period. Sure, get rid of
- monopolies that create <span class="quote">“<span class="quote">consumer harm</span>”</span> in the form of higher prices,
- but also, <span class="emphasis"><em>get rid of other monopolies, too.</em></span>
- </p><p>
- But this only prevents things from getting worse. To help them get
- better, we will have to build coalitions with other activists in the
- anti-monopoly ecology movement — a pluralism movement or a
- self-determination movement — and target existing monopolies in
- every industry for breakup and structural separation rules that
- prevent, for example, the giant eyewear monopolist Luxottica from
- dominating both the sale and the manufacture of spectacles.
- </p><p>
- In an important sense, it doesn’t matter which industry the breakups
- begin in. Once they start, shareholders in
- <span class="emphasis"><em>every</em></span> industry will start to eye their
- investments in monopolists skeptically. As trustbusters ride into
- town and start making lives miserable for monopolists, the debate
- around every corporate boardroom’s table will shift. People within
- corporations who’ve always felt uneasy about monopolism will gain a
- powerful new argument to fend off their evil rivals in the corporate
- hierarchy: <span class="quote">“<span class="quote">If we do it my way, we make less money; if we do it your
- way, a judge will fine us billions and expose us to ridicule and
- public disapprobation. So even though I get that it would be really
- cool to do that merger, lock out that competitor, or buy that little
- company and kill it before it can threaten it, we really shouldn’t —
- not if we don’t want to get tied to the DOJ’s bumper and get dragged
- up and down Trustbuster Road for the next 10 years.</span>”</span>
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="goto-10"></a>20 GOTO 10</h2></div></div></div><p>
- Fixing Big Tech will require a lot of iteration. As cyber lawyer
- Lawrence Lessig wrote in his 1999 book, <span class="emphasis"><em>Code and Other
- Laws of Cyberspace</em></span>, our lives are regulated by four
- forces: law (what’s legal), code (what’s technologically possible),
- norms (what’s socially acceptable), and markets (what’s profitable).
- </p><p>
- If you could wave a wand and get Congress to pass a law that
- re-fanged the Sherman Act tomorrow, you could use the impending
- breakups to convince venture capitalists to fund competitors to
- Facebook, Google, Twitter, and Apple that would be waiting in the
- wings after they were cut down to size.
- </p><p>
- But getting Congress to act will require a massive normative shift,
- a mass movement of people who care about monopolies — and pulling
- them apart.
- </p><p>
- Getting people to care about monopolies will take technological
- interventions that help them to see what a world free from Big Tech
- might look like. Imagine if someone could make a beloved (but
- unauthorized) third-party Facebook or Twitter client that dampens
- the anxiety-producing algorithmic drumbeat and still lets you talk
- to your friends without being spied upon — something that made
- social media more sociable and less toxic. Now imagine that it gets
- shut down in a brutal legal battle. It’s always easier to convince
- people that something must be done to save a thing they love than it
- is to excite them about something that doesn’t even exist yet.
- </p><p>
- Neither tech nor law nor code nor markets are sufficient to reform
- Big Tech. But a profitable competitor to Big Tech could bankroll a
- legislative push; legal reform can embolden a toolsmith to make a
- better tool; the tool can create customers for a potential business
- who value the benefits of the internet but want them delivered
- without Big Tech; and that business can get funded and divert some
- of its profits to legal reform. 20 GOTO 10 (or lather, rinse,
- repeat). Do it again, but this time, get farther! After all, this
- time you’re starting with weaker Big Tech adversaries, a
- constituency that understands things can be better, Big Tech rivals
- who’ll help ensure their own future by bankrolling reform, and code
- that other programmers can build on to weaken Big Tech even further.
- </p><p>
- The surveillance capitalism hypothesis — that Big Tech’s products
- really work as well as they say they do and that’s why everything is
- so screwed up — is way too easy on surveillance and even easier on
- capitalism. Companies spy because they believe their own BS, and
- companies spy because governments let them, and companies spy
- because any advantage from spying is so short-lived and minor that
- they have to do more and more of it just to stay in place.
- </p><p>
- As to why things are so screwed up? Capitalism. Specifically, the
- monopolism that creates inequality and the inequality that creates
- monopolism. It’s a form of capitalism that rewards sociopaths who
- destroy the real economy to inflate the bottom line, and they get
- away with it for the same reason companies get away with spying:
- because our governments are in thrall to both the ideology that says
- monopolies are actually just fine and in thrall to the ideology that
- says that in a monopolistic world, you’d better not piss off the
- monopolists.
- </p><p>
- Surveillance doesn’t make capitalism rogue. Capitalism’s unchecked
- rule begets surveillance. Surveillance isn’t bad because it lets
- people manipulate us. It’s bad because it crushes our ability to be
- our authentic selves — and because it lets the rich and powerful
- figure out who might be thinking of building guillotines and what
- dirt they can use to discredit those embryonic guillotine-builders
- before they can even get to the lumberyard.
- </p></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="up-and-through"></a>Up and through</h2></div></div></div><p>
- With all the problems of Big Tech, it’s tempting to imagine solving
- the problem by returning to a world without tech at all. Resist that
- temptation.
- </p><p>
- The only way out of our Big Tech problem is up and through. If our
- future is not reliant upon high tech, it will be because
- civilization has fallen. Big Tech wired together a planetary,
- species-wide nervous system that, with the proper reforms and course
- corrections, is capable of seeing us through the existential
- challenge of our species and planet. Now it’s up to us to seize the
- means of computation, putting that electronic nervous system under
- democratic, accountable control.
- </p><p>
- I am, secretly, despite what I have said earlier, a tech
- exceptionalist. Not in the sense of thinking that tech should be
- given a free pass to monopolize because it has <span class="quote">“<span class="quote">economies of scale</span>”</span>
- or some other nebulous feature. I’m a tech exceptionalist because I
- believe that getting tech right matters and that getting it wrong
- will be an unmitigated catastrophe — and doing it right can give us
- the power to work together to save our civilization, our species,
- and our planet.
- </p></div></div></body></html>
projects / text-destroy-surveillance.git / commitdiff