'Orwellian' technology no match for today's invasive corporate technology, assault on privacy

By Jon Katz
First Amendment Center scholar

11.20.98

Printer-friendly page

"Orwellian" is a description so over-used it has become an almost meaningless cliche. We hear it all the time, but no longer grasp its power or meaning.

In part, the power of the story comes from the author's despairing view of what technology might ultimately do to human dignity and independence. His hero Winston had to contend with techno-abuse by a remote, vicious and authoritarian government. The Big Screen was always watching him, even during forced daily exercise sessions. He was expected to think and behave in a certain way, and the punishment for trying to withdraw or remain private was death or life in prison.

In our time, the invasive and ubiquitous use of technology comes more from corporations than from heavy-handed government, so we perceive it to be less menacing.

But companies rapidly acquiring new digital behavior-monitoring technology can gather vastly more information about our personal lives or habits than Big Brother ever dreamed of turning on poor Winston and the citizens of Oceania.

The idea of Orwellian technology advanced so powerfully in 1984 (published in 1949) has never been as relevant or timely an issue as it is today, as decisions are being made all the time — and with almost no public debate or discussion — that will affect elemental ideas about choice, privacy and freedom.

Technology has long exceeded some of Orwell's gloomy predictions about it. We live with Orwellian dramas every day that would send Winston diving out of his monitored apartment window.

Remember all those TV screens beaming President Clinton's videotaped testimony, in which his image was on every wall in the country saying nothing? And wouldn't Big Brother have drooled over the invasive possibilities of Caller ID, which makes it impossible in our time to even change your mind about calling somebody and just hang up?

Then there was the journalistic Mindspeak surrounding those happy McCaugheys from Iowa, still being hailed by journalists for their seven "miracle" babies, even as medical alarms about the awful risks taken in bringing them into the world were largely ignored.

Unconsidered technology is dangerous technology, a point only madmen like the Unabomber seem willing to talk about much in our culture. The problem with our journalistic and media obsession about sex and technology isn't just that it's exaggerated or hysterical, but that it distracts us from considering the many serious issues technology does raise.

New software programs called Group or Collaborative Filtering (GF) are the latest example of unthinking and disturbing technology, advances in behavioral monitoring that cry out for public discussion and thought. GF is digital software designed not only to sell things, but to monitor our tastes and behavior so closely that companies can use the most intimate knowledge about our lives to alter what we do or don't buy; that is, to affect the behavior of acquisition, choice and taste.

Software developers are selling what even they call "Big Brother" Technologies.

The November issue of American Demographics Magazine has a cheerful piece about this new technology called "Reading Your Mind, Reaching Your Wallet," with this sub-hed: "This Big Brother technology records your buying habits and tastes online. And it could be coming to a store near you." Chances are, it already has.

As the online world continues to draw commercial retailers, many Web companies use personal recommendation programs to convert lookers into regular buyers. If you've ever been on Amazon.com, Barnes and Noble, Moviefinder, CDNow, or other established websites, you undoubtedly have noticed that these vendors have personalized product recommendations for you, based on the previous purchases you've made, or the purchases other people have made that are similar to the things you've bought.

When I first encountered this on Amazon.com, I was enchanted by the idea and impressed by the technology. This is the Gee Whiz approach to technology, epidemic in American life: "Gee whiz," I thought. "They can help me pick books!"

Most of the book recommendations were pretty close to target, and I was rattled to see that this Web site could choose books for me more precisely than my wife could. But I stopped chuckling when I read the American Demographics piece and saw that 40 per cent of all commercial websites now use recommendation technology, and 93 per cent of those that don't plan to add the application within the next year.

According to Jupiter Research, online sales could top $40.8 billion by 2002.

Suggestive selling, says the firm, could contribute 34 per cent of total sales revenues within the first year of implementation. "It's direct marketing to the n-th degree," says one New York City marketing analyst.

And to an Orwellian degree, too. Orwell's chilling vision was that ubiquitous forces would use technology to monitor every aspect of our outward and internal behavior to the point that we had no free will or privacy. In making government the villain, Orwell never quite imagined the American mega-corporation. Or did he?

American Demographic reports that collaborative filtering is soon to spread from cyberspace to stores and supermarkets near you, where the checkout cashier will be happy to remind you that the beets you always buy are on special today, or that you forgot the skim milk you usually get, or that your favorite laxative or toilet paper is on sale.

Reading this, I squirmed at my own lack of consciousness, as well as at the growing power of these corporations to gather so much information about myself and my family. For the longest time, it literally never crossed my mind that companies were collecting, building and undoubtedly sharing a growing digital file of all the things I buy, a file that was getting to know me better with each purchase. Did I really want to get this close to a Web site or supermarket, to share so many intimate personal and creative experiences with to many different retailers? I don't recall anybody ever asking me.

Nor is there any law that keeps these companies from sharing the behaviorial information they collect with other retailers. Nobody who's been online for long puts all that much stock in online security. An elemental law of technology is that humans can understand and undo whatever humans can make or do. It's hard to imagine it being worth anybody's time, but there are hundreds of thousands of people out there who could get my 1998 booklist in a nano-second if they wanted to. And soon, a complete list of the food I buy as well.

And while our government isn't in an especially menacing frame of mind right, it isn't too much of a stretch — especially this year — to picture what some overzealous prosecutor might do with my Collective Group Filtering Profile, should he or she decide to investigate me.

And what about the subtle directing of collaborative filtering on my behavior? I'm only looking at the books Amazon's software has decided to recommend to me. I might buy one of their recommended books without even bothering to browse or search, thus avoiding all sorts of books I might want, or that they might not know that I want (yet).

And do I want my hardware store manager to remind me that I'm almost out of light bulbs or that I haven't got salt yet for the first storms of winter? Like any good salesclerk, collaborative filtering sharpens its suggestions, thus increasing the likelihood of sales, as it learns more about the customer. But there's no law that keeps an online bookstore from sharing its many burgeoning profiles of customers with the local mall down the road orthat giant discount chain whose stores are getting to be as big as Delaware.

Advocates of collaborative filtering praise the technology's impact on sales and repeat traffic. Sixty-three per cent of Amazon.com customers are repeat buyers, according to Net Perceptions, while a typical commercial site averages 35 to 40 per cent.

And CF software makers are eager to move these new techno-marketing notions off the Net and the Web. According to American Demographic, one software company met recently with a major U.S. retailer that's evaluating the impact of technology in its stores. The chain's average customer drops by every 21 days, purchases an average of 13 or 14 items per trip, and spends approximately $75. If collaborative filtering software were in the store's computers, it would read items as they were scanned at the cash register — shampoo, nail polish remover, and deodorant, for example — and instantly determine which other products were bought with these items in the past: toothpaste, perhaps, or chocolate chip cookies. Printed on the back of the customer's receipt would be coupons for toothpaste and cookies, valid for the next 14 days.

Collaborative filtering is intriguing, even useful. But the fact that American Demographic can joke so casually about "Big Brother" technology is also revealing. It speaks to the American obsession with more sales all the time at any cost whatsoever. And to the fact that we've agreed to surrender so much information about ourselves without even giving the idea a tiny fraction of the discussion we have daily about keeping Jane and Johnny off the Playboy website.

The nature of the Internet is changing rapidly as middle-class consumers get wired and more and more people become comfortable with browsing the Web. According to Jupiter, 42.5 per cent of people who've been online for two years or more are shoppers; 28.8 per cent of people who've been online for less than two years are shoppers, but only 20.5 per cent of those who've only recently gotten online buy things there. This collision of so many potential consumers with so much gee-whiz behavioral technology can only mean the ability of corporations to observe and gather and share information about us will evolve rapidly.

It seems a bit knee-jerk, even paranoid, to worry that giant retail chains will gather vast amounts of personal data about all of us and use it to take over our lives. But it is undoubtedly true that technology is becoming less and less of a choice, and more of an overwhelming reality we can't escape from even if we want to.

Were the United States government gathering this material about our reading habits, libertarians of every stripe from both the left and right would be felling whole forests in outrage and protest. And there is nothing to keep candidates or political parties from buying information like this or gathering their own.

In the very linked age of computer networks, the trek from corporate to government computer isn't a very long trip. Police agencies are already using digitalized toll booth passes to monitor the comings of and goings of suspects in criminal cases.

In America, capitalism has become a religion and ideology that surpasses even technology. If we can possibly make more money, we do. Anything we can invent, create or deploy in the service of that ideal is okay and moves ahead with little attention from journalism or government.

Government de-regulation and the epidemic number of mergers, takeovers and acquisitions has led to larger, more powerful and wealthier corporations than would have been imaginable even just a few years ago. Their growth parallels the rise of the Digital Age, when techno- marketing is giving business news ways of studying and manipulating us that were not available before.

Even if ignored, the issues seem enormous. Do we have the right to buy books in private, and keep our choices to ourselves? Do we have the right to keep our choice of mouthwash and toilet paper personal? If we wish, can we keep some or all parts of ourselves and our identities private? Do we really want to surrender these choices without any public discussion at all? Can the individual possibly stand alone with dignity and independence against these gargantuan corporations and their overwhelming ability to observe us and affect the way we behave?

Suddenly, Winston's drama seems less remote, less of a cliche, especially his own exhausted reflections as he sat awaiting merciless justice in an Oceania jail cell: "There was truth and there was untruth, and if you clung to the truth even against the whole world, you were not mad. He was safe, everything was all right. He fell asleep murmuring "Sanity is not statistical," with the feeling that this remark contained in it a profound vision."