Four short links: 17 June 2009

Word Mining, Open Ideas, Power Meter BotNet, and Realtime Web Traffic Tracking

1. NY Times Mines Its Data To Identify Words That Readers Find Abstruse -- the feature that lets you highlight a word on a NY Times web page and get more information about it is something that irritates me. I'm fascinated by the analysis of their data: boggling that sumptuary is less perplexing than solipsistic. Louche (#3 on the list) has been my favourite word for two years, by the way, since I heard Dylan Moran toss it out in that uniquely facile way the Irish have with words. I think Irish citizens get this incredible competence with the English language for free, along with staggering house prices and beer you can walk on.
2. Open Ideas -- Alex Payne's blog of Concepts in the public domain, awaiting collaboration and appropriation.
3. Buggy 'smart meters' open door to power-grid botnet (The Register) -- Paul Graham said that we've found what we get when we cross a television with a computer: a computer. Similarly, intelligent power meters are computers, computers that apparently haven't been well-secured. To prove his point, Davis and his IOActive colleagues designed a worm that self-propagates across a large number of one manufacturer's smart meter. Once infected, the device is under the control of the malware developers in much the way infected PCs are under the spell of bot herders. Attackers can then send instructions that cause its software to turn power on or off and reveal power usage or sensitive system configuration settings.
4. Chartbeat -- the sexiest web analytics ever. It gives realtime count of users, whether they're reading or writing (based on whether focus is in a form element), where they're from, mentions on Twitter, and more and more and more. This is a different form of analytics than Google Analytics, which tells you trends and historical access. Love this for the pure sex appeal of a heads-up dashboard that can tell you exactly how many people are on your site and exactly what they're doing. (via Artur)

Credit card company data mining makes us all instances of a type

The New York Times has recently published one of their in-depth, riveting descriptions of how credit card companies use everything they can learn about us. Any detail can be meaningful: what time of day you buy things, or the quality of the objects you choose.

The way credit collectors use psychology reminds me of CIA interogators (without the physical aspects of pressure). In fact, they're probably more effective than CIA interogators because they stick to the basic insight that kindness elicits more cooperation than threats.

So who gave them permission to use our purchase information against us? What law could possibly address this kind of power play?

There's another disturbing aspect to the data mining: it treats us all as examples of a pattern rather than as individuals. Almost eleven years I wrote an article criticizing this trend. The New York Times article shows how much we've lost from what we consider essential to our identity--our individuality.

Update

This article drew six comments in a few hours--thoughtful and valid comments, which have made me set down attitudes into words. Now we can look put the attitudes under a light and see what makes sense, or doesn't, to readers.

The article contained two levels of criticism: a criticism of data mining to build up composite pictures of individuals, and a criticism of the use of data accumulated from routine transactions to manipulate those individuals.

Building up a composite picture

Of course, a company that reaches out and does any marketing has to target people. Someone who bought the O'Reilly book Even Faster Web Sites (sorry about the plug) might appreciate a notification about our upcoming Velocity conference, which was founded by the book's author and covers the same topics. Someone who bought a book on a totally different subject wouldn't want or respond to the notification. O'Reilly does this kind of targeting, like most companies, and until everybody participates in truly frictionless information exchanges, companies will have to continue doing it.

Aggregated information is useful too. Organizations that mine public data for evidence of health epidemics can identify likely sites and investigate them further. The data mining is understood to provide an approximation of the truth.

Where I see a problem is when the increasing quantity of constant information refinement shades over into a qualitative change. There's a difference between a campaign targeted to 500 likely customers and a campaign targeted to one.

At some point the composite portrait starts to look so much like a person that corporate decision makers can begin to believe it is the person. The portrait becomes like a replicant, or like the statues that came to life in myths from Pygmalion to Pinocchio.

Joseph Weizenbaum, creator of the classic Eliza program, was shocked to see that people treated his "doctor" program like a human interviewer. There were plenty of computer programs that prompted the user with questions and gave varied responses based on the answers, but none had imitated a person so realistically.

Nowadays, nobody would be drawn in by Eliza. And perhaps companies and customers alike will get used to composite portraits. Perhaps the companies will send their composite to each of us and we can update it to make it more accurate. That will be a very different world, though.

Now we can turn to the next level, manipulation.

Manipulation

I've read numerous accounts in biographies and articles about interrogations, and talked to a couple people who have undergone interrogations. I haven't been on either side of an interrogation, but I've been deposed for a court case. All these situations remind me vividly of the exchanges reported in the New York Times article.

In these exchanges, a well-armed caller is laying, like a silkscreen, a composite over the real person and trying to manipulate the result. It's not exactly a case of asymmetric knowledge (because at least in theory, a customer could also learn a lot about a company and use that knowledge to manipulate it). It's more insidious: an employee carrying out a precise initiative on behalf of a company--a machine in the service of a goal--approaching the targeted customer in an informal manner that brings out a natural, human, empathetic reaction in customer.

Interrogation always takes place in the context of an open or implied threat--there would be no reason for making the contact otherwise--but as I mentioned in the article, the interrogation goes best when the threat is raised only rarely and strategically. A feigned sympathy and heart-to-heart engagement is the path to the most desired outcome.

In a sense, now, the employee has become the replicant. He is using a careful counterfeit of human responses to induce the behavior he or she is paid to induce. This is ethical when dealing with a criminal, although even then US law limits (based on the Fourth Amendment) the gathering of relevant information by the interrogator beforehand. I question how ethical it is in a business situation, especially when exploiting information given by the customer for entirely different purposes.