Four short links: 20 November 2009

Social Network Search for Morons, Bulking Up Bio Data, Better E-Mail, Better Standards

by Nat Torkington | @gnat | comments: 1

1. Spokeo -- abysmal indictment of society, first prize in mankind's race to the bottom. Uncover personal photos, videos, and secrets ... GUARANTEED! Spokeo deep searches within 48 major social networks to find truly mouth-watering news about friends and coworkers. PS, anybody who gives their gmail username and password to a site that specializes in dishing dirt can only be described as a fucking idiot. (via Jim Stogdill, who was equally disappointed in our species)
2. Biologists rally to sequence 'neglected' microbes (Nature) -- The Genomic Encyclopedia of Bacteria and Archaea is project to sequence genomes from more branches of the evolutionary tree of life. Eisen's team selected and sequenced more than 100 'neglected' species that lacked close relatives among the 1,000 genomes already in GenBank. The researchers reported earlier this year at the JGI's Fourth Annual User Meeting that even mapping the first 56 of these microbes' genomes increased the rate of discovery of new gene and protein families with new biological properties. It also improved the researchers' ability to predict the role of genes with unknown functions in already sequenced organisms. (via Jonathan Eisen)
3. Mail Learning: The What and the How (Simon Cozens) -- a few things that a really good mail analysis tool needs to do. I hope that my mail client and server does these out of the box in the next five years.
4. Introducing the Open Web Foundation Agreement -- The Open Web Foundation Agreement itself establishes the copyright and patent rights for a specification, ensuring that downstream consumers may freely implement and reuse the licensed specification without seeking further permission. In addition to the agreement itself, we also created an easy-to-read "Deed" that provides a high level overview of the agreement. Applying the open source approach to better standards.

Four short links: 2 October 2009

Social Media Parasites, Open Government Data, Prime Numbers, Amazon Image Abuse

by Nat Torkington | @gnat | comments: 1

1. I'm Tired of Your Analogue Attitude -- hilarious animated clip about social media gurus, made using xtranormal. (via trib on twitter)
2. Three Laws of Open Government Data -- 1. If it can’t be spidered or indexed, it doesn’t exist; 2. If it isn’t available in open and machine readable format, it can’t engage; 3. If a legal framework doesn’t allow it to be repurposed, it doesn’t empower. (also see slide deck)
3. Structure and Randomness in the Prime Numbers -- paper about some of the fun mathematics around prime numbers. (via Hacker News)
4. Abusing Amazon Images -- decoding and doing fun things with the Amazon images API. The cool thing (if you want to generate unlikely Amazon images) is that you're not limited to one use of any of these commands. You can have multiple discounts, multiple shadows, multiple bullets, generating images that Amazon would never have on its site. However, every additional command you add generates another 10% to the image dimensions, adding white space around the image. And that 10% compounds; add a lot of bullets, and you'll find that you have a small image in a large blank space. (You can use the CR command to cut away the excess, however.) Note also that the commands are interpreted in order, which can have an impact on what overlaps what.

Four short links: 20 August 2009

DIY SPY, Screencasting, Social Network Analysis, Term Extraction

by Nat Torkington | @gnat | comments: 1

1. DIY SPY - a homebrew 2.4GHz wi-fi spectrum analyzer -- As proof of concept (and a cool toy for anyone who has one of these lying around), I have implemented a working Wi-Fi spectrum analyzer on TI’s ez430-RF2500 development kit ($50), a 2-part USB dongle which consists essentially of a CC2500 radio strapped to an MSP430 low-power microcontroller (detachable bottom half) and a USB interface which enumerates as a virtual serial port (top half). The top half doubles as a standalone MSP430 programmer, so this kit is a great cheap way to get started playing with them. (via joshua on Delicious)
2. Screenr -- Instant screencasts for Twitter. Flash-based, uploads to their site and tweets the URL. The whole "for Twitter" thing is going a little too far: who records screencasts only for Twitter? It's like having a spellchecker only for three-letter words.
3. Social Network Analysis in R -- video and slides for talk on doing social network analysis with R.
4. We're Keeping the Term Extraction Service -- Yahoo!'s useful API gets a stay of execution. OK, we heard you. You’ve made it clear to us that shutting down the Term Extraction Service would be a mistake. So, we’ve changed our plans. We're leaving the service up and running indefinitely. (via Simon Willison)

Four short links: 15 July 2009

A collection inspired by Science Foo Camp attendees

by Nat Torkington | @gnat | comments: 1

1. Endogenous steroids and financial risk taking on a London trading floor (PNAS) -- We found that a trader's morning testosterone level predicts his day's profitability. We also found that a trader's cortisol rises with both the variance of his trading results and the volatility of the market. Our results suggest that higher testosterone may contribute to economic return, whereas cortisol is increased by risk. Our results point to a further possibility: testosterone and cortisol are known to have cognitive and behavioral effects, so if the acutely elevated steroids we observed were to persist or increase as volatility rises, they may shift risk preferences and even affect a trader's ability to engage in rational choice.
2. The Origin of Universal Scaling Laws in Biology -- eye-opening paper that blew my mind. Highlight of Sci Foo was meeting the author and shaking his hand. Relates metabolic rate, size, heart rate, and lifespan by applying physics to biology.
3. Ushahidi -- open source software for managing disasters. The Ushahidi Engine is a platform that allows anyone to gather distributed data via SMS, email or web and visualize it on a map or timeline. Our goal is to create the simplest way of aggregating information from the public for use in crisis response.
4. Dissecting the Canon: Visual Subject Co-Popularity Networks in Art Research -- In this paper we analyze a classic da- taset of art research, which collects ancient art and architecture and their Western Renaissance documentation since 1947. [T]here is clearly a long tail of monument popularity.

Four short links: 30 June 2009

Military Open Source, Social Govwork, Dietbot, and US IT Dashboard

by Nat Torkington | @gnat | comments: 0

1. Military Open Source Software Conference -- 12-13 August 2009 in Atlanta.
2. Govloop -- a "Social Network for Gov 2.0". Gov 2.0 could easily become the intersection of talk radio and social media consultant inanity. As with the Web 2.0 lunacy, when everyone who could spell wiki tried to sell one, you should cultivate the art of identifying and sidestepping the bozos, the time-wasters, and the charlatans who use buzzwords as a convenient alternative to thought. (via cheeky_geeky on Twitter)
3. Introducing the Autom -- a personal robot to help you lose weight. Developed by Initiative Automata as an offshoot from MIT researcher Cory Kidd, Autom has conversations that encourage you to record your diet and exercise. The theory is that the added benefit of interaction will help you stick with the diet longer, increasing the chance that it will stick. Trials showed Autom users stick with their "weight loss regimen" twice as long as pencil-and-paper. (via So, Where's My Robot?)
4. USA Government IT Dashboard Launches -- Vivek Kundra's latest project, a dashboard giving insight into government spending. Contractors, CIOs, projects, schedules, and data via an API. Built in Drupal!

The Digital Panopticon

by Joshua-Michéle Ross | @jmichele | comments: 16

This post is part three of a series raising questions about the mass adoption of social technologies. Here are links to part one and two. These posts will be opened to live discussion in an upcoming webcast on May 27. (special guest to be announced shortly)

In 1785 utilitarian philosopher Jeremy Bentham proposed architectural plans for the Panopticon, a prison Bentham described as "a new mode of obtaining power of mind over mind, in a quantity hitherto without example." Its method was a circular grid of surveillance; the jailors housed in a central tower being provided a 360-degree view of the imprisoned. Prisoners would not be able to tell when a jailor was actually watching or not. The premise ran that under the possibility of total surveillance (you could be being observed at any moment of the waking day) the prisoners would self-regulate their behavior to conform to prison norms. The perverse genius of the Panopticon was that even the jailor existed within this grid of surveillance; he could be viewed at any time (without knowing) by a still higher authority within the central tower - so the circle was complete, the surveillance - and thus conformance to authority - total.

In 1811 the King refused to authorize the sale of land for the purpose and Bentham was left frustrated in his vision to build the Panopticon. But the concept endured - not just as a literal architecture for controlling physical subjects (there are many Panopticons that now bear Bentham’s stamp) - but as a metaphor for understanding the function of power in modern times. French philosopher Michel Foucault dedicated a whole section of his book Discipline and Punish to the significance of the Panopticon. His take was essentially this: The same mechanism at work in the Panopticon - making subjects totally visible to authority - leads to those subjects internalizing the norms of power. In Foucault’s words “…the major effect of the Panopticon; to induce in the inmate a state of conscious and permanent visibility that assures the automatic functioning of power. So to arrange things that the surveillance is permanent in its effects, even if it is discontinuous in its action; that the perfection of power should tend to render its actual exercise unnecessary” In short, under the possibility of total surveillance the inmate becomes self regulating.

The social technologies we see in use today are fundamentally panoptical - the architecture of participation is inherently an architecture of surveillance.

In the age of social networks we find ourselves coming under a vast grid of surveillance - of permanent visibility. The routine self-reporting of what we are doing, reading, thinking via status updates makes our every action and location visible to the crowd. This visibility has a normative effect on behavior (in other words we conform our behavior and/or our speech about that behavior when we know we are being observed).

In many cases we are opting into automated reporting structures (Google Lattitude, Loopt etc.) that detail our location at any given point in time. We are doing this in exchange for small conveniences (finding local sushi more quickly, gaining “ambient intimacy”) without ever considering the bargain that we are striking. In short, we are creating the ultimate Panopticon - with our data centrally housed in the cloud (see previous post on the Captivity of the Commons) - our every movement, and up-to-the-minute status is a matter of public record. In the same way that networked communications move us from a one to many broadcast model to a many to many - so we are seeing the move to a many-to-many surveillance model. A global community of voyeurs ceaselessly confessing to "What are you doing? (Twitter) or "What's on your mind? (Facebook)

Captivity of the Commons focused on the risks corporate ownership of personal data. This post is concerned with how, as individuals, we have grown comfortable giving our information away; how our sense of privacy is changing under the small conveniences that disclosure brings. How our identity changes as an effect of constant self-disclosure. Many previous comments have rightly noted that privacy is often cultural -- if you don't expect it - there is no such thing as an infringement. Yet it is important to reckon with the changes we see occurring around us and argue what kind of a culture we wish to create (or contribute to).

Jacques Ellul’s book, Propaganda, had a thesis that was at once startling and obvious: Propaganda’s end goal is not to change your mind at any one point in time - but to create a changeable mind. Thus when invoked at the necessary time - humans could be manipulated into action. In the U.S. this language was expressed by catchphrases like, “communism in our backyard,” “enemies of freedom” or the current manufactured hysteria about Obama as a “socialist”.

Similarly the significance of status updates and location based services may not lie in the individual disclosure but in the significance of a culture that has become accustomed to constant disclosure.

Captivity of the Commons

by Joshua-Michéle Ross | @jmichele | comments: 25

This post is part two of the series, “The Question Concerning Social Technology”. Part one is here. These posts will be opened to live discussion in an upcoming webcast on May 27.

In January 2002 DARPA launched the Information Awareness Office. The mission was to, “ imagine, develop, apply, integrate, demonstrate and transition information technologies, components and prototype, closed-loop, information systems that will counter asymmetric threats by achieving total information awareness (emphasis added)” The notion of a government agency achieving total information awareness was too Orwellian to ignore. Under criticism that this “awareness” could quickly migrate to a mass surveillance system the program was defunded.

Fast-forward to last week and my near-purchase of Libbey Duratuff Gibralter Glasses (the perfect bourbon glass one might speculate). Over the course of the next few days I was peppered with exact-match ads for Libbey Duratuff glassware on several other websites; A small example of information awareness at work.

Personal data is the currency of Web 2.0. Knowing what we watch, buy, click, own, what we think, intend and ultimately do confers competitive advantage. Facebook possesses your social graph, your personal interests and your full profile (age, location, relationship status etc.) not to mention your daily (or hourly) answer to their persistent question, “what’s on your mind?”. Reviewing the “25 Surprising Things Google Knows About You” should give anyone pause. And it’s not just the Web 2.0 set. Credit Card Companies, Telcos, Insurance , Pharma… all are collecting vast stores of personal data. If you watch the trendline it is moving toward more data and more analytic capability - not less.

So why is it that we seem to have more comfort when the capacity for total information awareness lies with corporations as opposed to government? Experience shows that there is a very thin barrier between the two. To wit, the release of thousands of phone records to the U.S. government - and, conveniently, government immunity for those same corporations after the breach. Google and Yahoo! and Microsoft have all been accused of cooperating with the Chinese government to aid censorship and repression of free speech. What happens if/when we encounter the next version of the Bush administration that sees no problem abrogating civil rights in pursuit of “evildoers”?

What's more, when we deliver our personal information over to corporations we are giving this data over to an institution that is amoral. Companies are not yet structured to deliver moral or ethical results - they are encouraged to grow and deliver “shareholder value” (read money) which is a numb and narrow measure of value. Do I want my data to be managed by an amoral institution?

To be clear - I want the convenience and miracles that modern technology brings. I love the Internet and I am willing to give over lots of data in the trade. But I want two fundamental protections:

First, change the corporation. The structure of the corporation continues to be driven by 20th century hard goals of efficiency and scale - not by more complex measures of environmental sustainability, value creation and the commonweal. These are simply not adequately factored into any structural, organizational, incentive or taxation systems of business today. Profit and profit motive are fine - but hiding social and environmental costs is no longer acceptable. I want to deal with institutions capable of morality. This is no small task - but if we can build the Internet….

Second. We need a right to privacy that matches the 21st century reality. As a friend of mine likes to say, “privacy is now a responsibility - not a right.” While it is pithy (and perhaps true), the reason we grant rights - and laws to enforce those rights in society is the simple fact that people do not generally have the wherewithal to protect themselves from large, institutional interests. In the same way that regulatory structures are needed to keep a financial system in balance (alas even the Ayn Rand acolyte Greenspan finally agrees with this truism), we need new rights and regulations governing the use of our personal data - and simple sets of controls over who has access to it.

The true work of the 21st century lies not in refining our technology - this we will achieve without any political will. The work lies in re-imagining our institutions.

Why I Love Twitter

by Tim O'Reilly | @timoreilly | comments: 99

If you care what I think, you know that Twitter is just about the best way to learn what I'm paying attention to. I pass along tidbits of O'Reilly news, interesting reading from mailing lists and blogs I follow, and of course, tidbits from the twitterers I'm following. These are all the things I could never find time to put on my blog, but that I spray via email like a firehose at editors, conference planners, and researchers within O'Reilly. A lot of my job is, as we say, "redistributing the future" - following interesting people, and passing on what I learn to others. And twitter is an awesome tool for doing just that.

Like a lot of people, I tried out Twitter early on, but didn't stick to it. Most of the early twitter conversation was personal, and I didn't have time for it. I came back when I noticed that about 5000 people were following my non-existent updates, waiting for me to say something. With that many listeners, I thought I'd better oblige. (There are now close to 16,000.) I soon realized that Twitter has grown up to become a critical business tool, ideal for following the latest news, tracking the ideas and whereabouts of people who will shape the future of technology, and sharing my own thoughts and attention stream.

I thought I should outline here some of the specific things I find so compelling about Twitter, with suggestions about architectural features to be emulated by other internet services.

1. Twitter is simple. Twitter does one small thing, and does it well. Folks like Robert Scoble sing the praises of Friendfeed, which you could think of as twitter++. After all, it's got comments and aggregation of data from multiple services. But despite its powerful premise, Friendfeed hasn't dented Twitter's growth. Personally, I don't have time to wade through the comments; for me, Twitter is about quick hits, not about extended discussion. And while I love the promise of service aggregation, I tend to think that trying to marry it to commenting obscures its potential. Less is more. New services like peoplebrowsr are reframing service aggregation in a richer way, as a way of learning more about the people you follow, browsing the social graph. (Peoplebrowsr is still in alpha, but I think it has real potential as a social graph explorer, rather than as yet another people feed-reader.)

2. Twitter works like people do. If I'm interested in someone, I don't have to ask their permission to follow them. I don't have to ask if they will be my friend: that is something that evolves naturally over time. If you're a public figure like I am, the metaphor of mutual "friending" is truly broken. I get tens of thousands of friend requests from people I don't know. Accepting would make it impossible for me to use a social tool to keep in touch with my real friends. Friend groups don't really help.

   Twitter's brilliant social architecture means that anyone can follow me, and I can follow anyone else (unless they want to keep their updates private.) Gradually, through repeated contact, we become friends. @ replies that can only be seen by people followed by both parties to a conversation create a natural kind of social grouping, as well as social group extensibility, as I gradually get more and more visibility into new people that my friends already know. Meanwhile, truly private direct messages are also supported.

   I don't know who first used the term "ambient intimacy" but it's a great description of what begins to happen on Twitter. I know not just what people are thinking about or reading, but enough about what they are doing that our relationship deepens, just like real-world friendships. People who follow me on Twitter learn that I'm making jam or pies, or gardening or riding my bike or feeding the horses, things that I'd never (or rarely, since I'm doing it here) share on my blog. I know a lot more about many of my professional contacts that makes them more into friends. And in the case of my family, who keep their updates private and visible only to a limited group of real friends, we can keep in touch in small ways that mean a lot. I get special moments of my wife or daughters' day that we might not have shared otherwise. It's truly lovely.

3. Twitter cooperates well with others. Rather than loading itself down with features, it lets others extend its reach. There are dozens of powerful third-party interface programs; there are hundreds of add-on sites and tools. Twitter even lets competitors (like FriendFeed or Facebook) slurp its content into their services. But instead of strengthening them, it seems to strengthen Twitter. It's the new version of embrace and extend: inject and take over. (Scoble recently noticed that 60%+ of his friends' updates on Facebook actually came from twitter. And as John Battelle noted in a recent tweet, "I noticed now that my FBook status is updated with Twitter, I get responses in Fbook, but would like to see them here." It might seem like a strength for Facebook to allow Twitter to update its status feed, but not the other way around, but I think Facebook will one day realize that Twitter has taken them over....)

4. Twitter transcends the web. Like all of the key internet services today, Twitter is equally at home on the mobile phone. Even on the PC, I find myself using a separate client (Twhirl is an Adobe Air program) that provides a rich, alternate interface.

5. Twitter is user-extensible.The @syntax for referring to users, hashtags, and whatever you call the use of $ as a special symbol for reference to financial instruments, were all user-generated innovations that, because of Twitter's simplicity, allowed for third party services to be layered not just on the API, but on the content.

6. Twitter evolves quickly. Perhaps because its features are so minimal, new user behaviors seem to propagate across Twitter really quickly. It's a bit like the reason that fruit flies are used for genetic research: the short lifespan compresses the time for mutations to take hold. Perhaps a better analogy would be the speed of cultural evolution among humans compared to biological evolution. The most fascinating evolution happening on Twitter isn't an evolution of the software, but an evolution in user behavior and in the types of data that are being shared.

   I saw this myself with retweeting, a behavior I picked up not from Twitter itself but from twhirl, the Twitter client I use. Because Twhirl actually has a button for retweeting, it seemed the most natural thing in the world to do it. I became one of the most prolific retweeters, figuring that I have more followers than most of the people I know, and that it would be good form to pass on the best of what they post. But it's fascinating to see the growth of retweeting by others, even those not using twhirl.

In many ways, Twitter is a re-incarnation of the old Unix philosophy of simple, cooperating tools. The essence of Twitter is its constraints, the things it doesn't do, and the way that its core services aren't bound to a particular interface.

It strikes me that many of the programs that become enduring platforms have these same characteristics. Few people use the old TCP/IP-based applications like telnet and ftp any more, but TCP/IP itself is ubiquitous. No one uses the mail program any more, but all of us still use email. No one uses Tim Berners-Lee's original web server and browser any more. Both were superseded by independent programs that used his core innovations: http and html.

What's different, of course, is that Twitter isn't just a protocol. It's also a database. And that's the old secret of Web 2.0, Data is the Intel Inside. That means that they can let go of controlling the interface. The more other people build on Twitter, the better their position becomes.

There's a real lesson to Facebook here about giving other services (like Twitter) access to their social graph. They have the best one going, but because they try to keep users coming back to their interface, and even the applications built on their service have to live in Facebook, they end up as a ghetto rather than a true internet service. It's the data, not the interface! Let other people use your data, build on it, and it will still belong to you. Hold it too tight, and they will compete with it.

Lots more to say, but the beach is calling on this sunny Saturday.

Google's Social Graph API Learns a New Trick

by David Recordon | @daveman692 | comments: 1

This past February at Social Graph Foo Camp, Google released the first version of their Social Graph API. (see past Radar coverage) This API was focused on making it easier for developers to understand who a user is and find their other accounts around the web via publicly declared data.

Today I'm driving up to Foo Camp along with Brad Fitzpatrick, the developer of the API, where he has just pushed a new API method live called "OtherMe". This new methods focuses on making it easier for developers to get a holistic view of a user, their feeds, and some basic profile information. You can see this for my profiles in a pretty form from the API. Unfortunately it hasn't been documented yet, but if you're familiar with the existing API it is pretty easy to figure out and real documentation is on its way. This is another large step forward when it comes to opening the social graph. Today it has become many times easier to welcome a new user to your service by presenting a list of people they already know versus asking them if you can scrape contacts from their email address book(s).

Part of Brad's announcement of these changes is below:

Note that this is simply a mechnical transformation on the /lookup method, not offering anything that wasn't possible before. It's just that the transformation was tedious and error-prone and silly to have to repeat in each client library. Hopefully putting it in the server makes it more convenient for everybody.

Why is this useful? A lot of websites are now letting you list your other websites/profiles on your profile, but it's just as annoying to repeat this information on every site as it is to redefine your friends everywhere. If the site incrementally hits this API in the background as you enter profile URLs, the site can then recommend you link/share your other websites. Examples of sites that let you enter your other profile URLs and could use this API include: Pownce, Digg, Vox, Typepad, Movable Type, Plaxo, Friendfeed, Mugshot. And that's just off the top of my head from sites I'm familiar with. I'm sure there's a bunch more.