The Library of the Commons: Rise of the Infodex

by Mark Sigal | @netgarden | comments: 8

Somewhere between the realm of Personal and Shared media lies the realm of the Universal.

The realm of the universal is the Library of the Commons, a global repository of user-generated and crowd-sourced media and information.

Services that logically nest in the Library include: Amazon, Yelp, YouTube, Craigslist, Wikipedia, Flickr, Twitter tweets, Bit.ly items, Scribd docs, Expedia, Google News, Google Maps, TripAdvisor, iTunes, the App Store and any other services and/or information sources that 'just work.'

In other words, these are services that have defined the 'IT' to the point that we can now pretty much take their utility and availability for granted (typically via API access and/or embed codes with some form of customization wizard).

The Genesis of a Library

So how did we get to this place in the story? What gave birth to the Library of the Commons?

No one formally deigned it so, but from the countless me-too services borne of the dotcom and Web 2.0 land rushes, the above-referred services are the ones that cultivated the biggest audiences, grew the richest ecosystems and inspired the deepest engagement levels.

In Darwinian terms, these are the survivors, whose structures and workflows have been defined and refined by time/experience.

As such, they are generally well thought out, holistic and integrated, but more to the point, have large, engaged user bases.

Thus, the Commons presents a riddle. Almost as if inspired by Herman Hesse's 'The Glass Bead Game', the riddle is this.

If all of these services yield a smorgasbord of best practices, why not systematically emulate them so as to...FEDERATE them?

Put another way, what if a time came when people ceased trying to perennially re-create the wheel, and instead, started to 'decompose' these services; to empty their function sets from whatever nesting they were contained within; and to re-apply them into new contexts supported by a now federated data flow proxied within the Cloud.

Couldn't the composite feature set be exposed switchboard-style to enable any number of custom services and client apps?

To put some meat on the conceptual skeleton, consider the following exercise that I recently did:

A decomposition of Craigslist and TripAdvisor yields deep profiles that are accessorized and interconnected via context traversal flows, such as categorization routines, places, events, airfares, posts, pages, ratings, discussion threads, offers, jobs, businesses, products and personal listings.

Craigslist offers up 36 different sub-types of items For Sale; Services represent another 19 sub-types; Jobs 41 more; Discussions, another 72. And so it goes (including Housing, Personals and Community) across 175+ geo-locales.

TripAdvisor is an instance of this model that overlays a set of time-tested workflows specific to the relatively complex task of planning a vacation.

These workflows make it easy to match a travel plan to specific tastes, requirements and budget - regardless of the information traversal path you pursued to being ready to get pricing on desired travel dates.

Could these same workflows be re-purposed for researching and then purchasing other similarly complex products or services?

I will come back to that thought, in a moment.

The Rise of the Infodex

What is de-composed, can be re-assembled, and thus begins the Infodex.

The Infodex is a kind of next-generation Rolodex, with aspirations to grow into a real-time marketplace.

What exactly is the Infodex? It is comprised of three parts.

Part one is a listing tool for linking to content, creating a metadata wrapper around media items and encapsulating the above-referenced services (i.e., Yelp, YouTube, WIkipedia) into listing containers that define and expose the methods that one can interface to the media item (framework integrity stuff).

Part two is an indexing engine so that, once simple rules are defined, your media libraries and the information in the listings themselves becomes 'self-organizing.'

Named picture types (globes, animals, historic or famous images), for example, could be a federation of multiple picture services (Flickr, Photobucket, Getty Images) and 'discovered' pictures from past queries.

Looked at from this perspective, the goal, in part, is to establish a cloud-based, crowd-sourced Dewey Decimal System built around the outcome of facilitating better searching, compositing, cross-indexing, sharing, archiving, and analytics functions for specific media and information 'types.'

Part three of the Infodex is a unified runtime player that is congruent with the information flows of the mobile broadband age; namely, iPhone, Twitter, Facebook and Web (Javascript/Flash embeds/Adobe AIR) based viewing/playback environments.

One simple example of a basic type of function that might be propagated across all of these environments is the Three Item Topical List (e.g., Top Three Favorites or Three Most Related Items). Define once, propagate everywhere.

A core assumption of the model is that both the media player and the service integration layers are open-sourced. This ensures that the user experience is uniformly good across all of these services, and pushes proprietary-ness higher up the stack, thus raising the floor for all comers.

A final thought. Google became Google by indexing the web. Couldn't the next generation extend this approach by being federated, crowd-sourced and context-specific (i.e., media, information and service aware)?

Are their obvious best practices for The Commons? Obvious gotchas? What about the Infodex?

Related Posts:

1. Pattern Recognition: Makers, Marketplaces and the Library of the Commons


2. Envisioning the Social Map-lication


3. The Mobile Broadband Era: It's About Messages, Mobility and The Cloud

Old Media, New Media and Where the Rubber Meets the Road

by Mark Sigal | @netgarden | comments: 11

Analog (old) media is all about managing scarcity by controlling distribution, the net effect of which is to enable publishers to price access to their “toll roads” as they see fit.

Digital (new) media, by contrast, is premised on the assumption that the tools for content creation, selling, distributing and marketing enable meta-professionals and prosumers to create a surplus of “good enough” content.

This content, in tandem with un-tethered distribution and pretty good search/retrieval functions, operates in complete disregard for the old media-based pricing models that preceded it.

As such, when the forces of analog media collide with digital media, as they have in music, newspapers, yellow pages, books and magazines (and are beginning to collide in television and movies), a brutally efficient “creative destruction” process occurs.

Simply put, if the digital forces can assemble a “good enough” version of the un-tethered content, then in most cases, the analog media provider is in deep trouble (read: devastating business model disruption).

Understanding Media Disruption

My once-beloved San Francisco Chronicle has been “hollowed out,” reduced to a thin pamphlet, thereby accelerating their subscriber attrition.

Why PAY for content that is less deep, less differentiated than I can get online elsewhere for FREE? It's a vicious cycle.

My once-favorite local news station, KRON, no longer has sports on the weekends; it runs more syndicated content and requires that its reporters operate their own cameras to minimize cost. It's definitely struggling. KNBR, which is the sports radio station that I listen to, tells a similar story.

Do you even know anyone who actually uses the Yellow Pages anymore? That would have been unfathomable when I was growing up.

Now, Google is the Yellow Pages.

On some level, it really is as simple as saying that Craigslist killed the classified ads business, which in turn, killed the newspaper business.

The music business was once supremely cool. Records were cool. The whole chain between record producers, tour promoters and record stores was pretty cool.

Remember record stores? Whither Tower Records. Heck, even Blockbuster is standing on some wobbly legs.

Strangely, it's not that the music suddenly is less good. In fact, I probably listen to as much music as I ever have.

It's just that the "disruption" cow has left the barn (and is living in my iPod), and there is no turning back.

In this case, there are just too many incentives for the performers to maximize their online availability and shift their monetization to other sources, like touring and merchandising.

As a result, the music producer/promoter has been pushed to the backseat (for now).

(Un)Differentiated Media

It seems that the only safe havens are highly differentiated media creators that can’t readily be replicated elsewhere, such as the type of original programming one sees on HBO (e.g., check out: True Blood); the vertical/demographically targeted cable channels (where old media distribution rules still promulgate); and big budget movies, where production values (and production costs) are out of the reach of meta-professionals.

That is what makes the furor playing out with AP, all the more interesting.

AP is a syndicated content and news distribution service that makes its money offering infill content to (traditionally) analog media sources.

In the online world, however, the digital form of AP’s fee-based media is fodder for enabling digital publishers to link to, reference and excerpt from these same stories, typically without paying a nickel to AP.

Now, AP wants to turn back the hands of time by limiting/restricting access to and usage of that content.

Meanwhile, digital media advocates are citing fair use, and you just know that this can’t end well for AP, as their product is fundamentally undifferentiated.

That is not to suggest that they have no case, at least karmically speaking, but it's akin to arguing about oxygen. This is the atmosphere that they operate within.

The media industry would have to exercise a collective re-set to turn the tide on this one. Maybe they will, but I am skeptical.

Re-thinking The Audience and Your Product

Extending the conversation further, Fred Wilson’s post, ‘Monetize The Audience, Not The Content’ (read the comments section) presents a conundrum.

On the one hand, I totally agree with the objective of building your business around your audience.

But, I also think that a true solution needs to reconcile how the product or service evolves to achieve differentiation in such a universe; and that is a bigger challenge.

Here, my specific assertion is that while not all content is created equal, a whole heck of a lot of it is fundamentally undifferentiated.

In the case of The New York Times (a high profile pub that Fred regularly writes about), there are a few star writers, but none of which are such must-reads as to drive users to pay for access to them (hence, the failure of NYT's Times Select).

I love reading Frank Rich; Maureen Dowd is pretty entertaining; and Thomas Friedman is thought-provoking. Plus, there are 6-7 other times throughout the month that I find myself reading a Times article.

But, I've seriously never considered paying for access to them, and when the Select thing was in effect, and folks like Friedman were behind lock and key, I mostly forgot about them.

Case in point, whatever happened to Howard Stern after he left broadcast radio? Is the King of All Media even relevant anymore?

Don't tell me how much he is worth now. Tell me this. What happened to his audience?

It's a hard truth, but while there are 10+ good “enough” quality news/opinions sources for every news story of the day (and they are easy to find and well-indexed vis-a-via Techmeme and Google News), there is no "good enough" cheap/free alternative to the Ridley Scott directed, Christian Bale starring action movie.

As such, the NYT’s of the world face a real paradox. Their brand is their content, and without continuing to cultivate their content and innovate the way it's presented, which costs money, they have no durable audience.

Thus, I think a better path is to:

1. Come up with well-defined linkages between online and offline workflows. For example, print subscribers get access to deeper analysis, better tools for saving, excerpting, sharing and finding related content;
2. Create new types of media/engagement units that reward loyalty, communit-ize it, perhaps game-ify it;
3. Re-think segmentation (and pricing) across high-end, low-end, hyper-local, and vertical-specific distinctions, and re-work the product accordingly.

Apple, Record Labels serve up 'Cocktail'

So it seems fortuitous, that as I am updating this post, word filters into the blogosphere that Apple's long-rumored Tablet computing device is due in September (the Friday rumors said Q1, 2010), and that Apple is working with the record labels to re-invent the packaged music experience for the digital realm. Smart!

Here's an excerpt from the article:

Apple wants to make bigger purchases more compelling by creating a new type of interactive album material, including photos, lyric sheets and liner notes that allow users to click through to items that they find most interesting.

Consumers would be able to play songs directly from the interactive book without clicking back into Apple’s iTunes software, executives said. “It’s not just a bunch of PDFs,” said one executive. “There’s real engagement with the ancillary stuff.”

New York Story

Ending where we began, creative destruction has had a field day with the media business (and by virtue of its association, advertising as well).

To get to the other side intact, the NYT’s of the world have to figure out what they are that a focused, less expensive blogger, prosumer or meta-creator can’t emulate.

With brutal efficiency, this truth will separate those that can meaningfully, unquestionably differentiate from those that can’t.

Prognosis: more hurting ahead; then the industry finds its footing, begins a renaissance, and gets back on offense.

Related Posts:

1. Digital Media Rules: The Open Sourcing of Information
2. Apple, the ‘Boomer’ Tablet and the Matrix
3. How Social Media Works: It's About Breadcrumbs and Conversations
4. The Programmable Fan Site: A New Media/Ad Unit Model
5. Flip Video News Network: Crowd-Sourcing meets CNN

Four short links: 11 June 2009

Trends, Graffiti, Games, and Streaming Video

by Nat Torkington | @gnat | comments: 1

1. Trending Topics -- full source code for trendingtopics.org, Wikipedia trend analysis. Rails app running on the Cloudera Hadoop Distribution on EC2. (via mattb on Delicious)
2. Graffiti from Pompeii -- I can't help but read these as Tweets. Herculaneum (on the exterior wall of a house); 10619: Apollinaris, the doctor of the emperor Titus, defecated well here (see also olde style Twitter) (via OvidPerl on Twitter)
3. Online Games Dominate Beijing Startonomics -- presentations from sessions on Chinese game business at Startonomics conference. Though there are many differences between the US and China games market, the one that stands out most is China’s ability to massively monetize games. Tencent, a leading Chinese web portal, social network and game developer, famously announced revenue of over $1 billion earlier this year, much of it coming from their avatar service. (via TinaTranT on Twitter)
4. Ustream's Audience for Apple iPhone Announcement Greater Than Cable News -- Ustream is amazing, you can take a consumer handycam and video broadcast live to a greater audience than many TV shows get.

Completing the circle on journalists and public participation

by Andy Oram | @praxagora | comments: 3

Journalists, politicians, and foundations are all tinkering with forms of amateur input: inviting bloggers to major events, quoting popular online sites in newspapers, etc. But Capital News Connection has really jumped in full-tilt with Ask Your Lawmaker. A creative combination of public input and ratings with professionals who have their boots on the ground in the US Capitol building, Ask Your Lawmaker is a case study in progress concerning how to get experts and the public to work together.

I heard a talk from CNC founder and executive director Melinda Wittstock this evening at the Ethos Roundtable, a forum for non-profits in Eastern Massachusetts. CNC gets consulting input from Ethos Roundtable organizer Deborah Elizabeth Finn, and Wittstock came looking for volunteer help with such matters as developing a Facebook or iPhone application. As Wittstock said, Ask Your Lawmaker is still working on how to complete the circle of public input, feedback, and outreach.

Step one is the simple form (on the web site's "Ask A Question" tab) for submitting a question to any Congressman or Senator of your choice. Step two is the simple voting mechanism, reminiscent of the pre-inauguration Change.gov site.

At this point, the journalists working for CNC--who have years of experience at leading media sites--take over. They don't merely choose the highest-rated questions. Sometimes a question shouldn't have to wait around and gather votes because the topic is hot. The reporters use their judgment in combination with votes to pick timely and provocative questions, and sometimes direct a question to a more appropriate lawmaker (such as the sponsor of a bill or the head of a committee).

The next step invokes the power of professional journalism. CNC sends its reporters into the Capitol and congressional office buildings daily. Although they have regular routines with their typical journalists' questions, they throw in citizen questions where appropriate and tell the lawmaker how many people voted for each question. Wittstock mentioned that it's very hard for a congressperson to dismiss a question that came from a constituent, especially one that got a lot of votes.

Videos are very hard to make in the Capitol, unfortunately, because filming is severely restricted there by law and the lawmakers are understandably leery of allowing themselves to be filmed any place at any time.

The next step goes from real-time back to the web site, along with conventional radio stations. Questions and answers are taped and transcribed so they can be offered as both audio and text. CNC has contracts with a number of PBS stations who work public questions into regular news broadcasts.

Podcasts and texts are posted on the web site and served through an RSS feed, but you can also follow AskYourLawmaker on Twitter or search for hashtag #ayl. (Right now they're discussing the talk I attended.) This can bring the answers back to those who asked the questions.

Ask Your Lawmaker also offers a feed that visitors can add to their own web sites, and an iframe for each individual report, suitable for embedding.

Most powerful at all, citizens' questions can change policies. Lobbyists harangue lawmakers day after day, but sometimes they're more impressed by a simple question revealing a deep-seated need in their communities. They have been heard walking away from journalist interviews saying to their staff, "Brief me about that issue."

All very impressive for an effort that's so provisional, the journalists run the web site themselves. Several weak points remain before the circle is complete.

• Ask Your Lawmaker doesn't get enough publicity. It may or may not be mentioned on the radio station that reports its results. Hardly any listeners, I wager, realize that questions were generated by ordinary citizens, much less realize that anyone can ask a question.
• The site needs a way to accept questions through SMS. Attendees at this evening's talk speculated about the power of accepting questions for US lawmakers from victims of wars or globalization policies around the globe.
• The site doesn't exploit the potential for social networking to let questioners promote the site. Someone whose question is chosen should be informed when the answer is posted or broadcast on the radio, and should be encouraged to invite her friends and fellow workers to view the answer.

CNC is looking for ways to complete the circle--and will gladly accept volunteer help, as I mentioned--but they're doing a lot in the meantime to firm up their appeal and raise funds. They plan to allow cobranding and to let sites select the length and subject matter of the material they post, just as they now serve up very customized reports to the radio stations they serve. They may start accepting advertising, and they're looking for fun contests that will publicize their work.

Ask Your Lawmaker demonstrates a unique solution to a situation whered for amateur input can augment expert practice and expertise can augment what the public has to offer. In this regard, Ask Your Lawmaker is worth comparing to the landmark Peer-to-Patent project and to two commercial ventures I analyzed a few months ago, uTest and TopCoder. The opportunity for a virtuous cycle of public input, professional processing, and listener loyalty--especially in a field whose death has been predicted by many--puts Ask Your Lawmaker into an intriguing category of its own.

Scribd Store a Welcome Addition to Ebook Market (and 650 O'Reilly Titles Included)

by Andrew Savikas | @andrewsavikas | comments: 7

The document-sharing site Scribd has launched a new "Scribd Store" selling view and download access to documents and books. As part of the launch, there are now more than 650 O'Reilly ebooks now available for preview and sale in the Scribd store, and all include DRM-free PDF downloads with purchase. (Scribd will soon be adding EPUB as a format, and we'll make that available as soon as possible.)

Many publishers (including O'Reilly) have kept Scribd at arm's length because the service was often used by people posting copyrighted material without permission. Though Scribd was reasonably responsive to takedown requests, that puts the onus for monitoring on the publisher, a whack-a-mole scenario that will consume as many resources as you throw at it if you let it. But Scribd has implemented a new system that uses the ebooks provided for sale to identify (and remove) any other unauthorized versions of that material, as well as prevent future unauthorized uploads. Like any technology it's far from perfect (for example, I suspect scanned images are more difficult to test than standard PDFs), but it's good enough for us to be comfortable participating, and is as good an example as any of turning lemons into lemonade.

For a publisher (and I use the term loosely) the terms for the Scribd store are impressive -- publishers set the sale price directly, and keep 80% of the revenue (compare that to Amazon's DTP program, where the standard terms are that Amazon gets to set the actual price, and the publisher only gets 35% of their "suggested" price). There's also an interesting "automated pricing" option in Scribd, which uses an (unspecified) algorithm to set the sale price. But the pieces of the Scribd store I'm most excited about is the real-time reporting (compared with a lag of a month or more with most ebook resellers, including Amazon), the option to easily provide free updates to existing content, and the variety of adjustable display options -- like preview amount, refreshingly optional DRM, and purchase-link images. Administering and understanding your sales in Scribd is downright delightful compared with the same for Kindle.

A service like Scribd further reduces the barriers to content creators interested in self publishing digital material (and again offers much better terms than Amazon's DTP program for Kindle), so in some ways absolutely a threat to existing publishers. But we also view it as an opportunity to get our books in front of interested readers, and a promising sign that the market for ebooks is large enough to continue attracting startups like Scribd who bring needed diversity and competition among resellers.

Four short links: 12 May 2009

Storage Superfluity, Data-Driven Design, Twit-Mapping, and DIY Biohacking

by Nat Torkington | @gnat | comments: 1

1. Lacie 10TB Storage -- for what used to be the price of a good computer, you can now buy 10TB of storage. Storage on sale goes for less than $100 a terabyte. This obviously promotes collecting, hoarding, packratting, and the search technology necessary to find what you've stashed away. Analogies to be drawn between McMansions full of Chinese-made crap and terabyte drive full of downloaded crap. Do we need to keep it? Are there psychological consequences to clutter? (via gizmodo)
2. In Defense of Data-Driven Design -- a thoughtful response to the "Google hates design!" hashmob formed around designer Douglas Bowman's departure from Google. When you’ve got the enormous traffic necessary to work out if miniscule changes have some minor, statistically significant effect, then sure, if you can do it quickly, why wouldn’t you? But that’s optimization that should happen at the very end of the design cycle. The cart goes after the horse. Put it the other way ‘round and you have a broken setup. It doesn’t mean horses suck. It doesn’t mean carts suck. Carts are not the enemy of horses. Optimization is not the enemy of design. Get them in the right order and you have something really useful. Get them the wrong way around and you have something broken.
3. Just Landed: Processing + Twitter + Metacarta + Hidden Data -- Jer searched Twitter for "just landed in", used Metacarta to extract the locations mentioned, and then used Processing to build visualizations.
4. Do It Yourself Genetic Sleuthing -- MIT is starting a hotbed of DIY biologists. The 23-year-old MIT graduate uses tools that fit neatly next to her shoe rack. There is a vintage thermal cycler she uses to alternately heat and cool snippets of DNA, a high-voltage power supply scored on eBay, and chemicals stored in the freezer in a box that had once held vegan "bacon" strips. Aull is on a quirky journey of self-discovery for the genetics age, seeking the footprint of a disease that can be fatal but is easily treated if identified. But her quest also raises a broader question: If hobbyists working on computers in their garages can create companies such as Apple, could genetics follow suit? It's unclear what those DIY-started "genetics" companies would look like--the potential is there, but it's yet to met the right problem. (via Andy Oram)

Just Landed - 36 Hours from blprnt on Vimeo.

Clay Shirky's "Newspapers and Thinking the Unthinkable"

by Tim O'Reilly | @timoreilly | comments: 37

Sometimes Clay Shirky astounds us by articulating something we've never thought of, and sometimes he astounds us by telling us something many have thought, but never so clearly and so compellingly. But always, he astounds.

Into the first category falls the claim that he made in his keynote at the last Web 2.0 Expo that "the critical technology of the 20th century...was the sitcom." Who would have thought that so penetrating an analysis could hinge on such a preposterous assertion! (If you haven't already done so, read the transcript or watch the video.)

Yesterday's piece, Newspapers and thinking the unthinkable, falls into the second category. When I said the other day that "Twitter is the most minimal newspaper," or when I talked to the New York Times about rediscovering what is essential in what they do, I was speaking to this same topic, but like so many others, I was still framing the dialogue around "saving the newspaper." By contrast, Clay cuts the Gordian knot:

That is what real revolutions are like. The old stuff gets broken faster than the new stuff is put in its place....

And so it is today. When someone demands to be told how we can replace newspapers, they are really demanding to be told that we are not living through a revolution. They are demanding to be told that old systems won’t break before new systems are in place. They are demanding to be told that ancient social bargains aren’t in peril, that core institutions will be spared, that new methods of spreading information will improve previous practice rather than upending it. They are demanding to be lied to.

There are fewer and fewer people who can convincingly tell such a lie.

This is a piece that anyone concerned with the future of publishing simply MUST read.

That being said, when I speak to this topic myself, I offer this hope: that while institutions may be overwhelmed by the tide of change, new institutions do arise. The deep needs that newspapers serve aren't going away. We will find new ways to serve those needs. As Clay says:

When we shift our attention from ’save newspapers’ to ’save society’, the imperative changes from ‘preserve the current institutions’ to ‘do whatever works.’ And what works today isn’t the same as what used to work.

And to be quite frank, we can already see the shape of that reinvention in specialized fields. In the mid-1990s, Michael Leeds, the CEO of CMP, in its day one of the titans of specialty computer newspapers, told me that if he couldn't get one of his papers to $50 million in revenue in 3 years, he would shut it down as not worth doing. Today, many of the papers he owned are gone, yet small firms like Techcrunch, Mashable, and ReadWriteWeb are successful (and doing at least as good a job of covering computer industry news) at an order of magnitude less revenue than CMP would once have thrown away.

Jobs that matter get done. But no one is guaranteed that their business as they conceive of it today will be preserved, especially at any given scale or profitability. So, have faith. The world as we know it is being broken. Now, let's get on with reinventing it!

Four short links: 10 Mar 2009

by Nat Torkington | @gnat | comments: 1

1. Norwegian Broadcasting Corporation Sets Up Its Own BitTorrent Tracker -- the money shot is not that they're using the same code as Pirate Bay, it's "By using BitTorrent we can reach our audience with full quality media files. Experience from our early tests show that if we’re the best provider of our own content we also gain control of it.". Finally, a broadcaster realizing that they have to jump into the conversation with customers even though they don't know how it ends. (via BoingBoing)
2. Sita Sings The Blues Released -- release of the movie that was mired in copyright strife, now freed under Creative Commons Attribute And No Damn DRM licensing. It still is copyright-entangled: some of the songs in the movie are restricted and if you want to reuse the songs in your reuse of the movie then you'll have to wrangle with the copyright overlords.
3. Crisis of Credit Explained in Infographics -- a great 10m movie explaining the whole disaster from cash to crash, with an infographic-meets-Flash-game feel to it. This is the future of educational films. I've embedded it below. (via Flowing Data)
4. Cowpox Smallpox -- very clear essay from Maciej Ceglowski about how the economic dramas and the climate dramas challenge our democracy in the same way. You might know Maciej from Argentina on two steaks a day or Dabblers and Blowhards. Complexity as a result of feedback loops caught my eye, as that's part of the talk I gave at Webstock, "Better Stronger Failures": "Feedback loops in the financial world are even worse, since the entities being modeled are aware of their behavior - and aware of the models being used to study them. Investors form strategies based not just on market conditions, but on their perceptions of others' perceptions of market conditions, and so on in a hall of mirrors effect. Any algorithm that can reliably predict the behavior of a financial market will be used by participants in that market to earn money, altering the system in a way that leaves you right back where you started. In this sense our ability to model economics will always be worse than our understanding of the weather, since we don't have to worry about a raindrop anticipating that it will hit the ground before it even forms, and taking steps to change the outcome."

The Crisis of Credit Visualized from Jonathan Jarvis on Vimeo.

A Climate of Polarization

by Gavin Starks | comments: 10

Guest blogger Gavin Starks is founder and CEO of AMEE, a neutral aggregation platform designed to measure and track all the energy data in the world. Gavin has a background in Astrophysics and over 15 years Internet development experience.

We're all aware of the emotive language used to polarize the climate change debate.

There are, however, deeper patterns which are repeated across science as it interfaces with politics and media. These patterns have always bothered me, but they've never been as "important" as now.

We are entering an new era of seismic change in policy, business, society, technology, finance and our environment, on a scale and speed substantially greater than previous revolutions. The sheer complexity of these interweaving systems is staggering.

Much of this change is being driven by "climate science", and in the communications maelstrom there is a real risk that we further alienate "science" across the board.

We need more scientists with good media training (and presenting capability) to change the way that all sciences are represented and perceived. We need more journalists with deeper science training - and the time and space to actually communicate across all media. We need to present uncertainty clearly, confidently and in a way that doesn't impede our decision-making.

On the climate issue, there are some impossible levers to contend with;

1. Introducing any doubt into the climate debate stops any action that might combat our human impact.
2. Introducing "certainty" undermines our scientific method and its philosophy.

When represented in political, public and media spaces, these two levers undermine every scientific debate and lead to bad decisions.

Pascal's Wager is often invoked, and this is entirely reasonable in this case.

It is reasonable because of what's at stake: the risk of mass extinction events. If there is a probability that anthropogenic climate change will cause the predicted massive interventions in our ecosystem, then we have to act.

The nature of our actions must be commensurate with both the cause and the effect. The causes are many: population, production, consumption - as are the effects: war, poverty, scarcity, etc.

Our interventions will use all our means to address both cause and effect, and those actions will run deep.

Equally, we must allow science to do what it's designed to do: measure, model, analyse and predict.

From a scientific perspective we must allow more room for theories to evolve, otherwise we'll only prove what we're looking for.

However, if we ignore the potential need to act, the consequences are not something anyone will want to see.

It's not something we can fix later (for me, "geo-engineering" is not a fix, it's a pre-infected band-aid).

Given the massive complexity of the issues, and that - really - anthropogenic climate change is only one of many "peak consumption" issues that we face, there is no way we can accurately communicate all the arguments that would lead to mass understanding.

However, the complexity issues are no different from those we face in politics. They are not solvable, but they are addressable.

We can communicate the potential outcomes, and the decisions that individuals need to make in order to impact the causes.

Ultimately it's your personal choice.

My choice is based on my personal exposure to the science, business, data, policy, media, and broader issues around sustainability. That choice is to do my best to catalyse change as fast as I possibly can.

We all need to actively engage in improving communication, so that everyone - potentially everyone on Earth - can make informed choices about the future of the planet we inhabit.

--
Recommended reading:

http://www.realclimate.org/ is a great resource.

Today, the UK Government launched a campaign "to create a more science literate society, highlighting the science and technology based industries of the future"

Four short links: 23 Jan 2009

by Nat Torkington | @gnat | comments: 3

Potty mouth, piracy, pointers to the future of the web, and Presidential technology woes, all in today's link roundup.

1. F*ck the Cloud - Jason Scott's brilliant (and profanity-strewn) rant about cloud computing and the things people throw away without thinking about. Jason, an Internet historian, has a unique perspective and I think what he says makes a lot of sense. "[I]f you’re not asking what stuff means anything to you, then you’re a sucker, ready to throw your stuff down at the nearest gaping hole that proclaims it is a free service".
2. Pirating the Oscars - Andy Baio summarizes online piracy of the Oscar-nominated movies, as he has done since 2003. It's interesting to see what's new this year: movies are taking longer to leak, but more of them are being leaked.
3. Webkit Owns Mobile - Alex Russell lays out the case that Webkit "has mobile all sewn up". I've been saying for the last umpty years that the Web is at a Windows 286 stage of development--we need 3.1 to come along and standarize the widgets that presently everyone reinvents. I recognized that in this line from Alex: "If we look at the APIs of Dojo, Prototype, or jQuery as a set of suggestions for the APIs that the web should expose, then it becomes pretty clear that we’ve still got a long long way to go".
4. New Staff Find White House Tech in Dark Ages - they've gone from a startup to The Enterprise (not Star Trek, alas, just a big company) and now are learning the pain of IT rules that are bigger than they are.

Four short links: 15 Jan 2009

by Nat Torkington | @gnat | comments: 1

Today we have Tom's Brain on Flickr, the Newspaper Industry's Death in Context, REST with friends, and Filthy Lucre from Twitter.

1. A map of my brain - Tom Coates mindmaps his interests as part of brainstorming for a Webstock talk. I'd love to see these for other geeks. I guess part of keeping up with your friends is building your own models of what their mindmaps look like.
2. How Newspapers Tried to Invent the Web - a Slate piece showing that newspapers have always reacted to new media, from buying up airways when radio became big, to videotex and the proprietary information sources that predated the mainstreaming of the Internet. Puts the modern handwringing in context. Sample quote from 1980: "What you're really worried about is an electronic Yellow Pages that will destroy your advertising base, isn't it?"
3. RESTful Django Practice - the full RESTian mindset is surprisingly difficult to grok, so one programmer has said "here's how I think it works for a sample app" and his readers are using the comments to describe the choices, drawbacks, pitfalls, and best practices. I love the Internet.
4. 1stfans - a mixture of social networking updates, Twitter feeds from artists, and in-person events that the museum hopes will be a good excuse for people to become members and support the museum's operation. Twitter Feed as Membership Benefit goes into detail on their reasons. It's always interesting to see people experimenting with finding things to charge for online.

Four short links: 12 Jan 2009

by Nat Torkington | @gnat | comments: 2

Brace yourself: kids, design, newspapers, and robots. It can only be another collection of four tasty links (or the key elements of the least successful Disney holiday movie ever).

1. Our Work So Far This Year - amazing blog entry about St Pauls high school in England, which has had exceptional technologists come to speak to their ITC class. Who? Oh, only the CEO of Arduino company tinker.it, Steven Johnson, Cory Doctorow, Gavin Starks, Phil Gyford, Tom Armitage, .... They recorded the talks and are building their own YouTube channel. The really interesting bit is at the top where they talk about the skills that their 13-year olds are coming into their ICT class with: last year they were teaching tabbed browsing to the first years, this year the first years are coming in with Firefox on a USB drive so they can keep their bookmarks wherever they go ....
2. Infinite Zoom Into Milk - a glimpse at a delectable series of books that drill into everyday items to reveal manufacturing and design decisions, materials, etc.
3. Things Our Friends Have Written On The Internet - a very classy compilation of favourite blog posts and Tweets, assembled into a tabloid-sized newspaper with tastefully typesetting. Are there sufficient numbers of offline people left that it's worth producing a mainstream magazine compiled, like this, from online material? If so, how long until those economics no longer apply?
4. Anybots Launches - congrats to Trevor Blackwell, whose presence and work have graced several Foo camps. Anybots, his telepresence robot company, revealed their products at CES last week.

Four short links: 9 Jan 2009

by Nat Torkington | @gnat | comments: 0

Four questions, one per link: what next, can it solve a big problem, what's the final boss for Python programming, and why on earth would anyone want yogurt that glows in the dark?

1. End Times - gloomy piece on the future of journalism, to be added to the large pile of other gloomy pieces on the future of journalism (e.g., Bad News, Good News). The critical problem is still how to pay for journalism if the new media revenues are significant lower than old, and if the new media economics decree that journalism is dead then who fills the social good role that journalism's death will leave?
2. Ward Cunningham's Visible Workings - an intriguing glimpse, from March last year, into the way Ward lays out web interactions. Nice system for laying out these interactions, but it's also fascinating for how it makes transparent what will happen as a result of the data you submit. How scalable is this? Could it tackle privacy?
3. Project Euler - fun programming exercises that require more than math to finish. We learn by doing, not by reading, so interesting exercises are part and parcel of training. It's interesting to see educators are moving from being authors to being game designers, providing a series of staged challenges that make us stronger by defeating them. I'm presently dieing in as many ways as I can while learning iterators and generators in Python, as a way of ensuring I have Python's "game physics" sussed.
4. Rise of the Garage Genome Hackers - more on hobbyist molecular biology. It mentions DIYBio, the Cambridge biohacker collective that I first heard about at BioBarCamp. (via Glynn Moody)

A Star is Born? NY Times syndicates outside blogs but that's not enough

by Joshua-Michéle Ross | @jmichele | comments: 0

Recently the New York Times announced that it will be syndicating content from three well-known blogs, Read/Write Web, Giga Om and Venture Beat. The New York Times is using these blogs as an extra-sensory organ; they can dial into what is happening in the tech sector (and particularly the West Coast with this trio) without allocating a lot of internal resources to it. Smart move.

As newspapers are locked in a desperate bid for survival I get the sense that we are watching the business equivalent of A Star is Born. In this case the iconic, shaggy-maned newsman falls for a feisty blogger with a horrific perm. Two strangers meeting in life's stairwell; One headed down - the other headed up... Let's hope this pairing has a better ending.

Syndication seems to have one goal - leverage that content to build online ad revenues. I hope the Times has more up its sleeve because this is just a “more of the same” strategy. Ad revenues for newspapers dropped $3 billion in the first six months of this year. With the Wall Street implosion it looks like revenues will continue to collapse as overall big-spend, ad budgets decline. I say this despite predictions of growth in '09 online ads because big print newspapers need both online/offline revenue to stay viable. Bumping online ad sales in place of plummeting offline ad revenue will not come close to solving the fundamental problem.

In order to survive newspapers need to:
1. Get better at their core revenue business: advertisting. Use behavioral targeting to maximize ad prices - better user profiles equal higher revenues. Currently traditional newspaper ad networks are outsourced, weak and generic. Understanding these technologies should be a core competence inside a modern news organization.

2. Aggressive online diversification. U.S. Newspapers missed a chance to claim the classified space. Craigslist got there first and locked the newspapers out (in most U.S. markets). Schibsted, a pioneering media company in Europe, has a very healthy online classified business because they got there first. (Similarly Schibsted is competing with Google for search b/c they have huge data assets in content and video etc. that they are using to compete). Online services like this feed off of network effects - leaving precious little for runner-ups. It is too late for US papers to compete with Google or Craigslist but there are other areas that have not been claimed. Newspapers need to move aggressively to create services that deliver value; all-things-local; niche classifieds, local real estate, political polling data, creating decision markets with their readers etc. Newspapers still have assets that are hard to rival: large sales teams with great relationships; top-notch content-creation teams; established brands with public trust and, last but not least an online readership that can help become a pillar of new innovation.

3. Gene Therapy: This is what I call “the harder stuff.” Traditional newspapers require a DNA transplant. Many of the tenets of the social web: innovation from the outside, publish-then-filter, rapid adaptive behavior (fail forward fast) and learning from failure all meet with stiff organizational resistance. If newspapers do not empower their online businesses, take more small risks and get out of the way there will be nothing left in a few years to reclaim.

Newsgathering organizations serve a vital civic function - but without a clear revenue model they will become an artifact of the last millennium. Time is not on their side.

What else do newspapers need to do to save themselves?

Numbers for Digital's Rise

by Nat Torkington | @gnat | comments: 7

I talk a lot to people who don't quite understand the scale of the media shift from atoms to bits (update: corrected), so I always have my eyes open for numbers and anecdotes that illustrate the point. The latest I found are from an article on Apple's threat to shut the iTunes store if it has to pay more to songwriters:

Digital downloads grew 38 per cent from 2006 to 2007 to become a $1.26 billion business, making up 23 per cent of the market for recorded music, according to the Recording Industry Association of America. Sales of physical music media such as CDs, cassettes and DVDs declined 19.1 per cent to $7.5 billion in the same one-year period.

I'm still looking for convincing numbers on film and TV movement to digital. For example, does anyone have numbers on how well Dr Horrible's Singalong Blog (the web-only offering from Buffy creator Joss Wedon) did?

Nice Visualization of Candidates' Speeches

by Tim O'Reilly | @timoreilly | comments: 0

Paul Kedrosky pointed to a great visualization of what the candidates are talking about, from the new Dow Jones Insight Election Blog:

Biggest one to note: Health care remains big for the Democrats (24% of all mentions) and small for the GOP (9%). The economy is No. 1 for everyone.

The difference in domestic issue coverage between Obama and Clinton is slight, with the concept of "terrorism" and "health care" being the only places where there is noticeable difference. Obama gets more mentions in close proximity to terrorism to Clinton's edge in health care....

Terrorism continues to be McCain's issue, second only to the economy.

Click on the image above to see a readable copy.

There aren't a lot of details about their methodology, but presumably this is a demonstration of what you get when you subscribe to Dow Jones Insight and can point it to your own topics of interest. Unfortunately, they suffer from the corporate inaccessibility that dooms so many great products to avoidance by early adopters. Rather than giving you any real information, they ask for your info so a sales person can contact you. (I remember having this argument with the folks at Microsoft about their "sign up for a sales call" mapping APIs, before Google blew them all away by providing simple self-service sign-up.)

Nonetheless, this is definitely a blog I'm going to be following from now on.

Great Blend: The Future of Media

by Nat Torkington | @gnat | comments: 13

This weekend I spoke on a panel at the Great Blend event, organized by my Kiwi Foo Camp compadre Russell Brown.  He'd lined up three courses of "food for thought" for the approximately 100 people who turned up, all roughly themed around the future of media.

The soup was William Cooper from InformITV.  He had to dash to catch a plane, but he and Russell used the available 45 minutes for a spirited conversation.  My notes are rather shaky thanks to the excellent wine that was available, but here are some points that I can make out:

• Interactive TV actually happened in the UK.  I only know it as the flying saucer car from late 1980s futurism books, where you can replace "interactive TV" with "the Internet" and get an accurate look at 2005, but apparently the Brits have a red button on all their remotes (left over from the Teletext days) that can be used to summon more interactive content.  However, PVRs defeat it--you don't see the prompt to hit the red button.  I found that an interesting point: two "make TV better" technologies clashing.
• He talked about "video as a social medium", namely that people endlessly point each other to clips on the web.  I realized "but they can't do anything else".  All you can do with video on the web is point people to it--it's a very passive object at the moment.  This is still good news for movie folks, whose only social media exposure beforehand was when people leaving theatres called friends with mobile phones to say "don't go to see this movie, it sucks!".
• A friend asked a question about income from online properties, namely how can you continue to make a $200k episode if you're making $60k from online distribution and the online is killing the broadcast distribution?  The question wasn't well answered other than a handwavy "it'll all work out".  It's a question I also thought about, and I figure if TV is uneconomic then we'll have a new show format besides the 22m episodic TV we're familiar with.  Serials in magazines have died out, short stories in print are almost dead, radio plays have passed their heyday ... formats change in response to economics and technology.

The main course was Mr Brown, a Singaporean blogger and podcaster.  He's building quite the media empire on the net, with daily, weekly, and monthly regular shows.  A lot of his success and his subject matter is due to Singapore's tight media regulation.

Mr Brown had a weekly humorous newspaper column in a major daily but the government called his editor and cancelled the column after one particularly pointed jab at the establishment.  That was his impetus to go beyond text and produce video and audio segments.  He's had 500k downloads of some of his pieces (impressive--Singapore's population is 3.5M) and was even mentioned by the President in a talk!  He, naturally, turned the shoutout into a humorous jab.

It was an entertaining presentation, and a look at how blogs are actually changing political institutions in a part of the world that doesn't have free media or particularly representative democracy.

Finally came a rather-too-meaty dessert, the panel that I was on: "the future of media" with Clare O'Leary of New Zealand On Air (the government body that contributes towards local program development), Mr Brown, and Eric Kearley from Television New Zealand (the state-run broadcaster).  Eric started with a quick plug for his work within TVNZ, launching the new digital free-to-air channels.  His message was that people panic about the Internet only until they have experience with it, then they stop holding conferences about the effect of the Internet on Media and instead just get on with putting their media on the Internet.

In particular, he advocated "brands not shows".  That is, you create a brand (like "Big Brother") that exists in multiple media (TV, online, print, figurines, interpretive dance), attempting to reuse your work as much as possible between media.  During Q&A I tried to probe how this model handles the declining revenues from TV, but he was handwavey.

I was asked what we at O'Reilly think about the future of media, and I replied it was personalization and social experience.  We love the web because my web experience can be different from yours, yet I can still influence and change your experience by simply sending you a link.  There are no real social TV experiences (the social stuff happens around the watercooler, rarely around the TV).  Schulze and Webb are working on a social radio prototype for the BBC, and I hope boffins are working on similarly connecting viewers around TVs.

I didn't mention it, but something else I've been pondering is the way that PVRs separate broadcast time from viewing time.  Here's a crazy idea: a web site lets people select what shows or movies they want to watch, those shows are scheduled over the satellite, and subscribers' PVRs pick them out and saves them for later viewing.  PVRs could be networked and automatically record the stuff your friends recorded so it's right there when they later say "wow, you have to see the latest Big Survivor House Race".  Think NetFlix for over-the-air TV.

One interesting point that Mr TVNZ had to make was that the traditional perception of ad revenue decreasing as the Internet steals viewers isn't necessarily true.  The true events that unite people around their TV in real time (in the US it's SuperBowl, in NZ it's rugby, in the UK it's probably the Queen's Christmas Message) will be all the more valuable for their rarity, the argument goes.  I'm not sure I buy it--it's an argument that total value will drop to zero, but unquantified it fails as an argument that ad revenue will persist at current levels.

Anyway, a fun evening was had by all (not least because the L.E.D.s played afterwards).  If you have any thoughts on the future of TV, please let me know what they are.  It all appears up for grabs at the moment.

(Update: Fake Steve Jobs has a great rant on why the Internet (and the Real Steve Jobs in particular) are the flaming comet to the TV networks' dinosaurs)