Turning Predictions into Opportunities

by Nat Torkington | @gnat | comments: 7

The view from the eye of a recession isn't great. When companies are going bust, unemployment growing, and everyone's scouring their budgets for costs to cut, it can be hard to see opportunities. However, when Tim pointed to Stephen O'Grady's fine set of 2010 predictions I found myself popping with "oh, so naturally this will happen next ..." thoughts. Think of this as a glimpse of the blue sky after the economic funnelspout that's demolished our economy. (Continuations of the tornado metaphor with "being sucked into the cloud", or "trailer park economics", or "we're not in Kansas any more, Tantek" left as an exercise to the reader)

1. As every cloud provider creates their own "open API" (itself a fraught term), look to see the rise of brokers who can migrate you from one cloud to another. Deltacloud is an early free project here from RedHat, but there are many business opportunities waiting. It's possible that companies will pay for assurance (you've tested your migration tool, you know it works on corner cases), service vs product (they don't want tools to run, they want to pay you to install and maintain the tools accessible through a web console), or premium services so that you're a partner helping them get the most from the cloud and not simply a vendor.
2. We're a long way from sated in the world of collaboration tools. The current rage is mail learning, applying machine learning techniques to email so as to better understand social networks and prioritise incoming email messages and these are largely server-based solutions because it's so hard to get access to the desktop/web clients. Should Google Mail create an app store environment with hooks into the backend, the game could be on for consumer plays around email analytics, prediction, and simply smarter behaviour (why does my email client still not tell me when I say "see attached" yet don't have an attachment in the message?).
3. Beyond email, many interesting tools have sprung up around the Gov 2.0 space that have applicability within organisations. Yammer has done well to bring Twitter to large companies, but there are still opportunities around simple document markup and suggestion gathering and filtering. Solve a real problem and there's money waiting.
4. Google's low overhead management is made possible by its automated intranet and the visibility into projects from public code repositories, public smoke builds, and public status blogs. The opportunities to sell this into large companies looking to be "more like Google" are huge.
5. If Stephen's right that datasets are increasingly viewed as "serious, balance sheet-worthy assets" then the world is going to need some serious balance sheet-worthy help in valuing those assets.
6. Big data is being democratised, but there's a lot of unmet need in businesses around data warehousing. The typical solution is to build a data warehouse team around a product like Oracle, but I've heard plenty of business people grizzling about the result. They want answers, they don't want the headaches and lag that a data warehouse involve. Big Data (or Cloud Analytics or whatever) may be the opportunity to figure out a new minimum viable product for these folks, and offer it without the "data warehouse" baggage. This might be back end, might be UIs, might be visualisation, but all of these have a lot of room for improvement.
7. The proliferation of developer targets immediately makes me think of the early PC era. It makes sense to proliferate: let the most useful ("successful") bubble to the top and survive naturally. At this point in the evolution of the scaleout of massively multiplayer online programming languages, we don't know exactly what winning looks like: it's a big feedback loop between the people who build the programming languages and the people with problems to solve (there are always more of the latter than former) and each time we go around it we know more about what is and isn't useful in this brave new world of coding for other people's data centres. Opportunity? Join the mob and write your own programming language, or simply take your commercial opportunity for a spin around the many different languages out there and be the first in your niche to find a good fit between problem space and solution tool.
8. Stephen's throwaway comment "I’ve never subscribed to the idea that only what can be measured can be managed - open source, in particular, belies that claim" seems like a thrown gauntlet on open source analytics. In particular, I suspect there's a tools opportunity around the nebulous "community manager" role that every company seems to need. It's part CRM, it's part developer tool, it's part tech support, and part camp mother. Usefully quantify aspects of open source development and help companies that are doing it to know how they're doing and what they could do better.
9. Marketplaces are big in mobile, but I look to other areas as ripe for the picking. For example, if Google Apps are catching on in many companies then a plugin marketplace is a natural extension. It would build out the Apps suite faster than Google can, would enable the tight loop between demand and supply that will drive the product along, and make Google's offering very different from other parties. This is also true of Microsoft and others, but I feel like momentum is more with Google's product than the others. (A feature can push a leader further in front, but rarely helps a laggard leapfrog to the lead)
10. Every marketplace thus far has been flawed. Apple's famously annoys many developers and blocks huge categories of product (the "don't be better than we are" rule, which is hard to justify as being in the customer's interest), but don't forget Palm's impedance mismatch with jwz's open source code. I think the final chapter on how marketplaces work is far from written.
11. NoSQL tools remain in their infancy and so there are huge opportunities here. Identify a niche ("fast accurate and timely web metrics for decision-making"), a tool that can solve it (MongoDB), and build the deployment, scaling, administration, reporting tools so you can sell a complete package into that niche. Rinse, lather, repeat.

Four short links: 3 September 2009

Smarter Eyes, Urinal Protocol Efficiency, Petabytes on a Budget, and LocaLondon

by Nat Torkington | @gnat | comments: 1

1. Many Eyes Make All Bugs Shallow, Especially When The Eyes Get Smarter (David Eaves) -- Mozilla released bug submission data, and David realizes with some minor investment (particularly some simpler vetting screens prior to reaching bugzilla) bug submitters could learn faster. For example, a landing screen that asks you if you've ever submitted a bug before might take newbies to a different page where the bugzilla process is explained in greater detail, the fact that this is not a support site is outlined, and some models of good "submissions" are shared (along with some words of encouragement). By segmenting newbies we might ease the work burden on those who have to vet the bugs.
2. Urinal Protocol Efficiency (xkcd blog) -- geeks are pattern-matching creatures that can count. This leads us to a question: what is the general formula for the number of guys who will fill in N urinals if they all come in one at a time and follow the urinal protocol? One could write a simple recursive program to solve it, placing one guy at a time, but there’s also a closed-form expression. If f(n) is the number of guys who can use n urinals, f(n) for n>2 is given by: [...] The protocol is vulnerable to producing inefficient results for some urinal counts. Some numbers of urinals encourage efficient packing, and others encourage sparse packing. (via Hacker News)
3. Petabytes on a Budget: 67Tb for $7,867 -- DIY cloud hardware. (via timhaines on Twitter)
4. LocaLondon (Chris Heathcote) -- informative, ingenious, and replicable (like all that Chris does), it's a Twitter feed of art exhibitions in London (when they open, when there's a week left, and on the last day) and a glorious horizontal touchscreen-friendly meta-reviews site so you can quickly see at a glance what's on now and what people think of it.

Four short links: 4 August 2009

NASA Cloudware, btrfs, eBook Editing, Exponential Death

by Nat Torkington | @gnat | comments: 1

1. NASA Nebula Services/Platform Stack -- The NEBULA platform offers a turnkey Software-as-a-Service experience that can rapidly address the requirements of a large number of projects. However, each component of the NEBULA platform is also available individually; thus, NEBULA can also serve in Platform-as-a-Service or Infrastructure-as-a-Service capacities. Bundles RabbitMQ, Eucalyptus, LUSTRE storage, Fabric deployment, Varnish front-end, MySQL and more. (via Jim Stogdill)
2. A Short History of btrfs -- Now for some personal predictions (based purely on public information - I don't have any insider knowledge). Btrfs will be the default file system on Linux within two years. Btrfs as a project won't (and can't, at this point) be canceled by Oracle. If all the intellectual property issues are worked out (a big if), ZFS will be ported to Linux, but it will have less than a few percent of the installed base of btrfs. Check back in two years and see if I got any of these predictions right!
3. Sigil -- open source WYSIWYG eBook editor. (via liza on Twitter)
4. Exponential Decay of Life -- This startling fact was first noticed by the British actuary Benjamin Gompertz in 1825 and is now called the “Gompertz Law of human mortality.” Your probability of dying during a given year doubles every 8 years. For me, a 25-year-old American, the probability of dying during the next year is a fairly miniscule 0.03% — about 1 in 3,000. When I’m 33 it will be about 1 in 1,500, when I’m 42 it will be about 1 in 750, and so on. (via Hacker News)

Cloud computing perspectives and questions at the World Economic Forum

by Andy Oram | @praxagora | comments: 10

The World Economic Forum started a research project at Davos 2009 concerning cloud computing, which they broadly define to include all kinds of remote services, from Software as a Service to virtual machines.

I was asked to provide some ideas on the implications of cloud computing for business as well as its future operating environment. To allow my colleagues and the O'Reilly community to help define the issues and provide references, I've put up a discussion forum as a wiki. Anyone with relevant and valid ideas can suggest points. I don't even mind people listing their businesses and information sources, so long as the information is relevant and is directed toward the larger educational goal of the wiki.

I'm holding a phone call with someone from WEF on Thursday, July 16 where I'll present my views, augmented by your suggestions.

Here are the categories of the notes I used to seed the wiki.

• Benefits and drawbacks for potential clients
• Benefits and drawbacks of offering software as a service or using a development environment
• Access
• Resilience
• Portability
• Environmental implications
• Software freedom
• Government use

Update, July 17:The phone conversation with a research director at World Economic Forum went well. She had already seen the wiki. We had a pretty free-ranging discussion during which i brought up my favorite points, including access issues and the need for different types of applications for different populations. I passed on some resources that readers sent me and gave her the URL of this blog with all its comments. I'll continue to collect ideas in this area and work on the wiki.

Four short links: 8 June 2009

3D Geometry, The Printable Web, Government Internet Fail, and Real World Cloud Computing

by Nat Torkington | @gnat | comments: 2

1. How to Project on 3D Geometry -- the fine art (and math) of distorting an image so that it looks undistorted when projected onto a non-flat 3D surface. Confused? See the images below. (via straup on Delicious)
2. ZinePal -- Create your own printable magazine from any online content. (via warrenellis on Delicious)
3. What The Government Doesn't Understand About The Internet And What To Do About It -- Tom Steinberg from MySociety lays it out. As true for US, NZ, and every other country as it is for the UK (for which it was written). Accept that any state institution that says “we control all the information about X” is going to look increasingly strange and frustrating to a public that’s used to be able to do whatever they want with information about themselves, or about anything they care about (both private and public). This means accepting that federated identity systems are coming and will probably be more successful than even official ID card systems: ditto citizen-held medical records. It means saying “We understand that letting train companies control who can interface with their ticketing systems means that the UK has awful train ticket websites that don’t work as hard as they should to help citizens buy cheaper tickets more easily. And we will change that, now.” What I like about Tom vs the US's Gov 2.0 is that Tom puts down philosophy that's hard to argue with, whereas the US is dangerously close to simply focusing on techniques and that's subvertible.
4. Real World Cloud Computing -- summary from a panel of startups who are using EC2. The lock-in is latency. Transfering data within the Amazon services is free. Transfering data to an Amazon competitor: not free.

3D Glasses: Virtual Reality, Meet the iPhone

by Mark Sigal | @netgarden | comments: 29

A light flickers from two distinct points in time. As a child in the early-1970s, one of my toys was a View-Master, a binoculars-like device for viewing 3D images (called stereograms), essentially a mini-program excerpted from popular destinations, TV shows, cartoons, events and the like.

The View-Master completely predated the advent of electronic toys (it was light powered and human click driven), but it was dumb simple to operate, and the 3D viewing experience was quirky cool. Plus, the content was customizable (just pop in a different program card) and for its time, it was engaging (sound could play on top of each image, making it even more so).

Flash forward, and it's 1992. I am reading Mondo 2000, a long since deceased magazine that was at the bleeding edge of the technology wave that was to come. Total reboot in terms of re-thinking and re-imaging the schema of the possible.

In one issue, Jaron Lanier is prophesizing his vision of virtual reality (VR). It's a revelation, but it's also too early, and so despite Mattel-scale efforts to commercialize "something" that embraces VR concepts, the VR industry is stillborn (except, as verbally rendered in Snow Crash and Neuromancer, two seminal novels to this day).

Alas, the MetaVerse would have to wait. In the intervening years, there would be RPGs (Role-Playing Games), MUDs (Multi-User Dungeons) and of
course, Second Life, but virtual reality was just that; more virtual than reality.

Flash forward to the present, and we are suddenly on the cusp of a game-changing event; one that I believe kicks the door open for 3D and VR apps to become mainstream. I am talking about the release of iPhone OS version 3.0.

Huh? Did I lose you? Well, let me take a step back.

Hardware Accessories Innovation: A Phoenix Rises

I have harped repeatedly (HERE and HERE) about the fact that the next version of the iPhone OS (and the underlying SDK) will allow third-party hardware accessory makers to build external hardware accessory offerings that take advantage of the software, service and hardware capabilities of the iPhone and iPod touch platforms.

Specifically, Apple is opening up the 30-pin connector at the base of the iPhone and iPod touch such that hardware accessory makers can create a software layer that is optimized to take advantage of the capabilities of the hardware in a way that cobbles together with and extends the capabilities of the iPhone Platform. (Side note: Bluetooth-accessible hardware accessories can also take advantage of these new capabilities.)

What this means is a rebirth of hardware-based innovation, a segment that Apple has historically played a leading role in fostering, first with postscript-based printing, then with Appletalk networking (pre-dates Ethernet) and now with Mobile Broadband Computing.

Look at it this way; very soon, hardware accessory makers will be able to leverage the same tools and marketplace functions that have resulted in more than 35K applications being built and more than 1 billion apps downloaded, all the while tapping into a 37 million device global footprint.

What's that worth? Consider this. The iPod accessory business itself is already a $2B market, and there has really been no such thing as "software value-add" to the hardware accessory itself. With iPhone 3.0, this changes. That's a big deal.

3D Mobility: The Rise of the Meta-Platform

Connecting the pieces, there is 3D/Virtual Reality and there is this macro trend of the iPhone Platform becoming more hardware extensible in a programmatic fashion.

My assertion is this: beginning with the assumption that View-Master 2.0 is an achievable baseline in terms of a "good enough" 3D viewing experience, and full immersive VR the design goal, 3D Glasses are a compelling hardware accessory to harness the power, extensibility and mobility functions of the iPhone Platform.

Framing the point a bit further, 3D Glasses are a kind of "meta" platform built to embrace and extend the iPhone Platform vis-à-vis application specific libraries, open APIs, custom tools and of course, the glasses themselves, to enable third-parties to create their own 3D/VR-powered applications.

The clearest way to think about the 3D Glasses Meta-Platform - let's call it X-Ray Specs - is as being a composite of a Runtime Layer, a Physical Glasses Layer and a Cloud Service Layer.

The Runtime Layer is an iPhone application that deals with items such as media control (search, activate, play/pause/forward/rewind, bookmark); communication handling between the 3D Glasses, the iPhone and the Internet; and can execute custom-built X-Ray Specs applications.

The Physical Glasses Layer is the actual hardware glasses that support viewing, playback and output resolution in a "good enough" fashion relative to mass-market cost economics. Augmenting the glasses would be headphones and microphone inputs, with physical controls for simple navigation, modal and settings adjustment.

Finally, the Cloud Service Layer is a hosted service where third-party applications are sourced and hosted, and social communications are sent, tracked, federated, filtered and routed back to the X-Ray Specs Runtime.

In other words, 3D Glasses would do more than 3D viewing. They would also support overlay and immersive viewing applications.

By overlay applications, I mean viewing applications that act as filters, enhancing, modifying or shaping whatever you are doing/viewing by adding information or inserting characters into viewed content.

Imagine watching a ballgame and being served up feeds, like sports scores, news items or other animated tickers, or having video highlights pop up in your view when key events occur (home run hit by a watched player, key match-up between players).

By contrast, immersive applications consume 100% of your viewing perspective, making them ideal for virtual reality types of applications, not to mention shared (social) and passive viewing experiences.

At its most basic, this means View-Master types of applications (i.e., 3D stereogram content), akin to slide shows on steroids; namely tuned to take advantage of easy access to the local media libraries resident on your iPhone/iPod touch and shared libraries accessible via the Internet (and App Store/iTunes).

But how exactly does one navigate virtual worlds in such an immersive realm?

One, they can use touch, tilt and shake as the primary controller mechanism.

Two, voice-based control can be embedded in the glasses, supporting a simple voice dictionary for audible-lizing actions, such as "GET," "PLAY," "NEXT," "PREVIOUS," "PAUSE," "SEARCH," "FORWARD," "BACKWARD," "TURN," "GRAB" and so on. Think: BASIC Programming language for humans.

On top of this, one could easily envision voice-powered shortcuts for accessing and controlling popular apps like YouTube, Hulu, Flickr, Scribd, SlideShare and the like.

So there you have it. The MetaVerse is virtually within reach (pun intended). No less, there is a proven platform the enables fairly creative application partitioning scenarios (between hardware, iPhone and cloud).

And oh yeah, that same platform plugs into a marketplace that can support subscriptions, chapters or levels, not to mention a digital distribution center and e-wallet that reaches 37M users.

It doesn't get much cooler than that. What do you think?

Four short links: 15 Apr 2009

by Nat Torkington | @gnat | comments: 0

Computer archaeology, Unix, mad science, and data mining:

1. NASA Images Saved By Volunteers -- Pictures from the mid-1960s Lunar Orbiter program lay forgotten for decades. But one woman was determined to see them restored. One woman and some keen hardware hackers who built Frankenstein's tape reader to recover the images. Not just a reminder of how ephemeral our media, but also the huge amount of useful work that falls outside the interest of Official Groups to fund. (via Tim's twitter stream)
2. The Art of Unix Programming and The UNIX-HATERS Handbook (PDF) -- one loves Unix, the other ... not so much. It's interesting to read both books consecutively and realize the vast gulf that existed between Good Enough and Perfect, and how Perfect has been well and truly vanquished by Good Enough. The original Unix solved a problem and solved it well, as did the Roman numeral system, the mercury treatment for syphilis, and carbon paper. And like those technologies, Unix, too, rightfully belongs to history. (TAoUP via bengebre's delicious bookmarks)
3. Theo Gray's Mad Science -- a book full of Make-like charismatic megascience that you could theoretically do if you were sufficiently patient, provisioned, and safe. Projects include making your own nylon, turning beach sand to steel, and making salt by spectacularly combining sodium and chlorine. (via BoingBoing)
4. Microsoft Offers Data Mining Tools in the Cloud (Byteonic) -- Microsoft offers some data mining functionality of SQL Server 2008 with no local analysis services server in the cloud. The service is offered in two flavors: a cloud service and as a plug-in for Excel. The tools are forecasting, prediction, and "analyze key influencers". Interesting to see Microsoft offering this higher-level service than the simple Spreadsheet-in-the-Sky offered by Google.

Salt from sodium and chlorine

Karmic Koalas Love Eucalyptus

by Simon Wardley | comments: 7

Guest blogger Simon Wardley, a geneticist with a love of mathematics and a fascination for economics, is the Software Services Manager for Canonical, helping define future cloud computing strategies for Ubuntu. Simon is a passionate advocate and researcher in the fields of open source, commoditization, innovation, and cybernetics.

Mark Shuttleworth recently announced that the release of Ubuntu 9.10 will be code-named Karmic Koala. Whilst many of the developments around Ubuntu 9.10 are focused on the desktop, a significant effort is being made on the server release to bring Ubuntu into the cloud computing space. The cloud effort begins with 9.04 and the launch of a technology preview of Eucalyptus, an open sourced system for creating Amazon EC2-like clouds, on Ubuntu.

I thought I'd discuss some of the reasoning behind Ubuntu's Cloud Computing strategy. Rather than just give a definition of cloud computing, I'll start with a closer look at its underlying causes.

The computing stack is comprised of many layers, from the applications we write, to the platforms we develop in and the infrastructure we build upon. Some activities at various layers of this stack have become so ubiquitous and well defined that they are now suitable for service provision through volume operations. This has led to the growth of the 'as a Service' industries, with providers like Amazon EC2 and Force.com.

Information Technology's shift from a product to a service-based economy brings with it both advantage and disruption. On the one hand, the shift offers numerous benefits including economies of scale (through volume operations), focus on core activities (outsourcing), acceleration in innovation (componentisation), and pay per use (utility charging). On the other hand, many concerns remain, some relating to the transitional nature of this shift (management, security and trust), while others pertain to the general outsourcing of any common activity (second sourcing options, competitive pricing pressures and lock-in). These concerns create significant adoption barriers for the cloud.

At Canonical, the company that sponsors and supports Ubuntu, we intend to provide our users with the ability to build their own clouds whilst promoting standards for the cloud computing space. We want to encourage the formation of competitive marketplaces for cloud services with users having choice, freedom, and portability between providers. In a nutshell, and with all due apologies to Isaac Asimov, our aim is to enable our users with 'Three Rules Happy' cloud computing. That is to say:

• Rule 1: I want to run the service on my own infrastructure.


• Rule 2: I want to easily migrate the service from my infrastructure to a cloud provider and vice versa with a few clicks of a button.


• Rule 3: I want to easily migrate the service from one cloud provider to another with a few clicks of a button.

(continue reading)

Cloud Computing defined by Berkeley RAD Labs

by Artur Bergman | comments: 5

I am pleased to finally have found a paper that manages to bring together the different aspects of cloud computing in a coherent fashion, and suggests the requirements for it to develop further.

Written by the Berkeley RAD Lab (UC Berkeley Reliable Adaptive Distributed Systems Laboratory) the paper succinctly brings together Software as a Service with Utility Computing to come up with a workable definition of Cloud Computing and is a recommended read.

The services themselves have long been referred to as Software as a Service (SaaS). The datacenter hardware and software is what we will call a Cloud. When a Cloud is made available in a pay-as-you-go manner to the general public, we call it a Public Cloud; the service being sold is Utility Computing. We use the term Private Cloud to refer to internal datacenters of a business or other organization, not made available to the general public. Thus, Cloud Computing is the sum of SaaS and Utility Computing, but does not include Private Clouds.

Exploring the difference between the raw service of Amazon EC2 to the high level web centered Google App Engine, the highlights are:

• Insight into the pay-as-you go aspect with no commits
• Analysis of cost with regards to peak and elasticity in face of unknown demand
• Cost of data transfers versus processing time
• Seamless migration of user to cloud processing
• Limits and problems with I/O on shared hardware

They raise the following obstacles and opportunities, echoing Tim's posts on Open Source and Cloud Computing and Web 2.0 and Cloud Computing.

• Availability of Service
• Data Lock-In
• Data Confidentiality and Auditability
• Data Transfer Bottlenecks
• Performance Unpredictability
• Scalable Storage
• Bugs in Large-Scale Distributed Systems
• Scaling Quickly
• Reputation Fate Sharing
• Software Licensing

I particularly find interesting the analysis of transportation cost versus computing cost; when is it more efficient to to use EC2 than your own individual processing? I predict speed of light and available of raw transfer capacity is going to become a even larger obstacle. (Both inside computers, between them on local LANs and on WANs.)

The paper reinforces my belief in the cloud, but that we need open source cloud environments and a larger ecosystem of providers.

Read more on the Above the Clouds blog.

Four short links: 23 Jan 2009

by Nat Torkington | @gnat | comments: 3

Potty mouth, piracy, pointers to the future of the web, and Presidential technology woes, all in today's link roundup.

1. F*ck the Cloud - Jason Scott's brilliant (and profanity-strewn) rant about cloud computing and the things people throw away without thinking about. Jason, an Internet historian, has a unique perspective and I think what he says makes a lot of sense. "[I]f you’re not asking what stuff means anything to you, then you’re a sucker, ready to throw your stuff down at the nearest gaping hole that proclaims it is a free service".
2. Pirating the Oscars - Andy Baio summarizes online piracy of the Oscar-nominated movies, as he has done since 2003. It's interesting to see what's new this year: movies are taking longer to leak, but more of them are being leaked.
3. Webkit Owns Mobile - Alex Russell lays out the case that Webkit "has mobile all sewn up". I've been saying for the last umpty years that the Web is at a Windows 286 stage of development--we need 3.1 to come along and standarize the widgets that presently everyone reinvents. I recognized that in this line from Alex: "If we look at the APIs of Dojo, Prototype, or jQuery as a set of suggestions for the APIs that the web should expose, then it becomes pretty clear that we’ve still got a long long way to go".
4. New Staff Find White House Tech in Dark Ages - they've gone from a startup to The Enterprise (not Star Trek, alas, just a big company) and now are learning the pain of IT rules that are bigger than they are.

Data Center Power Efficiency

by Jesse Robbins | @jesserobbins | comments: 8

James Hamilton is one of the smartest and most accomplished engineers I know. He now leads Microsoft's Data Center Futures Team, and has been pushing the opportunities in data center efficiency and internet scale services both inside & outside Microsoft. His most recent post explores misconceptions about the Cost of Power in Large-Scale Data Centers:

I’m not sure how many times I’ve read or been told that power is the number one cost in a modern mega-data center, but it has been a frequent refrain. And, like many stories that get told and retold, there is an element of truth to the it. Power is absolutely the fastest growing operational costs of a high-scale service. Except for server hardware costs, power and costs functionally related to power usually do dominate.

However, it turns out that power alone itself isn’t anywhere close to the most significant a cost. Let’s look at this more deeply. If you amortize power distribution and cooling systems infrastructure over 15 years and amortize server costs over 3 years, you can get a fair comparative picture of how server costs compare to infrastructure (power distribution and cooling). But how to compare the capital costs of server, and power and cooling infrastructure with that monthly bill for power?

The approach I took is to convert everything into a monthly charge. [...]

My mind is buzzing right now...

by Tim O'Reilly | @timoreilly | comments: 18

My mind is buzzing right now, full of ideas that are all demanding my attention, begging to be written up. I don't have time right now, but I thought I'd write down the headlines, both as a kind of "to do list" and also to give a sense of the stew of topics I'm spending time on:

• Why change.gov needs to use revision control

• Pathologies in the hive mind

• Why we need cloud data services, not just cloud database services

• Why our energy discussions will falter till we're all using the same math

• My priority list for technology initiatives in the next administration.

Hopefully I'll get a chance to write up all these posts this weekend, and get them out in the coming week.

Sprint blocking Cogent network traffic...

by Jesse Robbins | @jesserobbins | comments: 3

It appears that Sprint has stopped routing traffic (called "depeering") from Cogent as a result of some sort of legal dispute. Sprint customers cannot reach Cogent customers, and vice versa. The effect is similar to what would happen if Sprint were to block voice phonecalls to AT&T customers.

Here's a graph that shows the outage, courtesy of Keynote :

Rich Miller at DataCenterKnowledge has a great summary of the issues behind the incident, which has happened with Cogent before. Rich says:

At the heart of it, peering disputes are really loud business negotiations, and angry customers can be used as leverage by either side. This one will end as they always do, with one side agreeing to pay up or manage their traffic differently.

I think this is particularly Radar-worthy because it provides an example of the complex issues around Net Neutrality . In this case customers are harmed and most (especially Sprint wireless customers) will have no immediate recourse.

(continue reading)

Network Effects in Data

by Tim O'Reilly | @timoreilly | comments: 12

Nick Carr's difficulty in understanding my argument that cloud computing is likely to end up a low-margin business unless companies find some way to harness the network effects that are the heart of Web 2.0 made me realize that I use the term "network effects" somewhat differently, and not in the simplistic way many people understand it.

Here's Nick:

Let's stop here, and take a look at the big kahuna on the Net, Google, which O'Reilly lists as the first example of a business that has grown to dominance thanks to the network effect. Is the network effect really the main engine fueling Google's dominance of the search market? I would argue that it certainly is not....

The intelligence embedded in a link is equally valuable to Google whether the person who wrote the link is a Google user or not. In his new post, in other words, O'Reilly is confusing "harnessing collective intelligence" with "getting better the more people use them." They are not the same thing. The fact that my neighbor uses Google's search engine, rather than Yahoo's or Microsoft's, does not increase the value of Google's search engine to me, at least not in the way that my neighbor's use of the telephone network or of Facebook would increase the value of those services to me. The network effect underpins and explains the value of the telephone network and Facebook; it does not underpin or explain the value of Google. (Indeed, if everyone other than myself stopped using Google's search engine tomorrow, that would not decrease Google's value to me as a user.)

Ah, I say to myself: Nick only sees first order network effects, what you might call endogamous networks, those that require the user to be part of the tribe. Thus, phone networks, and networks like Facebook. But the internet is an exogamous network; its benefits increase by the extent to which it reaches out to new groups, increases cross-breeding, and thus the total robustness and variety of the gene pool. This is why links matter, why web services matter, because they extend the reach of the network. Understanding the benefit of exogamous networks requires a more subtle calculus than Nick is applying. It's not necessarily that you benefit directly from belonging, but the fact that you belong allows others to harvest the benefit of your participation.

Consider Google: The underlying network that Google is based on is one that they neither own nor control, the web itself. It has both endogamous end exogamous elements. No one controls it; its richness and diversity depends on that fact. And yet, there is a benefit to belonging. If there weren't, sites would use their robots.txt file to tell Google and other search engines to stop spidering them.

Yes, you might say: but other search engines have access to that same network. And here, of course, is the first lesson: Google is better at spidering that network than their competitors. They thus benefit more powerfully from the network that we are all collectively building via our web publishing and cross-linking. Nick correctly points out that Google has built superior systems, and that these are the source of their competitive advantage. But that's a diversion. Why did they build those superior systems? To harness the power that was hidden in the network more effectively than their competitors.

Google's second network effect advantage is PageRank. As Robert Scoble so insightfully noted back in 2003, we contribute to Google with every link. Google realized that there was an additional layer of meaning hidden in the network. Far from being a contradiction to my network-effect hypothesis, as Nick claims, this is a validation of it. Advantage came to Google for seeing more deeply into the nature of the network, and building tools to harvest and apply data that was hidden in the network graph.

Google's third (and most profitable) network effect insight was, of course, the ad auction. And once again, Nick misses the point. He says:

Now it's true that, if you want to define market liquidity as a type of network effect, Google enjoys a strong network effect on the advertising side of its business (which is where it makes its money), but it would be a mistake to say that the advertising-side network effect has anything to do with Google's dominance of the searches of web users.

It isn't that the advertising-side network effect has anything to do with Google's dominance of search, but rather, that Google's dominance of search is central to the design of their ad auction. You see, while Yahoo! (nee Overture) sold keyword advertising to the highest bidder, Google realized that they could mine their users' clickstream activity to predict which ads would be most likely to be clicked on, and by what ratio, and thus sell to the best combination of price and actual click through. Thus: higher revenue, more ability to invest in infrastructure, better results for advertisers and users, thus more users, thus better data, thus better results for both organic search and advertising (both of which do, in fact, matter to users, no matter what Nick thinks).

And of course, from there, you can also see other areas in which Google (and their competitors) are doing just this, from Google Docs and Spreadsheets (which exhibits the obvious kind of network effects that Nick is comfortable with), to mining clickstream data, to machine translation.

In short, Google is the ultimate network effects machine. "Harnessing collective intelligence" isn't a different idea from network effects, as Nick argues. It is in fact the science of network effects - understanding and applying the implications of networks.

I want to emphasize one more point: the heart of my argument about Web 2.0 is that the network effects that matter today are network effects in data. My thought process (outlined in The Open Source Paradigm Shift and thenWhat is Web 2.0?, went something like this:

1. The consequence of IBM's design of a personal computer made out of commodity, off- the-shelf parts was to drive attractive margins out of hardware and into software, via Clayton Christensen's "law of conservation of attractive profits." Hardware became a low margin business; software became a very high margin business.

2. Open source software and the standardized protocols of the Internet are doing the same thing to software. Margins will go down in software, but per the law of conservation of attractive profits, this means that they will go up somewhere else. Where?

3. The next layer of attractive profits will accrue to companies that build data-backed applications in which the data gets better the more people use the system. This is what I've called Web 2.0.

It's network effects (perhaps more simply described as virtuous circles) in data that ultimately matter, not network effects per se. Nick probably wouldn't think of Nuance as a network-effects driven company, but it is, because their applications and services depend on data that gets better the more people use it (or have their data harvested in one way or another.) More speech in more circumstances and more domains makes Nuance better for the next user. No user thinks that Nuance is better because of them (and because many of Nuance's products are standalone, this is in fact true.) Yet Nuance, like Google, has figured out how to harvest data contributed by millions to build a better product.

Nick also took exception to my characterization of Wikipedia as a network-effects driven success:

I would also take issue with O'Reilly's suggestion that Wikipedia's success derives mainly from the network effect; Wikipedia doesn't become any more valuable to me if my neighbor starts using it. Wikipedia's success is probably better explained in terms of scale and scope advantages, and perhaps even its nonprofit status, than in terms of the network effect.

How wrong can you be? If there weren't a network effect driving Wikipedia, Knol and Citizendium would be succeeding. Wikipedia got there first, to be sure, but they also built an infrastructure and a workflow and a philosophy that recognized that the collective of all users was smarter than any expert, and that barriers to participation would slow down improvement in the data. There isn't a Facebook-like benefit in "belonging" to Wikipedia, but the application understands something that its competitors don't about harnessing the network and its users to improve its data.

In short, Facebook is the obvious network effect case study. But we learn more by studying what is not obvious: the way internet sites and companies have derived competitive advantage by leveraging different kinds of network effects, most particularly (but not exclusively) to improve the data on which their services are built.

I'm making no claim, as Nick seems to think, that there are no other levers of competitive advantage in the internet era. Nor am I claiming that every network-effect business will be more successful than those that are not, precisely because there are other levers of competitive advantage, but also because some markets are more monetizable than others. But I am claiming that there will be significant differences in profitability between companies that find a network-effect sweet spot in a lucrative market, and those who embrace the commodity end of the business. And sorry, Nick, but I consider the cloud infrastructure business to be the commodity end of the business. It will look like the web hosting business, say, with a bunch of large, capital intensive providers, and not like the hugely profitable company extracting monopoly rents that Hugh Macleod (whose post The Cloud's Best-Kept Secret triggered my own) envisions. Monopoly rents, if they occur, will be at higher levels in the cloud stack.

Web 2.0 and Cloud Computing

by Tim O'Reilly | @timoreilly | comments: 56

A couple of months ago, Hugh Macleod created a bit of buzz with his blog post The Cloud's Best Kept Secret. Hugh's argument: that cloud computing will lead to a huge monopoly. Of course, a couple of weeks ago, Larry Ellison made the opposite point, arguing that salesforce.com is "barely profitable", and that no one will make much money in cloud computing.

In this post, I'm going to explain why Ellison is right, and yet, for the strategic future of Oracle, he is dangerously wrong.

First, let's take a look at Hugh Macleod's argument:

...nobody seems to be talking about Power Laws. Nobody's saying that one day a single company may possibly emerge to dominate The Cloud, the way Google came to dominate Search, the way Microsoft came to dominate Software.

Monopoly issues aside, could you imagine such a company? We wouldn't be talking about a multi-billion dollar business like today's Microsoft or Google. We're talking about something that could feasibly dwarf them. We're potentially talking about a multi-trillion dollar company. Possibly the largest company to have ever existed.

I imagine many of my friends who work for the aforementioned companies know all about this, and know how VAST the stakes are.

Windows vs Apple? Who cares? Kid's stuff. There's a much bigger game going on... And for some reason, its utter enormity seems to be a very well-kept secret, at least to non-combatants like myself.

The problem with this analysis is that it doesn't take into account what causes power laws in online activity. Understanding the dynamics of increasing returns on the web is the essence of what I called Web 2.0. Ultimately, on the network, applications win if they get better the more people use them. As I pointed out back in 2005, Google, Amazon, ebay, craigslist, wikipedia, and all other other Web 2.0 superstar applications have this in common.

Cloud computing, at least in the sense that Hugh seems to be using the term, as a synonym for the infrastructure level of the cloud as best exemplified by Amazon S3 and EC2, doesn't have this kind of dynamic. (More on different types of cloud computing later.)

Of course, it is true that the bigger players will have economies of scale in the cost of equipment, and especially in the cost of power, that are not available to smaller players. But there are quite a few big players -- Google, Microsoft, Amazon -- to name a few, that are already at that scale, with or without a cloud computing play. What's more, economies of scale are not the same as increasing returns from user network effects. They may be characteristic of a commoditizing marketplace that does not actually give outsize economic leverage to the winners.

I can't vouch for the authenticity of the following remark, since I heard it secondhand, but it was from a thoughtful, informed source: Jeff Bezos is reported to have said that he welcomes cloud competition from Google and Microsoft, because they'll subsidize their cloud services with profits from other part of their business, while Amazon will always have to make it pay. "We're good at commodity businesses," Jeff is reported to have said, and the facts bear him out.

If cloud computing is a commodity business, then the outsize profits that Hugh envisioned are not going to be there. This is a business that will be huge, but it may be more similar to the web hosting and ISP markets, which are also huge, but not hugely profitable. (See Rackspace's numbers for a taste.)

But because one of my goals at Radar is to help people think about the future, I wanted to spend some time on the possible futures and strategies that could turn cloud computing into the kind of massive monopoly that Hugh envisioned.

Types of Cloud Computing

Since "cloud" seems to mean a lot of different things, let me start with some definitions of what I see as three very distinct types of cloud computing:

1. Utility computing. Amazon's success in providing virtual machine instances, storage, and computation at pay-as-you-go utility pricing was the breakthrough in this category, and now everyone wants to play. Developers, not end-users, are the target of this kind of cloud computing.

   This is the layer at which I don't presently see any strong network effect benefits (yet). Other than a rise in Amazon's commitment to the business, neither early adopter Smugmug nor any of its users get any benefit from the fact that thousands of other application developers have their work now hosted on AWS. If anything, they may be competing for the same resources.

   That being said, to the extent that developers become committed to the platform, there is the possibility of the kind of developer ecosystem advantages that once accrued to Microsoft. More developers have the skills to build AWS applications, so more talent is available. But take note: Microsoft took charge of this developer ecosystem by building tools that both created a revenue stream for Microsoft and made developers more reliant on them. In addition, they built a deep -- very deep -- well of complex APIs that bound developers ever-tighter to their platform.

   So far, most of the tools and higher level APIs for AWS are being developed by third-parties. In the offerings of companies like Heroku, Rightscale, and EngineYard (not based on AWS, but on their own hosting platform, while sharing the RoR approach to managing cloud infrastructure), we see the beginnings of one significant toolchain. And you can already see that many of these companies are building into their promise the idea of independence from any cloud infrastructure vendor.

   In short, if Amazon intends to gain lock-in and true competitive advantage (other than the aforementioned advantage of being the low-cost provider), expect to see them roll out their own more advanced APIs and developer tools, or acquire promising startups building such tools. Alternatively, if current trends continue, I expect to see Amazon as a kind of foundation for a Linux-like aggregation of applications, tools and services not controlled by Amazon, rather than for a Microsoft Windows-like API and tools play. There will be many providers of commodity infrastructure, and a constellation of competing, but largely compatible, tools vendors. Given the momentum towards open source and cloud computing, this is a likely future.

2. Platform as a Service. One step up from pure utility computing are platforms like Google AppEngine and Salesforce's force.com, which hide machine instances behind higher-level APIs. Porting an application from one of these platforms to another is more like porting from Mac to Windows than from one Linux distribution to another.

   The key question at this level remains: are there advantages to developers in one of these platforms from other developers being on the same platform? force.com seems to me to have some ecosystem benefits, which means that the more developers are there, the better it is for both Salesforce and other application developers. I don't see that with AppEngine. What's more, many of the applications being deployed there seem trivial compared to the substantial applications being deployed on the Amazon and force.com platforms. One question is whether that's because developers are afraid of Google, or because the APIs that Google has provided don't give enough control and ownership for serious applications. I'd love your thoughts on this subject.

3. Cloud-based end-user applications. Any web application is a cloud application in the sense that it resides in the cloud. Google, Amazon, Facebook, twitter, flickr, and virtually every other Web 2.0 application is a cloud application in this sense. However, it seems to me that people use the term "cloud" more specifically in describing web applications that were formerly delivered locally on a PC, like spreadsheets, word processing, databases, and even email. Thus even though they may reside on the same server farm, people tend to think of gmail or Google docs and spreadsheets as "cloud applications" in a way that they don't think of Google search or Google maps.

   This common usage points up a meaningful difference: people tend to think differently about cloud applications when they host individual user data. The prospect of "my" data disappearing or being unavailable is far more alarming than, for example, the disappearance of a service that merely hosts an aggregated view of data that is available elsewhere (say Yahoo! search or Microsoft live maps.) And that, of course, points us squarely back into the center of the Web 2.0 proposition: that users add value to the application by their use of it. Take that away, and you're a step back in the direction of commodity computing.

   Ideally, the user's data becomes more valuable because it is in the same space as other users' data. This is why a listing on craigslist or ebay is more powerful than a listing on an individual blog, why a listing on amazon is more powerful than a listing on Joe's bookstore, why a listing on the first results page of Google's search engine, or an ad placed into the Google ad auction, is more valuable than similar placement on Microsoft or Yahoo!. This is also why every social network is competing to build its own social graph rather than relying on a shared social graph utility.

   This top level of cloud computing definitely has network effects. If I had to place a bet, it would be that the application-level developer ecosystems eventually work their way back down the stack towards the infrastructure level, and the two meet in the middle. In fact, you can argue that that's what force.com has already done, and thus represents the shape of things. It's a platform I have a strong feeling I (and anyone else interested in the evolution of the cloud platform) ought to be paying more attention to.

The Law of Conservation of Attractive Profits

A lot of my thinking about web 2.0 grew directly out of my thinking about open source. My argument in The Open Source Paradigm Shift was that what we learned from the history of the IBM personal computer -- a commodity platform built from off-the-shelf parts -- was that it drained value out of the hardware ecosystem, turning it into a low-margin business. But profits didn't go away. Instead, through something that Clayton Christensen calls "the law of conservation of attractive profits," value migrated elsewhere, from hardware to software, from IBM to Microsoft. Christensen:

When attractive profits disappear at one stage in the value chain because a product becomes modular and commoditized, the opportunity to earn attractive profits with proprietary products will usually emerge at an adjacent stage.

I believe strongly that open source and open internet standards are doing the same to traditional software. And value is migrating to a new kind of layer, which we now call Web 2.0, which consists of applications driven not just by software but by network-effects databases driven by explicit or implicit user contribution.

So when Larry Ellison says that cloud computing and open source won't produce many hugely profitable companies, he's right, but only if you look at the pure software layer. This is a lot like saying that the PC wouldn't produce many hugely profitable companies, and looking only at hardware vendors! First Microsoft, and now Google give the lie to Ellison's analysis. The big winners are those who best grasp the rules of the new platform.

So here's the real trick: cloud computing is real. Everything is moving into the cloud, in whole or in part. The utility layer of cloud computing will be just that, a utility, without outsized profits.

But the cloud platform, like the software platform before it, has new rules for competitive advantage. And chief among those advantages are those that we've identified as "Web 2.0", the design of systems that harness network effects to get better the more people use them.

If Oracle isn't playing that game, they will one day be doomed to irrelevance. Perhaps, like hardware giants of the past - Compaq, say - they will be absorbed by a bigger company. Or perhaps, like Unisys, they will linger on in specialized markets, too big to go away but no longer on the cutting edge of anything. Or they will understand that it's not the database software that matters, but the data that it holds, and the services that can be built against that data.

The company that creates the right platform for network effects in data may well achieve the scale that Hugh Macleod envisioned.

P.S. I will be doing two panels on cloud computing at the Web 2.0 Summit in San Francisco the week after next, one on the application layer, and one on the infrastructure layer. Panelists include Paul Maritz (CEO of VMware, who, by the way totally gets what I'm talking about here), Russ Daniels (CTO for cloud services at HP), Padmasree Warrior (CTO at Cisco), the inimitable Marc Benioff of Salesforce.com, Kevin Lynch, CTO of Adobe, and Dave Girouard, who is in charge of Google Apps for the Enterprise. Should be some interesting conversations on the subjects raised in this post!

Amazon's new EC2 SLA

by Jesse Robbins | @jesserobbins | comments: 7

Amazon announced a new SLA for EC2, similar to the one for S3. This is a notable step for Amazon and cloud computing as a whole, as it establishes a new bar for utility computing services.

Amazon is committing to 99.95% availability for the EC2 service on a yearly basis, which corresponds to approximately four hours and twenty three minutes of downtime per year. It's important to remember that an SLA is just a contract that provides a commitment to a certain level of performance and some form of compensation when a provider fails to meet it.

Here's the summary of the EC2 SLA (emphasis added):

Service Commitment AWS will use commercially reasonable efforts to make Amazon EC2 available with an Annual Uptime Percentage (defined below) of at least 99.95% during the Service Year. In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit as described below. [...]

• “Annual Uptime Percentage” is calculated by subtracting from 100% the percentage of 5 minute periods during the Service Year in which Amazon EC2 was in the state of “Region Unavailable.” If you have been using Amazon EC2 for less than 365 days, your Service Year is still the preceding 365 days but any days prior to your use of the service will be deemed to have had 100% Region Availability [...]
• “Unavailable” means that all of your running instances have no external connectivity during a five minute period and you are unable to launch replacement instances. [...]

To receive a Service Credit, you must submit a request by sending an e-mail message to aws-sla-request @ amazon.com. To be eligible, the credit request must [...] include your server request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks)

This new SLA does not appear to address the reliability of server instances individually or in aggregate. For example, if half of a customer's EC2 instances lose their connections or die every 6 minutes, EC2 would still be considered "available" even if it is essentially unusable.

If the entire EC2 service is down a cumulative four hours and twenty minutes, customers must furnish proof of the outage to Amazon to be eligible for the 10% credit. This seems like an onerous process for very little compensation, and isn't in-line with Amazon's famous "Relentless Customer Obsession". Amazon takes monitoring very seriously and should take the lead by tracking, reporting, and proactively compensating customers when it lets them down.

Open Source and Cloud Computing

by Tim O'Reilly | @timoreilly | comments: 41

I've been worried for some years that the open source movement might fall prey to the problem that Kim Stanley Robinson so incisively captured in Green Mars: "History is a wave that moves through time slightly faster than we do." Innovators are left behind, as the world they've changed picks up on their ideas, runs with them, and takes them in unexpected directions.

In essays like The Open Source Paradigm Shift and What is Web 2.0?, I argued that the success of the internet as a non-proprietary platform built largely on commodity open source software could lead to a new kind of proprietary lock-in in the cloud. What good are free and open source licenses, all based on the act of software distribution, when software is no longer distributed but merely performed on the global network stage? How can we preserve freedom to innovate when the competitive advantage of online players comes from massive databases created via user contribution, which literally get better the more people use them, raising seemingly insuperable barriers to new competition?

I was heartened by the program at this year's Open Source Convention. Over the past couple of years, open source programs aimed at the Web 2.0 and cloud computing problem space have been proliferating, and I'm seeing clear signs that the values of open source are being reframed for the network era. Sessions like Beyond REST? Building Data Services with XMPP PubSub, Cloud Computing with BigData, Hypertable: An Open Source, High Performance, Scalable Database, Supporting the Open Web, and Processing Large Data with Hadoop and EC2 were all full. (Due to enforcement of fire regulations at the Portland Convention Center, many of them had people turned away, as SRO was not allowed. Brian Aker's session on Drizzle was so popular that he gave it three times!)

But just "paying attention" to cloud computing isn't the point. The point is to rediscover what makes open source tick, but in the new context. It's important to recognize that open source has several key dimensions that contribute to its success:

1. Licenses that permit and encourage redistribution, modification, and even forking;
2. An architecture that enables programs to be used as components where-ever possible, and extended rather than replaced to provide new functionality;
3. Low barriers for new users to try the software;
4. Low barriers for developers to build new applications and share them with the world.

This is far from a complete list, but it gives food for thought. As outlined above, I don't believe we've figured out what kinds of licenses will allow forking of Web 2.0 and cloud applications, especially because the lock-in provided by many of these applications is given by their data rather than their code. However, there are hopeful signs like Yahoo! Boss that companies are at beginning to understand that in the era of the cloud, open source without open data is only half the application.

But even open data is fundamentally challenged by the idea of utility computing in the cloud. Jesse Vincent, the guy who's brought out some of the best hacker t-shirts ever (as well as RT) put it succinctly: "Web 2.0 is digital sharecropping." (Googling, I discover that Nick Carr seems to have coined this meme back in 2006!) If this is true of many Web 2.0 success stories, it's even more true of cloud computing as infrastructure. I'm ever mindful of Microsoft Windows Live VP Debra Chrapaty's dictum that "In the future, being a developer on someone's platform will mean being hosted on their infrastructure." The New York Times dubbed bandwidth providers OPEC 2.0. How much more will that become true of cloud computing platforms?

That's why I'm interested in peer-to-peer approaches to delivering internet applications. Jesse Vincent's talk, Prophet: Your Path Out of the Cloud describes a system for federated sync; Evan Prodromou's Open Source Microblogging describes identi.ca, a federated open source approach to lifestreaming applications.

We can talk all we like about open data and open services, but frankly, it's important to realize just how much of what is possible is dictated by the architecture of the systems we use. Ask yourself, for example, why the PC wound up with an ecosystem of binary freeware, while Unix wound up with an ecosystem of open source software? It wasn't just ideology; it was that the fragmented hardware architecture of Unix required source so users could compile the applications for their machine. Why did the WWW end up with hundreds of millions of independent information providers while centralized sites like AOL and MSN faltered?

Take note: All of the platform as a service plays, from Amazon's S3 and EC2 and Google's AppEngine to Salesforce's force.com -- not to mention Facebook's social networking platform -- have a lot more in common with AOL than they do with internet services as we've known them over the past decade and a half. Will we have to spend a decade backtracking from centralized approaches? The interoperable internet should be the platform, not any one vendor's private preserve. (Neil McAllister provides a look at just how one-sided most platform as a service contracts are.)

So here's my first piece of advice: if you care about open source for the cloud, build on services that are designed to be federated rather than centralized. Architecture trumps licensing any time.

But peer-to-peer architectures aren't as important as open standards and protocols. If services are required to interoperate, competition is preserved. Despite all Microsoft and Netscape's efforts to "own" the web during the browser wars, they failed because Apache held the line on open standards. This is why the Open Web Foundation, announced last week at OScon, is putting an important stake in the ground. It's not just open source software for the web that we need, but open standards that will ensure that dominant players still have to play nice.

The "internet operating system" that I'm hoping to see evolve over the next few years will require developers to move away from thinking of their applications as endpoints, and more as re-usable components. For example, why does every application have to try to recreate its own social network? Shouldn't social networking be a system service?

This isn't just a "moral" appeal, but strategic advice. The first provider to build a reasonably open, re-usable system service in any particular area is going to get the biggest uptake. Right now, there's a lot of focus on low level platform subsystems like storage and computation, but I continue to believe that many of the key subsystems in this evolving OS will be data subsystems, like identity, location, payment, product catalogs, music, etc. And eventually, these subsystems will need to be reasonably open and interoperable, so that a developer can build a data-intensive application without having to own all the data his application requires. This is what John Musser calls the programmable web.

Note that I said "reasonably open." Google Maps isn't open source by any means, but it was open enough (considerably more so than any preceding web mapping service) and so it became a key component of a whole generation of new applications that no longer needed to do their own mapping. A quick look at programmableweb.com shows google maps with about 90% share of mapping mashups. Google Maps is proprietary, but it is reusable. A key test of whether an API is open is whether it is used to enable services that are not hosted by the API provider, and are distributed across the web. Facebook's APIs enable applications on Facebook; Google Maps is a true programmable web subsystem.

That being said, even though the cloud platforms themselves are mostly proprietary, the software stacks running on them are not. Thorsten von Eicken of Rightscale pointed out in his talk Scale Into the Cloud, that almost all of the software stacks running on cloud computing platforms are open source, for the simple reason that proprietary software licenses have no provisions for cloud deployment. Even though open source licenses don't prevent lock-in by cloud providers, they do at least allow developers to deploy their work on the cloud.

In that context, it's important to recognize that even proprietary cloud computing provides one of the key benefits of open source: low barriers to entry. Derek Gottfried's Processing Large Data with Hadoop and EC2 talk was especially sweet in demonstrating this point. Derek described how, armed with a credit card, a sliver of permission, and his hacking skills, he was able to put the NY Times historical archive online for free access, ramping up from 4 instances to nearly 1,000. Open source is about enabling innovation and re-use, and at their best, Web 2.0 and cloud computing can be bent to serve those same aims.

Yet another benefit of open source - try before you buy viral marketing - is also possible for cloud application vendors. During one venture pitch, I was asking the company how they'd avoid the high sales costs typically associated with enterprise software. Open source has solved this problem by letting companies build a huge pipeline of free users, who they can then upsell with follow-on services. The cloud answer isn't quite as good, but at least there's an answer: some number of application instances are free, and you charge after that. While this business model loses some virality, and transfers some costs from the end user to the application provider, it has a benefit that open source now lacks, of providing a much stronger upgrade path to paid services. Only time will tell whether open source or cloud deployment is a better distribution vector, but it's clear that both are miles ahead of traditional proprietary software in this regard.

In short, we're a long way from having all the answers, but we're getting there. Despite all the possibilities for lock-in that we see with Web 2.0 and cloud computing, I believe that the benefits of openness and interoperability will eventually prevail, and we'll see a system made up of cooperating programs that aren't all owned by the same company, an internet platform, that, like Linux on the commodity PC architecture, is assembled from the work of thousands. Those who are skeptical of the idea of the internet operating system argue that we're missing the kinds of control layers that characterize a true operating system. I like to remind them that much of the software that is today assembled into a Linux system already existed before Linus wrote the kernel. Like LA, 72 suburbs in search of a city, today's web is 72 subsystems in search of an operating system kernel. When we finally get that kernel, it had better be open source.

Video of Rich Wolski's EUCALYPTUS talk at Velocity

by Jesse Robbins | @jesserobbins | comments: 1

Rich Wolski gave a truly impressive talk at Velocity about an open-source software infrastructure for cloud computing called EUCALYPTUS . The API is compatible with Amazon's EC2 interface, and the underlying infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain. Watch and learn...

You can see more videos from Velocity on Blip.tv.

Two new open source projects at Velocity

by Jesse Robbins | @jesserobbins | comments: 3

At Velocity next week there will be two significant open source projects debuting. The first is the Jiffy: Open Source Performance Measurement and Instrumentation tool created by Scott Ruthfield and his team at Whitepages.com.

Most tools for measuring web performance come in two flavors:

• Developer-installed tools (Firebug, Fiddler, etc.) that allow individuals to closely trace single sessions


• Third-party performance monitoring systems (Gomez, Keynote, etc.) that will hit your site occasionally and report back component-level metrics (for a fee)

Neither of these tools give you real-world information on what’s actually happening with your clients—how long are pages really taking to load, what’s the real cost of client-side execution, and what’s the impact of your loading or dependency chain. This is even more important when you don’t host all of your own assets, such as when you load ads or JavaScript from third parties, for example, and you need to monitor their performance.

Thus we built Jiffy—an end-to-end system for instrumenting your web pages, capturing client-side timings for any event that you determine, and storing and reporting on those timings. You run Jiffy yourself, so you aren’t dependent on the performance characteristics, inflexibility, or costs of third-party hosted services.

The second is project is EUCALYPTUS, the Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems, presented by Rich Wolski from UCSB. This project has already started getting attention. (Many thanks to Surj Patel of Structure08/GigaOM for connecting us!)

Eucalyptus is an open-source software infrastructure for implementing "cloud computing" on clusters. The current interface to EUCALYPTUS is compatible with Amazon's EC2 interface, but the infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain.

The talk will focus on the design, the implementation tradeoffs we have identified in implementing Eucalyptus as an exploratory tool, and the ways in which we have chosen to address these tradeoffs in the first version of the software.

CloudCamp gathering after Velocity

by Jesse Robbins | @jesserobbins | comments: 2

On Tuesday after Velocity closes there will be a CloudCamp gathering at Microsoft's San Francisco Office. I'll be going (unless I'm too exhausted to stand).

CloudCamp was formed in order to provide a common ground for the introduction and advancement of cloud computing

Through a series of local cloudcamp events, attendees can exchange ideas, knowledge and information in a creative and supporting environment, advancing the current state of cloud computing and related technologies. As an informal, member-supported gathering, we rely entirely on volunteers to help with meeting content, speakers, meeting locations, equipment and membership recruitment. We also have corporate sponsors that provide financial assistance with venues, software, books, discounts, and other valuable donations. To become a member, simply register for an upcoming event. Anyone may attend a meeting, there are no fees or dues.

It looks like there is now a London CloudCamp being planned for July 16th as well.

(PS: If you still haven't registered for Velocity and want to attend, you can use my 20% discount code "vel08js".)