Four short links: 5 November 2009

Heat Maps in R, EC2 Blackhat Tricks, Snickersome Unicode, and Decoding Statistics

by Nat Torkington | @gnat | comments: 0

1. Heat Maps in R -- We used financial data here because it's easier to access than the airline data, but it's actually a pretty interesting way of looking at a financial time series. Weekend and holiday effects are a bit more obvious, and it's a bit like being able to see the daily, weekly, monthly and yearly closes all at once (by scanning your eye over the calendar in different directions). Includes source code. (via migurski on Delicious)
2. BlackHat and EC2 -- Theft of resources is the red-headed step-child of attack classes and doesn't get much attention, but on cloud platforms where resources are shared amongst many users these attacks can have a very real impact. With this in mind, we wanted to show how EC2 was vulnerable to a number of resource theft attacks and the videos below demonstrate three separate attacks against EC2 that permit an attacker to boot up massive numbers of machines, steal computing time/bandwidth from other users and steal paid-for AMIs. (via straup on Delicious)
3. Funny Characters in Unicode -- I never get tired of the wacky stuff in Unicode. I love the thought of a Unicode committee somewhere arguing passionately about the number of buttons on the snowman .... (via Hacker News)
4. Statistics to English Translation -- The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate. This matters. Bandying around numbers with misleading labels, or misinterpreting numbers that have a precise and defined meaning, does not further understanding. (Said 78.4% of statisticians, with a 20% confidence factor probability of false positives)

 

Four short links: 8 October 2009

DIY Baby Rocker, Unix Systems Glory, Encrypting Ephemera, and Explaining Creative Joy

by Nat Torkington | @gnat | comments: 0

1. Linux Baby Rocker -- inventive use of a CD drive and the eject command ... (via Hacker News)
2. I Like Unicorn Because It's Unix -- forceful rant about the need to rediscover Unix systems programming. Reminds me of the Varnish notes where the author explains that it works better because it uses the operating system instead of recreating it poorly.
3. Encrypting Ephemeral Storage and EBS Volumes on Amazon -- step-by-step instructions. (via Matt Biddulph on Delicious)
4. You Have No Life -- if a video smacks even slightly of concentrated effort or advance planning, someone will inevitably scoff that the subject has a) "too much time on his hands" or b) "no life." Ten times out of ten. [...] After six years I lack a succinct, meaningful response to my students' defensive, clannish embrace of mediocrity, though I'm grateful for this tweet, which comes pretty close: dwineman: You say "looks like somebody has too much time on their hands" but all I hear is "I'm sad because I don't know what creativity feels like."

Four short links: 2 October 2009

Social Media Parasites, Open Government Data, Prime Numbers, Amazon Image Abuse

by Nat Torkington | @gnat | comments: 1

1. I'm Tired of Your Analogue Attitude -- hilarious animated clip about social media gurus, made using xtranormal. (via trib on twitter)
2. Three Laws of Open Government Data -- 1. If it can’t be spidered or indexed, it doesn’t exist; 2. If it isn’t available in open and machine readable format, it can’t engage; 3. If a legal framework doesn’t allow it to be repurposed, it doesn’t empower. (also see slide deck)
3. Structure and Randomness in the Prime Numbers -- paper about some of the fun mathematics around prime numbers. (via Hacker News)
4. Abusing Amazon Images -- decoding and doing fun things with the Amazon images API. The cool thing (if you want to generate unlikely Amazon images) is that you're not limited to one use of any of these commands. You can have multiple discounts, multiple shadows, multiple bullets, generating images that Amazon would never have on its site. However, every additional command you add generates another 10% to the image dimensions, adding white space around the image. And that 10% compounds; add a lot of bullets, and you'll find that you have a small image in a large blank space. (You can use the CR command to cut away the excess, however.) Note also that the commands are interpreted in order, which can have an impact on what overlaps what.

Rebooting the Book (One Apple iPad Tablet at a Time)

by Mark Sigal | @netgarden | comments: 30

"It is August, 1927, and Al Jolson is industriously, unwittingly, engaged in the destruction of one great art form and the creation of another...In four short years, the 'talkie' will completely subsume the silent movie." - from The Speed of Sound by Scott Eyman

The "Come to Jesus" Moment for the Book Business
In the age of the always on, it's fair to ask, do people read anymore?

Web content, video games, iPhone apps, Facebook sessions, YouTube videos, iTunes libraries, and Hulu media programming drive significant portions of our clickstream activity throughout the day.

Talking on the phone, emailing, and other forms of messaging sop up huge chunks of our free time, too.

As a consequence, book sales are stagnating, and have been for some time (this coincides with declines in all forms of print media - news and magazines included).

In big box retail land, Borders, the only real competitor to Barnes & Noble, is on life support. The independent bookstore is a shrinking breed, with less than 10% of the market.

Meanwhile, Amazon is the book industry's boogeyman, given their market share and proximity to the customer's wallet (the all important "billing relationship"). And the Kindle e-Book reader has the potential to entirely dis-intermediate the book publisher or, minimally, exert even stronger pricing power over them.

More terrifying, the book industry has no idea how to effectively market a book in a world devoid of bookstores, save for the hail-mary of an Oprah recommendation.

"Media doesn't matter, reviews don't matter, blurbs don't matter," says one powerful agent. "Nobody knows where the readers are, or how to connect with them."

And owing to a decades-old "consigment logic," unsold inventory is "remaindered." This is a euphemism for the practice of shredding unsold books and magazines. Not exactly green-friendly.

(continue reading)

Four short links: 19 August 2009

Survivor Bias, Algorithmic Trading, S3 Tools, DIY GSM

by Nat Torkington | @gnat | comments: 3

1. Business Advice Plagued by Survivor Bias -- "Burying the other evidence: [...] Doesn't most business advice suffer from this fallacy? Harvard Business School's famous case studies include only success stories. To paraphrase Peter, what if twenty other coffee shops had the same ideas, same product, and same dedication as Starbucks, but failed? How does that affect what we can learn from Starbucks's success? (via Hacker News)
2. A Bestiary of Algorithmic Trading Strategies -- insight into the algorithms used by quant traders. Statistical arbitrageurs are a sort of squishy area, similar to arbs, but distinct from them. They find “pieces” of securities which are theoretically equivalent. For example, they may notice a drift between prices of oil companies which should revert to a mean value. This mean reversion should happen if the drift doesn’t have anything to do with actual corporate differences, like one company’s wells catching on fire. What you’re doing here is buying and selling the idea of an oil company, or in other words, a sort of oil company market spread risk. You’re assuming these two companies are statistically the same, and so they’ll revert to some kind of mean when one of the prices move. (via Hacker News)
3. s3cmd -- commandline tool for moving files into and out of Amazon S3.
4. DIY GSM Network -- wow. How to build your own GSM network. Bit by bit, the telcos are getting pressured by the hobbyists. This barbarian is looking forward to the day when the walled gardens are sacked. (via Slashdot)

Amazon, Zappos and Buying What You Can’t Compete Against

by Joshua-Michéle Ross | @jmichele | comments: 0

Amazon bought Zappos. At first I was a bit surprised. Like an aging celebrity going to the “big theater in the sky” it is unexpected when you first hear about it - but upon reflection not surprising at all. It smacks of inevitability.

Amazon has consistently displayed a genius for pushing the boundaries of their business - for syndicating their strengths and making them a commodity for others to purchase. Are they a bookseller, an ecommerce platform, a cloud service provider, a consumer electronics company etc.?

Along the way Zappos has shown the same genius - taking Amazon’s competence syndication strategy one step further. When they launched Powered by Zappos they allowed smaller merchants to overlay their brand on top of the entire Zappos supply chain and, more importantly, on top of Zappos’ legendary customer service. Zappos was syndicating the entire experience.

Much of the online conversation is taken up with worries that Amazon will not abide Zappos’ culture. I feel quite the opposite. I believe culture is exactly what Jeff Bezos and Amazon were buying (after all it likely wasn’t fulfillment centers, logistics, inventory or even customers). Culture is the one asset Amazon couldn’t compete against.

It was a smart move.

Four short links: 8 June 2009

3D Geometry, The Printable Web, Government Internet Fail, and Real World Cloud Computing

by Nat Torkington | @gnat | comments: 2

1. How to Project on 3D Geometry -- the fine art (and math) of distorting an image so that it looks undistorted when projected onto a non-flat 3D surface. Confused? See the images below. (via straup on Delicious)
2. ZinePal -- Create your own printable magazine from any online content. (via warrenellis on Delicious)
3. What The Government Doesn't Understand About The Internet And What To Do About It -- Tom Steinberg from MySociety lays it out. As true for US, NZ, and every other country as it is for the UK (for which it was written). Accept that any state institution that says “we control all the information about X” is going to look increasingly strange and frustrating to a public that’s used to be able to do whatever they want with information about themselves, or about anything they care about (both private and public). This means accepting that federated identity systems are coming and will probably be more successful than even official ID card systems: ditto citizen-held medical records. It means saying “We understand that letting train companies control who can interface with their ticketing systems means that the UK has awful train ticket websites that don’t work as hard as they should to help citizens buy cheaper tickets more easily. And we will change that, now.” What I like about Tom vs the US's Gov 2.0 is that Tom puts down philosophy that's hard to argue with, whereas the US is dangerously close to simply focusing on techniques and that's subvertible.
4. Real World Cloud Computing -- summary from a panel of startups who are using EC2. The lock-in is latency. Transfering data within the Amazon services is free. Transfering data to an Amazon competitor: not free.

Four short links: 13 Apr 2009

by Nat Torkington | @gnat | comments: 1

Worms, sorting, languages, and infrastructure:

1. Twitter XSS Attacks (Lynne Pope) -- several incarnations of a worm spread quickly across Twitter this weekend. Twitter profiles are generated by themes, whose parameters users can change. The user-supplied value for the colour was used directly in the CSS color field without filtering, which the original worm strain used to end the CSS and begin Javascript to put the worm into the profile of any Twitter user who viewed the infected profile. Infected users were made to tweet about the worm, with links that would infect anyone who viewed. The worm spread quickly through RTing one of the worm's messages, which claimed to link to instructions on fighting the worm. Later variants use background-color and background parameters. Initial variations downloaded Javascript from mikeyylolz.uuuq.com, since closed down by its hosting company. Later variants download the code from stalkdaily.com, the site that the initial variation spammed about. I wonder whether the 17-year old author of the variants will be able to pay his inevitable legal bills through Google click dollars? (also interesting: Sophos and bdonews)
2. Visualising Sorting -- some beautiful and informative illustrations of how sorting algorithms work. (via @ajtowns)
3. Art and Code: Obscure or Beautiful? -- In the presentation called “50 in 50″ you can see Guy Steele rap about APL and later in the video about spelling keywords backwards. The song about God wrote in Lisp code is also a part of the presentation. Among the languages mentioned are APL, Cobol, AP/I, Scheme, IPL-V, AED, Madcap, Piet, SNOBOL, ADA, Algol60, Intercal, Logo, Perligata, Shakespeare, Lucid, Occam, HQ9+, MUMBLE, Rake, Perl and of course Lisp. It kicks in at about 3m20s and is rather a post-modern presentation. (via
4. Experiences Deploying Large-Scale Infrastructure in Amazon EC2 -- As an aside, I've been very impressed with the reliability of EC2. Like many other people, I didn't know what to expect, but I've been pleasantly surprised. Very rarely does an EC2 instance fail. In fact I haven't yet seen a total failure, only some instances that were marked as 'deteriorated'. When this happens, you usually get a heads-up via email, and you have a few days to migrate your instance, or launch a similar one and terminate the defective one. (via Simon Willison)

At Risk: Universal Online Access to All Knowledge

by Linda Stone | comments: 11

I’ve been following Brewster Kahle and Robert Darnton, a University Professor and director of Harvard’s Library, recently, and they’re concerned over the settlement of the lawsuit between Google and the authors and publishers, over the scanning and use of books in Google Book Search. In my experience, Brewster is extraordinarily thoughtful and takes a long view. Early in my career, I was a librarian. I love books. So while I’m not a lawyer and I find this settlement confusing, I’m writing about it because I think it merits awareness and a serious discussion.

The key issues appear to be whether the business model created by the settlement will lock up content that essentially belongs to the public domain (per Brewster) and whether the publishers’ and authors’ creation of a Google monopoly for books will harm access to knowledge in the future (per Darnton). Below, I’m relying on their words to explain this further.

Last week Brewster posted “It’s All About the Orphans” (http://www.opencontentalliance.org/2009/02/23/its-all-about-the-orphans/) on the blog of the Open Content Alliance, focusing on the plight of “orphan works” - that vast number of books that are still under copyright but whose authors can no longer be found:

"After digesting the proposed Google Book Settlement, it becomes clear that the dizzyingly complex agreement is, in essence, an elaborate scheme for the exploitation of orphan works… The upshot, if the Settlement is approved, would be legal protection for Google, and only for Google, to scan and provide digital access to the orphan works. Presto! … So, should the Settlement be approved, Google will be handed exclusive access to the orphans, and the public loses out… I, personally, am amazed at this creative use of class action law. The three parties have managed to skirt copyright law, bypass legislative efforts, and feather their own nests - all through the clever use of law intended to remedy harms. This Settlement, if approved by the judge, will accomplish things appropriate to a legislative body not to private corporate boardrooms. Let’s live under the rule of law, as arduous as that might be, and free the orphans, legitimately, not for one corporation but for all of us."

And in “Google & the Future of Books” (http://www.nybooks.com/articles/22281), an article that Darnton published in The New York Review of Books last month, the focus is slightly different but the upshot is the same:

"After reading the settlement and letting its terms sink in—no easy task, as it runs to 134 pages and 15 appendices of legalese - one is likely to be dumbfounded: here is a proposal that could result in the world's largest library… Moreover, in pursuing the terms of the settlement with the authors and publishers, Google could also become the world's largest book business - not a chain of stores but an electronic supply service that could out-Amazon Amazon… The class action character of the settlement makes Google invulnerable to competition… We are allowing a question of public policy - the control of access to information - to be determined by private lawsuit… As an unintended consequence, Google will enjoy what can only be called a monopoly - a monopoly of a new kind, not of railroads or steel but of access to information… The settlement creates a fundamental change in the digital world by consolidating power in the hands of one company… This is also a tipping point in the development of what we call the information society. If we get the balance wrong at this moment, private interests may outweigh the public good for the foreseeable future, and the Enlightenment dream may be as elusive as ever."

A lot seems to be at stake and the court may approve the settlement in June! I don't care if the settlement means that Google will get even richer (disclosure: I’m a Google shareholder). The question is: to what extent will WE become poorer?

Four short links: 26 Feb 2009

by Nat Torkington | @gnat | comments: 3

Three stories about old-media in new-media age, and some patent goblins to leave a bad taste in your mouth:

1. The Kindle Swindle -- the Authors Guild president argues that the robot voice of the Kindle does away with audiobook royalty streams, lucrative for some titles. Doesn't mention the vast majority of books for which there is no audiobook. Creators have attempted to regulate use with licenses, but I think the plasticity of bits argues against this being possible for much longer. Now "audiobook"-ness is a feature of the device, not a feature of the retailed artistic work, and the question is not only how to charge for it but whether it makes sense to continue to charge for it. Neil Gaiman, by the way, doesn't feel the same way as the head of the Author's Guild.
2. If You Want to Save Newspapers, Learn to Love Your iPhones -- a long Observer piece about the "future of newspapers", reinvention in the mobile age, subscription models, the curse of Google, etc. Many great quotes, for example: "Google is great for Google, but it’s terrible for content providers, because it divides that content quantitatively rather than qualitatively. And if you are going to get people to pay for content, you have to encourage them to make qualitative decisions about that content." -- Robert Thomson, the managing editor of The Wall Street Journal.
3. NYT ArticleSkimmer -- reminscent, vaguely, of Arts & Letters Daily, the original "big heap o' content" page. Between this and Big Picture, I'm enjoying the experimentation in online newspaper formats.
4. Microsoft Sues TomTom Over Patents, Including Linux Kernel -- Microsoft patented elements of the FAT filesystem, including the system for representing long filenames on systems that only handle 8.3 filenames like CRAPWARE.EXE. This filesystem is used in pretty much every digital camera and Flash filesystem device, and the TomTom system in question. This Ars Digita article raises the interesting possibility that the Open Invention Network could respond by flexing its patent portfolio muscles and make it clear that nobody wants a battle over patents (except lawyers who are paid by the hour).

Four short links: 25 Feb 2009

by Nat Torkington | @gnat | comments: 0

Amazon, Apple, Science, and Databases:

1. Amazon's Wheel of Growth -- a fascinating diagram in the middle, the flywheel of customer experience driving sales driving sellers driving selection which drives experience again, and all the while lower costs allows Amazon to deliver lower prices and thus lower selection.
2. iPhone Sketch -- stencils to use when sketching your iPhone app's screens.
3. The Importance of Stupidity in Scientific Research -- thoughts on the value of feeling stupid ("I actively seek out new opportunities to feel stupid") and how PhD programs don't prepare students for that feeling. "Science involves confronting our `absolute stupidity'. That kind of stupidity is an existential fact, inherent in our efforts to push our way into the unknown. Preliminary and thesis exams have the right idea when the faculty committee pushes until the student starts getting the answers wrong or gives up and says, `I don't know'." (via Titine's delicious stream)
4. First Key-Value Storage Meeting Held in Japan -- yet more work being done with modernized DBM technology. As Joshua said in his delicious comment, plenty of systems I've never heard of before. (via Joshua's delicious stream)

Google, WalMart, and MyBarackObama.com: The Power of the Real Time Enterprise

by Tim O'Reilly | @timoreilly | comments: 22

What do Google, WalMart, and MyBarackObama.com have in common, besides their extraordinary success? They are organizations that are infused with IT in such a way that it leads to a qualitative change in their entire business.

I get frustrated when I see people highlighting use of social media--blogging, wikis, twitter, customer feedback systems like Dell IdeaStorm or MyStarbucksIdea--as if they were exemplars of what has come to be called "Enterprise 2.0."

As I said in my keynote at the Web 2.0 Expo NY (and in a followup radar post), WalMart is a better example of Enterprise 2.0 than any of these more trendy examples of user contribution systems. If Google's key innovation with PageRank was to recognize that a link was a vote, which could be counted and measured to get better search results, so too, WalMart recognized early on that a purchase was a vote. Each company built real-time information systems to capture and respond to that vote. WalMart built a supply chain in which goods are automatically re-ordered as they go out the door, with algorithms based on rate of sale controlling the reorders. Google built a better search engine, in which pages that were "better linked" were given priority over the ones produced by pure keyword matches. They went on to build real-time systems to measure what John Battelle called the database of intentions, as expressed by people's queries and subsequent clickstream data, as well as an ad auction system that prices ads in real-time based on the predicted likelihood of the ad being clicked on.

I came to see just how closely MyBarackObama.com emulated these ideas of the real-time enterprise in accounts of the Houdini project, a bold program in which poll watchers eliminated the names from voters who had actually made it to the polling station from the "get out the vote" call lists:

While the hot line was too overwhelmed to be of much use, the source said the program itself still proved a smashing success....the campaign was able to clean 1.6 million voters from the call lists they distributed to canvassers that afternoon, making those lists 25 percent shorter on average.

While the infrastructure for data reporting broke down under the pressure of the election, the general trend is clear here: competitive advantage comes from capturing data more quickly, and building systems to respond automatically to that data.

Consider MyBarackObama.com as a kind of vast machine, with humans as extensions of the programmatic brain: volunteers log in to get their get-out-the-vote call lists. They place their calls, then use the web to report back their results. Those results modify the call lists for the next volunteer. At the other end, the Houdini volunteers are taking note of who is actually coming out to vote, allowing the system to dispatch additional attention to hot spots, for example where there is an undervote compared to the campaign's projections. Meanwhile, the pruned call lists make the volunteers more effective. Inside the machine, programmers are tuning the algorithms, while top campaign staffers are making key decisions to adjust the resource mix.

Now put these three examples, Google, WalMart, and MyBarackObama together, and ask yourself what they tell you about the future of business, military operations, or any large organization.

Sensing, processing, and responding (based on pre-built models of what matters, "the database of expectations," so to speak) is arguably the hallmark of living things. We're now starting to build computers that work the same way. And we're building enterprises around this new kind of sense-and-respond computing infrastructure. In this sense, you can argue that Microsoft's term "Live Software" is the best name yet for the kind of software-infused enterprise we're building.

It's essential to recognize that each of these systems is a hybrid human-machine system, in which human actions are part of the computational loop. Back in 1998, when I was trying to understand just how people were using Perl and other scripting languages on the web, I came to recognize that web applications, unlike desktop applications, still have the programmers inside them. Perl was called "the duct tape of the internet" precisely because it was used for programming that was only expected to last a short time; the programmers were still there, constantly tweaking the application. (I first began using the image of "the Mechanical Turk" in my talks about this aspect of web applications in 2003.)

What became clear in the ensuing decade is that humans are not just part of the programming, but also sensors and actuators for computers. Our aggregate behavior is measured, monitored, and becomes feedback that improves the overall intelligence of the system. That is why I've said that the defining characteristic of Web 2.0 applications is that they "harness collective intelligence."

Aside: I seem to have lost the battle to define Web 2.0 as" the use of the network as platform to build systems that get better the more people use them. Perhaps its the lure of the obvious: companies and products that harness explicit user contribution are easier to recognize than those that pursue the more subtle and difficult task of harnessing implicit contribution. Or perhaps it's the persistent gravitational tug of the idea that the heart of Web 2.0 is ad-supported business models; therefore, enterprise features that look like those of well-known companies featuring user contribution and ad-supported business models must by definition also be "2.0." For me, the far more profound and powerful systems come from harnessing both explicit and implicit human contribution.

Again, consider MyBarackObama.com. It definitely harnessed explicit contribution, providing a platform for volunteers to organize and host local calling parties, to blog, or perform other campaign activities. But ultimately, Obama's ground game--old fashioned precinct-level organizing, amped up to a new level by an army of distributed volunteers armed with mobile phones and coordinated via a web application--was the key to his victory. The "explicit" social media elements of MyBarackObama.com paled in impact compared to the development of a next generation electronic nervous system, in which volunteers were trained, deployed, and managed by a web application who used them, in John Sean McMullen's memorable phrase, as "souls in the great machine."

Web 2.0 and Cloud Computing

by Tim O'Reilly | @timoreilly | comments: 56

A couple of months ago, Hugh Macleod created a bit of buzz with his blog post The Cloud's Best Kept Secret. Hugh's argument: that cloud computing will lead to a huge monopoly. Of course, a couple of weeks ago, Larry Ellison made the opposite point, arguing that salesforce.com is "barely profitable", and that no one will make much money in cloud computing.

In this post, I'm going to explain why Ellison is right, and yet, for the strategic future of Oracle, he is dangerously wrong.

First, let's take a look at Hugh Macleod's argument:

...nobody seems to be talking about Power Laws. Nobody's saying that one day a single company may possibly emerge to dominate The Cloud, the way Google came to dominate Search, the way Microsoft came to dominate Software.

Monopoly issues aside, could you imagine such a company? We wouldn't be talking about a multi-billion dollar business like today's Microsoft or Google. We're talking about something that could feasibly dwarf them. We're potentially talking about a multi-trillion dollar company. Possibly the largest company to have ever existed.

I imagine many of my friends who work for the aforementioned companies know all about this, and know how VAST the stakes are.

Windows vs Apple? Who cares? Kid's stuff. There's a much bigger game going on... And for some reason, its utter enormity seems to be a very well-kept secret, at least to non-combatants like myself.

The problem with this analysis is that it doesn't take into account what causes power laws in online activity. Understanding the dynamics of increasing returns on the web is the essence of what I called Web 2.0. Ultimately, on the network, applications win if they get better the more people use them. As I pointed out back in 2005, Google, Amazon, ebay, craigslist, wikipedia, and all other other Web 2.0 superstar applications have this in common.

Cloud computing, at least in the sense that Hugh seems to be using the term, as a synonym for the infrastructure level of the cloud as best exemplified by Amazon S3 and EC2, doesn't have this kind of dynamic. (More on different types of cloud computing later.)

Of course, it is true that the bigger players will have economies of scale in the cost of equipment, and especially in the cost of power, that are not available to smaller players. But there are quite a few big players -- Google, Microsoft, Amazon -- to name a few, that are already at that scale, with or without a cloud computing play. What's more, economies of scale are not the same as increasing returns from user network effects. They may be characteristic of a commoditizing marketplace that does not actually give outsize economic leverage to the winners.

I can't vouch for the authenticity of the following remark, since I heard it secondhand, but it was from a thoughtful, informed source: Jeff Bezos is reported to have said that he welcomes cloud competition from Google and Microsoft, because they'll subsidize their cloud services with profits from other part of their business, while Amazon will always have to make it pay. "We're good at commodity businesses," Jeff is reported to have said, and the facts bear him out.

If cloud computing is a commodity business, then the outsize profits that Hugh envisioned are not going to be there. This is a business that will be huge, but it may be more similar to the web hosting and ISP markets, which are also huge, but not hugely profitable. (See Rackspace's numbers for a taste.)

But because one of my goals at Radar is to help people think about the future, I wanted to spend some time on the possible futures and strategies that could turn cloud computing into the kind of massive monopoly that Hugh envisioned.

Types of Cloud Computing

Since "cloud" seems to mean a lot of different things, let me start with some definitions of what I see as three very distinct types of cloud computing:

1. Utility computing. Amazon's success in providing virtual machine instances, storage, and computation at pay-as-you-go utility pricing was the breakthrough in this category, and now everyone wants to play. Developers, not end-users, are the target of this kind of cloud computing.

   This is the layer at which I don't presently see any strong network effect benefits (yet). Other than a rise in Amazon's commitment to the business, neither early adopter Smugmug nor any of its users get any benefit from the fact that thousands of other application developers have their work now hosted on AWS. If anything, they may be competing for the same resources.

   That being said, to the extent that developers become committed to the platform, there is the possibility of the kind of developer ecosystem advantages that once accrued to Microsoft. More developers have the skills to build AWS applications, so more talent is available. But take note: Microsoft took charge of this developer ecosystem by building tools that both created a revenue stream for Microsoft and made developers more reliant on them. In addition, they built a deep -- very deep -- well of complex APIs that bound developers ever-tighter to their platform.

   So far, most of the tools and higher level APIs for AWS are being developed by third-parties. In the offerings of companies like Heroku, Rightscale, and EngineYard (not based on AWS, but on their own hosting platform, while sharing the RoR approach to managing cloud infrastructure), we see the beginnings of one significant toolchain. And you can already see that many of these companies are building into their promise the idea of independence from any cloud infrastructure vendor.

   In short, if Amazon intends to gain lock-in and true competitive advantage (other than the aforementioned advantage of being the low-cost provider), expect to see them roll out their own more advanced APIs and developer tools, or acquire promising startups building such tools. Alternatively, if current trends continue, I expect to see Amazon as a kind of foundation for a Linux-like aggregation of applications, tools and services not controlled by Amazon, rather than for a Microsoft Windows-like API and tools play. There will be many providers of commodity infrastructure, and a constellation of competing, but largely compatible, tools vendors. Given the momentum towards open source and cloud computing, this is a likely future.

2. Platform as a Service. One step up from pure utility computing are platforms like Google AppEngine and Salesforce's force.com, which hide machine instances behind higher-level APIs. Porting an application from one of these platforms to another is more like porting from Mac to Windows than from one Linux distribution to another.

   The key question at this level remains: are there advantages to developers in one of these platforms from other developers being on the same platform? force.com seems to me to have some ecosystem benefits, which means that the more developers are there, the better it is for both Salesforce and other application developers. I don't see that with AppEngine. What's more, many of the applications being deployed there seem trivial compared to the substantial applications being deployed on the Amazon and force.com platforms. One question is whether that's because developers are afraid of Google, or because the APIs that Google has provided don't give enough control and ownership for serious applications. I'd love your thoughts on this subject.

3. Cloud-based end-user applications. Any web application is a cloud application in the sense that it resides in the cloud. Google, Amazon, Facebook, twitter, flickr, and virtually every other Web 2.0 application is a cloud application in this sense. However, it seems to me that people use the term "cloud" more specifically in describing web applications that were formerly delivered locally on a PC, like spreadsheets, word processing, databases, and even email. Thus even though they may reside on the same server farm, people tend to think of gmail or Google docs and spreadsheets as "cloud applications" in a way that they don't think of Google search or Google maps.

   This common usage points up a meaningful difference: people tend to think differently about cloud applications when they host individual user data. The prospect of "my" data disappearing or being unavailable is far more alarming than, for example, the disappearance of a service that merely hosts an aggregated view of data that is available elsewhere (say Yahoo! search or Microsoft live maps.) And that, of course, points us squarely back into the center of the Web 2.0 proposition: that users add value to the application by their use of it. Take that away, and you're a step back in the direction of commodity computing.

   Ideally, the user's data becomes more valuable because it is in the same space as other users' data. This is why a listing on craigslist or ebay is more powerful than a listing on an individual blog, why a listing on amazon is more powerful than a listing on Joe's bookstore, why a listing on the first results page of Google's search engine, or an ad placed into the Google ad auction, is more valuable than similar placement on Microsoft or Yahoo!. This is also why every social network is competing to build its own social graph rather than relying on a shared social graph utility.

   This top level of cloud computing definitely has network effects. If I had to place a bet, it would be that the application-level developer ecosystems eventually work their way back down the stack towards the infrastructure level, and the two meet in the middle. In fact, you can argue that that's what force.com has already done, and thus represents the shape of things. It's a platform I have a strong feeling I (and anyone else interested in the evolution of the cloud platform) ought to be paying more attention to.

The Law of Conservation of Attractive Profits

A lot of my thinking about web 2.0 grew directly out of my thinking about open source. My argument in The Open Source Paradigm Shift was that what we learned from the history of the IBM personal computer -- a commodity platform built from off-the-shelf parts -- was that it drained value out of the hardware ecosystem, turning it into a low-margin business. But profits didn't go away. Instead, through something that Clayton Christensen calls "the law of conservation of attractive profits," value migrated elsewhere, from hardware to software, from IBM to Microsoft. Christensen:

When attractive profits disappear at one stage in the value chain because a product becomes modular and commoditized, the opportunity to earn attractive profits with proprietary products will usually emerge at an adjacent stage.

I believe strongly that open source and open internet standards are doing the same to traditional software. And value is migrating to a new kind of layer, which we now call Web 2.0, which consists of applications driven not just by software but by network-effects databases driven by explicit or implicit user contribution.

So when Larry Ellison says that cloud computing and open source won't produce many hugely profitable companies, he's right, but only if you look at the pure software layer. This is a lot like saying that the PC wouldn't produce many hugely profitable companies, and looking only at hardware vendors! First Microsoft, and now Google give the lie to Ellison's analysis. The big winners are those who best grasp the rules of the new platform.

So here's the real trick: cloud computing is real. Everything is moving into the cloud, in whole or in part. The utility layer of cloud computing will be just that, a utility, without outsized profits.

But the cloud platform, like the software platform before it, has new rules for competitive advantage. And chief among those advantages are those that we've identified as "Web 2.0", the design of systems that harness network effects to get better the more people use them.

If Oracle isn't playing that game, they will one day be doomed to irrelevance. Perhaps, like hardware giants of the past - Compaq, say - they will be absorbed by a bigger company. Or perhaps, like Unisys, they will linger on in specialized markets, too big to go away but no longer on the cutting edge of anything. Or they will understand that it's not the database software that matters, but the data that it holds, and the services that can be built against that data.

The company that creates the right platform for network effects in data may well achieve the scale that Hugh Macleod envisioned.

P.S. I will be doing two panels on cloud computing at the Web 2.0 Summit in San Francisco the week after next, one on the application layer, and one on the infrastructure layer. Panelists include Paul Maritz (CEO of VMware, who, by the way totally gets what I'm talking about here), Russ Daniels (CTO for cloud services at HP), Padmasree Warrior (CTO at Cisco), the inimitable Marc Benioff of Salesforce.com, Kevin Lynch, CTO of Adobe, and Dave Girouard, who is in charge of Google Apps for the Enterprise. Should be some interesting conversations on the subjects raised in this post!

Amazon's new EC2 SLA

by Jesse Robbins | @jesserobbins | comments: 7

Amazon announced a new SLA for EC2, similar to the one for S3. This is a notable step for Amazon and cloud computing as a whole, as it establishes a new bar for utility computing services.

Amazon is committing to 99.95% availability for the EC2 service on a yearly basis, which corresponds to approximately four hours and twenty three minutes of downtime per year. It's important to remember that an SLA is just a contract that provides a commitment to a certain level of performance and some form of compensation when a provider fails to meet it.

Here's the summary of the EC2 SLA (emphasis added):

Service Commitment AWS will use commercially reasonable efforts to make Amazon EC2 available with an Annual Uptime Percentage (defined below) of at least 99.95% during the Service Year. In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit as described below. [...]

• “Annual Uptime Percentage” is calculated by subtracting from 100% the percentage of 5 minute periods during the Service Year in which Amazon EC2 was in the state of “Region Unavailable.” If you have been using Amazon EC2 for less than 365 days, your Service Year is still the preceding 365 days but any days prior to your use of the service will be deemed to have had 100% Region Availability [...]
• “Unavailable” means that all of your running instances have no external connectivity during a five minute period and you are unable to launch replacement instances. [...]

To receive a Service Credit, you must submit a request by sending an e-mail message to aws-sla-request @ amazon.com. To be eligible, the credit request must [...] include your server request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks)

This new SLA does not appear to address the reliability of server instances individually or in aggregate. For example, if half of a customer's EC2 instances lose their connections or die every 6 minutes, EC2 would still be considered "available" even if it is essentially unusable.

If the entire EC2 service is down a cumulative four hours and twenty minutes, customers must furnish proof of the outage to Amazon to be eligible for the 10% credit. This seems like an onerous process for very little compensation, and isn't in-line with Amazon's famous "Relentless Customer Obsession". Amazon takes monitoring very seriously and should take the lead by tracking, reporting, and proactively compensating customers when it lets them down.

Boycotting Amazon

by Allison Randal | comments: 14

In light of Amazon's attempts to lock print-on-demand publishers into their own printing services, I've made a personal decision not to buy from Amazon any more. Since the site first launched over a decade ago, I've spent thousands and thousands of dollars on Amazon feeding my addiction to tech books and fiction, on music, DVDs, electronics, and gifts for friends and family. I realize my spending is a tiny drop in the bucket of Amazon's total revenue, but it's a decision I feel good about, the same way I feel good about using low-energy lightbulbs, reusing plastic bags, and buying a car with environmentally friendly fuel economy and emissions ratings. One of the fundamental principles of capitalism is that when one source of goods and services isn't meeting your needs, you switch to another. The power to decide which businesses succeed and which fail lies in the collective hands of millions of individual consumers.

I've mainly switched to Books-A-Million for the prices (fair disclosure: I developed a good portion of the site back in the heady dot-com days), but I shop around at Barnes & Noble, Bookpool, Powell's, Alibris, BookFinder, and here in South Africa Exclusive Books. There's no shortage of alternatives, all over the world.

In my very first order, I bought some Xhosa language learning CDs, and on a whim added a print-on-demand book of Xhosa folk tales. It just goes to show that by restricting print-on-demand publishers, Amazon isn't only damaging the publishing ecosystem, it's also hurting its own business.

Amazon Accused of Anti-Trust Violations "Tied" to Print-On-Demand Terms

by Andrew Savikas | @andrewsavikas | comments: 2

Amazon's March announcement that POD publishers would be required to use Amazon's own POD service BookSurge in order to sell books directly on Amazon's site predictably rubbed quite a few folks the wrong way. (Technically speaking, publishers can use alternative POD providers, but must then provide Amazon with an inventory of at least five books, and the ability to offer many titles for sale without tying up capital in inventory is one of the appeals of a POD model.)

Now comes word that one of those POD publishers, Bangor, Maine-based BookLocker, has filed a class-action lawsuit against Amazon, alleging violations of the "tying" provisions of the Sherman Act -- in short, that Amazon has improperly predicated sales on use of printing services. From the complaint (PDF):

For example, on March 26, 2008, Amazon representative John Clifford notified Plaintiff that Amazon would only continue to sell BookLocker’s POD books through the Direct Amazon Sales Channel if Plaintiff agreed to print its books through BookSurge rather than Lightning Source. The Amazon representative further stated that books printed by Lightning Source or any other competing printer would have their “Add to Shopping Cart” buttons removed. (Amazon has also informed POD publishers that they may keep the Direct Amazon Sales Channel active if they agree to enroll in a program known as “Amazon Advantage.” However, the terms and conditions of participating in that program are so onerous so as to preclude it from being an economically viable option for POD book publishers.)

Amazon has continued through the present date to threaten POD publishers that unless they convert their inventory to BookSurge printing, their Direct Amazon Sales Channel will be discontinued.

The acrimony between publishers and Amazon is seemingly growing by the day, and this is hardly a surprising development. But it's also another sign that Amazon may be drifting toward damaging the very ecosystem it depends on for survival.

Amazon improves EC2 (by embracing failure)

by Jesse Robbins | @jesserobbins | comments: 5

Amazon just announced two big improvements to EC2:

• Multiple Locations
  Amazon EC2 now provides the ability to place instances in multiple locations. Amazon EC2 locations are composed of regions and Availability Zones. Regions are geographically dispersed and will be in separate geographic areas or countries. Currently, Amazon EC2 exposes only a single region. Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same region. Regions consist of one or more Availability Zones. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location.
  
  
• Elastic IP Addresses
  Elastic IP addresses are static IP addresses designed for dynamic cloud computing. An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it. Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by programmatically remapping your public IP addresses to any instance in your account. Rather than waiting on a data technician to reconfigure or replace your host, or waiting for DNS to propagate to all of your customers, Amazon EC2 enables you to engineer around problems with your instance or software by quickly remapping your Elastic IP address to a replacement instance.

Datacenters and geographic regions are Single Points of Failure (SPOF) too.  Failure Happens, and it's far better (and cheaper) to build services that are resilient to failure than to try to prevent them from happening.  This is a big step in the right direction.

Update: RightScale posted an excellent overview of how this works.