Understanding Web Operations Culture - the Graph & Data Obsession

by Jesse Robbins | @jesserobbins | comments: 8

We’re quite addicted to data pr0n here at Flickr. We’ve got graphs for pretty much everything, and add graphs all of the time.

-John Allspaw, Operations Engineering Manager at Flickr & author of The Art of Capacity Planning

One of the most interesting parts of running a large website is watching the effects of unrelated events affecting user traffic in aggregate. Web traffic is something that companies typically keep very secret, and often the only time engineers can talk about it is late at night, at a bar, and very much off the record.

There are many good reasons for keeping this kind of information confidential, particularly for publicly traded companies with complicated disclosure requirements. There are also downsides, the biggest being that is difficult for peers to learn from each other and compare notes.

John Allspaw recently created a WebOps Visualizations group on Flickr for sharing these kinds of graphs with the confidential information removed. Here’s an example of a traffic drop seen both by Flickr & by Last.FM that coincided with President Obama’s inauguration.

Similar traffic drop on Last.FM seen on the right

Google saw a similar drop as well

Was it because everybody went to Twitter?

Besides being an interesting story, sharing these kinds of graphs help people build better monitoring tools and processes. As just one example: How should the WebOps team respond to this dip in traffic? Is it an outage? The inaguration was a very well known event and so it’s easy to explain the drop in traffic… what happens when a similar drop in traffic occurs? Should the WebOps team be looking at CNN (or trends in twitter) along with everything else?

How do you tell when that unexpected 10% drop in traffic is really just people with something more important to do than browse your site?

(Note: Updated since original posting to add Google & Twitter graphs and annotations, and to switch the Last.FM graphic with an annotated one after I got permission.)

The Army, the Web, and the Case for Intentional Emergence

by Jim Stogdill | @jstogdill | comments: 19

Lt. Gen. Sorenson, Army CIO, at Web 2.0 Summit

I didn't make it to the Web 2.0 Summit in San Francisco in November last year so I didn't get to see Army CIO Gen Sorenson present this Higher Order Bit talk in person. However, I thought it was cool that the Army made the agenda and luckily someone posted the video. I finally got a chance to go through it. If you didn't see the talk, or don't have the 20'ish minutes to watch it now, here's a rough summary:

- Because of security and related concerns, it takes a very long time for the Army to take advantage of new generations of technology. We tend to deploy it widely about the time it's becoming obsolete.

- However, we are now beginning to take some advantage of Web 2.0 technologies in, for example, Stryker Brigade collaboration, battle command information sharing, and command and control.

I don't think that slow technology adoption is caused by fundamental first principles, so I don't think it has to remain true. But that's a long discussion for another time. In this post I'd like to focus on Army Battle Command, Web 2.0 and Gen Sorenson's connecting the two. Specifically I'd like to talk about lost opportunity and how the same technologies can constitute a generative platform in one setting and window dressing on a temple to determinism in another.

The lost opportunity I'm thinking of isn't whether Army Battle Command is Web 2.0 enough or not. It's that enterprises tend to see web technologies as an add on to whatever they already have. Plus, they tend to focus on specific technologies rather than the combination of technology, process and policy that make a collection of technologies viable as a generative platform. "Let's add some Web 2.0 to this system; we'll use REST instead of SOAP." But the fundamental question that the web answers isn't whether REST is better than SOAP, but whether emergence is more likely to create innovation than enterprise planning, and the answer to that question is yes.

General Sorenson says in the video that "CPOF brings in Web 2.0 capability, chat, video, etc..." and then comments on "graphics, chat, use of tools..." and stuff like that to reinforce the idea that Command Post of the Future (CPOF) and the Battle Command suite it is part of has Web 2.0 attributes. Like many enterprise technologists, General Sorenson appears to be focusing on rich user experience and collaboration as the attributes that give CPOF a Web 2.0 imprimatur. While that's not unexpected, I think it leaves most of the benefits on the table and untapped.

Putting aside for the moment that CPOF isn't primarily delivered through a browser, a first step toward webness, the reality is that CPOF and other systems like it neither leverage accessible platforms nor contribute to them. It is a standalone (though distributed) computing system with gee whiz collaboration and VoIP. And while it offers some enterprise-style data services, it has none of the features of a generative platform. If I'm in the field I can't readily extend it or build on it to solve different problems, modify its proprietary underpinnings to suit my local needs, or quickly incorporate its information into other applications. If an important aspect of Web 2.0 is enabling the long tail, then this isn't Web 2.0.

I should say, this isn't a post about web 2.0 semantics. However, it's important to understand that the web's power derives from its evolution as a platform. Otherwise it's hard to see what is being missed by the military's IT enterprise (and many other large enterprises).

From the beginning the web has been generative. It wasn't CompuServe. With some basic skills you could add to it, change it, extend it, etc. Jonathan Zittrain, in his excellent book The Future of the Internet - and How to Stop It, reflects on why the Internet has experienced such explosive innovation. He argues that it's the powerful combination of user-programmable personal computers, ubiquitous networking with the IP protocol, and open platforms. Today, the emergence of open source infrastructure, ubiquitous and cheap hosting for LAMP-based sites, open API's, and the intentional harnessing of crowd wisdom has ushered in the web 2.0 era. It's an era of high-velocity low-cost idea trying that leverages the web itself as the platform for building world changing ideas and businesses.

The Internet hosts innovation like it does because it is an unconstrained complex system where complex patterns can grow out of easy to assemble simple things. Simple things are not only permitted, but they are encouraged, facilitated, and often can be funded with a credit card.

I've subscribed to the notion of Gall's Law for longer than I knew it was a law:

"A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."

(continue reading)

Why Dell.com (was) More Enterprise 2.0 Than Dell IdeaStorm

by Tim O'Reilly | @timoreilly | comments: 9

In my keynote last week at Web 2.0 Expo New York, I made the comment that, cool as Dell Ideastorm is, the fundamental supply-chain approach behind dell.com is actually a better example of how Web 2.0 applies to the enterprise. I also made the provocative assertion that WalMart is a Web 2.0 company (or at least a model of how Web 2.0 principles apply to the enterprise.)

Based on questions I've heard since, I thought I should explain further.

Many people have been seduced into the ideas that Web 2.0 is all about explicit collaboration, contribution, and "the wisdom of crowds." So, for example, on his Web 2.0 Watch List, Seth Godin wrote: "For our purposes, my definition is that most of these companies are, as the wikipedia says, sites that 'let people collaborate and share information online in a new way.' So," he says, "Google doesn't make the cut, because most of their traffic comes to their search engine."

Now, I will say categorically that any Web 2.0 definition that excludes the Google search engine is broken. But it's broken in an instructive way, one that shows what the problem is.

Web 2.0 is ultimately about understanding the rules of business in the network era. I define Web 2.0 as the design of systems that harness network effects to get better the more people use them, or more colloquially, as "harnessing collective intelligence." This includes explicit network-enabled collaboration, to be sure, but it should encompass every way that people connected to a network create synergistic effects. So let's take Google:

• The very act of creating a search engine via spidering relies on a user-contributed network. Where does Google get its raw material except from us? When I publish this blog post, I'm contributing to Google (and to every other search engine.)

• Google's PageRank algorithm extracts an additional level of implied metadata contributed by links. All spiders follow links to discover new content; PageRank taught us that some links matter more than others. The engine became smarter by understanding a bit more about what people were contributing even when they didn't know they were doing so. Many a new breakthrough in Web 2.0 (e.g. Facebook's social graph) comes from making implicit contribution explicit in some way, gathering its benefit and amplifying it.

• Google's real-time ad auction is the heart of their economic engine. Their stroke of genius, which gave them their seemingly insuperable lead over the other search engines in ad monetization, was to understand that selling the top ad position to the highest bidder was actually leaving money on the table. Given that advertisers only pay for clicks, Google realized that if they could project the likely click-rate on an ad, they could sell top position to the best combination of price and click rate. A $5 cpc ad clicked on 3x as often as a $10 cpc ad will make $15. By instrumenting, measuring, and responding to the click rate, Google made ad auctions smarter - through harnessing implicit user contribution.

• Google also measures click-through behavior, surfing habits, and everything else they can get their hands on that will help them improve search results or ad performance. All of this is fed back into a real-time loop that is constantly trying to automate and optimize the user experience. Sounds a lot like the human brain, eh? (At least one with lots of learning plasticity.)

  If there is only one thing that enterprises ought to learn about Web 2.0, it's this one: building information systems that allow you to adjust in real time based on interaction with your customers is the true mark of a networked enterprise.

If you understand the following three things, you know everything you need to know about Web 2.0 and the enterprise:

1. Harvest every bit of user contribution, not just the explicit. Your business has thousands of touch points with customers. When they buy from you, they contribute data as well as money. When your suppliers increase their prices, or change their delivery times, they contribute data to you. When you advertise, and people respond (or don't), they contribute to you. When you introduce a new product, when you do something your customers love, or hate, and people talk about it, they contribute.

   Your data is one of your most critical business assets. Are you doing everything you can to wrest competitive advantage from it? I'll remind you again: PageRank and the real time Adwords auction were both hidden in plain sight. Understanding what data you have, and what meaning you can extract from it, is the holy grail of Web 2.0.

2. The era of IT as a back-office function is over. It's no longer good enough to gather data and analyze it, then propose and adjust strategies over the next budget cycle. You must infuse your organization with IT, so that, like Walmart, your supply chain responds every time a customer rings up an item at the cash register. This is how Walmart is like Google. No, not the website, but the live enterprise, which learns and responds.

   That's why in my enterprise 2.0 talks, I usually end by saying "turn your IT department inside out - or wait for some innovative startup to do it for you." Banks could be building something like Wesabe's Value Engine and tips feature, which extracts collective intelligence from credit card data; phone companies could be doing something like Skydeck's extraction of your social network from your phone bill. In fact, they'd be in a way better position to build integrated services against this data than startups that are having to first extract the data from corporate databases one customer at a time!

3. Web 2.0 thrives on network effects (also known as virtuous circles): data begetting more data, services getting better in such a way that they are used more often, until you are so far ahead of the next guy that he can't catch up. That network effect is enhanced by letting other people use and build on your data, not by keeping it private. What we've seen is that the first company to create network effects in a particular class of data tends to end up owning that data simply through having the biggest pile, or the best results, not because they have unique data. (Again, Google: Microsoft and Yahoo! have the same data for the most part; Google is better at creating value for others from it.)

I will note that Dell recently announced that they were abandoning their long-heralded build-to-order methodology in favor of standardized commodity boxes. But I still stand by my statement.

There are two possibilities: first, that Dell is wrong, and their new supply chain approach will not save them, just make them more like everyone else. It could be that their "live suppy chain" approach just got too crufty, too complex - the article linked above suggests more than 5000 possible configurations. Maybe what they needed to do was to make the system smarter again by streamlining and simplifying.

But it's possible too that the competitive advantage to be wrung from a live enterprise only takes you so far, and that in certain circumstances other advantages are more important. It may well be that the PC market has reduced itself to such commodity status that standardization trumps customization. It may well be that the costs of physical goods mean that the laws of virtual networks are only partly true in that realm.

I haven't studied Dell's situation and market sufficiently to have a fixed opinion. But the answer is knowable - a good field of study for business school cases. It's worth repeating something I once said about open source, but now for Web 2.0: This is science, not ideology. Our goal is to understand what works, and why.

P.S. I also talked about the ideas here in my piece from last year, What Would Google Do?