Four short links: 23 July 2009

Wave Fed, Fake Steve, Vanish and Reconnoiter

1. Google Wave Federation Protocol -- the interesting part of Wave for me is the system for keeping databases coherent. There's a reference implementationl.
2. I shouldn't have yelled at that Chinese guy so much -- the post that redeemed Fake Steve Jobs in my eyes. We all know that there's no fucking way in the world we should have microwave ovens and refrigerators and TV sets and everything else at the prices we're paying for them. There's no way we get all this stuff and everything is done fair and square and everyone gets treated right. No way. And don't be confused -- what we're talking about here is our way of life. Our standard of living. You want to "fix things in China," well, it's gonna cost you. Because everything you own, it's all done on the backs of millions of poor people whose lives are so awful you can't even begin to imagine them, people who will do anything to get a life that is a tiny bit better than the shitty one they were born into, people who get exploited and treated like shit and, in the worst of all cases, pay with their lives.
3. Vanish -- time-limited encryption in a Firefox plugin.
4. Reconnoiter -- holy cow web console and analytics for data centers, from the magic Theo Schlossnagle. He built the screenshots for his OSCON presentation, graphing streams of live performance data from dozens of data centers, while on a Virgin America flight.

Four short links: 20 July 2009

1. Apple's iPhone Wrecking the Cell Industry -- bleat bleat. Andy Oram's comment hits the mark: The music companies and AT&T were like travelers who refused to believe they were taking a long trip. They didn't pack warm clothing, and therefore had to buy it at disadvantageous terms when they came to need it. Apple was more sophisticated about where all companies are going technologically, so they had what others needed.
2. Fruux -- a lightweight and convenient system preference pane, that syncs your Address Book, Calendars, Tasks and Bookmarks between different Macs. (via Daniel Raffel)
3. Redflax -- notable not just for art, but for the Maori quote: He toi whakaaro, he mana tangata - roughly translates: where there is creativity/artistic expression, there is human dignity/prowess.
4. Google's Chiller-less Data Center -- Belgium has only 7 days (on average) when the ambient air temperature isn't enough to cool the data center. Finally, a business model for unpleasantly-cold climates.

Four short links: 20 May 2009

Cognitive Surplus, Data Centers=Mainframes, Django Microframework, and a Visit To The Future

1. Distributed Proofreaders Celebrates 15000th Title Posted To Project Gutenberg -- a great use of our collective intelligence and cognitive surplus. If I say one more Clay Shirkyism, someone's gonna call BINGO. (via timoreilly on Twitter)
2. Datacenter is the New Mainframe (Greg Linden) -- wrapup of a Google paper that looks at datacenters in the terms of mainframes: time-sharing, scheduling, renting compute cycles, etc. I love the subtitle, "An Introduction to the Design of Warehouse-Scale Machines".
3. djng, a Django powered microframework -- update from Simon Willison about the new take on Django he's building. Microframeworks let you build an entire web application in a single file, usually with only one import statement. They are becoming increasingly popular for building small, self-contained applications that perform only one task—Service Oriented Architecture reborn as a combination of the Unix development philosophy and RESTful API design. I first saw this idea expressed in code by Anders Pearson and Ian Bicking back in 2005.
4. Cute! (Dan Meyer) -- photo from Dan Meyer's classroom showing normal highschool students doing something that I assumed only geeks at conferences did. I love living in the future for all the little surprises like this.

Energy Savings, Strange Attractors, ...

... the Intrinsic Cost of State Change, Orbiting Alien Voyeurs, and 200 Square Kilometers of Solar Panels Somewhere in Texas

The Silicon Valley Leadership Group and Berkeley National Labs recently published the results of their first Data Center Demonstration Project (pdf). (Disclosure: My colleague Teresa Tung of Accenture R+D labs was the report's principal author). The study follows up on last year's publication of the EPA's report to Congress (pdf) on data center energy consumption. That report, among other things, estimated the range of savings that data center operators could achieve with varying degrees of technology and practice improvement. This more recent report is based on real world studies and was intended to validate the estimates in the EPA report.

Both reports are good reads if you are interested in reducing the megawatts being consumed in your organization's silicon (though the EPA report has been criticized as being a bit toothless). However, I should warn you that they are fairly long and detailed so the bedside table might not be the best home for them if you want to get through them, at least until the manga versions are released.

The EPA study estimated that "state of the art" technology and processes in the data center might cut energy usage by 55%, the more readily achievable "best practices" come in at 45% savings. State of the art includes a range of approaches including better server utilization through virtualization, better cooling techniques, improved power distribution, sensor networks, etc.

The more recent study, testing those techniques in working data centers, validates the EPA's estimates but also offers the initially surprising conclusion that legacy data centers can be retrofitted to achieve efficiencies close to that of new builds. That conclusion follows from the less surprising finding that the most bang for the buck comes from improvements on the "IT" side of the energy draw (energy efficient servers, virtualization, etc.) rather than from the harder to retrofit "site" side (cooling systems etc.). The dog wags the tail after all and if you can reduce the direct power consumption by the IT equipment, you will simultaneously reduce associated cooling costs whether in an old building with relatively inefficient HVAC or a shiny new one.

The last finding that I'll mention here is that it doesn't look like the time is right yet for widespread adoption of more advanced load management techniques outside of niche applications. The demonstration project had facilities that experimented with them, but the risk aversion that stems from high reliability requirements in production data centers has these experiments mostly restricted to centers that serve R+D rather than production functions.

Maybe one of the most interesting things about the report is what it doesn't (can't) say.

(continue reading)

Bill Coleman to keynote Velocity

Bill Coleman has twice transformed our industry, and I'm excited to announce that he will keynote Velocity later this month. Bill is most famous for being the "B" in BEA and for leading the creation of Solaris while at Sun. He is now the CEO of his new startup, Cassatt, which "makes Data Centers more efficient".

Bill is awesome and I'm really looking forward to his keynote. He is changing the way we think about and manage Data Centers and the software that runs within them.

When we spoke earlier this week he explained how vacuum tubes created the fear of powering down servers, and how funny it is that that fear persists with people that have never seen them. (I've never made that connection as I'm "part of the problem" ;-)

At Velocity, Bill will likely talk about virtualization & efficiency, where he thinks we're headed, and the questions we need to be asking now to get there.

(Many thanks to Tim for suggesting this to Bill and making the introduction.)

Special Purpose Computing Focuses on Energy Efficiency

To improve the climate models that predict global warming, climatologists are seeking model resolutions on the order of 1 km. Unfortunately, building the required 200 petaflop machine with today's commodity-hardware approach would cost $1B and would result in a staggering 40 megawatts of power consumption.

A group of researchers at Lawrence Berkeley National Labs, who must be aware of the irony inherent in using 40 continuous megawatts to better predict global warming, may be returning supercomputing to its specialized roots but along a new vector (yes, weak pun intended). In addition to the Cray-era focus on raw power, they are emphasizing energy efficient computation where floating point operations per watt is the key metric.

Their approach has already been described at EcoTech Daily and the lab's Research News so I'm just going to summarize it here. They are working on specialized hardware consisting of 20 million very low power embedded processors (of the sort used in iPods and cell phones) wired together with the specific climate calculations in mind. By trading flexibility for efficiency, the design should achieve a ten-fold improvement in the floating point operations per watt metric and the resulting 200 petaflop machine is predicted to require only 4MW of power and cost $75M to build.

Their motivations in their own words:

"What we have demonstrated is that in the exascale computing regime, it makes more sense to target machine design for specific applications," Wehner said. "It will be impractical from a cost and power perspective to build general-purpose machines like today's supercomputers."

Specialized problems are amenable to specialized solutions and scientific computation seems particularly suitable to this kind of approach. However, on the web and in corporate IT where computing is both more general and inefficiently deployed, the first wave of energy efficiency improvements are being addressed primarily through a combination of virtualization and incremental improvements in commodity chip design.

I don't think software carpooling will be the only game in town for long though. While virtualization and dynamic provisioning are facilitating better utilization of existing hardware, virtualization comes with a performance cost of its own and can be no better than the hardware it is running on. Once you get four passengers in a V-8 powered SUV further improvements have to come from changing driving habits and modifying the vehicle.

As virtualization initiatives pick the low hanging fruit, further gains will come from fundamental hardware improvements (which may include analogous specialization) in concert with "best efficiency" dispatching that targets optimal server utilization in a dynamic server pool. An interesting example of this kind of approach is described here (pdf).

As I've touched on before, beyond that, a "systems view" to optimize the whole data center as it operates under changing conditions and with heterogeneous hardware might come next. Returning one last time to the carpooling analogy, this would be like a smart traffic routing system that keeps each car-pooling hybrid moving at its most efficient speed. The end result might be an optimally-sized mixed pool of specialized and commodity hardware each dispatched to operate the data center holistically at its best unit of work per watt.