Four short links: 5 August 2009

Rebooting Britain, Revealing Errors, Reproducing Generators, Netflix Culture

by Nat Torkington | @gnat | comments: 0

1. Reboot Britain Video Archive -- video from the talks at Reboot Britain are online. The event also produced a essay set (PDF), CC-licensed. (via Paul Reynolds)
2. Revealing Errors -- Benjamin Mako Hill blog using computer errors as starting points for understanding how computers control the world around us. (via Dan Meyer)
3. New Microbe Strain Makes More Electricity, Faster -- University of Amherst researchers made current-generating bacteria work harder to live, and in five months had a strain that made an 8x larger current.
4. Netflix Culture -- readable slide deck which talks about the Netflix company culture. It's hard to read it and not nod in full agreement. (via joshua on Delicious)

Four short links: 29 July 2009

by Nat Torkington | @gnat | comments: 3

1. Bioweathermap -- crowdsourcing the gathering of environmental samples for DNA sequencing to study the changing distribution of microbial life. Another George Church project. (via timoreilly at Twitter)
2. We Are All African Now -- a great article about our genetic history and the computational genomics that makes it possible. (via Tim Bray)
3. Standing Out In The Crowd -- OSCON keynote by Kirrily Robert on women in open source. Excellent.
4. Energy Harvesting Powers Printed LED -- an interesting combination of two emerging technologies. Like an RFID, the circuit has a current induced by the presence of a changing RF field. The EL display and the RFID circuit are printed in organic compounds, whereas the power control is built with traditional circuit fabrication techniques. (via Freaklabs)

Four short links: 9 July 2009

by Nat Torkington | @gnat | comments: 1

1. Ten Rules That Govern Groups -- valuable lessons for all who would create or use social software, each backed up with pointers to the social science study about that lesson. Groups breed competition: While co-operation within group members is generally not so much of a problem, co-operation between groups can be hellish. People may be individually co-operative, but once put in a 'them-and-us' situation, rapidly become remarkably adversarial. (via Mind Hacks)
2. Yahoo! TrafficServer Proposal -- Yahoo! want to open source their TrafficServer product, an HTTP/1.1 caching proxy server. Alpha geeks who worked with it are excited at the prospect. It has a plugin architecture that means it can cache NNTP, RTSP, and other non-HTTP protocols.
3. App Engine Conclusions -- I've reluctantly concluded that I don't like it. I want to like it, since it's a great poster child for Python. And there are some bright spots, like the dirt-simple integration with google accounts. But it's so very very primitive in so many ways. Not just the missing features, or the "you can use any web framework you like, as long as it's django" attitude, but primarily a lot of the existing API is just so very primitive.
4. Microsoft Hohm -- Sign up with Hohm and we'll provide you with a home energy report and energy-saving recommendations tailored to your home. Wesabe for power at the moment, with interesting possibilities ahead should Microsoft partner with smartmetering utility companies the way Google Powermeter does. This is notable because this is a web app launched by Microsoft, with no connection to Windows or other Microsoft properties beyond requiring a "Live ID" to login. For commentary, see Microsoft Hohm Gets Green Light for Launch and PC Mag. (via Freaklabs)

Four short links: 19 June 2009

Cute Math, Fast Slo-Mo, Open Source HVAC, xkcd Hack

by Nat Torkington | @gnat | comments: 1

1. Inside-Out Multiplication Table -- very cool way to view the patterns of factors. Math is beauty with subscripts.
2. High-Speed Camera -- capture 100 frames at up to 1M frames/second. The sample videos, of a bullet liquefying on impact and a shotgun string boiling past, are stunning. The Makezine high-speed photography kit is the cheap amateur version.
3. Open Source Energy Management for Commercial Buildings -- open source project to enable interoperable applications for integrated Building Automation Systems (BAS). From NovusEdge. I wonder how they're planning to spread their open source and use it to disrupt. (via earth2tech and timoreilly on Twitter)
4. xkcd Knapsack Solution -- for those of you who like literal Python geeking with your comics. Have a great weekend!

Four short links: 17 June 2009

Word Mining, Open Ideas, Power Meter BotNet, and Realtime Web Traffic Tracking

by Nat Torkington | @gnat | comments: 0

1. NY Times Mines Its Data To Identify Words That Readers Find Abstruse -- the feature that lets you highlight a word on a NY Times web page and get more information about it is something that irritates me. I'm fascinated by the analysis of their data: boggling that sumptuary is less perplexing than solipsistic. Louche (#3 on the list) has been my favourite word for two years, by the way, since I heard Dylan Moran toss it out in that uniquely facile way the Irish have with words. I think Irish citizens get this incredible competence with the English language for free, along with staggering house prices and beer you can walk on.
2. Open Ideas -- Alex Payne's blog of Concepts in the public domain, awaiting collaboration and appropriation.
3. Buggy 'smart meters' open door to power-grid botnet (The Register) -- Paul Graham said that we've found what we get when we cross a television with a computer: a computer. Similarly, intelligent power meters are computers, computers that apparently haven't been well-secured. To prove his point, Davis and his IOActive colleagues designed a worm that self-propagates across a large number of one manufacturer's smart meter. Once infected, the device is under the control of the malware developers in much the way infected PCs are under the spell of bot herders. Attackers can then send instructions that cause its software to turn power on or off and reveal power usage or sensitive system configuration settings.
4. Chartbeat -- the sexiest web analytics ever. It gives realtime count of users, whether they're reading or writing (based on whether focus is in a form element), where they're from, mentions on Twitter, and more and more and more. This is a different form of analytics than Google Analytics, which tells you trends and historical access. Love this for the pure sex appeal of a heads-up dashboard that can tell you exactly how many people are on your site and exactly what they're doing. (via Artur)

Four short links: 16 June 2009

by Nat Torkington | @gnat | comments: 5

1. Dealing with Election Results Data -- taking the raw UK European election data into Google's Fusion Tables to try and make sense of it. More cloud-based tools for the data scientist within. (via Simon Willison)
2. Time for an Open 311 API -- "311" is the US number to call for non-emergency municipal services. There have been a lot of individual projects to hack together web sites that provide the single coherent view of government services that the government itself is unable to offer, but the individual projects have all built their own APIs. SeeClickFix suggest these be unified so tools can be written (e.g., iPhone apps) that run across multiple municipalities. (via timoreilly on Twitter)
3. Shoppers Cars Soon Able to Power Supermarkets (Daily Mail) -- At the Sainsbury's store in Gloucester, kinetic plates, which were embedded in the road yesterday, are pushed down every time a vehicle passes over them. A pumping action is then initiated through a series of hydraulic pipes that drive a generator. The plates are able to produce 30kw of green energy an hour - more than enough to power the store's checkouts. (via Freaklabs)
4. Humans Prefer Cockiness to Expertise (New Scientist) -- the blogosphere explained in one paper. (via Mind Hacks)

Four short links: 6 May 2009

Hamster Maps, Open Flu Data, Smart Grid Dollars, and Remixable Remix

by Nat Torkington | @gnat | comments: 1

1. Hamster Wheel Maps -- Jack Schulze has created an interesting way to see the world, in the form of "horizonless maps". The city unfolds in front of you like it was built on the inside of a hamster wheel and you're the hamster. Wired UK shipped an enormous foldout version.
2. Why Pig Flu is Better Than Bird Flu: Open Data (Glynn Moody) -- Glynn points to GISAID (Global Initiative on Sharing Avian Influenza Data), a system set up in 2006 because scientists were finding it hard to get timely H5N1 data. Following the correspondence letter in Nature, we have all pledged to share the data, to analyze the findings jointly, and to publish the results collaboratively, on the basis of open sharing of data respecting the rights and interests of all involved parties. This system has been used in the Mexican H1N1 outbreak.
3. IBM Plays Sugar Daddy to Smart Grid (CleanTech) -- IBM said it's making $2 billion available to jump-start IT projects, including the smart grid, because of the continued difficulty for partners to get project financing. The $2 billion would come from the company's lending and leasing arm, IBM Global Financing, in the form of low-rate loans, deferred payments, and other forms of project financing. The money is tied to projects authorized under the U.S. stimulus plan, which set aside $4.5 billion for smart grid projects. (via Freaklabs)
4. Lessig's "Remix" Book Now ccFree -- the latest book by Larry Lessig is now available under a CC-BY-NC license. (via Lessig blog)

Four short links: 1 May 2009

Smart Grids, Open Source, Stuff That Matters, and Global Culture

by Nat Torkington | @gnat | comments: 2

1. A Little Give and Take On Electricity (NY Times) -- Dennis L. Arfmann, a lawyer at the Boulder office of Hogan & Hartson who specializes in environmental law, said he had no idea how much electricity he and his wife, Dr. Julie Brown, had used before he filled his roof with solar panels producing 4.5 kilowatts of power. During the day he sells power to Xcel and at night he buys it back; his goal is to cut his use so his net sales rise. All hardware networked, everywhere!
2. Open Source World Map (Red Hat) -- very nice map showing the intensity of open source use in countries around the world. (via Flowing Data)
3. Imagine Cup -- Microsoft's contest to get students working on stuff that matters. The winners of the New Zealand leg, Team Think, tackled literacy: they devised a program for tablets that provides both handwriting recognition and audio output, eliminating the need for basic literacy to understand lessons or instructions. They hope to take this prototype to developing countries that have underutilised computers due to literacy issues. (via Idealog newsletter and Scoop)
4. UGT -- It is always morning when person comes into a channel, and it is always late night when person leaves. [...] The idea behind establishing this convention was to eliminate noise generated almost every time someone comes in and greets using some form of day-time based greeting, and then channel members on the other side of the globe start pointing out that it's different time of the day for them. Now, instead of spending time figuring out what time of day is it for every member of the channel, we spend time explaining newcomers benefits of UGT. (via migurski on delicious).

Four short links: 27 Apr 2009

Data centers, open research, Jeopardy!, and tombstones

by Nat Torkington | @gnat | comments: 5

1. Google Server and Data Center Details -- Greg Linden reports on a Efficient Data Center Summit. Google uses single volt power and on-board uninterruptible power supply to raise efficiency at the motherboard from the norm of 65-85% to 99.99%. There is a picture of the board on slide 17. (and this is a 2005 board). Greg has left Microsoft as Live Labs is dissolved.
2. The Economics of Open Access Publishing -- set of papers on the free distribution of research. Pointed to by the RePEc blog. RePEc is Research Papers in Economics, a collaborative effort of hundreds of volunteers in 67 countries to enhance the dissemination of research in economics. The heart of the project is a decentralized database of working papers, journal articles and software components. All RePEc material is freely available. (via Paul Reynolds)
3. Computer Program to Take On Jeopardy! (NY Times) -- move over Turing Test, IBM's working on the Trebek Test: a computer program to compete against human “Jeopardy!” contestants. If the program beats the humans, the field of artificial intelligence will have made a leap forward. Really? The system must be able to deal with analogies, puns, double entendres and relationships like size and location, all at lightning speed. Oh, ok. So it's more complex than inverting the hash table of questions and answers. (via ericries on Twitter)
4. The Value of Minimal Data (Powerhouse Museum) -- if you have the ability for passionate users to contribute their knowledge, they can turn "minimal" data into a delicious four course data feast with a vintage port to sip during the dessert course. (via sebchan on Twitter)

Four short links: 8 Apr 2009

by Nat Torkington | @gnat | comments: 2

Bias, RFCs, virus batteries, and a glimpse at life beyond record labels (the last item features profanity, beware):

1. Bias We Can Believe In (Mind Hacks) -- Vaughn asks the tricky question about the current enthusiasm for Behavioural Economics in government: where are the sceptical voices? As he points out, It's perhaps no accident that almost all the articles cite a 2001 study that found that simply making the US's 401(k) retirement savings scheme opt-out instead of opt-in vastly increased participation simply because it's a hassle to change and employees perceive the 'default' as investment advice. But it's probably true to say that this example has been so widely repeated but it's one of the minority of behavioural economics studies that have looked at the relation between the existence of a cognitive bias and real-world economic data from the population. And it's notable that behavioural economists who specialise in making this link, a field they call behavioural macroeconomics, seem absent from the Obama inner circle.
2. How The Internet Got Its Rules (NYTimes) -- about the first RFCs, which became IETF. The early R.F.C.’s ranged from grand visions to mundane details, although the latter quickly became the most common. Less important than the content of those first documents was that they were available free of charge and anyone could write one. Instead of authority-based decision-making, we relied on a process we called “rough consensus and running code.” Everyone was welcome to propose ideas, and if enough people liked it and used it, the design became a standard. (via Glynn Moody)
3. Viruses Could Power Devices (Science News) -- Ions and electrons can move through smaller particles more quickly. But fabricating nano-sized particles of iron phosphate is a difficult and expensive process, the researchers say. So Belcher’s team let the virus do the work. By manipulating a gene of the M13 virus to make the viruses coat themselves in iron phosphate, the researchers created very small iron phosphate particles. (via BoingBoing)
4. Amanda Palmer's Label-Dropping Game -- interesting email from Amanda Palmer to her fans about trying to get dropped from her label. i had to EXPLAIN to the so-called "head of digital media" of roadrunner australia WHAT TWITTER WAS. and his brush-off that "it hasn’t caught on here yet" was ABSURD because the next day i twittered that i was doing an impromptu gathering in a public park and 12 hours later, 150 underage fans - who couldn’t attend the show - showed up to get their records signed. no manager knew! i didn’t even warn or tell her! no agents! no security! no venue! we were in a fucking public park! life is becoming awesome. and then the times they are a-changing fucking dramatically, when pong-twittering with trent reznor means way more to your fan-base/business than whether or not the record is in fucking stores (and in my case, it ain’t in fucking stores).

Four short links: 20 Feb 2009

by Nat Torkington | @gnat | comments: 6

Accessibility, trails, Pacman, and power today. Have a fun weekend!

1. Social Accessibility Project -- clever IBM approach to solving web accessibility problems: a sidebar for Firefox that lets people with assistive devices like screenreaders say "hey, I had this problem with this page", and a crowd will help fix it. (via Derek Featherstone's Webstock talk, notes here)
2. Why I Want a Million Quid (mySociety) -- Tom's onto something. I am hooked by this vision of "systems where each person who is helped to solve a problem leaves a trail of advice, contacts, insider information and new user-friendly web services behind them". We're used to the data people leave behind being discrete and implicit (another purchase for the recommendation engine) rather than longitudinal and explicit (people who looked at this item eventually went on to find their answer here).
3. The PacMan Dossier -- everything there is to know about Pacman, from designer Toru Iwatani's inspiration and design process, through to the logic errors behind bugs and why it's better to move the joystick before you reach the turn. (via Grand Text Auto)
4. Two Stanford Students Rethink the Light Switch -- a power switch with a network connection and tactile feedback: teh awesome.

Google's PowerMeter. It's Cool, but don't Bogart My Meter Data

by Jim Stogdill | @jstogdill | comments: 17

Last week I read this piece in the New York Times about Google's PowerMeter, their entry into the smart meter game. The story was picked up in quite a few places but neither the NYT piece or related articles from other outlets expanded much on Google's underlying press release. Google's FAQ isn't very satisfying either; it has no depth so I didn't really know what to make of it. When I finished reading it I was left with an inchoate unsettled feeling and then I forgot about it. But on Friday evening I had a random conversation about it with a colleague who works in the meter data management (MDM) space. By the time we were through talking about what Google might be doing I had arrived at a position of love / hate. I'll explain the love first.

In terms of the attention this brings to energy consumption at the household level, I really love what Google is doing with this initiative. As they put it:

"But smart meters need to be coupled with a strategy to provide customers with easy access to near real-time data on their energy usage. We're working on a prototype product that would give people this information in an iGoogle gadget."

I agree completely. It's not exactly the same thing, but I've been amazed by how much my behavior behind the wheel changed once I started leaving average mpg permanently visible on my car's dashboard display. In short order I went from speed racer wannabe to one of those guys that gets harassed by co-workers for driving too slow. "Hey, can you hypermile on the way back from lunch? I'm starving."

While I am not sure that a gadget on the web will have the same right-there-in-front-of-my-eyes impact that my car's LCD display has, I'm convinced that Google has hit on something important. After all, today most of us have no idea how many kilowatts we use, what we use them for, or how much we're paying per kilowatt. We use power in our homes the way I used to drive my car.

Unfortunately, Google's FAQ doesn't really answer any questions about how the service works. But from statements like "Google is counting on others to build devices to feed data into PowerMeter technology" we can deduce that Google is proposing to correlate the total power reported by your smart meter with the data collected from individual loads inside the home. This is really cool, because not only does it make the information more generally accessible to you (in an easily accessible gadget), it proposes to tell you what it is in your house that is using that power, and when.

Google can do this because many national and state governments have begun to mandate smart meter programs. Most of us will probably have one on the side of our house pretty soon (especially if the stimulus bill speeds things up). Smart meters improve on their predecessors by automating meter reading, reporting consumption in intervals (typically 15 minutes), and they can send "last gasp" failure notifications in the event of power outages.

But, just like their dumb ancestors, they will be owned by the utility. This means that the data generated will ultimately be under control of the utility and hosted in their systems. The meter will talk to a utility data collector and from there its data will enter the utility's MDM system. The MDM will do a bunch of stuff with the data. However, from the point of view of you, the consumer, it will primarily send it to the billing system which will now be able to account for time of day pricing. Also, it will send those last gasp signals to the outage management system so that outage reporting will be automatic. This will make analysis and response faster and more accurate. Google appears to be leveraging their position and market power to make deals with the utilities to access that data on our behalf.

The biggest reason for smart meter initiatives is demand management. The utilities have to carry expensive excess capacity so that they can meet peak loads. If they can use interval metering coupled with better pricing and feedback systems, they may be able to change our usage patterns and smooth that load which will reduce the necessary peak capacity overhang. Also, as alternative energy sources with less predictable availability like wind power come on line the utilities will need more "load shaping" options. Ultimately they might be able to reach directly out to your smart appliances and turn them off remotely if they need to.

The laws that are mandating smart metering are focused on this demand side management. Practically speaking, most utilities will close the consumer feedback loop by offering a simple portal on the utility's web site that will let you monitor your usage in the context of your bill. However, this isn't the part of the system the utilities are excited about. The hardware and the meters are the sexy part. The contracts to build the consumer portals are probably going to go to low cost bidders who will build them exactly to low band pass requirements. In some cases they may provide provisions for customers to download historical data into a spreadsheet if they want to. A few enterprising customers will probably take advantage of this feature, but this is the hard way to do the kinds of correlations Google has in mind.

What should be apparant by now, is that the government is mandating a good idea, but they are mandating it from a utilty-centric rather than customer-centric point of view. There is naturally some overlap between utility and customer interests, but they are not identical. The utility is concerned about managing capital costs. They look at the interval data and the customer portal as a way to influence your time-of-use behaviors. They really don't care how much power you use, they just don't want your demand to be lumpy. On the other hand, we just want our bills to be low.

So, Google's initiative offers to take your data from the utility, combine it with data coming from devices in your home, and visualize it much more you-centrically. There offering will do a better job than the utility's portal illuminating structural efficiency problems in the home as well as usage pattern problems once utilities start implementing variable pricing. In short, while the utility is attempting to influence your "when I use it" decision making, Google is offering to help you make better "what I plug in" decisions along with the stuff the utility cares about.

So, what's not to like?

Google needs two distinct sources of data to make this initiative work. They need access to your data via the utility that owns your smart meter. Plus they need data from equipment manufacturers that are going to make your appliances smart or provide your home automation gadgets. It doesn't bother me at all that they get this data, as long as the utility makes it available for anyone else that might be able to innovate with it too, including me. You never know, I might want to use it for a home made gadget that sets up an electric shock on my thermostat any time my last eight averaged readings are above some arbitrary threshold, you know, just to make me think twice before turning it up.

The little bit of info that Google provides on this initiative is at their .org domain, but there is virtually no information about how to participate in data standards making, API specification, device development, or that kind of thing. If you want to participate, you pick whether you are a utility, device manufacturer, or government, fill out a form and wait for Google to get back to you. Imagine, the government fills out a form to participate in Google's initiative. Google has out governmented the government.

As I described already, governments are insisting on demand side management, but there don't appear to be any requirements to provide generic API's for meter readings or meter events. It's enterprise thinking rather than web platform thinking and we run the risk of your data being treated like utility "content." "In other news today HBO struck an exclusive deal with XYZ electric for all of their meter content, meanwhile Cinemax surprised industry watchers by locking up ABC Electric. As was reported last night, all of the remaining utility's signed with Google last week."

I'm guessing that Google is probably following the same pattern that they are using in the transit space and making (exclusive?) deals with the utilities to consume your data. You'll have to log into the utilty portal to approve their access (or check a box on your bill). But Google, or other big players that can afford to buy in, will probably be the only choice(s) you have. There is no evidence on Google.org that they are trying to create an eco-system or generalized approach that would let you, the owner of the data, share it with other value added service providers. If the utilities implement this under government mandate it will suck. If they install smart meters with stimulus package money and still don't provide eco-system API's it will worse than suck.

Any thoughts on how this plays out on the smart appliance / home automation side? Are there healthy open standards developing or is there danger of large scale exclusivity on that side of the equation too?

Google will be more innovative with this data than the electric utilities, I have no doubt about that. But I can easily imagine other companies doing interesing innovating things with my meter data as well. Especially as Google achieves utility scale themselves. If my electric utility is going to create a mechanism to share my data with companies like Google, I want them to make a generalized set of API's that will let me share it with anyone.

A quick note to policy makers in states who haven't yet finalized their programs. When you think about what to mandate, consider a more consumer-centric model (if it's easier, think of it as a voter-centric model). You should be shooting for a highly innovative and generative space where contributions and innovations can come from large and small firms alike, and where no one should be structurally locked out from participation. Don't lock us into a techno-oligarchy where two or three giant firms own our data and the possibility of innovation. If you insist on widely implemented consumer controlled API's and a less enterprise-centric model, you will not only encourage broader innovation at the consumer end, but you can use it to enhance competition on the generation side too.

Well, Google isn't really saying what they are doing, so maybe I got it wrong. Maybe they are about to get all "spectrum should be free" and roll out all kinds of draft API's specifications for comment. If you think I got it wrong, don't hesitate to let me know in the comments.

Update (2/17): Asa pointed out in the comments that Google does provide more about their intent in their comments to the California Public Utilities Commission. I missed that link before and it gives some useful hints.

Most interesting is the repeated reference to Home Area Networks (HAN). In the original post I assumed Google was taking current smart meters as a given and obtaining data from the utility MDM after it went through their data collectors. That looks like it was incorrect. Instead Google probably wants your meter to to talk to your HAN via wireless(?) and then on to them from there.

If Google can use their market position to make that data accessible off the HAN rather then from the utility MDM I think that's a good thing. Mostly because it makes possible the direct consumption and analysis of the data on my side of my home network's NAT / firewall. I didn't really touch on privacy considerations in the original post, but given that PowerMeter appears trivial from a computational point of view, I'd much rather run it locally rather than share my every light switch click with Google. If I want to know how I'm doing relative to peers I can share that data then, in appropriately summarized form.

The other point in the CPUC comments is this statement: "PowerMeter... we plan to release the technical specifications (application programming interfaces or API) so anyone can build applications from it."

This is great, but I would love to see the API's sooner rather than later. They aren't really PowerMeter API's after all, if I'm reading the situation correctly, these are proposed API's and data specifications for smart meters and smart devices. The API's that Google (and others) will be consuming, not the ones they are offering. If a whole ecosystem is going to be enabled through those API's, then the ecosystem should have a hand in developing them.

In summary, if Google manages to create a level playing field for the development of an ecosystem based on this data, I'll applaud them. Some people will use their service and, like they do with other Google services, trade privacy for targeted ads. Others will choose other approaches to using the data that provide those functions without exporting as much (or any) data.

Four short links: 21 Jan 2009

by Nat Torkington | @gnat | comments: 2

In today's edition: the spread of fake news, keeping track of your real power use, a Javascript library and a less-than-impressed take on mobile location apps.

1. Echo Chamber - the British tabloid The Sun posted a story that turned out to be fabricated. This site tracked that story's spread and uncritical acceptance by other news outlets and web sites.
2. Real Time Web-Based Power Charting - build the software and hardware to get a live chart in a web page that updates every 10 seconds with the instantaneous power usage for your entire house.
3. ActiveRecordJS - just what it sounds like, ActiveRecord for Javascript. AR is a complex subsystem of Rails, and it's interesting to see the functionality ported to Javascript.
4. I Am Here: One Man's Experiment with the Location-Aware Lifestyle - a reporter tries all the location apps, and discovers the future isn't all here yet. Interesting: only three paragraphs of this long story are about the good bits of location services, the rest question its implementation, privacy, and utility.

O'Reilly AlphaTech Ventures Invests in Amee

by Tim O'Reilly | @timoreilly | comments: 6

I'm pleased to announce that on Wednesday, O'Reilly AlphaTech Ventures, our VC affiliate, closed an investment in UK-based Amee, which bills itself as "the world's energy meter." Here's their description of what they do:

AMEE’s aim is to map, measure and track all the energy data on Earth. This includes aggregating every emission factor and methodology related to CO2 and Energy Assessments (individuals, businesses, buildings, products, supply chains, countries, etc.), and all the consumption data (fuel, water, waste, quantitative and qualitative factors).

It is a web-service (API) that combines measurement, calculation, profiling and transactional systems. Its algorithmic engine applies conversion factors from energy into CO2 emissions, and represents data from 150 countries.

AMEE aids the development of businesses and other initiatives - by providing common benchmarks for measurement, tracking, conversion, collaboration and reporting.

If you've been following my talks in which I urge software developers and entrepreneurs to "work on stuff that matters," you know that I consider getting a handle on carbon accounting is the first step in putting a stop to global warming. (If you're a warming skeptic, I consider global warming as a modern example of Pascal's wager: if we're wrong, and global warming is not human caused, the steps we'll take to address it are still worthwhile. We get off foreign oil, improve our energy security, build new industries, improve the environment.)

Even apart from the contribution to a critical world issue, Amee is interesting because it shows that the future of web services will involve a much broader range of data services than most people imagine. I've long argued that the subsystems of the emerging internet operating system are data subsystems. Some of those, like location and identity, are obvious, and thus hotly contested. Others, like carbon data, are sorely needed, and not yet built out. There's huge opportunity in finding and populating key databases, and then turning them into ubiquitous web services.

By the way, if you use dopplr, you've already seen Amee at work: it provides the data for dopplr's carbon calculator tab.

Union Square Ventures is also an investor in this round. Partner Albert Wenger gives his take on the investment on their blog.

Data Center Power Efficiency

by Jesse Robbins | @jesserobbins | comments: 8

James Hamilton is one of the smartest and most accomplished engineers I know. He now leads Microsoft's Data Center Futures Team, and has been pushing the opportunities in data center efficiency and internet scale services both inside & outside Microsoft. His most recent post explores misconceptions about the Cost of Power in Large-Scale Data Centers:

I’m not sure how many times I’ve read or been told that power is the number one cost in a modern mega-data center, but it has been a frequent refrain. And, like many stories that get told and retold, there is an element of truth to the it. Power is absolutely the fastest growing operational costs of a high-scale service. Except for server hardware costs, power and costs functionally related to power usually do dominate.

However, it turns out that power alone itself isn’t anywhere close to the most significant a cost. Let’s look at this more deeply. If you amortize power distribution and cooling systems infrastructure over 15 years and amortize server costs over 3 years, you can get a fair comparative picture of how server costs compare to infrastructure (power distribution and cooling). But how to compare the capital costs of server, and power and cooling infrastructure with that monthly bill for power?

The approach I took is to convert everything into a monthly charge. [...]

DIY Appliances on the Web?

by Jim Stogdill | @jstogdill | comments: 9

Or, My Enterprise is Appliancized, Why Isn't Your Web?

I wrote a couple of posts a while back that covered task-optimized hardware. This one was about a system that combined Field Programmable Gate Arrays (FPGA's) with a commodity CPU platform to provide the sheer number crunching performance needed to break GSM encryption. This one looked at using task-appropriate efficient processors to reduce power consumption in a weather predicting super computer. In these two posts I sort of accidentally highlighted two of the three key selling points of task-specific appliances, sheer performance and energy efficiency (the third is security). The posts also heightened my awareness of the possibilities for specialized hardware and some of my more recent explorations that focused on the appliance market in particular got me wondering if there might be a growing trend toward specialized appliances.

Of course, specialized devices have been working their way into the enterprise ever since the first router left its commodity Unix host for the task-specific richness of specialized hardware. Load balancers followed soon after and then devices from companies like Layer 7 and Data Power (now IBM) took the next logical step and pushed the appliance up the stack to XML processing. These appliances aren't just conveniently packaging intellectual property inside commodity 1U blister packs, they are specialized devices that process XML on purpose-built Application Specific Integrated Circuits (ASICS), accelerate encryption / decryption in hardware, and encapsulate most of an ESB inside a single tamper proof box whose entire OS is in firmware. They are fast, use a lot less power than an equivalent set of commodity boxes, and are secure.

Specialization is also showing up in the realm of the commodity database management systems. At last year's Money:Tech Michael Stonebraker described a column-oriented database designed to speed access to pricing history for back testing and other financial applications. In this case the database is still implemented on commodity hardware. However, I think it's interesting in the context of this conversation on specialized computing because it speaks to the inadequacy of commodity solutions for highly specific requirements.

A device from Netezza is also targeted at the shortcomings of the commodity DBMS. In this case the focus is on data warehousing, but it takes the concept further with an aggressive hardware design that is delivered as an appliance. It has PostgreSQL at its core but it takes the rather radical step of coupling FPGA's directly to the storage devices. The result, for at least a certain class of query, is a multiple order of magnitude boost in performance. I think this device is noteworthy because it puts the appliance way up the stack and is perhaps a harbinger for further penetration of the appliance into application-layer territory.

While appliances are expanding their footprint in the enterprise, it seems like the exact opposite might be happening on the web? Maybe the idea of a closed appliance is anathema to the open source zeitgeist of the web, but in any case, the LAMP stack is still king. Even traditional appliance-like tasks such as load balancing seem to be trending toward open source software on commodity hardware (e.g. Perlbal).

I can't help but wonder though, at the sheer scale that some web properties operate (and at the scale of the energy cost required to power them), can the performance and cost efficiency of specialized hardware appliances be ignored? Might there be a way to get the benefits of the appliance that is in keeping with the open source ethos of the web?

If you've ever uploaded a video to Youtube and waited for it to be processed you have an idea of how processor hungry video processing is on commodity hardware. I don't know what Google's hardware and energy costs are for that task but they must be significant. Same goes for Flickr's image processing server farm and I would guess for Google's voice processing now that its new speech services have launched. If the combination hardware and electricity costs are high enough, maybe this is a good place to introduce specialized appliances to the web?

But how to do that in a way that is consistent with the prevailing open source ethos and that still lets a firm continue to innovate? I think an answer might be sort of DIY writ large; a confluence of open source and open hardware that works like an undocumented joint venture based on the right to fork. Think Yahoo and the Hadoop community or JP Morgan and friends with AMQP but with hardware and you get the idea. Such a community could collaborate on the design of the ASICS and the appliance(s) that hosted them and even coordinate production runs in order to manage unit costs. Perhaps more importantly, specifying the components openly would serve cost sharing across these companies while still supporting flexibility in how they were deployed and ultimately, generativity and innovation for future uses.

There are probably a bunch of reasons why this is just silly speculation, but Google's efforts with power supply efficiency might be seen as at least a bit of precedent for web firms dabbling in hardware and hardware specifications. In fact, Google's entire stack, from it's unique approach to commodity hardware, to software infrastructure like GFS, might be thought of as a specialized appliance that suits the specific needs of search. It's just a really really big one that "ships" in a hundred thousand square foot data center form factor.

Radar Theme: Clean Energy Tech

by Nat Torkington | @gnat | comments: 1

[This is part of a series of posts that briefly describe the trends that we're currently tracking here at O'Reilly]

All civilization depends on energy, and always has done so. Oil is rising rapidly in price and alternative energy and energy consumption management have become viable businesses. We're interested in the IT use of energy technology (green data centers) and the energy industry's use of IT (smart monitors, intelligent grids, data center placement).

Watchlist: IBM/Sun/Google, Amazon/Microsoft, Wattzon, Vinod Khosla.

Energy Savings, Strange Attractors, ...

by Jim Stogdill | @jstogdill | comments: 4

... the Intrinsic Cost of State Change, Orbiting Alien Voyeurs, and 200 Square Kilometers of Solar Panels Somewhere in Texas

The Silicon Valley Leadership Group and Berkeley National Labs recently published the results of their first Data Center Demonstration Project (pdf). (Disclosure: My colleague Teresa Tung of Accenture R+D labs was the report's principal author). The study follows up on last year's publication of the EPA's report to Congress (pdf) on data center energy consumption. That report, among other things, estimated the range of savings that data center operators could achieve with varying degrees of technology and practice improvement. This more recent report is based on real world studies and was intended to validate the estimates in the EPA report.

Both reports are good reads if you are interested in reducing the megawatts being consumed in your organization's silicon (though the EPA report has been criticized as being a bit toothless). However, I should warn you that they are fairly long and detailed so the bedside table might not be the best home for them if you want to get through them, at least until the manga versions are released.

The EPA study estimated that "state of the art" technology and processes in the data center might cut energy usage by 55%, the more readily achievable "best practices" come in at 45% savings. State of the art includes a range of approaches including better server utilization through virtualization, better cooling techniques, improved power distribution, sensor networks, etc.

The more recent study, testing those techniques in working data centers, validates the EPA's estimates but also offers the initially surprising conclusion that legacy data centers can be retrofitted to achieve efficiencies close to that of new builds. That conclusion follows from the less surprising finding that the most bang for the buck comes from improvements on the "IT" side of the energy draw (energy efficient servers, virtualization, etc.) rather than from the harder to retrofit "site" side (cooling systems etc.). The dog wags the tail after all and if you can reduce the direct power consumption by the IT equipment, you will simultaneously reduce associated cooling costs whether in an old building with relatively inefficient HVAC or a shiny new one.

The last finding that I'll mention here is that it doesn't look like the time is right yet for widespread adoption of more advanced load management techniques outside of niche applications. The demonstration project had facilities that experimented with them, but the risk aversion that stems from high reliability requirements in production data centers has these experiments mostly restricted to centers that serve R+D rather than production functions.

Maybe one of the most interesting things about the report is what it doesn't (can't) say.

(continue reading)

Bill Coleman to keynote Velocity

by Jesse Robbins | @jesserobbins | comments: 0

Bill Coleman has twice transformed our industry, and I'm excited to announce that he will keynote Velocity later this month. Bill is most famous for being the "B" in BEA and for leading the creation of Solaris while at Sun. He is now the CEO of his new startup, Cassatt, which "makes Data Centers more efficient".

Bill is awesome and I'm really looking forward to his keynote. He is changing the way we think about and manage Data Centers and the software that runs within them.

When we spoke earlier this week he explained how vacuum tubes created the fear of powering down servers, and how funny it is that that fear persists with people that have never seen them. (I've never made that connection as I'm "part of the problem" ;-)

At Velocity, Bill will likely talk about virtualization & efficiency, where he thinks we're headed, and the questions we need to be asking now to get there.

(Many thanks to Tim for suggesting this to Bill and making the introduction.)

Africa's Energy Deficit: Energy Hacks Can Make A Difference

by Ben Lorica | @dliman | comments: 8

About six weeks ago I came across this quote from a Wall Street Journal article and I have been pondering it ever since:

Africa has the capacity to generate about 63 gigawatts of power for roughly 770 million people -- about what Spain produces for its population of 40 million. For most African countries, the World Bank estimates that universal access to electricity is at least 50 years away. But these days, even the few who have come to expect electricity are finding it increasingly difficult to come by -- or afford.

The article goes on to emphasize that the situation varies considerably from country to country, with internal conflicts and lack of investment among the primary reasons for low energy generation capacity in several countries. As India and China steadily modernize, we get constant reminders that they are going to be huge consumers of energy resources. The quote above is a reminder that large portions of Africa could eventually be major consumers of energy as well.

A reliable African energy infrastructure is necessary to attract large amounts of foreign direct investments. Electricity is vital in powering factories, offices, schools, hospitals, and other public services. While a modern infrastructure may be years away, there are “energy hacks” that can have tremendous impact over the short-term. One just needs to note the importance of cell phones in most African countries. Mobile phones have the added benefit that they can be easily powered using distributed renewable technologies.

At the recent Where 2.0 conference, Erik Hersman pointed to a paper he wrote that outlines the importance of cell phones in Africa (“Africa’s PC”) while highlighting software application areas (“search, news, community”) that could prove popular on the continent. Mobile phone initiatives similar to Grameen Phone’s Village Phone and Community Information Center could also be high-impact projects in Africa.

No doubt full-scale modernization requires the large energy infrastructure projects that take years to fund and build. Rather than get discouraged, I look to the work of people like Joel Selanikio (click here for details). Public health initiatives like Joel's have demonstrated that simple mobile devices go a long way towards improving the lives of the poorest people in the world.