Four short links: 29 July 2009

1. Bioweathermap -- crowdsourcing the gathering of environmental samples for DNA sequencing to study the changing distribution of microbial life. Another George Church project. (via timoreilly at Twitter)
2. We Are All African Now -- a great article about our genetic history and the computational genomics that makes it possible. (via Tim Bray)
3. Standing Out In The Crowd -- OSCON keynote by Kirrily Robert on women in open source. Excellent.
4. Energy Harvesting Powers Printed LED -- an interesting combination of two emerging technologies. Like an RFID, the circuit has a current induced by the presence of a changing RF field. The EL display and the RFID circuit are printed in organic compounds, whereas the power control is built with traditional circuit fabrication techniques. (via Freaklabs)

Four short links: 27 July 2009

1. Ignite OSCON -- 56m of video from Ignite OSCON. They're all great, but Dan Meyer remains the highlight for me.
2. gheat -- a maptile server in Python, delivering heatmaps to be superimposed on Google Maps. Handy for visualization fiends.
3. CaDNAno -- open source software for design of 3-dimensional DNA origami. One of George Church's projects. I love the combination of math, biology, and whimsy in open-source giftwrap. (via timoreilly on Twitter)
4. CommentPress -- an open source theme for the WordPress blogging engine that allows readers to comment paragraph by paragraph in the margins of a text. Annotate, gloss, workshop, debate: with CommentPress you can do all of these things on a finer-grained level, turning a document into a conversation. It can be applied to a fixed document (paper/essay/book etc.) or to a running blog. I'm taking a greater interest in tools that channel and focus participation rather than simply providing "edit this page". (via gov2.net.au's issues paper)

OSCON: Programmer Insecurity and the Genius Myth

Two of my favorite presenters, Ben Collins-Sussman and Brian Fitzpatrick, did an OSCON session on "Programmer Insecurity and the Genius Myth." Brian and Ben talked about how programmers' insecurities cause all manner of troubles in programming projects, and then presented a number of tips for how to avoid these problems. They also asserted that there are very few genius "lone ranger"programmers in the real world -- most highly successful and productive programmers work smart and collaborate well.

(continue reading)

OSCON: The saga of MySQL

At OSCON in 2006, I followed sessions that discussed how open source companies would fare when big corporations come in. Back then there were only a handful of examples of big companies purchasing small open source companies. Three years later, we've witnessed MySQL AB get swallowed by Sun, only to have Sun be swallowed by Oracle. Now there are more open questions than ever and at least three versions of MySQL that are jockeying to continue the MySQL blood-line. Yesterday I attended talks by two of these groups and I have to wonder how the MySQL game will play itself out over time.

(continue reading)

OSCON: Standing Out in the Crowd

Kirrily Robert gave the first keynote speech this morning, entitled "Standing Out in the Crowd." She spoke about the gender imbalance in open source and shared her experiences working on open source projects that have a higher-than-average percentage of women participants. She laid out statistics about the current gender balance of various projects, looked at trends in open source, and closed with a number of tips on how open source projects can get -- and keep -- more women contributors.

(continue reading)

OSCON: Building Belonging (in communities)

I dove right in to OSCON by attending Jono Bacon's "Building Belonging" community talk. Jono, who is the community manager for Ubuntu, started out his presentation by asking what communities can do to build and improve the sense of belonging that people have in their community. After talking a little about what belonging means, he threw out the first concrete concept that builds belonging: Stories.

(continue reading)

Tonight's Ignite OSCON Line-Up

Bring the duct-tape, your head will explode. Here's the line-up for tomorrow's Ignite OSCON, starting 7.30pm in Exhibit Hall 3 of the San Jose Convention Center. (It's just before the 8.30pm Google-O'Reilly Open Source Awards)

Jesse Vincent (@obra) - Hacking the Kindle
The Kindle is not a read-only platform. One intrepid explorer reports back from the frontier.

Skud (@skud) - Textiles
Far more than you ever wanted to know about far more than you ever wanted to wear.

Liz Henry (@lizhenry) - Your Flying Jetpack
Don't sit around waiting for the future to give you a flying jetpack, there's something you should be working on.

Julian Cash - Photographing Open Source
Stories from behind the lens.

Sandy Jen (@meebosandy) - How To Be Really Cheap
How Meebo didn't buy expensive stuff and you can too.

Dan Meyer (@ddmeyer) - Three Words on Teaching
Three words that made me a better teacher.

Selena Deckelmann (@selenamarie) - How A Bunch Of Normal People Used Technology To Repair a Rigged Election
Lessons learned from a trip to teach PostgreSQL in Akure, Nigeria.

Erica Olsen (@eriquita) - The Librarian Avengers Film Rating System
The A-Z (more or less) of film ratings.

Damian Conway - Titles Are For Wimps
Nobody knows what Damian will talk about it, but it will be a miracle if we can get him to stop at just 5 minutes.

How NPR is Embracing Open Source and Open APIs

Daniel Jacobson Will Talk About the NPR Open API at OSCON

News providers, like most content providers, are interested in having their content seen by as many people as possible. But unlike many news organizations, whose primary concern may be monetizing their content, National Public Radio is interested in turning it into a resource for people to use in new and novel ways as well. Daniel Jacobson is in charge making that content available to developers and end users in a wide variety of formats, and has been doing so using an Open API that NPR developed specifically for that purpose. Daniel will talk about how the project is going at OSCON, the O'Reilly Open Source Convention. Here's a preview of what he'll be talking about.

James Turner: Can you start by explaining what NPR Digital Media is and what your role with it involves?

Daniel Jacobson: Sure. NPR is a radio organization, of course, and the Digital Media Group, of which I'm a part, handles, essentially as I describe it, everything that is publishable by NPR that does not go to a radio. So that includes the website, podcasts, API, mobile sites, HD radios, anything that has some sort of visual component to it. So Digital Media as a group is responsible for producing that content, producing all of those distribution channels, managing all of those relationships.

James Turner: And what is your particular role there?

Daniel Jacobson: I manage the application development team that is responsible for all the functional aspects of all of the systems, which includes our CMS, all of the templating engines for the website, for the API, for the podcasts, all of the engines that drive that.

James Turner: Now NPR is an organization that consists of a lot of member stations kind of flying in close formation. What's your relationship with the content producers? To what extent do they have their own stuff, and to what extent do you work together?

Daniel Jacobson: Those member stations are really exactly that; they are members of NPR. They essentially buy NPR programming. They're distinct organizations from us. NPR is a content producer and distributor. They buy our programming and broadcast it out to the world. They also have their own corresponding web teams that can take NPR content and also produce their own content and create their own websites. So in the Digital Media Team, we take a lot of pride and effort in providing services that help those member stations better serve their communities and their listeners and audiences, using NPR content and using their own content. We work with them to try and satisfy their missions. And to the extent that they need NPR services or content, we work hard to try and provide those. The API is one massive step, I think, in making it much easier for them to do what they need to do without a whole lot of intervention from us, where previously they would have to pull in content in much more arduous ways. So the API, I think, is a step in the right direction to make it more of a self-service model.

James Turner: Since you've mentioned the API, that's what you're going to be talking about at OSCON. We've already talked to the New York Times and the way they're opening up their content through APIs. What are you doing with yours?

Daniel Jacobson: Well, we launched ours formally at OSCON last year. And at that time, we essentially opened up our entire archive. So anything that you can get on npr.org is available through the API, to the extent that we have the rights to distribute it. There are some rights restrictions, for example, for receiving photos or stories from sources that we have not cleared rights to redistribute. Those are getting suppressed through a rights filtering engine on our API. Everything else that you can get on npr.org, you can get through the API. That includes full text. It includes images, audio, video, everything like that. Throughout the last year, we have added more features. We included the layer of "mix your own podcast", for example, which allows people to not only get the content in audio form, but also to download it as a podcast-type item. And all of that is available through search terms or totally customized queries. So what the API really does is it enables people to take the content, make widgets, or do whatever they want with essentially everything that is on npr.org and get to audiences that we are not getting to.

(continue reading)

Making Government Transparent Using R

Danese Cooper thinks it will be an important tool in Open Gov

With Open Source now considered an accepted part of the software industry, some people are starting to wonder if we can't bring the same degree of openness and innovation into government. Danese Cooper, who is actively involved in the open source community through her work with the Open Source Initiative and Apache, as well as working as an R wonk for Revolution Computing, would love to see the government become more open. Part of that openness is being able to access and interpret the mass of data that the government collects, something Cooper thinks R would be a great tool for. She'll be talking about R and Open Government at OSCON, the O'Reilly Open Source Convention.

James Turner: Why don't you start by describing where you came from, and you're involved in, and what your interests are?

Danese Cooper: Okay. I'm Danese Cooper. I serve on the board of the Open Source Initiative. I have been serving for the last eight years. And I'm also currently employed by Revolution Computing, which is a start-up focusing on an open source language called R, as in the letter R, that is very useful for analytics and statistical analysis. I'm also an Apache member. And I also serve on an advisory board for Mozilla.

James Turner: One of the two panels you're going to be speaking on at OSCON is on open source and open government. If you could talk a little bit about what interests you about open government and also what open government means to you.

Danese Cooper: Sure. Well, along with a lot of open source people, I got interested in the Obama campaign and in helping President Obama get elected. And part of why he was so compelling was that the vision of how Washington needed to change is pretty close to the way that we think about working collaboratively in open source. The night that he was elected, there was a great little clip on CNET of a Republican commentator actually explaining open source as exactly what I just said. It was a really brilliant little two-minute clip. He pointed at The Cathedral and the Bazaar, that canonical document about how open source works. And he said, "Microsoft is the cathedral. It's their way or the highway. And the bazaar is a bunch of people working together grassroots to collaboratively build the things that they need. And so Obama's basically asking for the government to become open source, and the problem is Washington isn't really like that right now."

So anyway, that's the transformation that has to happen in order for government to really be transparent. To me, open source government is transparent government. There's been an awful lot of shenanigans in recent political history, like the last decade has been pretty crazy in terms of things happening that couldn't be traced back to any source. Even just the way we vote and the way that voting is managed, and the fact that the software that runs the machines that we vote on is not open source so it can't be inspected. And nobody knows quite what it does. There are all of these stories of weird updates to the software that happened right before major elections in states where there are strange results. Transparency, in the same way that it helped the software industry transform, could really help the government transform. So that's what I'm talking about. There's a bunch of other people on that panel. My good friend, Brian Behlendorf, and I co-proposed it. And he's actually taken the next step. He helped found Apache. And he's run off to Washington to work on projects that are interesting to the Obama government to try to figure out how to help them to more open source solutions. And he'll be talking about his progress on that panel. So I think it's a pretty exciting panel.

(continue reading)

Sequencing a Genome a Week

Radar Talks to OSCON Speaker David Dooling

The Human Genome Project took 13 years to fully sequence a single human's genetic information. At Washington University's Genome Center, they can now do one in a week. But when you're generating that much data, just keeping track of it can become a major challenge in itself. David Dooling is in charge of managing the massive output of the Center's herd of gene sequencing machines, and making it available to researchers inside the Center and around the world. He'll be speaking at OSCON, the O'Reilly Open Source Convention. His talk, titled The Freedom to Cure Cancer: Open Source Software in Genomics, will be about how he uses open source tools to keep things under control, and he agreed to talk about how the field of genomics is evolving.

James Turner: Can you start by describing what it is you do and how you came to be doing it?

David Dooling: Sure. I work at the Genome Center at Washington University in St. Louis. We are one of the handful or so of large scale genome sequencing centers around the world. What that means is essentially we participate in large genome sequencing projects that some people may have heard of, like the Human Genome Project, Thousand Genomes Project, things like that. And involved in that is a lot of data processing, laboratory processing, tracking and all sorts of things, so it's a rather large enterprise.

There are about 300 or so people that work here. And how I came to work here was that about eight years ago, I decided that I wanted to get more into programming and more into open science. So I took a job as a programmer here at the Genome Center and gradually worked my way around to where I am now, where I oversee all of the software development and IT infrastructure here at the Genome Center. And it's a fairly large IT infrastructure.

We have somewhere around three petabytes of storage online, and somewhere north of 3,000 cores in our computational cluster. And we're generating terabytes, tens of terabytes of data, per day with our current sequencing instruments. The sorts of things that we're doing now as we transition from more fundamental evolutionary types of projects, such as the Human Genome Project and subsequent projects like the Mouse Genome Project, we've done things like corn and things of that nature, now we're doing more and more sequencing projects related to medicine and medical sequencing.

Last year, we published the full cancer genome sequence. In doing both the cancer and the normal, we were able to determine the differences between those two genomes and begin to identify what might've possibly caused cancer in that individual. So projects like that. We're also doing projects with metabolic syndromes, like diabetes, and several other cancer projects as well. That's essentially what we're doing and how we're doing it and how I got here.

James Turner: Genomics is an area that seems to be on the steep part of the hockey stick curve right now. In just a decade, we've gone from sequencing one genome over a period of years to doing them routinely. Can you talk a bit about what's enabled this acceleration?

David Dooling: Well, a whole host of things. But I think really at the core was the changing fundamentals of sequencing itself. For a long time, DNA sequencing was based on a process invented by Sanger, sometimes called Sanger Sequencing, sometimes called capillary electrophoresis now because of the last revision of the instruments that were generated. But essentially with that approach, you did reactions in 96 plate wells. You processed sequence in these 96 plate well chunks. And you did reactions in there. You loaded them on the readers, and the readers read out sequence for each of those 96 wells. So that's sort of how you processed it. And at the height of that sort of sequencing, which was only a few years ago, we had about 130 or so of those instruments each churning about 15 to 20 runs per day. Each run gave you 100 pieces of sequences. You had 100 or so machines. And so you got on the order of a few thousand sequence reads, that's what we called them, because of the way the instrument read the information.

Now, since that time, 454 was first [of the new generation of sequencers] and then Solexa came, which was later bought-out by Illumina, and the ABI SOLiD has a platform. There's one from Helicos as well. And then several other third generation, those first being the second generation, sequencers have come out. And what those do is greatly increase the parallelism with which you're able to process DNA and sequence it. So instead of a few thousand runs per day, or a few thousand reads per day, you may get a few million reads per run. And these runs, for some of the platforms, do take a little bit longer. But the parallelism of it increases your throughput tremendously. And so now we have about 35 to 40 of these highly parallel instruments in-house. And with that, we're able to sequence the human genome to complete coverage in less than a week.

So the main driver has been this change in the sequencing technology and the parallelism of it. It's a fundamentally different chemistry, different physics. The flipside of it is that we talked about the hockey stick, and so that hockey stick is the sequencing hockey stick, but it's brought several other hockey sticks along with it, mainly the amount of data that these things generate. And the amount of processing power that is required to process that data has increased greatly as well. Much faster than Moore's Law over the last two years or so. Whereas with those original instruments, you would generate on the order of megabytes per day, now we're doing tens of terabytes per day with these new instruments. And then processing that, instead of taking a single processor a few minutes, it can take a small cluster a few days to actually analyze the data from each of these runs.

Those are the main things. The enabling technology was the change in the sequencing chemistry itself. And then what had to come along with that was building these infrastructures to be able to track these things and process these things and store all of this data as the instruments increased in their abilities.

(continue reading)

Open Source is Infiltrating the Enterprise

Forrester's Jeffrey Hammond Says There's Plenty of it Around, if You Look

There's a persistent perception that open source software is being ignored in the enterprise, that IT management fears it and it ends up being more costly to deploy than proprietary solutions. That's certainly the perception that some major software vendors would like you to have. But it's Jeffrey Hammond's job to dispel those perceptions, at least when they aren't accurate. As an analyst for Forrester Research, Hammond covers the world of software development as well as Web 2.0 and rich internet applications, so he sees how open source is being used on a daily basis. He'll be speaking at OSCON, the O'Reilly Open Source Convention, talking about the true cost of using open source, and he gave us a sample of what's going on in the enterprise at the moment.

James Turner: To begin, can you just talk about the areas you cover for Forrester?

Jeffrey Hammond: Sure. I'm in the Application Development and Program Management Group at Forrester, which means I write for folks like developers, architects and development managers. And open source is a little bit of an interesting topic for me because it kind of crosses roles. But what we tend to find is that a lot of time it's developers and maybe development managers that are pulling open source into the enterprise. And that, I think, is why I have it as a research area. But in addition to open source, I also cover Web 2.0. I cover rich internet application development. I cover software change and configuration management and application life cycle management, software modeling, mobile development, IDEs and programming languages. So a pretty wide-variety of development related stuff.

James Turner: There's obviously a lot of apocrypha and maybe even some downright misinformation about how open source is being used in the enterprise. Can you illuminate things a bit?

Jeffrey Hammond: Sure. You know, it crosses a wide-variety of spectrums. But anyone who tells you that it's not being used strategically for mission critical applications isn't talking to the right people, because I see it used that way as well as on individual projects at the departmental level. I'll give you one example: I have talked with a large organization that's using multiple open source products and projects and frameworks as the core of it's reservation systems, and they're pushing 30,000 transactions a second through what's essentially an open source infrastructure. So I defy anybody to say that that's not mission critical. And it certainly contributes revenue to their business. I've also talked to organizations that are almost completely open source in their development environment. So all of the software code that they write is being built with open source tools. I think all you have to do is look at the latest data from the Eclipse Community Survey, which they published about three weeks ago, and it's pretty hard to make the argument that open source adoption is not pervasive and accelerating in the market.

James Turner: What are some of the real success stories you've seen with open source in commercial settings?

Jeffrey Hammond: Well, I just mentioned the one about the travel and transport provider, but I've seen open source used as the basis for in-store sales systems that are PCI compliant. So that would counter another fallacy that you often hear; you can't build secure software with open source. And I defy you to find an example of a system which would need a higher level of governance and qualification than something that's handling credit card data. And, yet, we see organizations successfully deploying software with things like OpenSolaris and networking stacks that are based on open source software. I've also seen it used in financial services organizations and by a major airline in Europe, which is using open source at the core of their operating system strategy. And they're running their SAP installations on top of an open source operating system framework and saving a million dollars simply by doing that.

So the uses are many and varied, but generally, the goal is to save money. That's where organizations tend to start. And then what tends to happen is the more that they become comfortable with using open source, and the more that they apply it successfully, the more they start to realize that there are benefits other than cost savings that they can take advantage of. And that's when you start to see them turn from open source opportunists into open source advocates. It's interesting to watch that transformation happen over a year or a two-year period at a large company. They start looking for opportunities to replace commercial products, things at the app server level, things at the business intelligence level, things in the web content management space. And all of these are opportunities where there are real, credible, open source projects that are used by large organizations successfully.

(continue reading)

Patrick Collison Puts the Squeeze on Wikipedia

How to Cram the Wikipedia onto an 8GB iPhone

Think about Wikipedia, what some consider the most complete general survey of human knowledge we have at the moment. Now imagine squeezing it down to fit comfortably on an 8GB iPhone. Sound daunting? Well, that's just what Patrick Collison's Encylopedia iPhone application does. App Store purchasers of Collison's open source application can browse and search the full text of Wikipedia when stuck in a plane, or trapped in the middle of nowhere (or, as defined by AT&T coverage...) Collison will be presenting a talk on how he did it at OSCON, O'Reilly's Open Source Convention at the end of July, and he spent some time talking to me about it recently.

James Turner: Why don't you start by talking about your background a bit and how you got involved with working with the Wikipedia?

Patrick Collison: I guess I've always been pretty interested in Wikipedia, and I ran my own MediaWiki installations back when I was in school in Ireland. We had our own personal ones and all of the rest. Then in November of 2007, I went to visit my friend in Japan for a month. And in Japan they have all of this incredibly advanced cellular technology and all of the rest. And so because of that, they had very few wireless networks, and my phone didn't work. As a result, I actually had very little access to the Internet. I sort of realized without Wikipedia how little I really knew. And I had just got an iPhone, so I decided to try basically putting a copy of Wikipedia on the phone, so that I'd have it as I was walking around in Japan. Then basically, I spent a significant fraction of my time there in Japan, again, in 2007 writing those applications, say maybe two or three weeks, just firstly trying to decide if it was possible and putting it all together. And then it was released, I think, January of 2008.

James Turner: Now you've also worked on getting it onto the OLPC I understand. How did that occur?

Patrick Collison: I actually didn't do much of the work for this. It was actually a project led by Chris Ball who works both with FreeBSD and with the OLPC project. But I released the code to this application; it was open source from the very start. So it was pretty easy for them to take it and to port it to the OLPC. I mean there are already some applications that allowed you to put a copy of Wikipedia on your computer or something like that, but none had really been optimized for embedded or low power devices or anything like that, which obviously Wikipedia for the iPhone had to be. I think it took about two or three weeks to take the code that ran on the iPhone and then to bring it to the point where it'd run on the OLPC.

James Turner: There are obvious benefits to having Wikipedia on the OLPC, because connectivity is very important in some of those areas. So you'd want to have it local, but outside of the experience that you were just describing, isn't the point of the iPhone that you can just access Wikipedia? What are kind of the advantages of having it locally?

Patrick Collison: I actually find that you spend, or I certainly spend a surprising amount of my time without access to the internet, even with the iPhone. Say for start if you were abroad, I mean everyone knows the horror stories of the data changes AT&T will issue you with if you're roaming. But also just stuff like personally, I find that on a plane or something you have eight hours to not do much. And so I actually end up doing a lot of my Wikipedia browsing there. But even aside from connectivity issues, it actually turns out to be quite a bit faster to use the built-in, cached Wikipedia application as opposed to the website. I mean you can search in real-time with the applications. You just type a couple of characters and tap into your article, rather than firing up Safari or searching for the article in Google; then zooming in so you can tap in, et cetera, et cetera. I and most of the people I know who use the application actually end up using it even when they have internet connectivity. And maybe 20 percent of the time it's pretty useful because it's the only choice.

James Turner: Now just as a point of interest, is this an App Store app or do you have to have a jail-broken phone for it?

Patrick Collison: It was released back when only the jail-broken SDK existed. It was in that initial sort of surge of early applications. I guess the first jail-broken iPhone app, I think, happened in August, and so this was released just under six months later. And then when Apple announced the SDK, I actually originally did not intend to port it to the App store, just because I was just working on other things at the time and my company had just been bought and so it seemed like a lot of work. But then over the summer, I started getting a huge amount of email from people who had upgraded to the new version of the iPhone OS, and were now missing Wikipedia. And I started getting 20 or so emails from people per day saying they love this application and they were really missing it. Or even people saying they were continuing to use the old version of the OS just for this application. And they really hoped that I would port it so they could eventually upgrade. After receiving these emails for a while, I eventually felt too bad about not porting it. So I spent a couple of days porting it and then released it in the App Store. I wrote it and finished the port in August. And then it took about three months to wade through Apple's approval process. Around the end of October, it was released in the App Store.

(continue reading)

OSCON 2009 Highlights

OSCON 2009 is just around the corner, this year in San Jose, California. When I spoke at the Silicon Valley Linux Users Group last night, they asked me for a few highlights. It's tough to pick from over 200 sessions, all the best-of-the-best out of 800 submissions (and there were at least 100 more I wish I could have fit in). But, a few of the talks I'm particularly looking forward to are "Situation Normal, Everything Must Change" by Simon Wardley on the cloud computing apocalypse and its implications for open source, "Drizzle: Status, Principles, and Ecosystem" by Brian Aker, "Introduction to Animation and OpenGL on the Android SDK" by Satya Komatineni, "Security-Centered Design: Exploring the Impact of Human Behavior" by Chris Shiflett, "What Web App Design Can Learn From the Harpsichord" by Elaine Wherry, and "The Freedom to Cure Cancer" by David Dooling on open source software in the Human Genome Project. On a less serious note, Friday will be capped off with "The Art of Klingon Programming" by Paul Fenwick and "The Conway Channel" by Damian Conway, two speakers from the southern hemisphere who never fail to make me laugh and think all at the same time.

We've extended the early registration discount until June 23rd, so you can still save $250. And OSCamp—the unconference within OSCON, organized by participants for participants—is completely free (as in beer and speech) open to anyone registered for an Exhibit Hall pass. Hope to see you there.

Nominations For Google-O'Reilly Open Source Awards 2009

The 5th annual Google-O'Reilly Open Source Awards will be hosted at OSCON 2009 in San Jose, CA. The awards recognize individual contributors who have demonstrated exceptional leadership, creativity, and collaboration in the development of Open Source Software. Past recipients for 2005-2008 include Angela Byron, Karl Fogel, Pamela Jones, Gerv Markham, Chris Messina, David Recordon, Doc Searls, and Andrew Tridgell.

The nomination process is open to the entire open source community, closing May 22, 2009. Send your nominations to osawards@oreilly.com. Nominations should include the name of the recipient, any associated project/org, suggested title for the award ("Best Hacker", "Best Community Builder", etc.), and a description of why you are nominating the individual. Google and O'Reilly employees cannot be nominated.

Four short links: 17 Feb 2009

Four Tuesday quickies:

1. The Technology Behind Coraline -- 3D stop-motion movie used a 3D printer to make the dolls and things like drops of water.
2. Some OSCON Proposal Tips (Alex Russell) -- good advice for anyone submitting a talk to a technical conference.
3. Oscar Predictions You Can Bet On -- Nate Silver of FiveThirtyEight turns his attention to the Oscars.
4. Web Hooks and the Programmable Web of Tomorrow -- a epic presentation of different ways to offer and use callbacks, URLs on your site that a remote service can hit when something happens on their service. (via Stinky)

OSCON moves to San Jose

The official word is out, OSCON 2009 will be moving from Portland, Oregon to San Jose, California. We've received significant positive feedback on the move, and messages of welcome from Bay Area open source contacts, but also some messages of disappointment from the local Portland open source community, and from non-local attendees who enjoyed visiting Portland every year. We're also sad to leave, Portland has been an incredible incubator for the conference, as it is for many open source ventures. OSCON was first launched in Monterey, California, and then moved to San Diego, California. In 2003 we moved to Portland, to the Marriott Hotel on the Waterfront, and in 2005 we moved to the Oregon Convention Center on the other side of the river. During our time in Portland, we received incredible support from the local open source and technical communities, and from the Portland Development Commission. In Portland, OSCON grew from 1300 registered attendees in 2003, to 3000 registered attendees in 2008.

Clearly Portland was good for OSCON, but at the same time, we move most of our conferences every few years, to allow new local communities and organizations to participate and to provide new activities for non-local attendees. And while every conference planner likes to see yearly growth as a sign of a healthy conference, it's a challenge to find space for 3000 people plus a projected growth of ~20% (based on previous years). The largest available keynote space at the Oregon Convention Center (the largest conference facility in Oregon) holds approximately 2500 people in our current layout (with a stage and airwall space to divide into smaller rooms for day sessions), and has an absolute maximum limit of 3600 people. 2008 marked a record high of OSCON attendees being turned away from sessions they wanted to see, due to firecode restrictions on the maximum occupancy of the rooms. Even Tim O'Reilly was turned away from two sessions. Given a choice between restricting conference registrations to a pre-set limit (like we do with the Web 2.0 Summit) and finding a bigger space, we knew the right choice for a community-oriented conference like OSCON was to find a bigger space.

We're thrilled to see LinuxCon starting up in Portland, and hope that the energy of yearly Portland-local open source events that grew to complement OSCON will naturally migrate to LinuxCon. Jim Zemlin, who I've known for years, emailed me as soon as an article was released speculating that LinuxCon was started as a reaction to OSCON leaving Portland, to assure me that they had been planning LinuxCon in Portland long before they knew OSCON was moving, and reiterating the Linux Foundation's full and continued support for OSCON. We plan to do cross-promotion between OSCON and LinuxCon, and maybe even cross-conference discounts (if we can work out the practicalities of verifying registration at an unrelated event).

We don't know yet how long we'll stay in San Jose. Some aspects of the space are ideal: it's got a strong local open source community, it's certainly large enough to host us now and for a few years in the future, and it'll be convenient for the attendees and staff to have the conference hotels connected to the session space again. And, with the rising cost of fuel and travel, a huge local open source population in the greater Bay Area is a definite advantage. But, it's hard to tell exactly how good a fit any space is for a given conference until you've actually held the conference there. I'm sure some things about the space will be less-than-ideal, and we look forward to the feedback from OSCON 2009 attendees (both good and bad) to help us make an educated decision on the location of OSCON 2010 and beyond. We also welcome suggestions from open source communities in other cities (with conference facilities to host 3500+ people), as candidates for future years.

Thanks for your part in ten fabulous years of OSCON and open source. Here's to another fabulous ten ahead.

Random OSCON Tidbits

Some things I learned about at the Django/Python meetup in downtown Portland during OSCON:

• JS Bridge: a Python to Javascript bridge for all Mozilla applications, still under very active development (i.e., changing daily).
• 960.gs: a grid framework for Javascript (replacing Blueprint CSS) with a naming scheme that makes prototyping designs a lot less painful.
• Dojo has Django Templates: I take my eye off Dojo for a year and it suddenly grows the ability to have full Django templates in the browser. Holy CRAP.

Open Source and Cloud Computing

I've been worried for some years that the open source movement might fall prey to the problem that Kim Stanley Robinson so incisively captured in Green Mars: "History is a wave that moves through time slightly faster than we do." Innovators are left behind, as the world they've changed picks up on their ideas, runs with them, and takes them in unexpected directions.

In essays like The Open Source Paradigm Shift and What is Web 2.0?, I argued that the success of the internet as a non-proprietary platform built largely on commodity open source software could lead to a new kind of proprietary lock-in in the cloud. What good are free and open source licenses, all based on the act of software distribution, when software is no longer distributed but merely performed on the global network stage? How can we preserve freedom to innovate when the competitive advantage of online players comes from massive databases created via user contribution, which literally get better the more people use them, raising seemingly insuperable barriers to new competition?

I was heartened by the program at this year's Open Source Convention. Over the past couple of years, open source programs aimed at the Web 2.0 and cloud computing problem space have been proliferating, and I'm seeing clear signs that the values of open source are being reframed for the network era. Sessions like Beyond REST? Building Data Services with XMPP PubSub, Cloud Computing with BigData, Hypertable: An Open Source, High Performance, Scalable Database, Supporting the Open Web, and Processing Large Data with Hadoop and EC2 were all full. (Due to enforcement of fire regulations at the Portland Convention Center, many of them had people turned away, as SRO was not allowed. Brian Aker's session on Drizzle was so popular that he gave it three times!)

But just "paying attention" to cloud computing isn't the point. The point is to rediscover what makes open source tick, but in the new context. It's important to recognize that open source has several key dimensions that contribute to its success:

1. Licenses that permit and encourage redistribution, modification, and even forking;
2. An architecture that enables programs to be used as components where-ever possible, and extended rather than replaced to provide new functionality;
3. Low barriers for new users to try the software;
4. Low barriers for developers to build new applications and share them with the world.

This is far from a complete list, but it gives food for thought. As outlined above, I don't believe we've figured out what kinds of licenses will allow forking of Web 2.0 and cloud applications, especially because the lock-in provided by many of these applications is given by their data rather than their code. However, there are hopeful signs like Yahoo! Boss that companies are at beginning to understand that in the era of the cloud, open source without open data is only half the application.

But even open data is fundamentally challenged by the idea of utility computing in the cloud. Jesse Vincent, the guy who's brought out some of the best hacker t-shirts ever (as well as RT) put it succinctly: "Web 2.0 is digital sharecropping." (Googling, I discover that Nick Carr seems to have coined this meme back in 2006!) If this is true of many Web 2.0 success stories, it's even more true of cloud computing as infrastructure. I'm ever mindful of Microsoft Windows Live VP Debra Chrapaty's dictum that "In the future, being a developer on someone's platform will mean being hosted on their infrastructure." The New York Times dubbed bandwidth providers OPEC 2.0. How much more will that become true of cloud computing platforms?

That's why I'm interested in peer-to-peer approaches to delivering internet applications. Jesse Vincent's talk, Prophet: Your Path Out of the Cloud describes a system for federated sync; Evan Prodromou's Open Source Microblogging describes identi.ca, a federated open source approach to lifestreaming applications.

We can talk all we like about open data and open services, but frankly, it's important to realize just how much of what is possible is dictated by the architecture of the systems we use. Ask yourself, for example, why the PC wound up with an ecosystem of binary freeware, while Unix wound up with an ecosystem of open source software? It wasn't just ideology; it was that the fragmented hardware architecture of Unix required source so users could compile the applications for their machine. Why did the WWW end up with hundreds of millions of independent information providers while centralized sites like AOL and MSN faltered?

Take note: All of the platform as a service plays, from Amazon's S3 and EC2 and Google's AppEngine to Salesforce's force.com -- not to mention Facebook's social networking platform -- have a lot more in common with AOL than they do with internet services as we've known them over the past decade and a half. Will we have to spend a decade backtracking from centralized approaches? The interoperable internet should be the platform, not any one vendor's private preserve. (Neil McAllister provides a look at just how one-sided most platform as a service contracts are.)

So here's my first piece of advice: if you care about open source for the cloud, build on services that are designed to be federated rather than centralized. Architecture trumps licensing any time.

But peer-to-peer architectures aren't as important as open standards and protocols. If services are required to interoperate, competition is preserved. Despite all Microsoft and Netscape's efforts to "own" the web during the browser wars, they failed because Apache held the line on open standards. This is why the Open Web Foundation, announced last week at OScon, is putting an important stake in the ground. It's not just open source software for the web that we need, but open standards that will ensure that dominant players still have to play nice.

The "internet operating system" that I'm hoping to see evolve over the next few years will require developers to move away from thinking of their applications as endpoints, and more as re-usable components. For example, why does every application have to try to recreate its own social network? Shouldn't social networking be a system service?

This isn't just a "moral" appeal, but strategic advice. The first provider to build a reasonably open, re-usable system service in any particular area is going to get the biggest uptake. Right now, there's a lot of focus on low level platform subsystems like storage and computation, but I continue to believe that many of the key subsystems in this evolving OS will be data subsystems, like identity, location, payment, product catalogs, music, etc. And eventually, these subsystems will need to be reasonably open and interoperable, so that a developer can build a data-intensive application without having to own all the data his application requires. This is what John Musser calls the programmable web.

Note that I said "reasonably open." Google Maps isn't open source by any means, but it was open enough (considerably more so than any preceding web mapping service) and so it became a key component of a whole generation of new applications that no longer needed to do their own mapping. A quick look at programmableweb.com shows google maps with about 90% share of mapping mashups. Google Maps is proprietary, but it is reusable. A key test of whether an API is open is whether it is used to enable services that are not hosted by the API provider, and are distributed across the web. Facebook's APIs enable applications on Facebook; Google Maps is a true programmable web subsystem.

That being said, even though the cloud platforms themselves are mostly proprietary, the software stacks running on them are not. Thorsten von Eicken of Rightscale pointed out in his talk Scale Into the Cloud, that almost all of the software stacks running on cloud computing platforms are open source, for the simple reason that proprietary software licenses have no provisions for cloud deployment. Even though open source licenses don't prevent lock-in by cloud providers, they do at least allow developers to deploy their work on the cloud.

In that context, it's important to recognize that even proprietary cloud computing provides one of the key benefits of open source: low barriers to entry. Derek Gottfried's Processing Large Data with Hadoop and EC2 talk was especially sweet in demonstrating this point. Derek described how, armed with a credit card, a sliver of permission, and his hacking skills, he was able to put the NY Times historical archive online for free access, ramping up from 4 instances to nearly 1,000. Open source is about enabling innovation and re-use, and at their best, Web 2.0 and cloud computing can be bent to serve those same aims.

Yet another benefit of open source - try before you buy viral marketing - is also possible for cloud application vendors. During one venture pitch, I was asking the company how they'd avoid the high sales costs typically associated with enterprise software. Open source has solved this problem by letting companies build a huge pipeline of free users, who they can then upsell with follow-on services. The cloud answer isn't quite as good, but at least there's an answer: some number of application instances are free, and you charge after that. While this business model loses some virality, and transfers some costs from the end user to the application provider, it has a benefit that open source now lacks, of providing a much stronger upgrade path to paid services. Only time will tell whether open source or cloud deployment is a better distribution vector, but it's clear that both are miles ahead of traditional proprietary software in this regard.

In short, we're a long way from having all the answers, but we're getting there. Despite all the possibilities for lock-in that we see with Web 2.0 and cloud computing, I believe that the benefits of openness and interoperability will eventually prevail, and we'll see a system made up of cooperating programs that aren't all owned by the same company, an internet platform, that, like Linux on the commodity PC architecture, is assembled from the work of thousands. Those who are skeptical of the idea of the internet operating system argue that we're missing the kinds of control layers that characterize a true operating system. I like to remind them that much of the software that is today assembled into a Linux system already existed before Linus wrote the kernel. Like LA, 72 suburbs in search of a city, today's web is 72 subsystems in search of an operating system kernel. When we finally get that kernel, it had better be open source.

OSCON in 37 Minutes

The wonderful Gregg Pollack, of Rails Envy fame, wandered the halls and speaker room at OSCON with his video camera. He asked a pile of speakers to summarize their talks in 30 seconds or less, and has compiled the results into "OSCON in 37 Minutes". It's well worth watching even if you were at the conference—as anyone who's attended knows, it's impossible to see everything unless you have a clone army or a time machine. My favourite so far: R0ml's Exceptional Software Explained (video for the full talk of which is also available).

OSCON day 3: Reflections on OSCON 2008

Today was the last day of OSCON and I'm in the mood to think about the conference and share some of my random observations that didn't make it into any of my other blog posts.

First up is a comment that Brian Aker of MySQL fame made during the "Tim O'Reilly Interviews Monty Widenius & Brian Aker" interview:

Microsoft is irrelevant. ... We're more worried about Apple.

Woah. That's a tall statement! One that resonates with me since just a few weeks ago I realized that my life is now fully free of Microsoft. And I used to be a full time Win32 programmer 10 years ago. While Microsoft may not be fully irrelevant in all scopes, a comment like this shows that the open source movement has made an amazing amount of progress in the last 10 years. Consumers have a lot more operating choices today than they did 10 years ago. And to think that Apple had been written off for dead -- now they're back and they're feared!

(continue reading)