On Wikipedia, storms, teacups, and _why's notability

by Nat Torkington | @gnat | comments: 7

In which our hero ponders the Internet's underwear, the oxymoronic nature of social software, and that not only should you not hate the playa but you shouldn't even hate the game.

It must be a weekend, the interwebs have their panties in a bunch again. This time it's about the Wikipedia entry for _why the lucky stiff, one of the major Ruby hackers. For the backstory, see Deletionist Morons by Tim Bray. In short: Wikipedia editors want to delete _why's entry because he doesn't pass Wikipedia's Notability test.

Social software is a funny old thing, isn't it? On the one hand, we have the word "social" with its overtones of informality, emotion, and all those black turtleneck wearing arts graduates. Then we have the word "software" with its harmonics of precision, logical thought, and Aspies with intravenous caffeine. In fact, when you think of "software" you probably think of people who could easily be described as "antisocial". Is it any wonder, then, that the product of the two doesn't exactly mesh well with our view of the world?

Having read Wikipedia: The Missing Manual, I now know that Wikipedia is social software. Not the reading part, but the editing. There's a human process for humans to follow, whereby the humans use the software to debate (something humans do, not software) and arrive at a decision. This is a human, social, process ... not a software one. A lot of the rancour comes from misunderstanding this.

Perhaps an analogy to another social process would help. Wikipedia is like an open source software project where the great unwashed submit patches, the committers choose which to apply, and the core team make executive decisions when needed. There's no piece of code that determines worthiness to be committed to the source tree. Instead, there are people with judgement and human flaws in the way. The Linux kernel shouldn't grow e-mail protocol stacks, web server hacks, and a built-in relational database just because someone submits the patches. The project's committers are there to keep the software project on track. So too with Wikipedia.

Hating the humans or even hating the filtering process is a waste of time and energy. The deletionists and the inclusionists both have a role to play. Wikipedia has a lot of things that it is not and the humans are there to keep the project on track. Those who want to delete and want to keep are doing their bit, just as others did by creating a page for _why in the first place.

The creators of any piece of social software must carefully choose where to punch holes in pure computational deterministic perfection to let human attributes like intelligence or taste shine through. Their choices define the project. This "you want X, I want Y, we'll go back and forth citing Wikipedian principles and external sources until a decision emerges or must be made by an administrator" process isn't Wikipedia's weakness, or even its strength, it is Wikipedia.

In social software as in software projects, the human filters sometimes make poor decisions; you can't have the flexibility and intelligence of humans without their flaws. Using Wikipedia but becoming enraged when your favourite marginal entry is deleted is like going to an art gallery but being enraged that you saw something there you didn't like. It's a big waste of time and energy that could better be spent working on this patch I've got to add a relational database to the Linux kernel ....

O'Reilly School of Technology + Mathematica: What Do They Add Up To?

by Jimmy Guterman | comments: 0

Last month, the O'Reilly School of Technology and Wolfram Research announced that the school was licensing Wolfram's flagship math program Mathematica to create a web-based version of the system.

Right after the announcement, we ran an interview with Scott Gray, director of the O'Reilly School of Technology that gives a solid explanation of the deal and what it means. Tim O'Reilly and I had a few more Radar-oriented questions for Scott. (Disclosure: I'm a half-competent Mathematica hacker and a bleary-eyed fan of Mathematica creator Stephen Wolfram's A Different Kind of Science).

Q: There's already a webMathematica. How is what you're doing different from that?

Gray: webMathematica is the Web 1.0 version of Mathematica, using just HTML and web forms. It doesn't have any of the rich front-end capabilities of the Mathematica software. The in-line editing and computational functionality of Mathematica are not only some of the features of Mathematica that separate it from other symbolic mathematics software, but these features turn out to be very important when using Mathematica to teach mathematics. Hilbert will use Ajax and CSS to achieve high fidelity with the original software. Using Ajax and CSS to emulate rich GUI interfaces on the web is one of the trends we're seeing in this Web 2.0 era.

Q: Does this have any relation to the CalculuzWiz program from a few years back that used Mathematica to teach calculus?

Gray: No. Calculus Wiz is simply a repository of Calculus examples that utilize Mathematica. While that's a nice resource, it doesn't have the pedagogical and narrative structure we're seeking. We're going to be using the Calculus&Mathematica (C&M) content created by Prof. Jerry Uhl and Prof. Bill Davis of the University of Illinois and Ohio State. They've created over a dozen courses in mathematics using a unique narrative structure utilizing Mathematica. This structure uses Mathematica to engage learners in a conversation which encourages exploration and discovery. For learning theorists out there, the pedagogy is a computer-driven form of constructivism which has been used successfully at the University of Illinois, Ohio State, and many other universities since the early 1990's. This will allow us to increase the availability and decrease the overhead for teaching these wonderful courses.

Q: What's the long-term goal here? How are services like this going to change the school and -- more broadly -- education?

Gray: The overarching goal of the O'Reilly School of Techology (OST) is to improve science and engineering education by setting an example of what's possible using the computer and internet as catalyst. The education industry is one of the least disruptive industries on the planet. Although we hear a lot about technology being used in classrooms and the internet, the pedagogy is the same as it's been for hundreds of years. For the most part, nothing has changed about how subjects like mathematics are taught. Computers and the internet are simply being used to distribute the same boring lectures and presentations that haven't been working all along. Science and math education is still a mess, especially in America. It became clear to me long ago that change could not happen from within the educational establishment, so we're using OST as a platform to demonstrate new ways to teach and learn. We have a long way to go on our journey, but we're headed in the right direction.

Q: Are there any plans to license the Ajax version back to Wolfram?

Gray: If this works really well, look for Wolfram to consider other avenues for taking advantage of the potential of software as a service. Allowing this model in partnership with O'Reilly is a big policy step for them and they are curious to see if the benefits of business models like this can outweigh the risks. From a strickly idealistic point of view, everyone realizes that opening up and unleashing the power of Mathematica into the Web 2.0 world of mashups has enormous potential.

If you want to see more, Gray will be showing a preview at the Web 2.0 Expo in San Francisco in June.

Why Non-Obvious Brand Icons Work

by Sara Winge | @sarawinge | comments: 9

While pondering why names like Firefox, Fire Eagle, and firedog work for technology products, anthropologist and culture maven Grant McCracken concludes:

A Firefox and a Fire Eagle are counter intuitive in exactly the right proportions. These names resist comprehension but only just. They are counter intuitive, but not unintelligible. In the first moment of exposure, we don't quite get them...and this prevents them from washing over us and out into that sea of forgettable branding and marketing. Comprehension is held up just long enough for the new name to lock into memory.

Edie Freedman, O'Reilly's original Creative Director, knew this 20 years ago when she designed the first of our now-infamous animal covers:

As I started to look for imagery for the book covers, I came across some wonderful wood engravings from the 19th century. The strange animals I found seemed to be a perfect match for all those strange-sounding UNIX terms, and were esoteric enough to appeal to what I believed the UNIX programmer type to be.

Tim, against the advice of most everyone else in the office, gave the go-ahead to the quirky covers. Edie's intuition proved correct--UNIX geeks, an imaginative bunch who treasure a good story, loved the subtly non-obvious covers. Owning a shelf of "animal books" became a badge of honor for serious hackers.

In hindsight, we realized that the slight hurdle readers had to leap to "get" the animal brand made it stronger. Not only were people more likely to remember our books, but they were in on the secret of the covers, members of a select group of geeks in the know.

Our methodology

by Nat Torkington | @gnat | comments: 7

Thomas Lord posted an interesting question in a comment on one of my recent posts: "I have a question about how the "Radar" works. Are you tracking Erlang? or following the broader trend around the pi calculus? Is Erlang interesting to you as a technological idea? Or as a particular product?". I realized we haven't talked much about what we actually do here, so I thought today I'd take the time to talk about that.

Dale and Tim have great noses for the future, and were right in the thick of things with the commercialization of Unix, popular uptake of the Internet, Open Source, Peer-to-Peer, and Web 2.0. Their methodology is pretty simple: in hacks, research, and startups by alpha-geeks we can often catch early glimpses of what will later be mainstream products or trends. So when Dale saw Pei Wei working on an X11 viewer for this thing called the "World Wide Web", he thought "everyone can use this". The O'Reilly Radar is an attempt to scale this beyond Tim and Dale.

So we see trends like Web 2.0, the growing need for concurrency, ubiquitous machine learning, and the importance of operations. We look to see what alpha geeks are doing in those spaces, find the bits that resonate, hold them up and say "this is what the future holds". We do this on the Radar blog, in Release 2.0, in research reports, in conferences like Velocity, and in the talks we give.

O'Reilly's business model is obviously predicated on this kind of future thinking—typical animal books take nine months to hit the shelves and it's hard to launch a conference with shorter lead time (unless you're Dave McClure!). You might think we'd keep the best ideas for ourselves and publish the rest, but we don't. We hope the rest of O'Reilly listens to what we say, but we don't run departments like conferences or books. As Marc Hedlund is fond of quoting, "Don't worry about people stealing your ideas. If your ideas are any good, you'll have to shove them down people's throats".

Our process isn't scientific research, where you come up with a hypothesis and then conduct experiments to disprove that hypothesis. What would an experiment to disprove the hypothesis "concurrency is moving from a niche to the mainstream but it's still largely an unsolved problem" look like? We try to quantify trends wherever we can, but at its heart this is an attempt to train and employ our instincts.

So Erlang and Haskell are interesting to us because we see alpha-geeks learning and playing with them and they have a "we make parallel code easier" story that fits with the trend we see of people struggling to figure out how to take advantage of multicore systems. We look for datapoints like "Amazon built SimpleDB in Erlang" that would confirm the hypothesis "Erlang can be used in the mainstream", and we also look for failures that might disprove that hypothesis. Such a failure might be "we build this in Erlang but couldn't keep a team together to run it, so had it rewritten in C++". In this mindset, Yahoo! Stores is a failure for Lisp and not a success (sorry, Paul!).

That's why we don't try to break every story. We leave that to our friends at ReadWriteWeb and TechCrunch. As Tim's said on many occasions, we "amplify the faint signals of the alpha-geeks." It's fun and we get to meet interesting people and think about the way things should be instead of the way they are. Hope that answers your question, Tom!

Sex, math, and scientific achievement

by Nat Torkington | @gnat | comments: 14

Scientific American have an interesting article about gender balance, bias, and abilities. The danger in talking about whether ability is sex-linked is that people want to simplify the science and your position down to "girls' brains can't do this stuff" but reality is more complex and inoffensive than this. (update: changed some of what I said about the distribution to reflect the comments)

I'm taking two things away from the Scientific American article: (1) on average, abilities have different distributions across the two sexes; and (2) although these tendencies are probably influenced by hormones, targeted training can lift skills. The distributions are the important bits here: mathematical ability in girls tends to be quite tightly clustered, whereas boys tend to divide into either are more extreme, including the very good and the very poor. The bimodal (correction: broader) distribution of boy math talent puts the lie to "boys are better at math", a misconception that came from the way we select the best at math. The very good boys have, on average, better skills than the tightly-clustered girls, so when we select "the best at math", we get mostly boys even though there are huge numbers of girls not very far below them and a huge reservoir of more unskilled boys (correction: than girls) at the bottom end of the distribution.

And we do select "the best at math"—the article talks about kids choosing disciplines based on what they're best at. In general, boys and girls look at their abilities and if they're better at numbers go into sciences and if they're better at words go into arts. So there are girls going into the arts that have better math skills than the boys going into sciences (the girls just happened to have even better verbal skills). This will always be true in individual cases, but the studies show this is an overall tendency rather than anecdotal evidence from specific cases.

What does this mean? I think it shows we need to do a better job of emphasizing that science and technology can be verbal as well as numerical: Larry Wall, the creator of Perl, is a linguist by training, and there's a similar elegance in great code as in great poetry. If we finally acknowledged that science and technology are fields where words are critical and a keen mind for meaning can go far, rather than pretending it's all math with syntactic sugar, we might get better computer programmers not to mention a better gender balance. And finally, first year classes should have catch-up skills-building options for those boys and girls who weren't at the top of the curve. Do readers know of computer science departments (or senior high schools) that test for specific aptitudes and offer remedial courses for those lacking? Drop me a note in the comments if you do.

The significance of Google's Android

by Nat Torkington | @gnat | comments: 2

My friend Rod Drury pointed to this great piece explaining the significance of Google's Android mobile platform. There are lots of quotable sound-bites, but I'll tease you with three:

1. [Android is] a platform for building and channeling inventory, much like a web browser
2. Every application on Android is a Web 2.0 citizen.
3. $10 million [Google's Android programming contest prize pool] exceeds the yearly marketing budget of most operating system vendors

Read the piece, it's fascinating!

What does Google's Open Handset Alliance announcement tell us about iPhone third-party apps?

by Marc Hedlund | comments: 32

I'm listening in on Google's press call about the Open Handset Alliance, which Google announced today. It's hilarious to hear all of the big wireless companies speaking about open platforms and software. Good for Google.

This announcement and the focus on open platforms make me think back to Apple's recent, seemingly rushed announcement that it will finally be supporting third-party apps on the iPhone. If Apple had made that announcement after Google made this one, it would have fallen very flat. By announcing beforehand, they were able to tell an "open platform" story while they still had the whole stage to themselves. Did Apple announce iPhone third-party apps as an aside in their "Hot News" column (instead of on Steve Jobs' home court, a conference keynote) in order to get the news out fast -- before Google?

It's interesting to note that Google and the Open Handset Alliance are starting out by shipping the platform first, and shipping phones with that platform on it a year later. Andy Rubin mentioned that an SDK will be available in one week (Apple won't have an SDK until February), and that it will be shipped with the Apache v2 license. Starting with developers -- what a great way to compete with Apple. Someone asked if a manufacturer could create a "completely locked-down Android device," and Andy Rubin responded, sure, the Apache license lets you do whatever you want, but Eric Schmidt chimed in, why would you bother? The point is having access to the applications. As he said later, "This is fundamentally a developer platform announcement."

It's also interesting to remember that Apple was rumored to be considering a Sidekick-like model of application delivery -- that is, all apps would need be downloaded through something like iTunes, that Apple would control, rather than being installed by the user directly, as on Palm OS. Rubin, co-founder of Sidekick maker Danger and now leader of the Google effort, must be rooting for them to make that mistake. It certainly didn't seem to make developers excited about the Sidekick. Maybe this competitive pressure will spur Apple and AT&T to give that up, if they were considering it; here's hoping.

All in all, very interesting. It's remarkable to see Apple once again in the position of selling a whole-stack platform (software and hardware, at least -- network sold separately), competing with a broad coalition of commodity hardware companies using a common software platform. I think they'll repeat history -- they are already repeating history -- by not doing whatever they can to bring developers to their platform. I wonder if Google will teach them what they should have already learned from Microsoft.

Infiltrating the privacy movement

by Marc Hedlund | comments: 5

I had a fantastic teacher in high school named Rick Takagaki, who once played a class of mine two speeches in a row: Martin Luther King Jr.'s "I Have a Dream," and Malcolm X's "Message to the Grassroots." The speeches, while both incredibly compelling, couldn't be more different (and certainly couldn't be more different from what passes for rhetoric today). "Grassroots" contains a famous passage in which Malcolm X derides the March on Washington (at which King gave his "Dream" speech) as a watered-down revolution, infiltrated and controlled by the white power structure:

It's just like when you've got some coffee that's too black, which means it's too strong. What you do? You integrate it with cream. You make it weak. If you pour too much cream in, you won't even know you ever had coffee. It used to be hot, it becomes cool. It used to be strong, it becomes weak. It used to wake you up; now, it'll put you to sleep.

While the topic is far less momentous, I always think of that quote when I read privacy stories like the one in today's New York Times: "Online Marketers Joining Internet Privacy Efforts." Marketers joining a privacy movement?

It's not like there's a privacy revolution in the United States; there never has been. But there are certainly a lot of "Chief Privacy Officers" whose Orwellian role seems to be spinning encroachments on privacy to look like a revolution of freedom. The closest the U.S. has gotten to a privacy uprising is the National Do Not Call Registry. According to a January, 2007 Times article, since its launch in 2003, "more than 137 million phone numbers have been placed on the list by people tired of interruptions during dinner or their favorite TV show." 137 million! The seeds of a movement are there, at least. While probably nothing else has risen to that level of response, news coverage of ChoicePoint, identity theft, and the like make privacy a popular topic of lip service -- but usually, unfortunately, little else.

The significant quote in today's Times piece comes in the fifth paragraph:

There is a silver lining for marketers, however: the AOL site will try to persuade people that they should choose to share some personal data in order to get pitches for products they might like. Most Web sites, including AOL, already collect data about users to send them specific ads — but AOL is choosing to become more open about the practice and will run advertisements about it in coming months.

I don't have a problem with AOL's effort -- it seems like a good development to me. Explicit labels are good. It's definitely interesting to see search engine providers competing on how comfortable they can make people with their practices. But this isn't a privacy effort. The goal here is to find tracking that consumers will accept.

Back to Malcolm, who warns that the only one who would resent his teachings would be "a wolf, who intends to make you his meal." Real privacy comes from removing tracking altogether, not adding small labels to it, festooning it with compliance badges, and providing an opt-out from it buried somewhere on a site. (I've put my money where my mouth is on this topic; see "Super Ninja Privacy Techniques for Web App Developers" in (IN)SECURE number 11 [PDF], pages 47-53, which I co-wrote with my colleague Brad Greenlee.) Marketers can't join a privacy movement without it being an infiltration; the headline of the Times piece is in itself an impossible contradiction. Having more options for types of marketing is fine, but don't mistake this wolf, or his intentions, for anything else. My old teacher, Takagaki, would never have let me get away with calling something what it isn't, and this isn't privacy. It's marketing.

A Visit to Vanuatu

by Nat Torkington | @gnat | comments: 4

My friend Skud recently went to Vanuatu, and I caught up with her as she passed through Auckland on her way to SF. Vanuatu is a Micronesian nation with no real industry other than tourism. She has a Flickr set of photos of a "solidly middle class" family that'll blow your mind if you've never been to a developing nation (her commentary on those photos provides some context).

She's been organizing technical books for Vanuatu, helping the Vanuatu IT User Society build out their library. She's got some amazing stories about the kids, and also the adults who have potential and curiosity and the same hacker sense we see in ourselves but don't have instruction or even a computer of their own to build it out on. She's also helping them to get OLPCs, not trivial because Vanuatu is so small it doesn't rate on the scale against Africa.

I had a great afternoon with Skud, and was really inspired by what she's doing. O'Reilly has helped similar projects in the past, getting books in the hands of those who need them and we're going to send VITUS a box of books per Skud's request. It was uplifting to see Skud's passionate and pragmatic approach to helping the people she'd been staying with. It does us good to remember that the Internet can be used to connect people in more ways than Facebook friending.

The Faint Signals of Concurrency

by Nat Torkington | @gnat | comments: 18

We've been looking at the world of concurrent programming lately. You might have seen Tim's posts on Erlang and Haskell, or my post on alternative systems to threading. Here, in a nutshell, is why we're interested in this stuff and what we see.

1. CPUs aren't really getting faster. The CPUs in your brand new machine are the same speed they were two years ago. Moore's Law has expired. Now all the manufacturers can do is stuff more CPUs into your machine. Sure, someday there'll be a breakthrough in the speed of light or subnano manufacturing or we'll finally get a 64-qubit Intel Quarkium, but until then we're stuck with more slow CPUs in each new box. The only way to get faster execution is to parallelize your code.
2. Google's been doing it. An 8-CPU machine is analogous to an 8-machine cluster. Google have been building clusters with hundreds of thousands of machines because their industry has an embarrassment of parallel problems to solve. We're big believers in watching the alpha geeks, and it's harder to get more alpha or more geekier than Google. For this reason we're watching MapReduce and Hadoop. Other early leaders in this area are the big data sciences (astronomy, genomics/proteomics, nuclear physics) each with different problems (image processing, sequence alignment, 3d alignment, detailed simulation).
3. Barrier to entry is coming down. Your Playstation 3 can kick ass at protein folding, and for $13 you can buy the Parallax Propeller, a chip with 8 cores in it to slap into a board and play. And, of course, off-the-shelf commodity PC hardware has never been cheaper. Hell, your video card GPU has a ton of processing elements, although everyone has their own interface to the goodies.
4. It's a hard problem. If it were easy then it wouldn't be a problem. Computer science has been busy for years developing techniques for clusters and for multicore machines, figuring out how to distribute computation across multiple processors.
5. The alpha geeks have their hands in it. Before you needed a high-budget lab to work on these theoretical problems. Now you can buy your lab at Wal-Mart, the problems being solved are your problems, and the programming environment is familiar (Perl, Ruby, C) or worth learning (Erlang, Haskell's STM). When Audrey Tang went to develop a Perl 6 interpreter, she did it in Haskell. The Pragmatic Programmers are releasing an Erlang book and we're doing one on Haskell.
6. Did I mention it's hard? The mainstream systems for concurrency are baby talk, coding with crayons and fingerpaint. Threading is a nightmare for those who use it, and if 10% of programmers can really code well without bugs then only 1% of those can do so in a threaded environment. This pain (and the alternatives to threading that people are investigating) is on our radar.

My friend Bryan O'Sullivan is one of the alpha-geeks we're watching (he's co-author on our Haskell book). I pinged him about concurrency and he had this to say:

What's going to really fuck people up, out in the real world, is the programming model. Programming is hard; parallel programming is way the hell harder; compsci courses have turned into votech Java pap; and enrollments in compsci are in any case as lively as the waiting list for the Lusitania the week after it was torpedoed. People want their programming to be easier and more casual, and they're about to have it jammed into their eyesockets on bamboo stakes instead.

I wrote this post in March but for some reason failed to post it. In light of recent events it still seems timely. I'd love to know what you think. Is concurrency on your radar? Are you perfectly happy writing threaded code? Are you perfectly happy writing code without any concurrency whatsoever? Tell Unca Nat all ...

Listen to O'Reilly Radar

by Tim O'Reilly | @timoreilly | comments: 12

Years ago, I remember a fascinating alpha-geek moment. I had organized a peer-to-peer summit, and one of the people I'd invited was Kevin Lenzo, one of the founders of Festvox, the open source speech synthesis project and Cepstral, a speech synthesis company. One of the other attendees said to him, "Your voice sounds familiar." When Kevin told him who he was, the developer said, "Oh, that's it. I listen to your voice all the time when I pipe IRC channels to festvox so I can listen to them while I code." (Kevin's voice was the basis for one of Cepstral's synthetic voices.)

Well, Allen Noren, who has had an interest in having our books and sites read aloud for years, has just rolled out speech synthesis on our sites and blogs via a partnership with a Swedish company called ReadSpeaker.

Allen talks more about the program on the O'Reilly FYI blog (a good place to keep up on O'Reilly product- and company-related news). But all you really need to know is pretty obvious: click on the "listen" button next to this entry, and a synthetic voice will read it to you. We've just rolled out this feature on Radar. Let us know what you think.

(P.S. A small note on Kevin. He was one of the inspirations for my original idea of the "alpha geek." I kept running into him on so many interesting tracks to the future, that anything Kevin was interested in, I figured was worth paying attention to. In addition to being a pioneer in speech synthesis, he was the creator of a fascinating "infobot" that participated in the Perl IRC channel, one of the first people I heard about climbing on rooftops to set up homebrew WiFi antennas, and founder of the self-organized Yet Another Perl Conference... When I saw that Allen had put up this project with ReadSpeaker, I wondered why we'd never done it with Kevin. A missed opportunity from many years ago.)

Don't Miss Nat's Last Post

by Tim O'Reilly | @timoreilly | comments: 3

If you didn't bother to read below the fold on Nat's last post, go back and do so now. You'll be rolling on the floor laughing.

Jedi build their own lightsabers

by Marc Hedlund | comments: 11

I was down at Stanford recently with Adam, and sat in on one of the classes he's taking there. Later on, I looked around at some of the resources Stanford makes available on the web. They provide a lot of fantastic material for free. One thing I found was this video series of Larry Page and Eric Schmidt from Google, speaking at Stanford in 2002. I'd highly recommend it for entrepreneurs. (Each video is a few minutes long, and the whole set is about an hour.)

Larry's segment on tips for entrepreneurs, for instance, is fantastic. As he says, the advice is counter to a lot of what you'd hear as "standard wisdom" about technology startups. His point about the value of solving hard problems is one of the most difficult things for me to get through to people who take my Entrepreneuring for Geeks tutorial. A lot of investors and advisors will tell entrepreneurs to do the least amount of work possible, so that they can get going quickly. You can get going more quickly this way, but then you're stuck trying to build a real business, instead of just a thin UI on top of someone else's business. If you're making all your money off of AdSense (and you're not Google!), who really owns the relationship with your customers? Most people don't think this way, and they should.

Joel Spolsky has a great article on this called, "In Defense of Not-Invented-Here Syndrome." He talks about the Excel team making their own compiler and the benefits they got from that. He proposes a great test for making decisions like this:

If it's a core business function -- do it yourself, no matter what.

I've always referred to this idea, geekily, as, "Jedi build their own lightsabers." If you're going to depend on your lightsaber as your principal tool and weapon, you'd better know that it works.

We faced this decision at Wesabe when it came to syncing data from banks and credit cards. There are a couple of existing businesses that will do that for you, but when we looked at them, we realized that there were huge problems with going that route. First, we'd have to completely give up any control over our users' privacy, since those companies would need to hold all of our users' bank and credit card passwords, and they insisted on keeping a full copy of our users' transaction data. That wasn't compatible with our Data Bill of Rights, which we hear positive responses to every day on our support line. Having control ourselves leads directly to being able to provide control to our users, and that's what our users want. Second, syncing transaction data is central to our business. Let's say that we started to become successful -- if we had a single-source provider for that data, that would mean they would have full price control over us, and would get the full benefit of any value we created. It would be like installing a black hole at the bottom of our own bank account. Finally, we would have to pass that cost onto our users in one form or another -- either by charging them a high price, as some of our competitors do, or by selling their data to advertisers and marketers, which would completely corrupt our vision, to help people get control of their money.

So, we built our own transaction syncing infrastructure, and we did it our own way. We can provide those services to our users for free (in both senses -- for no money, and with freedom to control their own data), and with much higher privacy protection -- by giving people a client program that keeps their bank username and password on their own computer. Since we started with that as the first option, we now can extend to other syncing formats, including ones that are more convenient than installing a client download. People who took the easy route will be left with only one option for how to sync data, while we will have greater range of motion.

As Larry says, this isn't the common perspective, and others will surely disagree in the comments. But hearing Larry and Eric speak about this is great, and if you're interested in building a business, I think their talk is a fantastic resource.

The Disappointment of John von Neumann's Unopened Box

by Tim O'Reilly | @timoreilly | comments: 1

I saw my friend George Dyson last night. He's working on a book based on the letters between von Neumann and his wife Klara, one of the first modern programmers. I'd heard from George about a box that was among von Neumann's papers, with a note that it should not be opened till 50 years after his death.

We were all awaiting news of the contents of the box -- and it turns out not to have been von Neumann's at all, but rather, from Klara's next husband! Schroedinger's cat was nowhere to be found.

(P.S. The Mad Times blog has a summary of one of George Dyson's recent talks on John von Neumann. There's also lots of coverage elsewhere of George's talks at OScon about the early history of computing at Princeton. And if you haven't read it yet, George's essay about Google, Turing, and von Neumann, Turing's Cathedral, is one of the most thought-provoking documents you'll read in many a year.)

Radar Redesign and New Features

by Tim O'Reilly | @timoreilly | comments: 36

OK, how do you like it? (P.S. Be sure to refresh your browser. Shift-reload.)

We've been noodling for a while on a redesign of Radar. Our initial design was thrown together in a weekend, and we've lived with it happily for a couple of years. But it really hasn't given us the framework we need to pull together all the related activities of our group. You see, Radar isn't just this blog, it's a whole group at O'Reilly focused on the premise that "the future is here, it's just not evenly distributed yet."

Our radar bloggers are also typically our conference chairs; they help bring leads and do additional diligence for our venture fund. Our group organizes foo camp and now publishes the Release 2.0 Newsletter. We have built a technology trend data warehouse and publish research reports.

Part of the goal for the site redesign was to create more room to feature all these activities, and bring them together into a single framework. Kudos to Brady Forrest, who led the effort, George Humphries and Matthew Woodruff who created the new design, and Gabriel Williams, who did the hard work of actually making a design into a working site.

Speaking of new features -- the first issue of Release 2.0 is now available -- and we're finally set up to take new subscribers. It's a bit of a nail-biter to take over a publication like Release 1.0 from someone as respected as Esther Dyson. We hope you like our take on the mission of the newsletter. This first issue is a bit of an introduction to our editorial philosophy as much as it's a real working issue. As I wrote in the press release:

Our blog, the O'Reilly Radar, focuses on the disruptive innovations coming from "alpha geeks" and other early adopters. Our newsletter, Release 2.0, will focus on the business impact of those innovations.

So many of the innovations we've been deeply involved with over the years -- from the commercialization of the internet to open soruce software and most recently Web 2.0 -- have gone from being the playground of enthusiasts to the workhorse of big business. In the past, we've tended to abandon these technologies as they moved toward the mainstream. With Release 2.0, we're hoping to continue to provide insightful thinking about the future to entrepreneurs, investors, and corporate technology strategists.

O'Reilly AlphaTech Ventures

by Tim O'Reilly | @timoreilly | comments: 8

It's been leaking out around the edges that O'Reilly has gotten into the venture business, but by SEC rules, we weren't allowed to say so publicly until we had closed our fundraising. We did a first close late last year, and have already made four investments, but needed to stay mum until the final close. But now I can say it formally: we've launched a new venture fund called O'Reilly AlphaTech Ventures.

O'Reilly has always done a bit of opportunistic venture investing, and actually had a small internal venture fund called O'Reilly Ventures. Some of our most successful investments over the years included GNN, sold to AOL, Likeminds, sold to Andromedia (which was sold in turn to Macromedia), Pyra (blogger.com), sold to Google, and ActiveState, sold to Sophos. We're also investors in Collab.Net, Intalio, Sendmail, and Zope. This is the first time, though, that we've raised money from outside investors -- a total of $51 million. Mark Jacobsen and Bryce Roberts are the managing partners. Our first three investments include Instructables, Chumby, and Wesabe.

We're focused on the same kinds of opportunities we've always seen at O'Reilly: potentially disruptive technologies coming from outside the mainstream, often from enthusiasts who only later realize that they've created a new industry. Our company mission is "changing the world by spreading the knowledge of innovators." We do that through books, magazines, online publishing and conferences, but sometimes, the right way to do it is to fund an entrepreneur to build something new and earthshaking.

If you have world changing ideas that you think can be turned into great companies, we'd love to hear from you. You can write to us here at Radar, or contact OATV directly.

P.S. We will always disclose when we've made an investment, and when we talk about the companies we're invested in. But even more importantly, just as we've always done, we try to keep a church/state line between our technology activism and our investing or other business relationships. We're far more interested in growing the whole of a new market than in giving particular advantage to any one company we're affiliated with. Of course, we invest in companies because we think they're doing something cool and important that will make them successful, and so we're likely to talk about them, write about them, and feature them in conferences -- but we'll do the same for competitors that are just as interesting.

2008 Presidential Technology Race: Who's Using What?

by Tim O'Reilly | @timoreilly | comments: 4

Tim O'Brien wrote:

"The web site is the new campaign handshake, and I was curious to see what technologies were powering candidate web sites. Whoever puts these sites together has to choose a site that can be developed quickly and can handle intense bursts of traffic, so while you might think this is irrelevant, it tells you a little bit about what technology that the experts trust. And, it is also interesting, because this is one of the only statements that a candidate makes without even knowing about it. McCain doesn't know ASP.NET from Dodd's J2EE, but to us it's interesting. Read on for a table of results..."

Among the headlines:

• "Linux and Apache are Presidential Material"
• "Urchin for President in 2008"
• "Who's Hip to Video?"
• "Waiting for a Rails candidate?"

And Tim added in email: "And, this weekend, I cooked up this basic Rails mashup: presidentfeed.com. I'm adding the ability to login and "cast your vote". Was going to have a page that tracked the technology of each of those sites similar to the previous post (which is a little out dated... these politicians are fast moving technologists - especially Obama)"

Early History of the Web Pipes Concept

by Tim O'Reilly | @timoreilly | comments: 6

A few days ago, I gave credit to Jon Udell for the talk at the Perl Conference that first introduced me to the idea of pipes and filters for the web. One reader, Ed, asked in the comments whether it wasn't Andrew Schulman who gave that talk?

After a bit of digging down memory lane with Jon and Andrew, it's still not clear to me whether it was Jon or Andrew who gave the talk I was thinking of, and I should certainly have given Andrew some credit, because I'm pretty sure it was he who brought Jon's ideas to my attention. They both spoke at the conference, and likely on related topics, though there is only a record of Andrew's talk.

However, to complicate things further, Andrew was talking about ideas that started with Jon and some of the hacks he'd been writing about in his Byte column. And being precise, it wasn't first at the conference that I heard about those ideas. Since I was the one who put together the program for the conference, I invited both Andrew and Jon to speak based on what I already knew about their ideas.

In fact, Andrew reminded me that he had proposed a book on the new web programming model, and dug up some outline files from April of 1997. (The conference was in July, if I recall.) And the outline and notes that he dug up indicate some amount of discussion with both me and Jon by that time.

For those of you who are interested in the details of the history, and just how forward thinking both Jon and Andrew were, read on. (I've already posted their memories in the comment thread on the original post, but thought that it was worth giving them both a little more visibility.)

As a side note, it really is amazing how easily we forget the details of the past, and how important it is for future history for us to keep our notes. It gives real perspective on more distant history when you realize how hard it is to remember the sequence of events, and who influenced whom, for something that happened only ten years ago....

But it's also great how ideas seem obvious in retrospect. (And how ideas that seem obvious to the alpha geeks can take years or decades to become "common sense.") Jon wrote:

But really, the whole web pipeline idea was (I assume) completely obvious to the Roy Fieldings of the world who created HTTP and the toolkits like libwww. So obvious that they didn't bother to articulate it, which is all that I did.

(Jon articulated this idea more fully in a blog post this morning entitled Annotate the web, then rewire it.) And Andrew wrote:

Yes, the "web as an API", "web sites are subroutines", etc. theme all did seem pretty obvious at the time, and today even more so. In fact, the idea of a self-contained PC looks more and more like a detour from the main stream of computing. A necessary one, probably (had to get several hundred million boxes out there to bring down costs?), but a detour nonetheless.

Read on for Jon and Andrew's complete notes.

(continue reading)

From Walled Garden to Green Fields

by Nat Torkington | @gnat | comments: 3

Before Foo Camp last year, the Radar group had a meeting to share and find common threads in the trends that we'd individually been noticing. I pointed out that one of the characteristics of things that interest us is that they remove artificial barriers to behaviour (we like open source, we don't like DRM, we have a lot of problems with patents, we even grumble about copyright law) and I referred to this as "breaking down the walled garden". I was just browsing this year's Emerging Telephony (ETel) conference and I'm delighted to see they're running with that theme.

I can't make it (I'll be in Bangalore, India), but the sessions that I'll be looking for on podcasts or blog writeups are:

• Kitchen Table Alienware: Software Radio for the Hardware Hacker by Matt Ettus, of GNU Radio fame. Matt's Foo Camp session in 2006 was packed full of O'Reilly Radarites, and Quinn Norton (whose antennae quiver at cool technology) profiled him for Wired News. Matt is building the equivalent of gcc for radio, the tool that will put users in charge of spectrum again. Fantastically disruptive, incredibly powerful, and eagerly anticipated by those in the know. He'll make the FCC redundant and give us infinite spectrum. He's like Santa Claus with a Linux box and aerial.
• Searching Calls: Indexing, Searching, and Retrieving Recorded Speech. If you can record all your calls, then what? Finding a specific call is a needle-in-a-haystack situation. This session will give us the tools for a GMail for our voicemail. There's no walled garden broken here, just some sorely needed technology that we aren't getting from the telcos and so will have to invent ourselves. And on another Foo note, Kevin Lenzo of Cepstral is a long-time Interesting Person: he started the Perl Foundation, and wrote one of the first infobots on IRC.
• Asterisk Hard Hat Session: Deep Hacks and Shortcuts. (1) Jim and the Brians know Asterisk inside-out. (2) Asterisk is by far the most popular piece of open source voice software. (3) Who doesn't like deep hacks and shortcuts? Today's hacks grow into tomorrow killer apps. Asterisk lets companies escape the walled garden of their PBX provider.
• Blackbag VoIP Security Briefings. As Spiderman said, with great power comes great responsibility. Telcos are like a police state, but if we overthrow the regime then we better make sure it's not replaced with anarchy and lawlessness. Nobody will leave their walled garden if it means entering a world of the VoIP equivalents of spyware, spam, and phishing attacks. I'm curious to learn how vulnerable the tools we use are, and what can be done to make our systems more secure.
• Communication Product Design Workshop. I hate my phone with a passion, because it feels like the person who designed it hated me. They made it difficult to use, and slightly different in behaviour from all the other phones I've had. They made the battery case easy to lose, the charger incompatible with every other device, and the ring volume easily raised by unlockable keys while it bounces in my pocket. I look forward to someone more articulate and learned than myself ripping the morons who built it a new one. I feel like my phone was designed in Communist Russia by nine committees on a fifteen year plan, and I would expect usability would be an immediate dimension of differentiation were the carrier lock on handsets released.
• MSR's SMS Server for Rural India. Not only is it Sean Blagesvedt (Head of Program Management and Advanced Prototyping for Microsoft Research India) whom I've wanted to meet for a long time, but he's built something that lowers the barriers to technology world-wide. Oh that's right, I'm not going to meet him--he'll have flown from India to USA about the same time I fly from NZ to India. Bugger.
• Empowering People and the Coming Identity Layer of Everything. Phones are strongly intertwined with identity (GMail wanted a phone number before it'd let you sign up, Caller ID, even personalizing your phone with ringtones and desktop images), and for voice apps to progress into the truly convenient world we'll need to solve several pressing identity problems. Fortunately, the rest of the Internet is working on them too, and Kaliya will talk about the work being done on identity systems that serve us rather than the carriers. OpenID, for example, is distributed and doesn't privilege the builders of walled gardens.
• Tapping in to the Open Source for Innovation. TrollTech has real experience with Linux-based mobile phones. I wonder whether energy consumption or memory needs proved a problem. Open source handset software will lower the barrier to entry, the same way that the LAMP stack lowered the barrier to building a web site. Everyone built their useful differentiated software on top of the commodity open source. It is so time for this to happen on handsets.
• Linux Wi-Fi Telephony Handset - Fonav. No link for this one yet, but it looks interesting. Phil Torrone's been playing with wifi Skype handsets for a while now, but the twin combination of (a) open source programmable/extensible, and (b) open protocols make it sound very enticing. Open protocols are the "get out of the walled garden free" card of telephony.
• MySpace Mobile: Portable Communities Empowering Users. Given how addicted kids are to MySpace, it seems like MySpace Mobile would be the tech equivalent of a portable meth lab. This will be one of the first public reports on the uptake. I'm intensely curious. And yes, okay, it's a walled garden in a walled garden. But what fun would we be if we were purely ideological?
• Wifi and Low Cost Communication Networks in Kenya. One of the highlights of last year's conference was Brian Capouch talking about his rural networking in America. I imagine Quentin (this year's presenter) saying "oh, so you think you had it tough!". Oh, and Quentin invented the webcam. And VNC.
• Tribox - VoIP Hacking at Home. Brian Aker is a genius, an evil evil genius. He's the man who hacked his Asterisk system to remember telemarketers and answer their subsequent calls with loud monkey screeches. I want to know what he's been doing lately.

A Microsoft Timeline via Tags

by Brady Forrest | @brady | comments: 1

Tag clouds are a great way to get a quick summary of the concepts in a piece of text and their relative importance. When combined with a timeline and a set of chronologically-ordered content you can quickly see how ideas morphed overtime. This is a great way to track the changes of a company, industry or even a country - heck it would be interesting to see my Del.icio.us and Flickr tags this way.

Todd Bishop of the Seattle-Pi's Microsoft blog has done just this for a look at Microsoft's history. He compiled a set of 90 Microsoft memos, emails, book introductions, advertisements, and speeches. He then used Chirag Mehta's open-sourced and CC-licensed Tagline Generator to analyze and display them. The result is an interesting peek into the morphing of Microsoft's (and the tech industry's) goals and challenges (for another interesting application check out Chirag's US Presidential Speeches Tag Cloud Timeline).

It's interesting to watch the rise of the internet, search, google, and linux via the tag cloud. There's also a marked fall in the focus on the PC over the years.

Update: In the comments, Brent Fitzgerald posts about a Del.icio.us tag-timeline generator he recently wrote.

Hi, what a coincidence. I actually wrote a little timeline webapp called Yummy for my del.icio.us tags last week. You can view your own tags as well. It uses a double slider to allow adjustment of the date range.
Check it out:http://plw.media.mit.edu/people/brent/yummy/

Also see the blog post and rough screencast.