Four short links: 24 June 2009

Open Source Kids, Crowdsourcing Lessons, Flickr Secrets, Hadoop Spatial Joins

1. The Digital Open -- The Digital Open is an online technology community and competition for youth around the world, age 17 and under. Building a community of young open source hackers.
2. Four Crowdsoucing Lessons from the Guardian's Spectacular Expenses Scandal Experiment -- Your workers are unpaid, so make it fun. How to lure them? By making it feel like a game. "Any time that you’re trying to get people to give you stuff, to do stuff for you, the most important thing is that people know that what they’re doing is having an effect," Willison said. "It’s kind of a fundamental tenet of social software. … If you’re not giving people the ‘I rock’ vibe, you’re not getting people to stick around." (via migurski on delicious)
3. 10+ Deploys/Day: Dev & Ops Cooperation at Flickr -- John Allspaw and Paul Hammond's talk from Velocity. You tell any mainstream company in the world "10 deploys/day" and you'll be met with disbelief.
4. Reproducing Spatial Joins using Hadoop and EC2 -- bit by bit the techniques for emulating important operations from trad databases are being discovered and shared in the new database scene. (via straup on delicious)

Four short links: 14 May 2009

Open Source Ebook Reader, Libraries and Ebooks, Life Lessons, and Government Licenses

1. Open Library Book Reader -- the page-turning book reader software that the Internet Archive uses is open source. One of the reasons library scanning programs are ineffective is that they try to build new viewing software for each scan-a-bundle-of-books project they get funding for.
2. Should Libraries Have eBooks? -- blog post from an electronic publisher made nervous by the potential for libraries to lend unlimited "copies" of an electronic work simultaneously. He suggests turning libraries into bookstores, compensating publishers for each loan (interestingly, some of the first circulating libraries were established by publishers and booksellers precisely to have a rental trade). I'm wary of the effort to profit from every use of a work, though. I'd rather see libraries limit simultaneous access to in-copyright materials if there's no negotiated license opening access to more. Unlike the author, I don't see this as a situation that justifies DRM, whose poison extends past the term of copyright. (via Paul Reynolds)
3. Lessons Learned from Previous Employment (Adam Shand) -- great summary of what he learned in the different jobs he's had over the years. Sample:
   • More than any other single thing, being successful at something means not giving up.
   • Everything takes longer than you expect. Lots longer.
   • In a volunteer based non-profit people don't have the shared goal of making money. Instead every single person has their own personal agenda to pursue.
   • Unfortunately "dreaming big" is more fun and less work than "doing big".
4. Flickr Creates New License for White House Photos (Wired) -- photos from the White House photographer were originally CC-licensed (yay, a step forward) but when it was pointed out that as government-produced information those photos weren't allowed to be copyright, the White House relicensed as "United States Government Work". Flickr had to add the category, which differs from "No Known Copyright", and it's something that all sharing sites will need to consider if they are going to offer their service to the Government.

Four short links: 20 Mar 2009

Space, Space, Micromanufacturing, and Sensors:

1. Teens Capture Images of Space With £56 Camera and Balloon (Telegraph) -- DIY/MAKE culture at its best, four 18-19 year old Spanish students (with guidance of a teacher) rigged a balloon to carry a camera over 100,000 feet (that's twelve trillion and seven Canadian meters) above the earth, take pictures, and return to the ground. Here's their project's web page with a Google gadget to translate it into English. (via @erikapearson)
2. The Robot Who Helps Astronomers Identify Stars - IO9 interviewed Fiona Romeo, about the Royal Observatory's Astronomical Photograph of the Year contest and the astrotagging bot I linked to earlier.
3. Clive Thompson on the Revolution in Micromanufacturing -- talks about his experiences with Etsy. I was aware of the site but had dismissed it as some sort of urban-hipster thing—until I started seeing chatter about it on discussion boards for wealthy professionals and stay-at-home moms.
4. How The FitBit Algorithms Work -- The Fitbit’s primary method of collecting data is an accelerometer. Its accelerometer constantly measures the acceleration of your body and algorithms convert this raw data into useful information about your daily life, such as calories burned, steps, distance and sleep quality. How do we develop these algorithms? Our approach is that we have test subjects wear the Fitbit while also wearing a device that produces a “truth” value. [...].

Four short links: 19 Feb 2009

Art, astronomy and more fun for you in today's four short links:

1. Found in Space -- there's an astronomy bot on Flickr that identifies stars in the night sky, and from the unique positions of the stars figures out what bit of the night sky is looked at and then adds notes for interesting parts of the sky visible in the shot. A brilliant use of computer vision techniques to add value to existing data. (via Stinky).
2. 99 Secrets Twittered -- Matt Webb is posting a secret a day from Carl Steadman's 99 Secrets, an early piece of art on the web. Matt's explanation is worth reading. Ze Frank really made me realize that every web app is a medium for art, for provoking human responses, and now I keenly watch for signs of art breaking out.
3. Internet Ephemera -- a brief muse on "if we start with the assumption that everything we put online is ephemeral, how does that change what we put online?"
4. Pockets of Potential (PDF) -- a 52-page PDF talking about opportunities for supporting learning with the mobile devices already in kids' lives (via Derek Wenmoth).

Understanding Web Operations Culture - the Graph & Data Obsession

We’re quite addicted to data pr0n here at Flickr. We’ve got graphs for pretty much everything, and add graphs all of the time.

-John Allspaw, Operations Engineering Manager at Flickr & author of The Art of Capacity Planning

One of the most interesting parts of running a large website is watching the effects of unrelated events affecting user traffic in aggregate. Web traffic is something that companies typically keep very secret, and often the only time engineers can talk about it is late at night, at a bar, and very much off the record.

There are many good reasons for keeping this kind of information confidential, particularly for publicly traded companies with complicated disclosure requirements. There are also downsides, the biggest being that is difficult for peers to learn from each other and compare notes.

John Allspaw recently created a WebOps Visualizations group on Flickr for sharing these kinds of graphs with the confidential information removed. Here’s an example of a traffic drop seen both by Flickr & by Last.FM that coincided with President Obama’s inauguration.

Similar traffic drop on Last.FM seen on the right

Google saw a similar drop as well

Was it because everybody went to Twitter?

Besides being an interesting story, sharing these kinds of graphs help people build better monitoring tools and processes. As just one example: How should the WebOps team respond to this dip in traffic? Is it an outage? The inaguration was a very well known event and so it’s easy to explain the drop in traffic… what happens when a similar drop in traffic occurs? Should the WebOps team be looking at CNN (or trends in twitter) along with everything else?

How do you tell when that unexpected 10% drop in traffic is really just people with something more important to do than browse your site?

(Note: Updated since original posting to add Google & Twitter graphs and annotations, and to switch the Last.FM graphic with an annotated one after I got permission.)

Four short links: 2 Feb 2009

1. Songs off the Charts -- Johannes Kreidler's audio visualizations using Microsoft Songsmith. Reminds me of Douglas Adams in Dirk Gently's Holistic Detective Agency where the amazing spreadsheet program could produce happy jingles or funereal dirges based on a company's revenues. (via Ben Fry)
2. PWN! YouTube -- elegant URL hack: replace "www." with "pwn" in a YouTube movie URL and you'll be given links to the Google content server location of the movie so you can download it.
3. Apple iPhone and Microsoft Surface -- the interesting folks at Stimulant have written the code to connect an iPhone to a Microsoft Surface. It recognizes one or more iPhones on the Surface and lets you display different things on the iPhone. In the demo you see an iPhone on a photo showing you a sketch version of the subject of the photo. The zoom is very smooth.
4. Flickr, Getty, and the Greater Good (Phil Gyford) -- "Flickr and Getty Images, the stock photography giant, are launching a new scheme which enables people to market some of their Flickr photos as stock photography through Getty." Phil points out that CC-licensing and Getty-listing are mutually exclusive, and Flickr will switch the licensing on a photo to "All Rights Reserved" if you list with Getty. The first way people think of to profit from commons are to enclose and sell them. But the commons are a lot healthier when you make money by adding to them, not taking from them.

Four short links: 15 Jan 2009

Today we have Tom's Brain on Flickr, the Newspaper Industry's Death in Context, REST with friends, and Filthy Lucre from Twitter.

1. A map of my brain - Tom Coates mindmaps his interests as part of brainstorming for a Webstock talk. I'd love to see these for other geeks. I guess part of keeping up with your friends is building your own models of what their mindmaps look like.
2. How Newspapers Tried to Invent the Web - a Slate piece showing that newspapers have always reacted to new media, from buying up airways when radio became big, to videotex and the proprietary information sources that predated the mainstreaming of the Internet. Puts the modern handwringing in context. Sample quote from 1980: "What you're really worried about is an electronic Yellow Pages that will destroy your advertising base, isn't it?"
3. RESTful Django Practice - the full RESTian mindset is surprisingly difficult to grok, so one programmer has said "here's how I think it works for a sample app" and his readers are using the comments to describe the choices, drawbacks, pitfalls, and best practices. I love the Internet.
4. 1stfans - a mixture of social networking updates, Twitter feeds from artists, and in-person events that the museum hopes will be a good excuse for people to become members and support the museum's operation. Twitter Feed as Membership Benefit goes into detail on their reasons. It's always interesting to see people experimenting with finding things to charge for online.

Flickr Community Fills Gap

In the recent round of Yahoo! layoffs was someone I'd just met, George Oates. She started the Flickr Commons, where galleries, libraries, archives, and museums can post photos and the community can tag them. She was a tireless ambassador, as well, with a gruelling travel schedule to bring the word to other institutions on what's possible. Her blog post about how she found out about being laid off is moving.

Still shocked. So fucking brutal.

Within the sadness of that, however, comes some good news. Flickr group and George is taking heart from it.

I'm reminded of Google's philosophy around groups. They were so happy with the way that the Google Maps community rallied to support apps developers, they've rolled out a similar "help-less" support system for their other developer products. The thinking is: if we hold their hands then the work for us will grow faster than we can meet it, so it's better to stand back even at launch and let the community form and do the work. From other experiences I've had (Google Docs, I'm looking at you) this community support is sketchy at best, and Google may well have learned the magic ingredient that creates a vibrant helpful community as opposed to a question-filled answer-free ghetto. With the passion for Flickr Commons that I saw at National Digital Forum last month, I'm sure the community's on the right track.

(hint on the magic ingredient: Maps apps were public works, so the reward for helping was seeing another cool app appear. Google Docs are generally private, so there's rarely any direct reward for helping someone with their DNS configuration or email client setup. Flickr Commons: very public)