Four short links: 5 November 2009

Heat Maps in R, EC2 Blackhat Tricks, Snickersome Unicode, and Decoding Statistics

1. Heat Maps in R -- We used financial data here because it's easier to access than the airline data, but it's actually a pretty interesting way of looking at a financial time series. Weekend and holiday effects are a bit more obvious, and it's a bit like being able to see the daily, weekly, monthly and yearly closes all at once (by scanning your eye over the calendar in different directions). Includes source code. (via migurski on Delicious)
2. BlackHat and EC2 -- Theft of resources is the red-headed step-child of attack classes and doesn't get much attention, but on cloud platforms where resources are shared amongst many users these attacks can have a very real impact. With this in mind, we wanted to show how EC2 was vulnerable to a number of resource theft attacks and the videos below demonstrate three separate attacks against EC2 that permit an attacker to boot up massive numbers of machines, steal computing time/bandwidth from other users and steal paid-for AMIs. (via straup on Delicious)
3. Funny Characters in Unicode -- I never get tired of the wacky stuff in Unicode. I love the thought of a Unicode committee somewhere arguing passionately about the number of buttons on the snowman .... (via Hacker News)
4. Statistics to English Translation -- The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate. This matters. Bandying around numbers with misleading labels, or misinterpreting numbers that have a precise and defined meaning, does not further understanding. (Said 78.4% of statisticians, with a 20% confidence factor probability of false positives)

Four short links: 12 October 2009

DSL for NLP Task, Insider Tradespotting, Outsource Fail, Cloud Fail

1. Snowball -- a small string processing language designed for creating stemming algorithms for use in Information Retrieval. (via straup on delicious)
2. Insider Trades -- a Yahoo! Hack Day app that turned out to be worth continuing. Scans SEC systems every 30 seconds and alerts you if the stock you track has been traded by an insider. (via straup on delicious)
3. Air New Zealand Slams IBM -- central point of failure in the outsourced IT. "In my 30-year working career, I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client's customers is not the glowing endorsement you want.
4. Danger/Microsoft Loses Sidekick Customers' Data -- Regrettably, based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. This cloud had a brown lining.

Four short links: 5 October 2009

Bozo Cloud Talk, Annotation Fail(ish), Python MySQL Slash, and Infinite Books

1. Brown Cloud Marketing -- advertorial "interviewing" GM of a company offering "DNS in the cloud". This might be a worthwhile service, but the way he markets it (by saying open source is "freeware" and the market leader is "legacy") reveals a rich vein of bozo. Freeware legacy DNS is the internet's dirty little secret (actually, it's the reason we have a functioning DNS), Nominum software was written 100 percent from the ground up, and by having software with source code that is not open for everybody to look at, it is inherently more secure. (security through obscurity is equating clothing with being naked yet blind). The Internet kindly did the poor man's homework: screenshot of a cross-site scripting vulnerability in their customer portal, a Nominum security advisory from 2008, and the Nominum web server is running Linux, Apache, and PHP (all legacy freeware yet apparently not the Internet's dirty little secret). (via Bert Hubert and Securosis)
2. Public Annotations on Healthcare Bill -- using technology from SharedBook, Congressman Culberson hoped to get citizens marking up the healthcare bill. They're using the software but many are just commenting on page 1--turning the hosted annotation platform into a forum with an odd user interface. It's a UI challenge: designing a way to let focused people comment on specific things, while also permitting impatient unfocused people to comment on the general topic. It's like asking for a SmartCar that seats 80. See also OpenCongress and their annotation system which also has hundreds of comments on the first few lines of the bill (including 39 on the one line "111th Congress"--apparently more contentious than you'd think!).
3. MyConnPy -- pure-Python MySQL client library, useful because it requires no C compilation to install (and thus can work on systems without C compilers installed, e.g. mobile). (via Simon Willison)
4. The Infinite Book -- design concept for an ebook reader (not a product you can buy yet). Sexy. (via Gizmodo)

Four short links: 28 July 2009

UI Library, 3rd Party Wave Server, Mobile Phones + Parasites, Single API to Cloud Providers

1. CNMAT Resource Library -- The CNMAT Resource Library is our fast growing collection of materials, sensors, gestural controllers, interface devices, tools, demos, prototypes and products - all organized and annotated to support the design of physical interaction systems, "new lutherie" and art installations. (via egoodman on Delicious)
2. PyGoWave Server -- first third-party Google Wave server, based on Django.
3. Mobile Phones Identify Parasites and Bacteria -- UCB Researchers developed a cell phone microscope, or CellScope, that not only takes color images of malaria parasites, but of tuberculosis bacteria labeled with fluorescent markers.. The sensor network is built out, and the computers in our pockets surprise us with their uses. (via BoingBoing)
4. libcloud -- a unified interface to cloud providers, written in Python and open source. Covers EC2, EC2-EU, Slicehost, Rackspace, Linode, VPS.net, GoGrid, flexiscale, Eucalyptus. (via joshua on Delicious)

Announcing: Spike Night at Velocity

Guest blogger Scott Ruthfield is a Program Committee member of the O'Reilly Velocity: Web Performance & Operations Conference. 

Velocity 2009 - Big Ideas (early registration deadline)

My favorite interview question to ask candidates is: "What happens when you type www.(amazon|google|yahoo).com in your browser and press return?"

While the actual process of serving and rendering a page takes seconds to complete, describing it in real detail can take an hour. A good answer spans every part of the Internet from the client browser & operating system, DNS, through the network, to load balancers, servers, services, storage, down to the operating system & hardware, and all the way back again to the browser. It requires an understanding of TCP/IP, HTTP, & SSL deep enough to describe how connections are managed, how load-balancers work, and how certificates are exchanged and validated... and that's just the first request!

Web Performance & Operations is an emerging discipline which requires incredible breadth, focusing less on specific technologies and more on how the entire system works together. While people often specialize on particular components, great engineers always think of that component in relation to the whole. The best engineers are able to fly to the 50,000 foot view and see the entire system in motion and then zoom in to microscopic levels and examine the tiny movements of an individual part.

John Allspaw recently described this interconnectedness on his blog:

With websites, the introduction of change (for example, a bad database query) can affect (in a bad way) the entire system, not just the component(s) that saw the change. Adding handfuls of milliseconds to a query that’s made often, and you’re now holding page requests up longer. The same thing applies to optimizations as well. Break that [bad] query into two small fast ones, and watch how usage can change all over the system pretty quickly. Databases respond a bit faster, pages get built quicker, which means users click on more links, etc. This second-order effect of optimization is probably pretty familiar to those of us running sites of decent scale.

Working with these systems requires an understanding not only of the way technology interacts, but the way that people do as well. The structure, operation, and development of a website mirrors the organization that creates it, which is why so many people in WebOps focus on understanding and improving management culture & process.

Organizing a conference like Velocity is a wonderful challenge because it requires the same sort of thinking. We focus on the big concepts that everyone needs to know and then go deep into the technologies that change our understanding of the system. We find ways to share the unique experience that can only be gained by operating at scale. We make it safe to share as much of the "Secret Sauce" as we can.

Please join us at Velocity this year, we have an amazing lineup of speakers & participants. Early registration ends on Monday, May 11th at 11:59 PM Pacific. (Radar readers can use "vel09cmb" for an additional 15% discount.)

Four short links: 13 Apr 2009

Worms, sorting, languages, and infrastructure:

1. Twitter XSS Attacks (Lynne Pope) -- several incarnations of a worm spread quickly across Twitter this weekend. Twitter profiles are generated by themes, whose parameters users can change. The user-supplied value for the colour was used directly in the CSS color field without filtering, which the original worm strain used to end the CSS and begin Javascript to put the worm into the profile of any Twitter user who viewed the infected profile. Infected users were made to tweet about the worm, with links that would infect anyone who viewed. The worm spread quickly through RTing one of the worm's messages, which claimed to link to instructions on fighting the worm. Later variants use background-color and background parameters. Initial variations downloaded Javascript from mikeyylolz.uuuq.com, since closed down by its hosting company. Later variants download the code from stalkdaily.com, the site that the initial variation spammed about. I wonder whether the 17-year old author of the variants will be able to pay his inevitable legal bills through Google click dollars? (also interesting: Sophos and bdonews)
2. Visualising Sorting -- some beautiful and informative illustrations of how sorting algorithms work. (via @ajtowns)
3. Art and Code: Obscure or Beautiful? -- In the presentation called “50 in 50″ you can see Guy Steele rap about APL and later in the video about spelling keywords backwards. The song about God wrote in Lisp code is also a part of the presentation. Among the languages mentioned are APL, Cobol, AP/I, Scheme, IPL-V, AED, Madcap, Piet, SNOBOL, ADA, Algol60, Intercal, Logo, Perligata, Shakespeare, Lucid, Occam, HQ9+, MUMBLE, Rake, Perl and of course Lisp. It kicks in at about 3m20s and is rather a post-modern presentation. (via
4. Experiences Deploying Large-Scale Infrastructure in Amazon EC2 -- As an aside, I've been very impressed with the reliability of EC2. Like many other people, I didn't know what to expect, but I've been pleasantly surprised. Very rarely does an EC2 instance fail. In fact I haven't yet seen a total failure, only some instances that were marked as 'deteriorated'. When this happens, you usually get a heads-up via email, and you have a few days to migrate your instance, or launch a similar one and terminate the defective one. (via Simon Willison)

AT&T Fiber cuts remind us: Location is a Basket too!

The fiber cuts affecting much of the San Francisco Bay Area this week are similar to the outages in the Middle East last year (radar post), although far more limited in scope and impact.   What I said last year still holds true and is repeated below: 

From an operations perspective these kinds of outages are nothing new, and underscore why having "many eggs in few baskets" is such a problem. I believe we will see similar incidents when we have the first multi-datacenter failures where multiple providers lose significant parts of their infrastructure in a single geographic area.

Remember: Don't put all your eggs in one basket... and Location is a basket too!

To really understand the issue, I recommend Neal Stephenson's incredible (and lengthy) Wired article from 1996 entitled "Mother Earth Mother Board":

[...] It sometimes seems as though every force of nature, every flaw in the human character, and every biological organism on the planet is engaged in a competition to see which can sever the most cables. The Museum of Submarine Telegraphy in Porthcurno, England, has a display of wrecked cables bracketed to a slab of wood. Each is labeled with its cause of failure, some of which sound dramatic, some cryptic, some both: trawler maul, spewed core, intermittent disconnection, strained core, teredo worms, crab's nest, perished core, fish bite, even "spliced by Italians." The teredo worm is like a science fiction creature, a bivalve with a rasp-edged shell that it uses like a buzz saw to cut through wood - or through submarine cables. Cable companies learned the hard way, early on, that it likes to eat gutta-percha, and subsequent cables received a helical wrapping of copper tape to stop it.

[...] There is also the obvious threat of sabotage by a hostile government, but, surprisingly, this almost never happens. When cypherpunk Doug Barnes was researching his Caribbean project, he spent some time looking into this, because it was exactly the kind of threat he was worried about in the case of a data haven. Somewhat to his own surprise and relief, he concluded that it simply wasn't going to happen. "Cutting a submarine cable," Barnes says, "is like starting a nuclear war. It's easy to do, the results are devastating, and as soon as one country does it, all of the others will retaliate."

As the capacity of optical fibers climbs, so does the economic damage caused when the cable is severed. FLAG makes its money by selling capacity to long-distance carriers, who turn around and resell it to end users at rates that are increasingly determined by what the market will bear. If FLAG gets chopped, no calls get through. The carriers' phone calls get routed to FLAG's competitors (other cables or satellites), and FLAG loses the revenue represented by those calls until the cable is repaired. The amount of revenue it loses is a function of how many calls the cable is physically capable of carrying, how close to capacity the cable is running, and what prices the market will bear for calls on the broken cable segment. In other words, a break between Dubai and Bombay might cost FLAG more in revenue loss than a break between Korea and Japan if calls between Dubai and Bombay cost more.

The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is $30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark. [Link]

(Image source: http://www.flickr.com/photos/mundane_joy/2301368102/)

Four short links: 27 Mar 2009

Design, Perl, Heresy, and Ephemera:

1. Product Panic: 2009 -- Bruce Sterling essay on design for recession-panicked consumers. As is usual with Bruce, I can't tell whether he's wryly tongue-in-cheek or literally advocating what he says. Great panic products are like Roosevelt’s fireside chats. They’re cheery bluff. The standard virtues of fine industrial design—safety, convenience, serviceability, utility, solid construction … well, when you’re heading for the lifeboats, you can overlook those pesky little details. For designers, the ideal panic product in 2009 is a 99-cent iPhone application. Something like an iPhone ocarina or lava lamp.
2. Chuck vs Camel -- Programming Perl makes an appearance on mainstream TV. (thanks Allison!)
3. The Civil Heretic (NY Times) -- a fascinating portrait of Freeman Dyson.
4. FileFront Closes -- "48 terabytes of data, historical and user-generated, gone." Does our every upload deserve eternity? Who would want, take, or be able to support the continued existence of 48T of unprofitable blahblah? If 48T of user-generated content falls in the cloud, does it make a sound? (via waxy)

Google App Engine Lets Your Web App Grow Up

Google released App Engine less than a year ago (Radar post). It was the first chance for external developers to use the power of Google's servers. The powerful platform supported Python and was free (within limits). It now supports 45,000 apps and those apps get over 100 million page views per month day. Those pageviews were all free, but they had limits.

That's going to change. After today developers can pay to have more storage, more bandwidth, more CPU time and send more email. The costs as of this morning are listed below with a comparison to the AWS equivalent cost.

• 10 cents per cpu core hour (AWS charges $.10/hr for a small, standard Linux instance and up to $1.20/hr for an XL, Hi-CPU Windows instance in EC2)
• $.10 per gigabyte transferred into AE (AWS charges $.10 for all data transferred into S3)
• $.12 per gigabyte transferred out of AE (AWS charges $.17 for the first 10 TB/month transferred out ofS3)
• $.15 per gigabyte stored per month (AWS charges $.15 for the first 50 TB/month stored onS3)
• .0001 dollars per email (AWS does not have an equivalent)

Without running a more advanced cost calculation it seems that App Engine is slightly cheaper for smaller web apps. Pete Koomen, an App Engine Product Manager, would not say if they would add tiered pricing. I can't imagine it happening until after they add background processing for applications.

Developers will only pay after their app surpasses the free limits (and there are several AE apps that already have like buddypoke.com and mentalfloss.com). The existing free quotas will be reduced in 90 days. The new paid quotas do have unpublished limits, so if you need to support more than 500 requests/sec you'll need to contact Google.

Developer will use their Google Checkout account to cover their apps costs. This means that the new features are restricted to the U.S. and U.K. only. Hopefully the Checkout team expands their scope soon. Google Checkout appears to be available almost everywhere, but doesn't have a page that lists all of the countries. You can check the dropdown of the sign-up page to see if your country is supported and thus you can use the Google App Engine features.

This is a significant step for App Engine. By setting a theoretical limit above the dreams of many web apps they are now putting out the call for serious applications. They've already added Google App Engine Apps to Google Apps For Your Domain. I could envision AE becoming the backbone of an Enterprise Dev Marketplace.

AE still has many significant to-do items on their public Roadmap. Not the least of them is the support for more languages (Java is a good bet as it is used a lot internally) and to add support for background processes.

A huge concern with App Engine is platform lock-in. Google provides a lot of powerful, but non-standard APIs and features that make switching platforms difficult. Developers can extract themselves from App Engine via projects like AppDrop, but it is still risky to use their platform without an SLA. Without a guarantee Google could theoretically decide to raise prices unreasonably. Is it likely? No, but it is something that developers need to think about before committing to any platform.

Google I/O already has many sessions listed for App Engine developers. I'll give a free pass to I/O to the person who suggests the best AE app in the comments. Vic Gundotra, who runs the App Engine team, will be at the Web 2.0 Expo SF in April.

Updated: Correct mistakes about the current pageviews and the availability of the new features.

Two new open source projects at Velocity

At Velocity next week there will be two significant open source projects debuting. The first is the Jiffy: Open Source Performance Measurement and Instrumentation tool created by Scott Ruthfield and his team at Whitepages.com.

Most tools for measuring web performance come in two flavors:

• Developer-installed tools (Firebug, Fiddler, etc.) that allow individuals to closely trace single sessions


• Third-party performance monitoring systems (Gomez, Keynote, etc.) that will hit your site occasionally and report back component-level metrics (for a fee)

Neither of these tools give you real-world information on what’s actually happening with your clients—how long are pages really taking to load, what’s the real cost of client-side execution, and what’s the impact of your loading or dependency chain. This is even more important when you don’t host all of your own assets, such as when you load ads or JavaScript from third parties, for example, and you need to monitor their performance.

Thus we built Jiffy—an end-to-end system for instrumenting your web pages, capturing client-side timings for any event that you determine, and storing and reporting on those timings. You run Jiffy yourself, so you aren’t dependent on the performance characteristics, inflexibility, or costs of third-party hosted services.

The second is project is EUCALYPTUS, the Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems, presented by Rich Wolski from UCSB. This project has already started getting attention. (Many thanks to Surj Patel of Structure08/GigaOM for connecting us!)

Eucalyptus is an open-source software infrastructure for implementing "cloud computing" on clusters. The current interface to EUCALYPTUS is compatible with Amazon's EC2 interface, but the infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain.

The talk will focus on the design, the implementation tradeoffs we have identified in implementing Eucalyptus as an exploratory tool, and the ways in which we have chosen to address these tradeoffs in the first version of the software.

CloudCamp gathering after Velocity

On Tuesday after Velocity closes there will be a CloudCamp gathering at Microsoft's San Francisco Office. I'll be going (unless I'm too exhausted to stand).

CloudCamp was formed in order to provide a common ground for the introduction and advancement of cloud computing

Through a series of local cloudcamp events, attendees can exchange ideas, knowledge and information in a creative and supporting environment, advancing the current state of cloud computing and related technologies. As an informal, member-supported gathering, we rely entirely on volunteers to help with meeting content, speakers, meeting locations, equipment and membership recruitment. We also have corporate sponsors that provide financial assistance with venues, software, books, discounts, and other valuable donations. To become a member, simply register for an upcoming event. Anyone may attend a meeting, there are no fees or dues.

It looks like there is now a London CloudCamp being planned for July 16th as well.

(PS: If you still haven't registered for Velocity and want to attend, you can use my 20% discount code "vel08js".)

The battle for the cloud

Andy Kessler has a great op-ed in the Wall Street Journal, The War for the Web:

Microsoft was smart to walk away (for now) from its $44 billion bid for Yahoo. It's never good to overpay. But the software giant - whose stock has flatlined for eight years - was onto the right strategy in looking to the Web for growth....

With the Microsoft/Yahoo deal breakdown, everyone assumes Google walks away with the prize. Not so fast. This contest is just starting. For Microsoft or Google or anyone else to win, they need four key elements of an end-to-end strategy:

- The Cloud. The desktop computer isn't going away. But as bandwidth speeds increase, more and more computing can be done in the network of computers sitting in data centers - aka the "cloud."...

- The Edge. The cloud is nothing without devices, browsers and users to feed it....

- Speed. - Speed. Once you build the cloud, it's all about network operations....

- Platform. ...Having a fast cloud is nothing if you keep it closed. The trick is to open it up as a platform for every new business idea to run on, charging appropriate fees as necessary....

Andy's analysis is all in those ellipses. Succinct, on-point, and refreshingly insightful about the true drivers of Web 2.0. And I can't help pointing out that the Wall Street Journal has now noticed the fundamental premise of our Velocity conference: "Once you build the cloud, it's all about network operations."

If Velocity were a movie, don't you think that quote might be on the movie poster?