Four short links: 5 November 2009

Heat Maps in R, EC2 Blackhat Tricks, Snickersome Unicode, and Decoding Statistics

1. Heat Maps in R -- We used financial data here because it's easier to access than the airline data, but it's actually a pretty interesting way of looking at a financial time series. Weekend and holiday effects are a bit more obvious, and it's a bit like being able to see the daily, weekly, monthly and yearly closes all at once (by scanning your eye over the calendar in different directions). Includes source code. (via migurski on Delicious)
2. BlackHat and EC2 -- Theft of resources is the red-headed step-child of attack classes and doesn't get much attention, but on cloud platforms where resources are shared amongst many users these attacks can have a very real impact. With this in mind, we wanted to show how EC2 was vulnerable to a number of resource theft attacks and the videos below demonstrate three separate attacks against EC2 that permit an attacker to boot up massive numbers of machines, steal computing time/bandwidth from other users and steal paid-for AMIs. (via straup on Delicious)
3. Funny Characters in Unicode -- I never get tired of the wacky stuff in Unicode. I love the thought of a Unicode committee somewhere arguing passionately about the number of buttons on the snowman .... (via Hacker News)
4. Statistics to English Translation -- The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate. This matters. Bandying around numbers with misleading labels, or misinterpreting numbers that have a precise and defined meaning, does not further understanding. (Said 78.4% of statisticians, with a 20% confidence factor probability of false positives)

Four short links: 16 October 2009

Audio Geotagging, SF Open Data Stories, Wave Use Cases, Hadooped Genomes

1. Wiimote Audio Geotagging -- match audio with the map movement and annotations made with an IR pen and a Wiimote. Very cool! (and from New Zealand)
2. San Francisco: Open For Data -- Two months after it launched, the project is already reaping rewards from San Francisco's huge community of programmers. Applications using the data include Routesy, which offers directions based on real-time city transport feeds; and EcoFinder, which points you to the nearest recycling site for a given item.
3. Google Wave's Best Use Cases (Lifehacker) -- not cases where people are using Wave, but where they want to. Read this as "the Web has not provided all the tools to solve these problems". Something will solve them, and Wave is trying to. (via Jim Stogdill)
4. Analyzing Human Genomes with Hadoop -- case study from the Cloudera blog. Performs alignment and genotyping on the 100GB of data you get when you sequence a human's genome in about three hours for less than $100 using a 40-node, 320-core cluster rented from Amazon’s EC2. (via mndoci on Twitter)

Four short links: 8 October 2009

DIY Baby Rocker, Unix Systems Glory, Encrypting Ephemera, and Explaining Creative Joy

1. Linux Baby Rocker -- inventive use of a CD drive and the eject command ... (via Hacker News)
2. I Like Unicorn Because It's Unix -- forceful rant about the need to rediscover Unix systems programming. Reminds me of the Varnish notes where the author explains that it works better because it uses the operating system instead of recreating it poorly.
3. Encrypting Ephemeral Storage and EBS Volumes on Amazon -- step-by-step instructions. (via Matt Biddulph on Delicious)
4. You Have No Life -- if a video smacks even slightly of concentrated effort or advance planning, someone will inevitably scoff that the subject has a) "too much time on his hands" or b) "no life." Ten times out of ten. [...] After six years I lack a succinct, meaningful response to my students' defensive, clannish embrace of mediocrity, though I'm grateful for this tweet, which comes pretty close: dwineman: You say "looks like somebody has too much time on their hands" but all I hear is "I'm sad because I don't know what creativity feels like."

Four short links: 22 September 2009

Cities, How Things Work, Stylish Google, EC2 Numbers

1. The City is a Battlesuit for Surviving the Future (IO9) -- a great essay by Matt Jones, based on his talk at Webstock this year. Urban design is how we created alternate realities before we had iPhones, and the new technology lets us choose which science fiction future we want to inhabit. We are now a predominantly urban species, with over 50% of humanity living in a city. The overwhelming majority of these are not old post-industrial world cities such as London or New York, but large chaotic sprawls of the industrialising world such as the "maximum cities" of Mumbai or Guangzhou. Here the infrastructures are layered, ad-hoc, adaptive and personal - people there really are walking architecture, as Archigram said. Hacking post-industrial cities is becoming a necessity also. [...]
2. How and Why Machines Work (MIT Open Course Ware) -- Subject studies how and why machines work, how they are conceived, how they are developed (drawn), and how they are utilized. Students learn from the hands-on experiences of taking things apart mentally and physically, drawing (sketching, 3D CAD) what they envision and observe, taking occasional field trips, and completing an individual term project (concept, creation, and presentation). Emphasis on understanding the physics and history of machines. (via Hacker News)
3. Google Style Guide -- how Google codes. Useful if you're working on their code, starting a job there, or want to mock them for not specifying K&R braces/four space tabs/<insert One True Way here>. (via Hacker News)
4. EC2 Usage Guessed From Sequential IDs -- The Superseries ID changes so rarely that originally I had assumed it was some kind of checksum. This would have been odd as it limits the total available IDs to 2²⁴ = 16.8 million. Up to very recently, the Superseries ID for all resource types - instances, images, volumes, snapshots, etc. - was 69 (in the us-east-1 region (for eu-west-1 the Superseries ID is 74). These days, new instances use the Superseries ID 68. This subtle change, unnoticed by the industry, may hint at an astonishing achievement: 8.4 million instances launched since EC2’s debut! (Instance IDs are even so 8.4M = 16.8M / 2.) (via mattb on delicious)

Four short links: 24 June 2009

Open Source Kids, Crowdsourcing Lessons, Flickr Secrets, Hadoop Spatial Joins

1. The Digital Open -- The Digital Open is an online technology community and competition for youth around the world, age 17 and under. Building a community of young open source hackers.
2. Four Crowdsoucing Lessons from the Guardian's Spectacular Expenses Scandal Experiment -- Your workers are unpaid, so make it fun. How to lure them? By making it feel like a game. "Any time that you’re trying to get people to give you stuff, to do stuff for you, the most important thing is that people know that what they’re doing is having an effect," Willison said. "It’s kind of a fundamental tenet of social software. … If you’re not giving people the ‘I rock’ vibe, you’re not getting people to stick around." (via migurski on delicious)
3. 10+ Deploys/Day: Dev & Ops Cooperation at Flickr -- John Allspaw and Paul Hammond's talk from Velocity. You tell any mainstream company in the world "10 deploys/day" and you'll be met with disbelief.
4. Reproducing Spatial Joins using Hadoop and EC2 -- bit by bit the techniques for emulating important operations from trad databases are being discovered and shared in the new database scene. (via straup on delicious)

Four short links: 11 June 2009

Trends, Graffiti, Games, and Streaming Video

1. Trending Topics -- full source code for trendingtopics.org, Wikipedia trend analysis. Rails app running on the Cloudera Hadoop Distribution on EC2. (via mattb on Delicious)
2. Graffiti from Pompeii -- I can't help but read these as Tweets. Herculaneum (on the exterior wall of a house); 10619: Apollinaris, the doctor of the emperor Titus, defecated well here (see also olde style Twitter) (via OvidPerl on Twitter)
3. Online Games Dominate Beijing Startonomics -- presentations from sessions on Chinese game business at Startonomics conference. Though there are many differences between the US and China games market, the one that stands out most is China’s ability to massively monetize games. Tencent, a leading Chinese web portal, social network and game developer, famously announced revenue of over $1 billion earlier this year, much of it coming from their avatar service. (via TinaTranT on Twitter)
4. Ustream's Audience for Apple iPhone Announcement Greater Than Cable News -- Ustream is amazing, you can take a consumer handycam and video broadcast live to a greater audience than many TV shows get.

Four short links: 8 June 2009

3D Geometry, The Printable Web, Government Internet Fail, and Real World Cloud Computing

1. How to Project on 3D Geometry -- the fine art (and math) of distorting an image so that it looks undistorted when projected onto a non-flat 3D surface. Confused? See the images below. (via straup on Delicious)
2. ZinePal -- Create your own printable magazine from any online content. (via warrenellis on Delicious)
3. What The Government Doesn't Understand About The Internet And What To Do About It -- Tom Steinberg from MySociety lays it out. As true for US, NZ, and every other country as it is for the UK (for which it was written). Accept that any state institution that says “we control all the information about X” is going to look increasingly strange and frustrating to a public that’s used to be able to do whatever they want with information about themselves, or about anything they care about (both private and public). This means accepting that federated identity systems are coming and will probably be more successful than even official ID card systems: ditto citizen-held medical records. It means saying “We understand that letting train companies control who can interface with their ticketing systems means that the UK has awful train ticket websites that don’t work as hard as they should to help citizens buy cheaper tickets more easily. And we will change that, now.” What I like about Tom vs the US's Gov 2.0 is that Tom puts down philosophy that's hard to argue with, whereas the US is dangerously close to simply focusing on techniques and that's subvertible.
4. Real World Cloud Computing -- summary from a panel of startups who are using EC2. The lock-in is latency. Transfering data within the Amazon services is free. Transfering data to an Amazon competitor: not free.

Amazon's new EC2 SLA

Amazon announced a new SLA for EC2, similar to the one for S3. This is a notable step for Amazon and cloud computing as a whole, as it establishes a new bar for utility computing services.

Amazon is committing to 99.95% availability for the EC2 service on a yearly basis, which corresponds to approximately four hours and twenty three minutes of downtime per year. It's important to remember that an SLA is just a contract that provides a commitment to a certain level of performance and some form of compensation when a provider fails to meet it.

Service Commitment AWS will use commercially reasonable efforts to make Amazon EC2 available with an Annual Uptime Percentage (defined below) of at least 99.95% during the Service Year. In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit as described below. [...]

• “Annual Uptime Percentage” is calculated by subtracting from 100% the percentage of 5 minute periods during the Service Year in which Amazon EC2 was in the state of “Region Unavailable.” If you have been using Amazon EC2 for less than 365 days, your Service Year is still the preceding 365 days but any days prior to your use of the service will be deemed to have had 100% Region Availability [...]
• “Unavailable” means that all of your running instances have no external connectivity during a five minute period and you are unable to launch replacement instances. [...]

To receive a Service Credit, you must submit a request by sending an e-mail message to aws-sla-request @ amazon.com. To be eligible, the credit request must [...] include your server request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks)

This new SLA does not appear to address the reliability of server instances individually or in aggregate. For example, if half of a customer's EC2 instances lose their connections or die every 6 minutes, EC2 would still be considered "available" even if it is essentially unusable.

If the entire EC2 service is down a cumulative four hours and twenty minutes, customers must furnish proof of the outage to Amazon to be eligible for the 10% credit. This seems like an onerous process for very little compensation, and isn't in-line with Amazon's famous "Relentless Customer Obsession". Amazon takes monitoring very seriously and should take the lead by tracking, reporting, and proactively compensating customers when it lets them down.

Video of Rich Wolski's EUCALYPTUS talk at Velocity

Rich Wolski gave a truly impressive talk at Velocity about an open-source software infrastructure for cloud computing called EUCALYPTUS . The API is compatible with Amazon's EC2 interface, and the underlying infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain. Watch and learn...

You can see more videos from Velocity on Blip.tv.

Two new open source projects at Velocity

At Velocity next week there will be two significant open source projects debuting. The first is the Jiffy: Open Source Performance Measurement and Instrumentation tool created by Scott Ruthfield and his team at Whitepages.com.

Most tools for measuring web performance come in two flavors:

• Developer-installed tools (Firebug, Fiddler, etc.) that allow individuals to closely trace single sessions


• Third-party performance monitoring systems (Gomez, Keynote, etc.) that will hit your site occasionally and report back component-level metrics (for a fee)

Neither of these tools give you real-world information on what’s actually happening with your clients—how long are pages really taking to load, what’s the real cost of client-side execution, and what’s the impact of your loading or dependency chain. This is even more important when you don’t host all of your own assets, such as when you load ads or JavaScript from third parties, for example, and you need to monitor their performance.

Thus we built Jiffy—an end-to-end system for instrumenting your web pages, capturing client-side timings for any event that you determine, and storing and reporting on those timings. You run Jiffy yourself, so you aren’t dependent on the performance characteristics, inflexibility, or costs of third-party hosted services.

The second is project is EUCALYPTUS, the Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems, presented by Rich Wolski from UCSB. This project has already started getting attention. (Many thanks to Surj Patel of Structure08/GigaOM for connecting us!)

Eucalyptus is an open-source software infrastructure for implementing "cloud computing" on clusters. The current interface to EUCALYPTUS is compatible with Amazon's EC2 interface, but the infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain.

The talk will focus on the design, the implementation tradeoffs we have identified in implementing Eucalyptus as an exploratory tool, and the ways in which we have chosen to address these tradeoffs in the first version of the software.

CloudCamp gathering after Velocity

On Tuesday after Velocity closes there will be a CloudCamp gathering at Microsoft's San Francisco Office. I'll be going (unless I'm too exhausted to stand).

CloudCamp was formed in order to provide a common ground for the introduction and advancement of cloud computing

Through a series of local cloudcamp events, attendees can exchange ideas, knowledge and information in a creative and supporting environment, advancing the current state of cloud computing and related technologies. As an informal, member-supported gathering, we rely entirely on volunteers to help with meeting content, speakers, meeting locations, equipment and membership recruitment. We also have corporate sponsors that provide financial assistance with venues, software, books, discounts, and other valuable donations. To become a member, simply register for an upcoming event. Anyone may attend a meeting, there are no fees or dues.

It looks like there is now a London CloudCamp being planned for July 16th as well.

(PS: If you still haven't registered for Velocity and want to attend, you can use my 20% discount code "vel08js".)

Automated Infrastructure Podcast on IT Conversations

Adam Jacob and I did an IT Conversations podcast with Phil Windley last week, which I really enjoyed. We started with a summary of Adam's excellent Web2.0 Expo session, covered the phases of startup growth using virtual infrastructures like EC2 and 3tera, and discussed how Puppet shifts us to "Infrastructure as Code". We even got into the challenges and opportunities of Sarbanes-Oxley compliance for startups.

Adam also talked a lot about iClassify, his open source systems management tool. He announced iClassify at the Web2.0 Expo, and will be discussing it in-depth at Velocity next month.

You can download the podcast here.

Amazon improves EC2 (by embracing failure)

Amazon just announced two big improvements to EC2:

• Multiple Locations
  Amazon EC2 now provides the ability to place instances in multiple locations. Amazon EC2 locations are composed of regions and Availability Zones. Regions are geographically dispersed and will be in separate geographic areas or countries. Currently, Amazon EC2 exposes only a single region. Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same region. Regions consist of one or more Availability Zones. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location.
  
  
• Elastic IP Addresses
  Elastic IP addresses are static IP addresses designed for dynamic cloud computing. An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it. Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by programmatically remapping your public IP addresses to any instance in your account. Rather than waiting on a data technician to reconfigure or replace your host, or waiting for DNS to propagate to all of your customers, Amazon EC2 enables you to engineer around problems with your instance or software by quickly remapping your Elastic IP address to a replacement instance.

Datacenters and geographic regions are Single Points of Failure (SPOF) too.  Failure Happens, and it's far better (and cheaper) to build services that are resilient to failure than to try to prevent them from happening.  This is a big step in the right direction.

Update: RightScale posted an excellent overview of how this works.