Video of Rich Wolski's EUCALYPTUS talk at Velocity

Rich Wolski gave a truly impressive talk at Velocity about an open-source software infrastructure for cloud computing called EUCALYPTUS . The API is compatible with Amazon's EC2 interface, and the underlying infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain. Watch and learn...

You can see more videos from Velocity on Blip.tv.

Hyperic CloudStatus service dashboard launches at Velocity!

Javier Soltero just launched CloudStatus during his Hyperic sponsor session today at Velocity. CloudStatus is a public health dashboard for web services like Amazon's EC2/S3, and Google's App Engine.

Javier called to tell me about this last week after I declared that "Service Monitoring Dashboards are mandatory". This comes right after Amazon and Google had visible outages, and couldn't have happened at a better time. I'm really excited to see this idea take off, as it's something that is critical to the broad adoption of web services and cloud computing.

Two new open source projects at Velocity

At Velocity next week there will be two significant open source projects debuting. The first is the Jiffy: Open Source Performance Measurement and Instrumentation tool created by Scott Ruthfield and his team at Whitepages.com.

Most tools for measuring web performance come in two flavors:

• Developer-installed tools (Firebug, Fiddler, etc.) that allow individuals to closely trace single sessions


• Third-party performance monitoring systems (Gomez, Keynote, etc.) that will hit your site occasionally and report back component-level metrics (for a fee)

Neither of these tools give you real-world information on what’s actually happening with your clients—how long are pages really taking to load, what’s the real cost of client-side execution, and what’s the impact of your loading or dependency chain. This is even more important when you don’t host all of your own assets, such as when you load ads or JavaScript from third parties, for example, and you need to monitor their performance.

Thus we built Jiffy—an end-to-end system for instrumenting your web pages, capturing client-side timings for any event that you determine, and storing and reporting on those timings. You run Jiffy yourself, so you aren’t dependent on the performance characteristics, inflexibility, or costs of third-party hosted services.

The second is project is EUCALYPTUS, the Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems, presented by Rich Wolski from UCSB. This project has already started getting attention. (Many thanks to Surj Patel of Structure08/GigaOM for connecting us!)

Eucalyptus is an open-source software infrastructure for implementing "cloud computing" on clusters. The current interface to EUCALYPTUS is compatible with Amazon's EC2 interface, but the infrastructure is designed to support multiple client-side interfaces. EUCALYPTUS is implemented using commonly-available Linux tools and basic Web-service technologies making it easy to install and maintain.

The talk will focus on the design, the implementation tradeoffs we have identified in implementing Eucalyptus as an exploratory tool, and the ways in which we have chosen to address these tradeoffs in the first version of the software.

Understanding Web Operations Culture (Part 1)

“You don’t choose the moment, the moment chooses you. You only choose how prepared you are when it does.” - Fire Chief Mike Burtch

(Note: I became a Firefighter-1 and EMT in 2000. My experiences in the fire service profoundly influence my efforts in technology. Much of my work over the past few years has been translating and distilling my knowledge from these two worlds, teaching others, and finding ways to apply it in the service of both.)

Last week I came upon a truck vs. scooter accident on my way home. I could hear a woman yelling in pain from underneath the truck (a good sign!) and could see a guy in the cab looking panicked and touching his controls. I stopped my car and “surveyed the scene” looking for things that might kill me (traffic, hazmat, downed power lines) or make the situation worse if undetected (additional victims, deflating tires, fires).

It looked like the driver was about to move his truck, which would have definitely made things worse. I used my ‘command voice’ to yell “Put it in park! Stop your engine! Set your brake! Get out and wait!” as I approached the truck.

A city crew came over, and one of them told me “We’ve called 911 and they are on their way.”

I asked them to handle traffic control as I approached my patient. I then introduced myself and asked her if I could help. (I have to obtain consent before assisting an injured person, and a response means I know they have still have their Airway, Breathing, and Circulation intact.)

Her legs were entangled in her scooter which was trapped underneath the truck. While she probably had broken her leg, it didn’t look all that bad. She was still wearing her helmet and it wasn't seriously damaged which meant her head was probably okay too. I did a quick check for bleeding and other serious injuries and did a “mental status check” by asking her name, where she was (“on my way to school”), and what had happened (“I was riding and that a**hole RAN OVER ME!”). This meant she was alert and oriented, which was good.

Now that I was sure there weren’t any other life threatening injuries, I prepared to hold her head for c-spine stabilization. (Once you start holding stabilization, you cannot move again until you are ready to put the patient on a backboard.)

As I positioned myself on the ground and took hold of her head, I explained “I’m going to hold your head now to protect your neck and back. Once the fire department gets here, they are going to get your legs unstuck and then we’ll get you on a backboard. Your job is to keep still and keep talking to us. There will be a lot of commotion and noise around you, and that’s okay. Everyone will be watching out for you and so there is no reason to be scared. We’ve got you.”

(continue reading)

Bill Coleman to keynote Velocity

Bill Coleman has twice transformed our industry, and I'm excited to announce that he will keynote Velocity later this month. Bill is most famous for being the "B" in BEA and for leading the creation of Solaris while at Sun. He is now the CEO of his new startup, Cassatt, which "makes Data Centers more efficient".

Bill is awesome and I'm really looking forward to his keynote. He is changing the way we think about and manage Data Centers and the software that runs within them.

When we spoke earlier this week he explained how vacuum tubes created the fear of powering down servers, and how funny it is that that fear persists with people that have never seen them. (I've never made that connection as I'm "part of the problem" ;-)

At Velocity, Bill will likely talk about virtualization & efficiency, where he thinks we're headed, and the questions we need to be asking now to get there.

(Many thanks to Tim for suggesting this to Bill and making the introduction.)

TLS Report grades and reports on site security

My friend Ben Black just released TLS Report, a free (ad-supported) tool that evaluates SSL/TLS configurations across websites and assigns letter grades. In the example below, Facebook gets a D because it accepts several keys that are below 128-bits and relies on MD5:

Ben explains: Cryptography is arcane and complex. Cryptography is also the basis for the various protocols that secure online commerce, ensure privacy of communication, and provide for integrity of data. Transport Layer Security (TLS), formerly SSL, is the de-facto standard for secure communication on the web, and it, naturally, relies on some rather sophisticated cryptographic techniques. Properly implemented, TLS all but guarantees the security of the communication channel.

It's that properly implemented part that catches folks out. Whether from poor defaults in software, poor understanding of best practices, or a weak grasp on the various trade-offs between security and performance, TLS, as most often deployed on the web, is in a sorry state. We hope to change that.

The tls report delivers the tools, information, and visibility to reveal problems in TLS configurations and offer better alternatives so folks can improve their security posture and make sure it stays improved. Everybody wins.

Ben has received a few early complaints from sites getting low grades. This seems to be common with most new rating systems, and I think the discussion is often more important than the scores themselves. You can check out the top/bottom 20 sites, search, and add new ones to be included in the report.

Twitter Availability & Response Times: A Mixed Bag

From an all-time low of 76% registered on May 20th of this year, Pingdom.com’s uptime statistics for Twitter has slowly been increasing. Ideally, you want extended stretches of near 100% uptime, something that hasn’t occurred since mid April:

Daily uptime has hovered around 98% over the last week. Sound impressive? To put things in perspective, 2% downtime means that twitter.com/home is unavailable for 115 minutes every 4 days. The leading social networking sites Myspace and Facebook were each down a total of 20 minutes for the entire month of April! Through the first five days of June, twitter.com/home has already been down a total of 2.85 hours.

From a usability perspective, response time has been slow through the first five days of June. Waiting for (close to) 4 seconds for a page to load does not make for a pleasant user experience. I know that lots of users access Twitter through the API, but the web site is still important to a large contingent of members.

Several people on the Radar team now spend more time on Twitter than on blogs. The service itself is amazing, but for the moment, it may have attracted more users than it can handle. Nevertheless, I remain an avid user and an optimist. The trend for uptime indicates things are improving, albeit slowly. We are still in the early part of June, and so far uptime has been much better than May. On the other hand, if someone came along and offered better response times and reliability, along with similar API's, I sense some users would quickly flock to that service. Personally, I have been playing around a lot more with FriendFeed while waiting for my Twitter pages to reload.

Automated Infrastructure Podcast on IT Conversations

Adam Jacob and I did an IT Conversations podcast with Phil Windley last week, which I really enjoyed. We started with a summary of Adam's excellent Web2.0 Expo session, covered the phases of startup growth using virtual infrastructures like EC2 and 3tera, and discussed how Puppet shifts us to "Infrastructure as Code". We even got into the challenges and opportunities of Sarbanes-Oxley compliance for startups.

Adam also talked a lot about iClassify, his open source systems management tool. He announced iClassify at the Web2.0 Expo, and will be discussing it in-depth at Velocity next month.

You can download the podcast here.

Structure and Velocity

Several people have asked me about the differences between Om Malik's Structure conference and our Velocity Web Performance & Operations conference.  Velocity is on June 23 & 24th at the SFO Mariott, and Structure follows on June 25th in San Francisco. 

The conferences are complementary: Structure discusses what is changing in internet infrastructure, and Velocity teaches how to make that change happen.

I've been recommending that anyone considering Structure make sure their engineering teams are going to Velocity.  For many technical leaders I think there is value in attending both, and I definitely plan on doing so.

The knowledge and skills learned at Velocity can be put to immediate use and will have significant impact on your business.  The reason for this is simple:

Faster, scalable, and highly available websites serve more pages to more customers in the same amount of time.

That's why we've worked hard to make Velocity the best resource for engineers to learn how to build and operate at web scale.  Here are a few examples:

Adam Jacob will give a step-by-step overview of Building an Automated Infrastructure, and then Luke Kanies will follow up with an in-depth session on Puppet.  This is the exact combination I used to explain how effective operations is a huge competitive advantage:

Luiz Barroso will describe Google's approach to energy-efficient datacenter design and management.  Applying these lessons can ultimately save millions of dollars, increase your operational agility, and decrease your environmental footprint.

Mandi Walls will teach how actionable logging can mean the difference between a 20-minute outage and a 2-hour outage while esoteric error codes are deciphered or developers are contacted to investigate.

Eric Lawrence, Program Manager for Internet Explorer, and Mike Connor, lead developer for Mozilla Firefox will explain how to optimize page performance for their respective browsers.  We'll also have demos of leading performance testing tools: HTTPwatch, Fiddler, AOL PageTest, and Firebug.

John Allspaw from Flickr will be be giving a talk about Capacity Management.  John's way of explaining both the problem and the opportunity is wonderfully straightforward:

You can check out the rest of the program and register on the Velocity site.  (Hint: You can use the code "vel08js" for a 20% discount.)  I'll be posting frequently as we add speakers and events.  I hope to see you at Velocity!

The battle for the cloud

Andy Kessler has a great op-ed in the Wall Street Journal, The War for the Web:

Microsoft was smart to walk away (for now) from its $44 billion bid for Yahoo. It's never good to overpay. But the software giant - whose stock has flatlined for eight years - was onto the right strategy in looking to the Web for growth....

With the Microsoft/Yahoo deal breakdown, everyone assumes Google walks away with the prize. Not so fast. This contest is just starting. For Microsoft or Google or anyone else to win, they need four key elements of an end-to-end strategy:

- The Cloud. The desktop computer isn't going away. But as bandwidth speeds increase, more and more computing can be done in the network of computers sitting in data centers - aka the "cloud."...

- The Edge. The cloud is nothing without devices, browsers and users to feed it....

- Speed. - Speed. Once you build the cloud, it's all about network operations....

- Platform. ...Having a fast cloud is nothing if you keep it closed. The trick is to open it up as a platform for every new business idea to run on, charging appropriate fees as necessary....

Andy's analysis is all in those ellipses. Succinct, on-point, and refreshingly insightful about the true drivers of Web 2.0. And I can't help pointing out that the Wall Street Journal has now noticed the fundamental premise of our Velocity conference: "Once you build the cloud, it's all about network operations."

If Velocity were a movie, don't you think that quote might be on the movie poster?

Velocity preview at Web2.0 Expo

At the Web2.0 Expo this month we have a small preview of some of the topics and speakers at the Velocity Web Performance & Operations conference.  (Radar readers get a 20% discount by using "vel08js" as a discount code... and yes it works with the $300 early registration discount!).

Failure Happens
Friday @ 11:00 am, Room 2009

Artur Bergman and I will kick off the day with an entertaining/informative/eye-opening review of the year’s biggest failures, disasters, and painful lessons learned.

We'll review incidents by underlying root cause with a focus on what could have been done to prevent it. We promise not to be too harsh on anybody, although we will give special attention to particularly ironic failures or those that are "entertainingly coupled" to absurd marketing claims.

(Hint: Send your boss to this talk if they don't understand why you and your whole team need to go to Velocity.)

Even Faster Web Sites
Friday @ 1:30 pm, Room 2012

Steve Souders is the co-chair of Velocity and author of the bestselling book High Performance Web Sites. At the Expo last year Steve gave an incredibly popular talk on the 14 best practices he developed while working as the Chief Performance Yahoo!.

(continue reading)

What is Web Operations?

Theo Schlossnagle wrote a brilliant summary of one of the biggest challenges we discussed at the Velocity Summit in January:

What is this Velocity Summit thing? It was a bunch of web architects from highly trafficked sites sitting around talkin' smack. It was operated in Foo style. However, one thing that made me really appreciate this meet-up was the lack of self-importance displayed by attendees. Everyone was just there to talk -- not to make people understand how much they knew. We were talking about The O'Reilly Velocity Web Performance and Operations Conference: what it should be and why.

Two things that I walked away with were (1) a realization of the lack of a career path for people who do what we do (no standard titles, no standard roles and responsibilities and certainly a lack of sex appeal) and (2) a clear lack of terminology for the technology requirements that are so common in these environments. Terminology is easy, in my opinion -- you just argue until someone wins. Of course, arguing is a hobby of mine, so I have bias. On the other hand, defining a career path that is an industry accepted path is hard.

The term Web Operations was used a lot during this event. While it isn't awful, I really don't like this term. The hard part is that the captains, superstars, or heroes in these roles are multidisciplinary experts. They have a deep understanding of networks, routing, switching, firewalls, load-balancing, high availability, disaster recovery, TCP & UDP services, NOC management, hardware specifications, several different flavors of UNIX, several web server technologies, caching technologies, several databases, storage infrastructure, cryptography, algorithms, trending and capacity planning. The issue: how can we expect to find good candidates that have fluency in all of those technologies? In the traditional enterprise, you have architects which are broad and shallow and their team of experts which are focused and deep. However, in the expectation is that your "web operations" engineer be both broad and deep: fix your gigabit switch, optimize your MySQL database and guide the overall architecture design to meet scalability requirements.

I struggle with this. Not everyone can be a superstar. More importantly, no one can really start as a superstar. If we use an apprentice model (which is common in industries without institutional support) we limit the total number of able workers in this field. So, how do we (re)define the requirements for a junior web operations person?  [read more]

One of the reasons I'm excited about Velocity is that we're increasing the pool of great operations people.  We're getting inquiries from companies interested in sending groups of 30-40 people, and I expect more as we confirm speakers and sessions.  You can secure a spot now and get a $350 early registration discount.

Amazon improves EC2 (by embracing failure)

Amazon just announced two big improvements to EC2:

• Multiple Locations
  Amazon EC2 now provides the ability to place instances in multiple locations. Amazon EC2 locations are composed of regions and Availability Zones. Regions are geographically dispersed and will be in separate geographic areas or countries. Currently, Amazon EC2 exposes only a single region. Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same region. Regions consist of one or more Availability Zones. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location.
  
  
• Elastic IP Addresses
  Elastic IP addresses are static IP addresses designed for dynamic cloud computing. An Elastic IP address is associated with your account not a particular instance, and you control that address until you choose to explicitly release it. Unlike traditional static IP addresses, however, Elastic IP addresses allow you to mask instance or Availability Zone failures by programmatically remapping your public IP addresses to any instance in your account. Rather than waiting on a data technician to reconfigure or replace your host, or waiting for DNS to propagate to all of your customers, Amazon EC2 enables you to engineer around problems with your instance or software by quickly remapping your Elastic IP address to a replacement instance.

Datacenters and geographic regions are Single Points of Failure (SPOF) too.  Failure Happens, and it's far better (and cheaper) to build services that are resilient to failure than to try to prevent them from happening.  This is a big step in the right direction.

Update: RightScale posted an excellent overview of how this works.

Paging systems and Conference Bridges for startups & small teams

Early registration for the Velocity Web Performance & Operations Conference has opened. To help spread the word, I've written this "simplest thing that will work" hack to a common Operations need: Paging systems and Conference Bridges.

Step 1: Establish a team contact list with SMS email addresses

Create a Google Spreadsheet to create a team roster like this one. My recommendation is to let people enter and manage their own information. Most cell providers have an email to SMS gateway of some kind. In the US, these are:

• ATT: phonenumber@txt.att.net
• Nextel: phonenumber@messaging.nextel.com
• Sprint: phonenumber@messaging.sprintpcs.com
• T-Mobile: phonenumber@tmomail.net
• Verizon: phonenumber@vtext.com

Step 2: Set up a notification email list

Set up an email alias and add people by email address and SMS gateway address. If you don't have a way of creating an alias, you can use a mailing list provider such as Google Groups.

Step 3: Set up the Conference Bridges

I am really happy with FreeConferenceCall.com which, amazingly, provides free conference call bridges. I recommend setting up three different bridges, and naming them by color so you can refer to them as the "Red Line", "Blue line", etc.

Step 4: Test your notification & conference bridges

Test your notification system to make sure people get the pages and can dial in and use the conference bridges as expected. I've found that it's easier just to give everybody the "host code" instead of having some people using the "participant code". Your mileage may vary. Once you have verified that people are getting pages and can dial into the conference bridges you should...

(continue reading)

Steve Souders asks: "How green is your web page?"

Steve Souders, my Velocity conference Co-Chair and author of High Performance Websites, gave me permission to repost this great analysis:

How green is your web page?

Writing faster web pages is great for your users, which in turn is great for you and your company. But it’s better for everyone else on the planet, too.

Intrigued by an article on Radar about co2stats.com, I looked at my web performance best practices from the perspective of power consumption and CO2 emissions. YSlow grades web pages according to how well they follow these best practices. What if it could convert those grades into kilowatt-hours and pounds of CO2?

Let’s look at one performance rule on one site. Wikipedia is one of the top ten sites in the world (#9 according to Alexa). I love Wikipedia. I use it almost every day. Unfortunately, it has thirteen images in the front page that don’t have a far future Expires header (Rule 3). Every time someone revisits this page the browser has to make thirteen HTTP requests to the Wikipedia server to check if these images are still usable, even though these images haven’t changed in over seven months on average. A better way to handle this would be for Wikipedia to put a version number in the image’s URL and change the version number whenever the image changes. Doing this would allow them to tell the browser to cache the image for a year or more (using a far future Expires or Cache-Control header). Not only would this make the page load faster, it would also help the environment. Let’s try to estimate how much.

• Let’s assume Wikipedia does 100 million page views/day. (I’ve seen estimates that are over 200 million/day.)
• Assume 80% of those page views are done with a primed cache (based on Yahoo!’s browser cache statistics). We’re down to 80M page views/day.
• Assume 10%, no, 5% of those are for the home page. We’re down to 4M page views/day for the home page with a primed cache. Each of those contains 13 HTTP requests to validate the images, for a total of 52M image validation requests/day.
• Assume one web server can handle 100 of these requests/second, or 8.6M requests/day. That’s six web servers running full tilt year-round to handle this traffic.
• Assume a fully loaded server uses 100W. Six servers, year-round, consume 5,000 kilowatt-hours per year or approximately 500-1000 pounds of CO2 emissions.

I think this is a conservative estimate, but there are a lot of assumptions above. And six servers doesn’t sound like a lot. 5,000 kilowatt-hours is a drop in the bucket if you look at data center power consumption. But this was just one rule on one page on one site. Think about the impact of not gzipping, not minifying JavaScript, wasteful redirects, and bloated images. If we extrapolate this across all the performance rules across all sites the numbers are much bigger.

Make your pages faster. It’s good for your users, good for you, and good for Mother Earth.

-Steve

Steve has a SXSW Bookreading on Saturday @11 AM, and will be at the O'Reilly booth on Sunday from 3:30-4:30. Stop by and say hello!

WebOps Hack #1: Simple Availability Report for Busy Teams

I created this spreadsheet for tracking availability and "days since last outage". Along with the availability and uptime calculations, it asks the following questions:

• What broke?
• Why?
• What fixed It?
• What did we learn?
• How can we prevent recurrence?
• Who owns follow-up?

I've found this to be the "simplest thing that could possibly work" for identifying problems and tracking issues before a formal incident tracking system is in place, or with vendors or other teams who you want to keep honest. Please let me know if it's helpful for you and how it might be improved. (Feel free to improve upon it yourself too -- it's Creative Commons Attribution Share Alike.)

Link to the Google doc is here. You need to "Copy to a new spreadsheet" to be able to use it.

Technorati Tags: infrastructure, operations, secretsauce, startups