AT&T Fiber cuts remind us: Location is a Basket too!

by Jesse Robbins | @jesserobbins | comments: 3

The fiber cuts affecting much of the San Francisco Bay Area this week are similar to the outages in the Middle East last year (radar post), although far more limited in scope and impact.   What I said last year still holds true and is repeated below: 

From an operations perspective these kinds of outages are nothing new, and underscore why having "many eggs in few baskets" is such a problem. I believe we will see similar incidents when we have the first multi-datacenter failures where multiple providers lose significant parts of their infrastructure in a single geographic area.

Remember: Don't put all your eggs in one basket... and Location is a basket too!

To really understand the issue, I recommend Neal Stephenson's incredible (and lengthy) Wired article from 1996 entitled "Mother Earth Mother Board":

[...] It sometimes seems as though every force of nature, every flaw in the human character, and every biological organism on the planet is engaged in a competition to see which can sever the most cables. The Museum of Submarine Telegraphy in Porthcurno, England, has a display of wrecked cables bracketed to a slab of wood. Each is labeled with its cause of failure, some of which sound dramatic, some cryptic, some both: trawler maul, spewed core, intermittent disconnection, strained core, teredo worms, crab's nest, perished core, fish bite, even "spliced by Italians." The teredo worm is like a science fiction creature, a bivalve with a rasp-edged shell that it uses like a buzz saw to cut through wood - or through submarine cables. Cable companies learned the hard way, early on, that it likes to eat gutta-percha, and subsequent cables received a helical wrapping of copper tape to stop it.

[...] There is also the obvious threat of sabotage by a hostile government, but, surprisingly, this almost never happens. When cypherpunk Doug Barnes was researching his Caribbean project, he spent some time looking into this, because it was exactly the kind of threat he was worried about in the case of a data haven. Somewhat to his own surprise and relief, he concluded that it simply wasn't going to happen. "Cutting a submarine cable," Barnes says, "is like starting a nuclear war. It's easy to do, the results are devastating, and as soon as one country does it, all of the others will retaliate."

As the capacity of optical fibers climbs, so does the economic damage caused when the cable is severed. FLAG makes its money by selling capacity to long-distance carriers, who turn around and resell it to end users at rates that are increasingly determined by what the market will bear. If FLAG gets chopped, no calls get through. The carriers' phone calls get routed to FLAG's competitors (other cables or satellites), and FLAG loses the revenue represented by those calls until the cable is repaired. The amount of revenue it loses is a function of how many calls the cable is physically capable of carrying, how close to capacity the cable is running, and what prices the market will bear for calls on the broken cable segment. In other words, a break between Dubai and Bombay might cost FLAG more in revenue loss than a break between Korea and Japan if calls between Dubai and Bombay cost more.

The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is $30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark. [Link]

It's also worth mentioning the outages to multiple service providers hosted in a single colocation facility when the FBI sized all the equipment in the facility, the big outage at 365 Main from two years ago, and many others (see: Radar posts & comprehensive coverage at Data Center Knowledge).

Experience Syndication: Powered by Zappos

by Joshua-Michéle Ross | @jmichele | comments: 5

I have been thinking a lot about the new Powered by Zappos service.

According to Zappos:

Powered by Zappos (PBZ) is a feature Zappos.com offers to its partners where we design, host, fulfill and own a partners web site. Our goal is to provide Zappos customers as well as our partner's customers with the best possible service experience. By building partnerships through PBZ we can deliver great service to more people. Ultimately if you are purchasing through a PBZ site you are making a purchase from Zappos, your package with free shipping will even arrive in a Zappos.com box and you will receive all the great benefits Zappos has to offer.

For lack of a better term, I am calling this “experience syndication” since PBZ is essentially syndicating the value of the entire experience - not just one aspect such as content or business process or infrastructure. A quick Google search reveals that would-be competitors such as Clarks Shoes, Stuart Weitzman, Bostonian Shoes etc. are already utilizing PBZ.

I am of two minds on PBZ. As a business strategy I think it is a brilliant play. Zappos is syndicating the very thing that makes them great - the entire experience; from browsing to buying and especially post-sales support. In the hyper-competitive world of ecommerce, individual, mid-market brands like Clarks simply can’t compete with that so they better join. It also raises an interesting question. What other companies might look at syndicating their experience?

On the negative side: I am a big fan of Zappos but I am not blind to the fact that the more successful the PBZ offering gets, the more power they will wield over individual companies that depend on them for survival. In that sense I fear that Zappos may ultimately do to shoe companies what Amazon appears headed to do to book publishers - and what Walmart has already done to countless small brands - put them in a death grip and squeeze the life out of them. Let's hope I am wrong.

Why We're Failing in Math and Science

by Tim O'Reilly | @timoreilly | comments: 45

Norman Mailer's brilliant novel Why Are We in Vietnam? doesn't talk explicitly about the Vietnam war; it tells a story about American culture and the American psyche, thereby producing a devastating critique of the war with the title and last line alone.

In a similar way, it may be easier to understand why America is falling behind at math and science with a few simple stories.

Last week, Robert Bruce Thompson, author of An Illustrated Guide to Home Chemistry Experiments, wrote a guest blog post on makezine.com, Home Science Under Attack, which told the sad story of how a retired chemist was arrested and his lab confiscated because he was doing experiments:

The Worcester Telegram & Gazette reports that Victor Deeb, a retired chemist who lives in Marlboro, has finally been allowed to return to his Fremont Street home, after Massachusetts authorities spent three days ransacking his basement lab and making off with its contents. Deeb is not accused of making methamphetamine or other illegal drugs. He's not accused of aiding terrorists, synthesizing explosives, nor even of making illegal fireworks. Deeb fell afoul of the Massachusetts authorities for ... doing experiments.

Authorities concede that the chemicals found in Deeb's basement lab were no more hazardous than typical household cleaning products. Despite that, authorities confiscated "all potentially hazardous chemicals" (which is to say the chemicals in Deeb's lab) from his home, and called in a hazardous waste cleanup company to test the chemicals and clean up the lab.

Pamela Wilderman, the code enforcement officer for Marlboro, stated, "I think Mr. Deeb has crossed a line somewhere. This is not what we would consider to be a customary home occupation."

Allow me to translate Ms. Wilderman's words into plain English: "Mr. Deeb hasn't actually violated any law or regulation that I can find, but I don't like what he's doing because I'm ignorant and irrationally afraid of chemicals..."

I forwarded this message to Dave Farber's IP list (which is now searchable via markmail, the amazing mailing list search engine!), and got back some great stories that I wanted to share.

Armando Stettner wrote one story that illustrates just how much our culture has changed. His story also involves the cops, but here, they understand and support science. Too bad that was 40+ years ago:

When I was about 13 or so, I also had a chemistry set in my basement. I was living on Long Island - Freeport, to be exact. I also remember the hobby shop with ALL sorts of glassware and little labeled bottles of chemicals. I had some really neat stuff: all sorts of chemicals - I seem to remember potassium ferrocyanide with which I did some chemoluminescence (I think that's one of the ingredients), sodium in liquid form, various acids, a few rolls of magnesium - not to mention all the paraphernalia: lots of pyrex stuff, triple beam balances, etc. All the chemicals were neatly arranged in this cabinet.

One day, I had mixed a concoction and was carrying it (premixed!) in a tin coffee can. Myself and a friend were carrying the stuff to the train tracks to test it out (light it) where it was relatively safe. The stuff started getting warm but I thought it was the sun heading the can up. Then it started getting REALLY warm. As it got hot, I dropped it in the middle of the street. The stuff flashed over. It was VERY cool.

But, I decided I didn't want to stay around any more and left.

Unfortunately for me, this all occurred in front of the house of someone who knew me (she was a 'friend' of my parents). She called the cops.

The Freeport police came to my house questioned me and my parents, joined in a little while by some county detectives. They were very polite. We took them down to the basement where I showed them all the stuff. The uniformed police left and the detectives continued to look at all the stuff and ask questions. They called somebody to ask some advice. It turns out they called the county labs. The guy got off the phone and asked 'you're not making any drugs down here are you?" I said no!! He smiled - he winked at my parents. Then he said the most unexpected thing: he said the gang at the labs offered to give me a tour of the labs anytime I wanted.

Then they left asking me to be careful. For me, it was actually a positive experience.

Today, I'm sure I'd face a visit from the Hazmat teams and the DHS. And, because of the triple beam balance, my house (or my parents') would be confiscated under the forfeiture rules.

At Maker Faire earlier this year, Robert Bruce Thompson gave a talk (video unfortunately truncated at both ends) that highlighted how attitudes towards chemistry have changed since he was a kid, starting with a tour of the powerful chemistry sets available in 1964 (courtesy of the Sears Catalog), and tracing the dumbing down and rising fear of liability that doomed them, until, as Kevin Kelly noted in a recent review of Robert's book, we reached "the so-called chemistry sets today which boldly (and insanely) advertise they contain 'No Chemicals!'" (Review sent out in Cool Tools email, up on the Cool Tools site soon.)

Why are we failing at math and science? Because it isn't fun any more. When you put safety on the highest altar, what do you give up? When fear of lawsuits -- not to mention fear of technology -- drives product design, marketing, and public policy, you eliminate science at its roots, in the natural experimentation of kids who want to know how the world works.

Kaminsky DNS Patch Visualization

by Jesse Robbins | @jesserobbins | comments: 4

Dan Kaminsky has posted the details of the widespread DNS vulnerability. Clarified Networks created this visualization of DNS patch deployment over the past month:

Red = Unpatched
Yellow = Patched, "but NAT is screwing things up"
Green = OK

The new internet traffic spikes

by Jesse Robbins | @jesserobbins | comments: 5

Theo Schlossnagle, author of Scalable Internet Architectures, gave a great explanation of how internet traffic spikes are shifting:

Lately, I see more sudden eyeballs and what used to be an established trend seems to fall into a more chaotic pattern that is the aggregate of different spike signatures around a smooth curve. This graph is from two consecutive days where we have a beautiful comparison of a relatively uneventful day followed by long-exposure spike (nytimes.com) compounded by a short-exposure spike (digg.com):

The disturbing part is that this occurs even on larger sites now due to the sheer magnitude of eyeballs looking at today's already popular sites. Long story short, this makes planning a real bitch.

[...]What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.

[Link]

Amazon Gets Demanding with Print-on-Demand Publishers

by Andrew Savikas | @andrewsavikas | comments: 21

We often hold up Amazon as an example of one of the original Web 2.0 companies. Their survival amid the tech meltdown was driven largely by the value of the data they'd acquired through thousands of reader reviews, recommendations, and "people who bought this bought that" collaborative filtering. Amazon was a system that grew more valuable with more users: a network-effect-driven data lock-in.

That kind of lock-in is implicit: publishers were free to sell their books elsewhere, and readers were free to buy them elsewhere. Such implicit lock-in is characteristic of other Web 2.0 success stories, like eBay and craigslist. These sites relied on the value of the unique data/marketplace they were building to implicitly raise enormous barriers of entry. Not much fun if you're a newspaper, but a boon for buyers and sellers.

But today's news from Amazon about Print-on-Demand is the latest move from Amazon revealing a trend toward much more aggressive explicit lock-in attempts. (Not that it's an entirely new strategy from the folks that brought you the "one-click" patent). Amazon has effectively told publishers that if they wish to sell POD books on Amazon, they must use Amazon as the POD printer. Small/self publishers are unsurprisingly feeling bullied.

Let's look at four levels of lock-in at play here:

(continue reading)

Goodbye, New York Times

by Jimmy Guterman | comments: 32

I love The New York Times. I've read it almost every day of my life since I was in high school. For all its recent flaws -- the weirdo profiles of the major presidential candidates are the most high-profile -- it is still full of the most outstanding reporting. And, on the days that Gail Collins files, it offers up the most penetrating and entertaining opinion.

What's that? It's the last print copy of the Times I'll ever have delivered to my front door. Over the years, I've slowly weaned myself off subscriptions to physical newspapers, but it was hard to say no to the Times. The quality was high, the thump of the paper on the sidewalk was a pleasant sound to hear first thing in the morning, I liked the serendipity of walking through a print section, and I felt obligated to pay for the paper at a time when print subscribers were becoming an endangered species. But, after years of wavering, I'm done. The environmental argument alone should have been enough for me, but the simple fact is that I do more and more of my reading on a screen (the only holdouts: fiction and poetry). And plenty of that reading has been from the Times. What finally made me give in to the inevitable was realizing, one barely-dawn morning last week when I was reading the paper at our kitchen table, that I had already read much (most?) of it online. For all the pleasure of holding and print, the Times on paper is just too late. In 2008, today's paper is yesterday's news.

So now I'm a freeloader, although you could argue that my personal information, sent to the Times in return for a username and password, may have some value. I rarely, if ever, click on an ad on the Times's website. I would gladly pay for the pleasure and convenience of reading the paper online, just as I do for The Wall Street Journal, but I don't have that option. In this era of advertising-is-the-only-business-model, management at the Times Company has decided that I've decided that the value of what it sends to me is zero. I disagree -- and I'm not going to pay a premium for the proprietary and little-used Times Reader to make my point.

I'll miss the paper on paper, and I bet I'll buy it when I'm on vacation, as a treat, an indulgence. But if even people like me -- who adore The New York Times -- can no longer justify a print subscription, how can its print version survive, except as a high-priced, scarce product for an increasingly elite audience?

Trendalyzer view of the banking crisis

by Jesse Robbins | @jesserobbins | comments: 3

The team at "And Still I Persist" has created their own version of Hans Rosling's "Trendalyzer" (see: Radar post) to visualize the current US banking crisis.

"First lets look at the top 8 banks and their mortgages that are 90+ days late. Below is a flash charting system, feel free to use the controls and experiment. We chart the total assets of the bank along the horizontal axis, the value of loans that go 90+ days late on the vertical, and the size of the circles represent the total loan portfolio for that bank. You can set the charts in motion by hitting the “Play” button and stop them at any time. Hovering over a circle will show you the value for that data point.

Our charts step forward in time for Q1-2002 one quarter at a time, reading directly from the bank’s own FDIC reports. "
Bank Portfolios - 90+ Days Late

See the original article for more about this visualization and the team that created it.

Update: Bruce Henderson invites anybody interested in working with a larger data set to take a look at the OSG Boomerang tool.

Today's ETech Hack is Tomorrow's Critical Infrastructure...

by Jesse Robbins | @jesserobbins | comments: 0

My friend Jordan Schwartz just gave me the perfect example of how quickly a cool hack can turn into Critical Infrastructure.  Jordan wrote "How to build an SMS Service" and created SwaggleSMS as a demonstration of how to do group chat with SMS.  It's a hack that he created as an experiment (it's super-useful for conference afterparty coordination).

Jordan and I were talking about some of the interesting ways that Twitter is being used by mainstream emergency management (see: FactoryJoe, Radar post).  Jordan then showed me a message he discovered while checking logs after an upgrade:

"Tom1132 to OurTownFD: Possible drowning in bay"

If it's not obvious... this is fire department who has apparently been using the service for a while.  It's a perfect example of how quickly a hack can become critical infrastructure without the creator knowing, let alone being prepared for it.  The picture to the right is the "Swaggleplex"... fully operational.

Mikel Maron and I are presenting at ETech on Disaster Tech: What's Working, What's next and we'll be diving into this and other examples of just how quickly the world is changing.

US Judge censors WikiLeaks.org by ordering DNS records removed

by Jesse Robbins | @jesserobbins | comments: 6

The BBC and many others report that the international whistle-blower website WikiLeaks.org has been taken down as of this morning. Judge Jeffery White ordered that the WikiLeaks.org domain be removed at the request of Julius Baer Bank & Trust. Not only does the judge order that the site be removed, he orders that the whois privacy protections be turned off and, of course, that the log files be handed over.

Court Orders can be used as an effective Denial of Service attack and can circumvent otherwise strong privacy protections.

(continue reading)

Understanding the undersea cable cuts... (updated: "fifth cable cut")

by Jesse Robbins | @jesserobbins | comments: 5

The Fiber Cuts in the Middle East are getting a lot of attention. The economic damage is real and the geopolitical issues are extremely complex (which is why I edited my earlier post).

From an operations perspective these kinds of outages are nothing new, and underscore why having "many eggs in few baskets" is such a problem. I believe we will see similar incidents when we have the first multi-datacenter failures where multiple providers lose significant parts of their infrastructure in a single geographic area. (Remember: location is a basket too!)

To really understand the current issue, I recommend Neal Stephenson's incredible (and lengthy) Wired article from 1996 entitled "Mother Earth Mother Board":

[...] It sometimes seems as though every force of nature, every flaw in the human character, and every biological organism on the planet is engaged in a competition to see which can sever the most cables. The Museum of Submarine Telegraphy in Porthcurno, England, has a display of wrecked cables bracketed to a slab of wood. Each is labeled with its cause of failure, some of which sound dramatic, some cryptic, some both: trawler maul, spewed core, intermittent disconnection, strained core, teredo worms, crab's nest, perished core, fish bite, even "spliced by Italians." The teredo worm is like a science fiction creature, a bivalve with a rasp-edged shell that it uses like a buzz saw to cut through wood - or through submarine cables. Cable companies learned the hard way, early on, that it likes to eat gutta-percha, and subsequent cables received a helical wrapping of copper tape to stop it.

[...] There is also the obvious threat of sabotage by a hostile government, but, surprisingly, this almost never happens. When cypherpunk Doug Barnes was researching his Caribbean project, he spent some time looking into this, because it was exactly the kind of threat he was worried about in the case of a data haven. Somewhat to his own surprise and relief, he concluded that it simply wasn't going to happen. "Cutting a submarine cable," Barnes says, "is like starting a nuclear war. It's easy to do, the results are devastating, and as soon as one country does it, all of the others will retaliate."

As the capacity of optical fibers climbs, so does the economic damage caused when the cable is severed. FLAG makes its money by selling capacity to long-distance carriers, who turn around and resell it to end users at rates that are increasingly determined by what the market will bear. If FLAG gets chopped, no calls get through. The carriers' phone calls get routed to FLAG's competitors (other cables or satellites), and FLAG loses the revenue represented by those calls until the cable is repaired. The amount of revenue it loses is a function of how many calls the cable is physically capable of carrying, how close to capacity the cable is running, and what prices the market will bear for calls on the broken cable segment. In other words, a break between Dubai and Bombay might cost FLAG more in revenue loss than a break between Korea and Japan if calls between Dubai and Bombay cost more.

The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is $30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark. [Link]

Update Feb-06 @ 08:52 GMT: I am aware of five cable segments that are experiencing problems, including one that was reported on January 23rd which had a repair already underway. I don't think this is a "fifth cut" as some people are starting to report, and I'll post an update if that changes.

A lot of needless confusion and worry could be avoided if FLAG Telecom and the other carriers involved would provide timely and useful updates on their website. It appears that they are doing a good job of restoring connectivity, but they are terrible job of telling an increasingly concerned public exactly what is going on. This kind of confusion resulted in false reports that "Iran was completely offline", which was corrected by the Renesys blog team after the story spread to influential blogs, Slashdot, Digg, and the mainstream media.

Failure Happens: Transcontinental fiber-optic submarine cables

by Jesse Robbins | @jesserobbins | comments: 9

The Guardian published a summary of the ongoing impact from the transcontinental fiber-optic submarine cable cuts along with a map from Telegeography.com:

According to reports, the internet blackout, which has left 75 million people with only limited access, was caused by a ship that tried to moor off the coast of Egypt in bad weather on Wednesday. Since then phone and internet traffic has been severely reduced across a huge swath of the region, slashed by as much as 70% in countries including India, Egypt and Dubai. [...]

"It will depend on how bad the damage is, but they'll find the sections in question and bring them up onto a ship for repair before sinking them again," said Mauldin. "It could take a week or possibly two weeks."

The fibre optic wires in question - called Flag Europe-Asia and Sea-Me-We 4 - are some of the most vital information pipelines between Europe and the east. The latter, which runs in an uninterrupted line from western Europe to Singapore, had only recently been opened after a mammoth £500m, three-year installation project. Between them, the two lines are responsible for around 75% of all connectivity in the Middle East and south Asia.

(continue reading)

One Laptop Per Child will succeed even if it "fails"

by Jimmy Guterman | comments: 22

The way people are dismissing the One Laptop Per Child (OLPC) project this week reminds me of how people were treating Hillary Clinton during the five days between her Iowa defeat and her New Hampshire comeback. To many observers, the inevitable has become the disaster in record time.

Some of the anti-OLPC notes that have appeared since Intel was kicked out of the project have been well-reasoned (read the Economist's near-obituary and Nikolaj Nyholm on Radar) -- but much of the anti-OLPC opining has deteriorated to personal attack on OLPC head Nicholas Negroponte. There are plenty of forces that want OLPC to fail commercially. And, for a variety of reasons, it might.

But what does "fail" mean in the market OLPC is trying to serve? Regardless of whether it's the XO laptop, Intel's Classmate, Pixel Qi, or some other endeavor, it's now far more likely that ultra-low-cost PCs are going to be made available in quantity for a developing world that needs them. (It needs clean water and vaccines more, of course, but it needs inexpensive and efficient IT as well.) And, most important, even if the XO laptop fails in the marketplace, none of this activity -- commercial and otherwise -- would have happened without the breakthrough OLPC project to start it.

P.S. To learn more about the XO laptop's technology, I recommend this post from "Bunnie" Huang. To understand an unexpected example of its utility, see Mike Hendrickson, here on Radar.

Does Facebook own this blog post?

by Jimmy Guterman | comments: 20

Facebook, apparently, owns my birthday. Yours too.

At least that's one way to interpret why blogger Robert Scoble got kicked off Facebook. While testing an upcoming version of Plaxo Pulse, Scoble scraped information on his contacts (name, address, and birthday, so he could move them to Outlook, he says), which turns out to violate Facebook's terms of service.

Self-promotion is certainly an aspect of this made-for-the-blogosphere event. It's not like getting kicked off Facebook has prevented Scoble from broadcasting his every micromove. (And it's not like scraping isn't something social networks have to monitor. There are Black Hat scrapers, too. See Dare Obasanjo for more on this angle.) But there are serious issues here deeper than Scoble's behavior. We are strong believers in projects that open up the social graph, we've expressed disappointment when early attempts to do so have delivered less than promised, and we've noted when social networks run amuck with our data. Facebook is locking in its customers. Mark Zuckerberg may be young, but he's not too young to remember how AOL fared with a similar lock-in strategy when the open web challenged it.

The question is simple: Is it your data -- or is it Facebook's? Facebook has given its opinion. What do you think -- and what are you going to do about it?

(See Kara Swisher for more on the topic. Nick Carr covers it, too. He disagrees with my take, and Kara's, but his post made me laugh. In a good way.)

'Computing in the Cloud' workshop hosted by Princeton University - January 14-15

by Jesse Robbins | @jesserobbins | comments: 1

Marc Hedlund and I will be speaking at the 'Computing in the Cloud' workshop hosted by the Center for Information Technology Policy at Princeton on January 14-15. The sessions look very interesting and registration is free.

Panel 1: Possession and ownership of data - In cloud computing, a provider's data center holds information that would more traditionally have been stored on the end user's computer. How does this impact user privacy? To what extent do users own this data, and what obligations do the service providers have? What obligations should they have? Does moving the data to the provider's data center improve security or endanger it?

• Joel Reidenberg, (home page), Professor of Law, Fordham University
• Timothy B. Lee, blogger at Technology Liberation Front and adjunct scholar, Cato Institute
• Marc Rotenberg, Executive Director, Electronic Privacy Information Center

Panel 2: Security and risk in the cloud - How does the move to centralized services affect the security and reliability of users interactions with technology? What new threats are likely to emerge? How might provider behavior, user behavior, or government policy need to change in response to those threats? How does the open source ethos work in a cloud computing environment?

• Marc Hedlund, founder and chief product officer, Wesabe.com
• Mihai Christodorescu (home page), researcher at IBM TJ Watson Research Center
• Benjamin Mako Hill researcher at MIT Media Lab and Free Software Foundation

Panel 3: Civics in the cloud - How and where can cloud computing best improve public knowledge and engagement in political issues? What has been achieved so far? What is possible in the long run? What moves by private actors, and what policy changes, might do the most to harness the power of cloud computing for civic engagement?

• Josh Tauberer (home page), founder of Govtrack.us
• Andrew Page, associate director, MAPLight.org
• John Wonderlich, Program Director, Sunlight Foundation

Panel 4: What’s next? What new services might develop, and how will today’s services evolve? How well will cloud computing be likely to serve users, companies, investors, government, and the public over the longer run? Which social and policy problems will get worse due to cloud computing, and which will get better?

• Andrea LaPaugh (home page) Professor of Computer Science, Princeton University
• Reihan Salam The Atlantic Monthtly
• Jesse Robbins O'Reilly Radar

Updated on 1/21/08. Here is the the video of my panel:

"Privacy is protected because it is essential to liberty" - Senator Dodd blocks Telco Immunity

by Jesse Robbins | @jesserobbins | comments: 2

Senator Chris Dodd (D-CT) has temporarily defeated an attempt to pass the Foreign Intelligence Surveillance Act (FISA) which would have provided immunity to telecommunications companies who cooperated with the Bush administration’s secret wiretapping program.

"After nearly a full day spent on the Senate floor, Senator Chris Dodd (D-CT) defeated an attempt to pass the Foreign Intelligence Surveillance Act (FISA) reform legislation that would grant immunity to telecommunications companies who cooperated with the Bush administration’s secret wiretapping program. Dodd objected to the motion to proceed to the bill early this morning and remained on the floor for almost ten hours, taking a stand for the rule of law and the Constitution with his statements throughout the day. At approximately 7:30 P.M. Majority Leader Reid announced the FISA reform bill would be pulled from the Senate calendar and reconsidered in January."

Update: Cory Doctorow explains why this issue is important:

Here's the thing: EFF and others are suing the telecoms for participating in the wiretapping program. These lawsuits are the best chance we have of getting the details of the program into the public, so we can finally find out what the NSA have been doing to us all these years. The reason the government wants to grant the telecoms immunity is to keep the dirty laundry in the closet -- to keep us from finding out how they've been breaking the law.

Coverage: Wired News, The EFF, AP, New York Times, ZDnet, Slashdot, Cnet News, Huffington Post, BoingBoing

Technorati Tags: att, chrisdodd, data, eff, fisa, infrastructure, lawsuit, politics, privacy, surveillance, wiretapping

Condescending Customer Service

by Jimmy Guterman | comments: 21

When Gmail recently added IMAP to its features, I sent a note to the customer service box of the web-based email service I use for personal mail to ask whether IMAP was coming to that service. I received the following reply:

I understand that you are want to use IMAP. I welcome the opportunity to assist you with your concern.

So far, so good.

Jimmy, I regret that the feature is currently not available but I appreciate that you have given us a wonderful idea to improve our service. I hope that the feature will be available in future. Since, you have given us this idea, only your brilliant idea will be the reason for this.

I guess I'm not entirely surprised that IMAP wasn't coming any time soon, but that final sentence really bothered me. Was that sentence necessary? Was it going to satisfy any customers? Or was it just condescending? I thought about that for two seconds and went back to my life.

As it turns out, that email vendor also hosts a non-O'Reilly-related website for me. A few days ago, the site went down. So I wrote again to the customer-service box, reporting that attempts to reach any page on my site was resulting in 404's. The problem went away several hours later and the next day I received an email from customer service.

I am sorry that you had to go through this unpleasant experience and I apologize for the inconvenience that you are facing with this issue. I welcome the opportunity to assist you with this concern.

Again, so far, so good. The email did have some useful information, including the results of some pings and what I might want to do if the problem continues. And then...

Jimmy, I visited your site using three different web browsers and found that you site is resolving at a normal pace. Moreover, it is one of the most attractive site [sic] which I have ever seen. I am sure that other visitors are also able to access your website properly.

This time it's the next-to-last sentence that set me off. First of all, it's a lie. The site is attractive much in the same way Alfred E. Neuman is attractive: i.e., not at all. Worse, that condescending tone casts a shadow over the whole interaction.

Two interactions with a company: In one, it claims that it can't help me despite my "wonderful" idea being "brilliant." In the second, it ruins a perfectly useful response by delivering empty flattery. At a time when the switching costs between email and web-hosting services are not all that high, why is a company presenting such off-putting scripts?

Failure Happens: Taser-wielding thieves steal servers, attack staff, and cause outages at Chicago colocation facility

by Jesse Robbins | @jesserobbins | comments: 7

Dan Goodin at The Register reports that C I Hosts' Chicago facility was robbed last month for the third time... fourth time... second time (the other two times were merely "break-ins where things were stolen")

In the most recent incident, "at least two masked intruders entered the suite after cutting into the reinforced walls with a power saw," according to a letter C I Host officials sent customers. "During the robbery, C I Host's night manager was repeatedly tazered and struck with a blunt instrument. After violently attacking the manager, the intruders stole equipment belonging to C I Host and its customers." At least 20 data servers were stolen, said Patrick Camden, deputy director of news affairs for the Chicago Police Department.

The Chicago location has been hit by similar breaches in the past, according to police reports. One report detailing an occurrence on September 23, 2005, recounts a "hole cut through the wall coming out onto the hallway of third floor." During a September 20, 2006 incident, an intruder "placed a silver + blk handgun to [victim's] head and stated 'lay down on the floor.'" The victim, a C I Host employee, was then blindfolded, bound with black tape and struck on the head with a weapon, according to the report.

Wow... I hope that everybody is now okay. There is some interesting discussion by affected customers over on the WebHostingTalk forums.

I'll be doing a post-incident report using the Simple Availability Report format I introduced last week. (If you would like to contribute please post in the comments or email me directly jesse AT oreilly.com)

Updated: Anastasia Tubanos (theWHIR.com) has posted her interview and followup with James Eckles, chief corporate counsel for CI Host. (link)

"There's no resolution really," he says. "We're dealing with the situation on a customer-by-customer basis. We've got nothing to hide, even though people have been saying otherwise online. The forums have been a bed of misinformation - extortion compounded with defamation. One of the biggest mistakes is that people are talking about four robberies. A robbery means than property has been seized through violence or intimidation. C I Host has technically only been robbed twice in two years. The other two were break-ins where things were stolen, but not robberies."

Technorati Tags: crime, downtime, facilities, failurehappens, infrastructure, operations, outages, procurement, security, web20, webhosting

Disaster Telecom after the earthquake in Peru

by Jesse Robbins | @jesserobbins | comments: 4

The BBC is reporting that over 500 people were killed and thousands of people left injured and homeless after the earthquakes in Peru earlier this week.

The 24 hour Skype outage started shortly after the earthquake and contributed to the initial chaos. Skype's Villu Arak has claimed that the problems are resolved and promised to provide details about the cause on Monday.

On a positive note, Telecoms Sans Frontières (Telecoms without Borders) is on the ground in Peru and has deployed their first communications center in Pisco. They will then deploy two additional centers in Ica and Chincha. These centers provide communication resources for government & aid organizations along with free calls anywhere in the world for affected civilians.

TSFI is looking for volunteers & donors.

Your browser is a tcp/ip relay

by Artur Bergman | comments: 11

I've been a longtime fan of fellow hacker Dan Kaminsky, best known for his work in tracking down the spread of the sony rootkit. Recently I spoke with him about his current work, and he summed it up by saying, "I can turn your web browser into an VPN concentrator." When I stared at him in disbelief he explained that using DNS rebinding he can get the browser to connect to any IP he chooses.

The technique originates in the browser security model, based on same-origin policy. This allows a web browser, either using JavaScript or Flash, to connect back to the same host that the content came from. If the attacker changes where the hostname is pointing to, the browser can connect there. For example, the next time you connect to attacker.com, the DNS server actually serves you a 192.168.1.1 address, allowing the webapp to connect to your internal IP.

At Black Hat in his Black Ops 2007: Design Reviewing The Web talk, Dan released his slirpie tool, a framework for allowing you to tunnel traffic through a person's web browser.

I will demonstrate an extension of RSnake and Boneh's work, that grants full IP connectivity, by design, to any attacker who can lure a web browser to render his page.

I have had for a while a lurking feeling that the Web 2.0 world is full of surprising attack vectors that no one has come around to exploiting. Work like this doesn't exactly fill me with confidence that the environment is secure.

If you have ever wondered what framework a certain site uses, Dan also gives us p0wf.

But the web has become almost an entirely separate OS layer of its own, and especially with AJAX and Web 2.0, new forms of RPC and marshalling are showing up faster than anyone can identify. p0wf intends to analyze these streams and determine just which frameworks are being exposed on what sites.

I am really happy Dan is on our side.

** Update 6:03 PM PST **

Megginson Technologies has more details and how it can affect you.

** Update 6:00 PM Thursday, Augusust 2 **

Dan finally posted his slides. He also discusses breaking audo captchas and busting provider hostility, the opposite of network neutrality.