User Mediated Trans-Enterprise-Web Mashups?

There has been an on again off again discussion behind the scenes at Radar about the nature of the enterprise vs. the web and how they are defined not just by their technologies, but by their frames of reference. For my part, I think the enterprise view is defined implicitly by a planning mindset and a perceived scope of control that ends at the enterprise boundary. Whereas the web is too large for effective control so it tends to be an environment more conducive to serendipity and emergent behaviors. The web and the enterprise also differ in obvious cultural ways. Web culture tends toward speed and "good enough" while enterprise culture is informed by enterprise concerns like mission criticality, legal frameworks such as HIPPA and Sarbox, security, transaction volumes, and the like. These thoughts were still rattling around in my head last month when I arrived in Montreal for the weekend.

Just like every other year, as soon as I cleared customs I skipped the crappy exchange rates at the arrival area exchange vendors and headed upstairs to my favorite ATM machine in the departure hall. I needed to get cash for my cab ride to the center of town, only this time, the machine spit my card back out like day old sushi. I tried another ATM further down the hall with the same result.

After ten frustrating minutes of IVR traversal and the international roaming fees that went with it, I was talking to a Wachovia Bank customer service representative who politely suggested "you should have called us before you left the country, then we wouldn't have automatically blocked your card."

Apparently Wachovia (like many other banks) has decided the best way to reduce their risk of fraudulent transactions is to convert that risk to customer hassle with an algorithm that looks something like: IF Loc <> Home Country/County/City SET CardStatus to Blocked. My bank is now my mom and I have to call it and get permission before I am allowed to go out and play.

The funny thing about all this is that even though Wachovia suspected I hadn't accompanied my ATM card to Montreal, plenty of others knew that I had, including at least: AT+T (my cell phone provider), Verizon (my blackberry provider), Dopplr, USAA (I booked my flight with their credit card), Travelocity (where I booked the flight), US Airways, Plazes, Yahoo Fire Eagle (fed from Plazes and Dopplr), and naturally, the U.S. and Canadian Border Authorities.

(continue reading)

An ESB for the Web?

I spend a great deal of my time encouraging "enterprise people" to think more like "web people." Focus on adoption, use platforms to enable emergent capability, build the "generative enterprise," and that sort of thing.

So, imagine my surprise when I saw the web acting a bit like the enterprise with the launch of Gnip.

As the web moves toward a network of widespread transactional API's, each with it's own vocabulary, it is starting to look a lot like a legacy enterprise writ large or maybe like an industry eco-system. So we shouldn't be surprised to see web developers turning to solutions that their enterprise colleagues would find familiar.

Anyone who has spent more than five minutes in the enterprise world talking about SOA in the last five years (or spent time building "trading platforms" for industry consortiums prior to that) has probably drawn a picture on a whiteboard that looks something like this (see, almost identical):

Whether you have integrated line of business applications inside the enterprise or connected trading partners within an industry, that N squared connection problem will resonate with your experience. Webs of poorly documented point-to-point integrations are expensive to build, expensive to maintain, and impossibly brittle when the business changes.

And now the N squared problem seems like it might be beginning to resonate with web developers too now that they have to integrate to an ever growing population of API's. Plus, on the web, the additional limitations of a port 80 based infrastructure add to the nightmare by throwing the expense of constant API polling into the mix.

So, what to do?

(continue reading)

Hyperic CloudStatus service dashboard launches at Velocity!

Javier Soltero just launched CloudStatus during his Hyperic sponsor session today at Velocity. CloudStatus is a public health dashboard for web services like Amazon's EC2/S3, and Google's App Engine.

Javier called to tell me about this last week after I declared that "Service Monitoring Dashboards are mandatory". This comes right after Amazon and Google had visible outages, and couldn't have happened at a better time. I'm really excited to see this idea take off, as it's something that is critical to the broad adoption of web services and cloud computing.

BarCampBank is spreading

When Ben Black and I organized the first BarCampBank in North America last year, we hoped that it would spread. According to William Azaroff's post on NetBanker, the movement is there and growing:

What's all this about BarCampBanks? From a North American premiere in Seattle almost a year ago, we've witnessed two more in the last few months, and eight more are either scheduled, or in the planning process.

Well, maybe not exactly “planned.” BarCampBanks emerge more than they are planned.

[...]It started as a technology summit, an un-conference where developers and technology geeks could share exploits, connect, and find like-minded companions to extol the virtue of open-source and emerging technologies over pizza and wine.

And then someone decided that this forum would be a perfect place to talk about banking and finance. Weird. And yet it works.

The next event will be BarCampBankDallas on June 21-22nd at the American Bank of Texas Building in Frisco, Texas. William has details on the other events this year on his blog, and a current list can always be found on the main BarCampBank wiki.

Special Purpose Computing Focuses on Energy Efficiency

To improve the climate models that predict global warming, climatologists are seeking model resolutions on the order of 1 km. Unfortunately, building the required 200 petaflop machine with today's commodity-hardware approach would cost $1B and would result in a staggering 40 megawatts of power consumption.

A group of researchers at Lawrence Berkeley National Labs, who must be aware of the irony inherent in using 40 continuous megawatts to better predict global warming, may be returning supercomputing to its specialized roots but along a new vector (yes, weak pun intended). In addition to the Cray-era focus on raw power, they are emphasizing energy efficient computation where floating point operations per watt is the key metric.

Their approach has already been described at EcoTech Daily and the lab's Research News so I'm just going to summarize it here. They are working on specialized hardware consisting of 20 million very low power embedded processors (of the sort used in iPods and cell phones) wired together with the specific climate calculations in mind. By trading flexibility for efficiency, the design should achieve a ten-fold improvement in the floating point operations per watt metric and the resulting 200 petaflop machine is predicted to require only 4MW of power and cost $75M to build.

Their motivations in their own words:

"What we have demonstrated is that in the exascale computing regime, it makes more sense to target machine design for specific applications," Wehner said. "It will be impractical from a cost and power perspective to build general-purpose machines like today's supercomputers."

Specialized problems are amenable to specialized solutions and scientific computation seems particularly suitable to this kind of approach. However, on the web and in corporate IT where computing is both more general and inefficiently deployed, the first wave of energy efficiency improvements are being addressed primarily through a combination of virtualization and incremental improvements in commodity chip design.

I don't think software carpooling will be the only game in town for long though. While virtualization and dynamic provisioning are facilitating better utilization of existing hardware, virtualization comes with a performance cost of its own and can be no better than the hardware it is running on. Once you get four passengers in a V-8 powered SUV further improvements have to come from changing driving habits and modifying the vehicle.

As virtualization initiatives pick the low hanging fruit, further gains will come from fundamental hardware improvements (which may include analogous specialization) in concert with "best efficiency" dispatching that targets optimal server utilization in a dynamic server pool. An interesting example of this kind of approach is described here (pdf).

As I've touched on before, beyond that, a "systems view" to optimize the whole data center as it operates under changing conditions and with heterogeneous hardware might come next. Returning one last time to the carpooling analogy, this would be like a smart traffic routing system that keeps each car-pooling hybrid moving at its most efficient speed. The end result might be an optimally-sized mixed pool of specialized and commodity hardware each dispatched to operate the data center holistically at its best unit of work per watt.

GSM Cracking: Coming Soon to a Computer Near You via a Web Service

A web service that will make it easy and inexpensive to crack the GSM A5/1 encryption protocol, quickly enough for a call that is still in progress, is slated to launch at the end of April. Living right at the intersection of open hardware, open source software, software as a service, and cryptography, the service will reduce the cost and effort of cracking GSM call encryption by at least an order of magnitude.

The service is being developed by members of the GSM Software Project and demonstrates just how much things have changed in the world since the GSM system was designed. Various approaches to cracking both A5/1 (the European standard) and A5/2 (the weaker US standard) have been available for some time but this one is unique in that it should be available to researchers and hackers at the end of April in hosted api form instead of pdf.

Back in 1997 this overview of the GSM system declared that "Enciphering is an option for the fairly paranoid, since the signal is already coded, interleaved, and transmitted in a TDMA manner, thus providing protection from all but the most persistent and dedicated eavesdroppers." After all, such a radio encoding scheme made the signals invisible to typical radio band scanners.

Today, however, the availability of the Universal Software Radio Peripheral (USRP), an open hardware software defined radio that sells for about $700, combined with work being done at GNU Radio project to codify the GSM waveform (also targeted for the end of this month), makes this once reasonable point of view seem quaint. Good encryption is now a must and it appears that A5 no longer qualifies.

With USRP and GNU Radio making the waveform and encrypted frames cheaply accessible and the A5 Hacking Project's service easily breaking A5/1, anyone will be able to make a cheap GSM scanner. Today neither the complexity of the waveform or the encryption in use is adequate to keep a GSM call private.

(continue reading)

Disaster Telecom after the earthquake in Peru

The BBC is reporting that over 500 people were killed and thousands of people left injured and homeless after the earthquakes in Peru earlier this week.

The 24 hour Skype outage started shortly after the earthquake and contributed to the initial chaos. Skype's Villu Arak has claimed that the problems are resolved and promised to provide details about the cause on Monday.

On a positive note, Telecoms Sans Frontières (Telecoms without Borders) is on the ground in Peru and has deployed their first communications center in Pisco. They will then deploy two additional centers in Ica and Chincha. These centers provide communication resources for government & aid organizations along with free calls anywhere in the world for affected civilians.

TSFI is looking for volunteers & donors.

Patent Reform Project Looking for RoR Help

We don't normally broadcast job postings, but Andy Oram writes about one that could be of interest to our audience:

The Community Patent Review (a phase of the better-known Peer to Patent project) is seeking to hire a top-notch Ruby on Rails programmer.

This is not an ordinary job posting; it’s a chance to get paid for public service. The project is developing software for the U.S. Patent and Trademark Office, and is being closely watched in the UK too. It implements a process to let experts in multiple fields to submit prior art and other information related to patents.

This project has the potential to alleviate some of the problems universally recognized with patents (particularly patents that should have rejected because of prior art) and can serve as a model for other projects that open up government to the public....

As a Rails programmer, job candidates are expected to handle basic JavaScript, CSS, and Ajax work as well. Residency in San Francisco is preferred....The creator and coordinator of the project is Beth Simone Noveck, a New York Law School professor. If you’re interested, please reach her at bnoveck (at domain) nyls.edu."

Read Andy's post for more details.

IBM Wants Many Eyes on Visualization

IBM today announced Many Eyes, a site for sharing and commenting on visualizations. Martin Wattenberg, who developed the original version of the treemap we use for our book market visualizations as well as the awesome baby name voyager, and Fernanda Viegas, who worked with him on the equally awesome history flow visualizations of Wikipedia, are the geniuses behind this project.

As with swivel, users can upload any data set, but the tools for visualizing and graphing the data are much richer. The visualization options include US and World maps, line graphs, stack graphs, bar charts, block histograms, bubble diagrams, scatter plots, network diagrams, pie charts, and treemaps. The site isn't yet live, but should be very shortly. Meanwhile, you can get a good sense of the types of graphs available by checking out the visualization gallery.

I asked Martin and Fernanda how they compared themselves to swivel, and Fernanda replied:

You also asked if we see our site as "Swivel for visualization". That phrase isn't quite accurate (any more than Swivel is "Many Eyes for data" ;-). Both our site and Swivel are examples of a broader phenomenon, which we call "social data analysis," where playful, social exploration of data leads to serious analysis. At the same time the two sites fall on different ends of a spectrum. Swivel seems to have some neat data mining technology that finds correlations automatically. By contrast, we've placed our emphasis on the power of human visual intelligence to find patterns. My guess is that both approaches will be successful because social data analysis is a powerful idea.

Martin added:

In Many Eyes our goal is to "democratize" visualization by offering it as a simple service. We also think that there's something special about visualizations that gets people talking, so we placed a big emphasis in design and technology to let people have conversations around the visualizations.

Personally, I'd love to see swivel and manyeyes working together, as swivel already has some great data sets, but has only a limited number of graphing tools. But that's an exercise for the future. For now, data wonks can just rejoice that both sites exist, and should start exploring, and as Martin says, conversing about what they find. I love both of these sites.

Worldmapper Visualizations

"Worldmapper is a collection of world maps, where territories are re-sized on each map according to the subject of interest." [via] See how the map of the world changes when you look not just at land area, but at other factors, such as population, GDP, import/export flows, or HIV infections. There are literally hundreds of map projections. This would make a cool demo for Where 2.0, I think..

If you like this kind of thing, also look at Hans Rosling's amazing visualizations at gapminder.org.

Onto play

Justin Hall's Passively Multiplayer Online Game concept is an interesting spin on some of the issues with data ownership and the transparent society that we have been blogging about recently.

Hall's point is that one of the reasons the web experience is not nearly as absorbing or fulfilling as playing games, is that the web is in fact not paying attention to what you do when you surf. It does not adapt to your experience of the web or your surfing habits. It's an interesting thought, even if there are some obvious counterquestions: If we always live in a world adapted to us, when will we ever meet other people in a meaningful way? Is game-like immersion really a good global goal? We all know that flow is a pleasing and important experience, but is it really the only experience we need?

Some concrete ideas that touch on some of the same notions can be found in presentations by Matt Webb on ideas for a sensory web browser at reboot8 and 'a zoology for web applications' at EuroFoo.

NSF Grant for Wearable Computing

Ross Stapleton-Gray sent me a pointer to a recent NSF grant to support research on the infrastructure needed for wearable computing. From the grant abstract:

The next-generation of embedded computing systems will be increasingly human-centric. Embedded devices will operate in an environment where they monitor and react to human activity as opposed to engineering artifacts. Transparent interfaces and implicit modes of human-computer interaction will be enabled by sensors that are unobtrusively embedded in our immediate environment. These sensors and other embedded devices will accompany their (human) host to ensure uninterrupted service. Applications of such devices will include augmenting human capabilities, logging past activities (e.g., as a memory aid or to provide longitudinal monitoring), ensuring safety, and enhancing social connections (by offering a new form of remote access to an individual). Personal effects (such as attire) are a prime target for instrumentation to enable the human-centric, embedded-computing vision. Smart wardrobes equipped with distributed sensing, computing and memory resources are thus likely to become an increasingly popular platform for embedded computing. The objective of this project is to investigate the software infrastructure required to enable and support future wearable embedded systems such as those that reside in smart wardrobes and other personal effects. The project identifies the challenges encountered in building such a software infrastructure, and develops a new embedded operating system and middleware service architecture to address these challenges.

New version of Safari Books Online

Safari Books Online version 5.0 went live last week. The new version includes a design overhaul and is much more usable than previous versions, in my opinion. Not only do we now have Ajax-based paging (no more tedious page-refresh), we've also adjusted the amount of content shown in both preview and full access mode so that you aren't having to page anywhere near as often. (The original, far-too-limited amount of content shown on each page was driven by concerns from authors and other publishers about possible piracy. The net result, as is often the case with excessive fear-based security, was to sacrifice usablility. We now have enough experience under our belt to persuade doubters that it's in everyone's best interest to make the site as usable as possible. I think we've taken huge steps forward with this release, and I'd love your feedback about whether you agree.) As part of this family of fixes, we've also made substantial improvements to the Safari search capabilities, including searching of books that aren't on your bookshelf and more satisfying and useful search results for non-subscribers.

Also as part of the new revision, we've added "Graphically Rich Books" including the Head First series, Peachpit's Visual Quickstarts, and various web and graphic design books. These books were previously not available in Safari as it was difficult to present their content in the old HTML-only version.

A few other things I especially like in the new version:

(continue reading)

Where 2.0: Location Sensing Lightning Talks

The first lightning talk session at Where 2.0 on location sensing:

• Christopher Schmidt on Geolocation Using GSM Cells. GSM Loc allows you to collaboratively map GSM cell towers using GPS allowing other non-GPS users to approximate position based on signal strength of individual cell towers. OpenStreetMap for cell towers.

• Jason Kaufman on Gumspots and GSPS. Gumspots is a fantastic crowdpleasing service which maps bubble-gum spots on sidewalks in Manhattan to a geographic location. Gumspots works like semacodes (see image on right).

Where 2.0: Socialight Podcast

In the final Where 2.0 podcast I speak Michael Sharon, CTO of Kamida, a New-York-based mobile company. We focused on Socialight a mobile tagging app which allows users to leave sticky shadows. These are notes, pictures, and audio messages that are tied to a specific location that are accesible from a phone or their website. At Where 2.0, Michael will talk more about Socialight and the future of mobile apps. (MP3)

Superoyster is in alpha

SuperOyster, the Seattle startup that is creating marketplaces for spots in line, has gone alpha. They've created 3 fake marketplaces where you can get a spot in line. "sell" your spot in line, and "buy" a spot in line. The marketplace for the fictional L.A. Stars has over 200 people playing so far.

The alpha provides a good feel for how the service will work and, although the code is still a bit rough around the edges, it's clear that they've done a good job of keeping waiting lists simple. If you want to play around, i recommend reading the caveats first.

Où trouver De l'art de programmer en Perl ?

Laurie Petrycki writes: "A cool mapping mashup that links our French book database with Google maps, hacked by Julien, the French office's sys admin. The book icons are color coded to show the probability of finding the book (click on an icon to see store details). Our sales rep in France has already used the map to encourage bookstores to maintain appropriate stock. In your mind's eye, you can see yourself wandering through the bookshops in the 6th Arrondissement." Nice work, Julien!

Xavier Cazin, Editor in Chief of Editions O'Reilly (O'Reilly France) adds: "It works only for french books yet, and french zip codes, because these are the only sales and geo data we have currently in our database. The sales are updated daily, so the color of the bookstore may change depending of when was its last order."

I've asked our US staff to look into the possibility of a similar mashup for our US retail presence. But I should warn that it's a harder problem. In the US, a huge percentage of our sales are through chain stores, and we don't get per-store data on what books they carry. Even in France, as you'll note, we don't show the absolute presence or absence of a book, just the likelihood that they have it in stock.

TrafficGauge rocks

Last year, I accepted membership in a buzz marketing experiment called the Silicon Valley 100, in which people like me who are deemed to be "influencers" are offered various free products to test. I accepted with some misgivings -- was I selling out? -- but accepted after being assured that there was absolutely no expectation of endorsement of any of the products.

In the year or so since, I've received a fair share of products to test out. But I've never been inclined to say anything about any of them until I got my TrafficGauge. I was initially skeptical: one more special-purpose device to tote around. But I was immediately won over.

How stupid does this thing look? A small handheld device about the size of a Treo or a Blackberry. On its face a low-res map of Bay Area freeways (other cities also available). Turn it on, and it stays on. When there's no traffic, the map shows nothing. If traffic is slow, heavy dashed lines appear on the freeways affected. If it's stop and go, the dashed lines flash. That's it.

Now I know I could get this information on my phone, or even get real time traffic fed to my Garmin Streetpilot GPS. But having a dedicated device is surprisingly useful. I keep it in my glove compartment, and pull it out when I'm in doubt about which route to take. It's always on, and so there's no user interface to fiddle with, just a quick glance. I find it particularly useful at times when you wouldn't normally expect traffic -- for example, when I was heading home from Startup School (Palo Alto to Sebastopol) on a Saturday afternoon. It saved me from going home through Berkeley (my usual preferred route to the North Bay) by warning me of a slowdown (most likely due to an accident) at a point where traffic is normally light.

What's also interesting about this device is the very fact of its single purpose design, and the fact that its low-power display and sensors stay always on, running for months on a couple of AA batteries. It's a sign that special purpose hardware providing access to internet data services may eventually be commonplace. (I remember a nice presentation years ago by someone from IBM in which he compared pervasive computing to the spread of clocks: the clock tower was equivalent to the mainframe, the grandfather clock to the minicomputer, the clock-radio to the PC, and the embedded clocks you see everywhere to the future of computing.)

Sell Your Place In Line

Recently, I got together with Matt Kowalczyk, the founder and CEO of Seattle-based SuperOyster.com. They are building a merchant service that allows people to sell their spot in line. This sounds whacky until you hear him rattle off the markets where this is applicable: pro football tickets, high-end automobiles, condos, fashion accessories, and exclusive memberships. It also applies very nicely to hot, scarce products like the Xbox 360 or concert tickets.

Pro football, in particular, is ripe for this type of service. Half of the teams have waiting lists for season tickets. The wait on these lists vary from 2 - 40 (40!) years. There is no consistency in the way these are managed. Some require paper submissions, some are managed with Excel, some a simple database. Some of them allow you to sell your place in line, but most do not. They need someone to bring them onto the web.

They plan to offer list management services to merchants that will not only provide basic waiting list features but will create an online market where these list positions can be bought and sold. The companies can choose between a hosted version or a webservice. Superoyster charges the spot seller a transaction fee and gives some of this to the client. The client will really win in this scenario. They no longer have to handle maintenance on their waiting lists and they are able to develop a new revenue stream.

This is not a difficult technology. This is a business that is going to live and die by biz dev. Though no one else is currently in the market there are a couple of companies that are in a position to take advantage of this business plan. Both Razorgator and Stubhub have deals with various NFL teams and it may be easy for them to develop this in-house and leverage their existing relationships. If the market becomes lucrative enough bigger players such as Ticketmaster or eBay might consider similar services.

They are a small team, currently self-funded looking for additional funding. They plan on going Alpha this month, with a July Beta, and launch in September. You can track their progress on their blog.

Rhapsody Distributes Their Music

The other week I sat down with Leo Dirac of Rhapsody Webservices and he told me a bit about their upcoming plans. They want to be the music listening platform of choice for site-owners. Yesterday, they made a significant step towards this vision when Rhapsody (Real's subscription service) enabled anonymous users to listen to whole albums on demand for free with links from third-party sites.

The RESTful links used to enable this are very easy to read and write. The format, http://play.rhapsody.com/artistname/albumname (example: http://play.rhapsody.com/massiveattack/mezzanine) is simple to construct on the fly (paving the way for a multitude of Greasemonkey scripts and on-demand bookmarklets). When a user clicks on the link an ActiveX control or Firefox plugin is called and whole songs from that artist or album (new with this release) will start playing - for free. Anonymous users get 25 full song plays a month, any number beyond that degrades to the industry-standard 30sec clips; existing Rhapsody users get unlimited plays. Besides the download, the other unfortunate aspect of this implementation is that you are redirected away from the originating site to the associated album page on Real.

(continue reading)