Four short links: 12 October 2009

DSL for NLP Task, Insider Tradespotting, Outsource Fail, Cloud Fail

1. Snowball -- a small string processing language designed for creating stemming algorithms for use in Information Retrieval. (via straup on delicious)
2. Insider Trades -- a Yahoo! Hack Day app that turned out to be worth continuing. Scans SEC systems every 30 seconds and alerts you if the stock you track has been traded by an insider. (via straup on delicious)
3. Air New Zealand Slams IBM -- central point of failure in the outsourced IT. "In my 30-year working career, I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client's customers is not the glowing endorsement you want.
4. Danger/Microsoft Loses Sidekick Customers' Data -- Regrettably, based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. This cloud had a brown lining.

Four short links: 20 August 2009

DIY SPY, Screencasting, Social Network Analysis, Term Extraction

1. DIY SPY - a homebrew 2.4GHz wi-fi spectrum analyzer -- As proof of concept (and a cool toy for anyone who has one of these lying around), I have implemented a working Wi-Fi spectrum analyzer on TI’s ez430-RF2500 development kit ($50), a 2-part USB dongle which consists essentially of a CC2500 radio strapped to an MSP430 low-power microcontroller (detachable bottom half) and a USB interface which enumerates as a virtual serial port (top half). The top half doubles as a standalone MSP430 programmer, so this kit is a great cheap way to get started playing with them. (via joshua on Delicious)
2. Screenr -- Instant screencasts for Twitter. Flash-based, uploads to their site and tweets the URL. The whole "for Twitter" thing is going a little too far: who records screencasts only for Twitter? It's like having a spellchecker only for three-letter words.
3. Social Network Analysis in R -- video and slides for talk on doing social network analysis with R.
4. We're Keeping the Term Extraction Service -- Yahoo!'s useful API gets a stay of execution. OK, we heard you. You’ve made it clear to us that shutting down the Term Extraction Service would be a mistake. So, we’ve changed our plans. We're leaving the service up and running indefinitely. (via Simon Willison)

Four short links: 13 August 2009

1. Under the Hood of App Inventor for Android -- regular readers know I'm a big fan of visual programming language Scratch, and apparently Google are too. They've got twelve university classes testing App Inventor for Android, a visual connect-the-bits programming environment for Android. University classes probably because one of the co-creators is Hal Abelson, coauthor of the definitive programming textbook. Also found online: the PR-type announcement, a Professor using it, and @AppInv (nothing juicy on Twitter--it looks like might be a channel for tech support for the students). (via Hacker News)
2. Google Web Optimizer Case Study (Four Hour Work Week) -- GWO manages A/B tests for you, with a lot of statistical analysis. It's a fascinating read to see how these should be done. Every equation may halve the readership of a book, but every table of numbers and relevancy analysis doubles the value of a post like this. (via Hacker News)
3. Opening Up The BBC's Natural History Archive -- the BBC are releasing programme segments and a whole lot of metadata around their programming. Audio and video segmented, tagged with DBpedia terms, and aggregated into a URI structure based on natural history concepts: species, habitats, adaptations, etc. Gorgeous!
4. Yahoo! Term Extraction API to Close -- Internally, both services share a backend data source that is closing down, so the publicly-facing YDN services will be closing as well. I think it's the most significant casualty of Y! outsourcing search to MSFT, as this API was used by a lot of projects. (via Simon Willison)

Four short links: 31 July 2009

NoSQL, Goldman Sachs, Yahoo! Developer Products and Bing, and Alternate Reality

On this day in history, Mt Fuji exploded (781), Daniel Defoe was put in the stocks for seditious libel but was pelted with flowers (1703), the first U.S. patent was issued (1790), and the radio show The Shadow aired for the first time (1930).

1. Tokyo Cabinet: Beyond Key-Value Store -- description of Tokyo Cabinet and code examples in Ruby. More on the nosql move to leave relational databases behind for certain modern problems (such as scaling).
2. The Great American Bubble Machine (Rolling Stone) -- I know it's old hat, but read it for the poetry if for nothing else. The first thing you need to know about Goldman Sachs is that it's everywhere. The world's most powerful investment bank is a great vampire squid wrapped around the face of humanity, relentlessly jamming its blood funnel into anything that smells like money.
3. Yahoo!'s Developer Program and Bing -- note from Yahoo! to developers, saying that YQL, YUI, and Pipes are safe. For SearchMonkey and BOSS they currently do not have anything concrete to tell you. I assume (and hope) that Delicious is a top-level product, not something under "search". (via Simon Willison)
4. Preparing Us for AR -- (Schulze & Webb) round up of some apps and toys that show what AR might be, unfettered by current day technological constraints.

Four short links: 14 July 2009

Twenty Questions, CC Pix, INSERT INTO WEB, and Wash Your Hands!

1. Twenty Questions about GPLv3 (Jacob Kaplan-Moss) -- twenty very challenging questions about the GPLv3. foo.js is a JavaScript library released under the GPLv3. bar.js is a library with all rights reserved. For performance reasons, I would like to minimize all my site’s JavaScript into a single compressed file called foobar.js. If I distribute this file, must I also distribute bar.js under the GPL?
2. CC Searching within Google Image Search -- what it seems. (via waxy)
3. YQL INSERT INTO -- insert into {table} (status,username,password) values ("new tweet from YQL", "twitterusernamehere","twitterpasswordhere"). That's too cool. (via Simon Willison)
4. CleanWell -- very low-cost recyclable enviro-friendly antimicrobials to battle third-world disease. Met the founder at Sci Foo. He said women wash hands more than men, because women enter bathrooms in pairs. Single easiest way to increase handwashing compliance is to put sinks and basins outside the room, in public view.

Four short links: 9 July 2009

1. Ten Rules That Govern Groups -- valuable lessons for all who would create or use social software, each backed up with pointers to the social science study about that lesson. Groups breed competition: While co-operation within group members is generally not so much of a problem, co-operation between groups can be hellish. People may be individually co-operative, but once put in a 'them-and-us' situation, rapidly become remarkably adversarial. (via Mind Hacks)
2. Yahoo! TrafficServer Proposal -- Yahoo! want to open source their TrafficServer product, an HTTP/1.1 caching proxy server. Alpha geeks who worked with it are excited at the prospect. It has a plugin architecture that means it can cache NNTP, RTSP, and other non-HTTP protocols.
3. App Engine Conclusions -- I've reluctantly concluded that I don't like it. I want to like it, since it's a great poster child for Python. And there are some bright spots, like the dirt-simple integration with google accounts. But it's so very very primitive in so many ways. Not just the missing features, or the "you can use any web framework you like, as long as it's django" attitude, but primarily a lot of the existing API is just so very primitive.
4. Microsoft Hohm -- Sign up with Hohm and we'll provide you with a home energy report and energy-saving recommendations tailored to your home. Wesabe for power at the moment, with interesting possibilities ahead should Microsoft partner with smartmetering utility companies the way Google Powermeter does. This is notable because this is a web app launched by Microsoft, with no connection to Windows or other Microsoft properties beyond requiring a "Live ID" to login. For commentary, see Microsoft Hohm Gets Green Light for Launch and PC Mag. (via Freaklabs)

Four short links: 18 June 2009

Weaker Copyright Good, YQL.gov, GeoSPARQL, Happiness

1. Harvard Study Finds Weaker Copyright Protection Has Benefited Society (Michael Geist) -- Given the increase in artistic production along with the greater public access conclude that "weaker copyright protection, it seems, has benefited society." This is consistent with the authors' view that weaker copyright is "uambiguously desirable if it does not lessen the incentives of artists and entertainment companies to produce new works." (read the original paper)
2. Using Public Data for Good With the Power of YQL -- The first part is a new batch of YQL tables providing data on the U.S. government, earthquake data, and the non-profit micro-lender Kiva. The second part is an incredibly easy way to render YQL queries on websites. After all, what good is data that no one can see?
3. GeoSPARQL -- RDF meets geo goodness. SELECT ?s ?p ?o WHERE { ?s gn:name "Dallas" . ?s ?p ?o } (via the geowanking mailing list)
4. How To Be Happy in Business -- this Venn diagram makes me happy. (via Ned Batchedler)

FBML, YML, OSML oh my! HTML, meet Social

This morning Yahoo! launched the first fourteen OpenSocial applications for users of My Yahoo!, though as TechCrunch pointed out they did a bit of forking OpenSocial for their HTML-ish markup. It's not all that surprising considering that OpenSocial's support for this sort of markup (OSML) is relatively new, Yahoo! has been working on their application platform for quite awhile and OSML is just a bit strange.

For instance, the “small view” (i.e. the widgets which actually appear on the MyYahoo page) must be developed using “Yahoo! Markup Language” (YML), which is an extension of HTML with more bells and whistles. Yahoo is trying to bring together YML and the OpenSocial Markup Language (OSML), but right now they are forked. But turning an OpenSocial app into one that works inside Yahoo is getting easier.

Beware, the next few paragraphs get a bit geeky. YML (more info) is a lot like FBML and OSML (more info) in that they are all social markup languages. OSML is a bit different though, unlike YML which only works inside of Yahoo! and FBML in Facebook, OSML is part of the OpenSocial project and is designed to work inside of many different social network containers. If I wanted to display a user's name inside of my application, here's what it would look like:

• FBML: <fb:name uid="4" />
• YML: <yml:name uid="QPR12345" />
• OSML: <os:Name person="${User}"/>

In this simple example, FBML and YML are nearly identical; you pass in a userid. OSML is a bit different, they've created a rich templating language and you're passing in a user object instead of just a userid.

XFBML is the evolution of FBML but designed for use via Facebook Connect. Given that XFBML is designed to work for sites outside of Facebook.com, I'm much more interested in the ideas behind it and how they will ultimately be useful across social networks. Today XFBML is powered by JavaScript, though in the future I can imagine having actual HTML tags for this sort of social content. One of the large benefits of this approach is that a user's privacy settings can be maintained easily across sites (see Thoughts on dynamic privacy, though note that Chris' closing is no longer accurate).

Today XFBML works in such a way that I include Facebook's JavaScript loader in my page, the JavaScript walks the page's DOM looking for tags like <fb:profile-pic uid="4" />, uses your browser (and thus your current cookied session) to request the user's photo, and then based on the user's privacy settings and your relationship to the user fills in their photo (or doesn't). This provides two main benefits: 1) if you only share your photo with your friends, a non-friend browsing this page would not see the photo and 2) if you change your photo on Facebook it will change on this page as well.

Given how quickly the Social Web is coming together, I believe that HTML will need to support social elements someday soon. It's great to see this type of innovation by Facebook running in the wild, but the web itself ultimately evolves best when multiple competing approaches come together. Just as OAuth brought together the best practices from AOL, Flickr, Google, Yahoo! and others, there is a similar opportunity to bring together FBML, YML and OSML along with the client-side benefits of XFBML.

Google's Sneaky Launch of Latitude's Location-Sharing API

Google has extended their location sharing service Latitude (Radar post) with the first set of Latitude Apps. One of them is a blog badge for sharing your location publicly on a website. The other updates your GTalk status for sharing your location to your IM network. Both have to be turned on explicitly and allow you to share your location at the city level or as accurately as possible. These have generated a lot of press, but I haven't seen much discussion about the first sighting of the Latitude API.

On the blog badge page, below the fold there are two URLs that will show up *after* you turn on the blog badge (so it wasn't that sneaky, you just had to scroll below the fold or look on their Geo Developers blog to learn about it). Developers can get access to their own or other users' location with these URLs. The URLs let you access the data as KML or JSON. They look like:

KML -> http://www.google.com/latitude/apps/badge/api?user=<ID>&type=kml
JSON -> http://www.google.com/latitude/apps/badge/api?user=<ID>&type=json

I've included the JSON file and some documentation after the jump.

These are just the first signs of Google's Latitude API. This is a barebones release that I think is designed to see what developers do with it and how people react to public sharing (geo-broadcasting). These initial feeds do support multiple users (just add another ID separated by a comma) and could be used to build many types of apps. Let me know if you build one for yourself or others.

This is not going to be the end all be all of the Latitude API. I theorize that we will see some of the following features added for a formal release of the API.

User Control & OAuth - Right now a user can only turn the sharing on or off. Once they've shared out their badge or their user ID there is no way to know who else is tracking them. You can't even currently reset your User ID if you inadvertently share it - your only option to block a rogue app is to turn off sharing completely. I am hoping that in a future version Google adds OAuth and let's you turn on/off access to different apps. These apps should also be able to receive different levels of accuracy (some should get my city, others my exact location).
iPhone Client - This isn't tied to the release of the API, but it's got to be sticking in Google's craw that Latitude isn't on the iPhone yet. The Latitude site has said "Coming Soon" for months. I am hoping that it will be sneaked into the 3.0 release or shortly thereafter. (my understanding is that it is not in the current Developer release).
Location History - If Google is tracking my location then I want them to store it and analyze it. I want to know where I've been and be able to reflect upon the trends in my life. They can use this for targeting ads and services at me all they want. Google does not currently store my data and if they don't decide to then many will just turn to third-party developers and sites for the service.
Data Control - I want Google to track and store my location for its and my use, but I also want to be able to export it or delete it. I want to be able to delete a certain period of time or all of my data. When I press delete it should immediately be purged from their servers. I like the principles of data ownership that were developed by MIT/SenseNetwork's Sandy Pentland. I hope Google (and all other LBS providers) follow their lead.
Additional Inputs - Currently Google Latitude only accepts Google client inputs. Those won't always be the most accurate way to track my location. Yahoo's Fire Eagle, a location brokering service, will accept inputs from any service I authorize. This is the right way to do it.

For a developer release this has several important features that deserve more discussion:

Accuracy Control - You can currently set your location to be shared at the city-level or as accurately as possible. Personally I'd like to have neighborhood added as an option. I can't personally picture ever needing more privacy than city-level and still wanting to have Latitude on -- plus you can always lie via Latitude.
Reminders - Google will remind you that you are sharing your location on a weekly or monthly basis (or never). This is an important feature as location-sharing is realtime and not something that people should leave running without being checked.
App Gallery - There is already a Latitude App Gallery. It's populated with the two new releases. I am sure it will be filing up shortly.

This is just the first step in a Google location-sharing API and Apps. Latitude will end up in a place very similar to FireEagle, Yahoo's location broker (Radar post). Latitude is significant not just because it comes from Google, but because of its wide reach. Google has clients available for almost every mobile platform and many browsers. People just want a service like Latitude to work and currently Google is one of the only companies that can deliver (though I am sure Facebook and MySpace are working on something internally).

Google's Steven Lee of the Latitude team is speaking at Where 2.0 in two weeks. Use whr09rdr for 25% off registration. I am sure that there will be announcements about Latitude at either Where 2.0 or Google I/O.

(continue reading)

Where 2.0 Preview - Tyler Bell on Yahoo's Open Location Project

Location can be a vague concept to pin down. To a surveyor, location means latitude and longitude accurate to a few millimeters, while to a cab driver, a street address would be much more useful. If you're German, I can tell you that I live in the United States. To a Californian, I live in New Hampshire. And to someone from Manchester, I live in Derry. Unfortunately, the way that location is currently stored and presented online is both non-uniform and frequently at a level of precision inappropriate for the end-user. That's part of what Open Location is trying to fix. Tyler Bell, who took his doctorate from Oxford to Yahoo, is currently the product lead for the Yahoo Geo Technology Group. At O'Reilly's Where 2.0 Conference, he'll be discussing Open Location.

James Turner: So first off, can you describe what the Geo Technologies Group does?

Tyler Bell: The Geo Technologies Group at Yahoo oversees all technologies that relate to geography and geographic information. So it's largely self-evident. But this is what I mean by that: it's really we own and oversee the maps and mapping technologies. So the visualizations and placements of geographically informed data. We also own user location technologies. So here, we're dealing with different methods of detecting user location, managing user location, and ensuring that users receive geo-relevant results whenever they log onto Yahoo or use a Yahoo service. And then lastly, we have something which is slightly more esoteric. It's called the Geoinformatics Group. And that's the organization which uses geography to inform data. And we do this without ever showing a map. So it's really how we add value and power to information wholly based upon where things are and where our users are.

JT: That's like returning relevant search information to what you know about the user's location.

TB: That's correct. That's the end product of search groups consuming the geo technologies services on the back-end. But what we also need to do is actually organize the geographic information. So instead of searches, they're the specialists at Yahoo about matching user intent to the results that are returned; it's our job on the Geoinformatics Group, for example, to say that when a user queries against Springfield or they're searching for Springfield, which of the countless Springfields in the United States, in the world do you mean? So we need to be able to recognize that this is a place. We need to identify all of the places of a particular place name. And then we need to be able to do a so-called geo-geo disambiguation to ensure that when you mean Springfield, when you mean Campbell, when you give us a city name, which is otherwise nonspecific, we are very likely to return the most direct and accurate results.

(continue reading)

Four short links: 4 Feb 2009

Data, climate change, and location:

1. Details on Yahoo's Distributed Database (Greg Linden) -- summary of Yahoo!'s PNUTS, "a massively parallel and geographically distributed database system for Yahoo!'s web applications." Greg keeps up with the papers from the search engine companies, and the insights he offers are great. For example, "Second, as figures 3 and 4 show, the average latency of requests to their database seems quite high, roughly 100 ms. This is high enough that web applications probably would incur too much total latency if they made a few requests serially (e.g. ask for some data, then, depending on what the data looks like, ask for some other data). That seems like a problem.".
2. Google Latitude -- app and service for mobile phones (G1, Blackberry, Windows Mobile, Symbian) and desktops, where your location is tracked and displayed on a map which you can share with your friends. Interesting use of the map to get some Dodgeball-like functionality, but without programmatic access it's less functional than FireEagle. I'm still not sure I really understand the use cases for this, and assume that over time it will evolve into something more practical.
3. Without Hot Air -- the full text of an excellent book on global warming is available. Well written and well thought. I look forward to the inevitable flood of foot-stamping carbon polluters harrumphing about flawed science and the inevitable final triumph of the flat earth geocentric cosmology.
4. Is Big Data at a Tipping Point? -- Tim pointed me to this a while ago, but I don't think he's blogged about it. Thesis is that as more and more open data gets out there, it'll eventually be cross-related into something big and useful. The author asks how close we are to that. If the premise is true (and I'm not sure I buy the phase change metaphor), I think we're definitely not going to be saying within 12 months "remember when we didn't have enough useful plentiful accurate mashable data? thank goodness those days are past!".

Flickr Community Fills Gap

In the recent round of Yahoo! layoffs was someone I'd just met, George Oates. She started the Flickr Commons, where galleries, libraries, archives, and museums can post photos and the community can tag them. She was a tireless ambassador, as well, with a gruelling travel schedule to bring the word to other institutions on what's possible. Her blog post about how she found out about being laid off is moving.

Still shocked. So fucking brutal.

Within the sadness of that, however, comes some good news. Flickr group and George is taking heart from it.

I'm reminded of Google's philosophy around groups. They were so happy with the way that the Google Maps community rallied to support apps developers, they've rolled out a similar "help-less" support system for their other developer products. The thinking is: if we hold their hands then the work for us will grow faster than we can meet it, so it's better to stand back even at launch and let the community form and do the work. From other experiences I've had (Google Docs, I'm looking at you) this community support is sketchy at best, and Google may well have learned the magic ingredient that creates a vibrant helpful community as opposed to a question-filled answer-free ghetto. With the passion for Flickr Commons that I saw at National Digital Forum last month, I'm sure the community's on the right track.

(hint on the magic ingredient: Maps apps were public works, so the reward for helping was seeing another cool app appear. Google Docs are generally private, so there's rarely any direct reward for helping someone with their DNS configuration or email client setup. Flickr Commons: very public)

Tools for the Equity Research Toolbox

When I was a kid, I would always remember commercials for a school called Apex Tech. One of their taglines was "look over the professional tools you get to keep when you finish your training". It's a lot like that today. Google News, along with Yahoo! Pipes are two tools that analysts, traders, and salespeople are discovering and using.

Today the NYTimes pointed to the slow growth of ad revenue as a disappointment to Google on it's news site see - "At Google, Slow Growth in News Site"

In digging through the article I saw this quote that really outlines what Google News is about for people like me, those of us that live in the news.

Google executives defend the news site, saying traffic is not a paramount goal. Google News, they say, helps the company produce better search results and helps users find news sources that they might not know about otherwise.

“For us, news is about search and helping people find information,” said Marissa Mayer, Google’s vice president for search and user experience, who oversees Google News.

This idea of finding "other" news sources is the key.

One of the paramount abilities of a good analyst is to spot trends early and realize their potential impact on a company or industry. What analysts are usually searching for is any hint of weakness or strength in competitive advantage. Sometimes the smallest trends start in the local newspapers. Google News makes locating those topics and stories much easier.

If you pair Google News with the enhanced filtering ability of Yahoo! Pipes, and your favorite feed reader; you can create some worthwhile tools that help your trend seeking abilities.

I am a big fan of some of the ideas behind Microtrends by Mark Penn. Mark is known for spotting the 'soccer moms' impact on the 1996 Clinton campaign. Basically his idea centers around identifying trends that begin when 1% of a population begins to adopt. So if 1% of the US population (3 million people) starts saying something you need to pay attention. The trick as an analyst is to identify topics that may be on their way to that benchmark.

My current interest is in the debate on energy policy as it relates to Oil exploration. What I'm really concerned with is the indirect results and unintended consequences of a change in the current energy policy. Any change from today's policy will cause all companies to respond. This will allow me to set up a group of scenarios which I can watch and be prepared for as a trend develops. A cardinal rule in investment research is to make sure you have as much relevant data as possible. You never want to be blindsided. You can discount information at your own discretion -- but being ignorant is hazardous to your portfolio.

Here is an example I've been working on as part of a wider range of investment ideas on Oil.

My first approach was to set up a search in Google News that highlighted anytime OIL was in the title of a story. You can do that with the 'allintitle' operator and since I wanted US based sources I added the 'location' operator with USA as the source- it looks like this in the Google search window.
http://news.google.com/news?hl=en&ned=us&q=allintitle:oil+location:USA&ie=UTF-8&scoring=n

To see more useful operators check out the Google Cheat Sheet

There are choices on the page to make this search into an RSS feed. Clicking a link on the page will create a feed url in either RSS 2.0 or Atom. You can then take that feed and do further refining in Yahoo! Pipes. I like to create a broad search from Google News and then apply a layer of filters in Pipes for key terms that I think are important. Once I have configured Pipes to my liking it becomes a feed for my RSS reader. I also created a pipe that looks at the opinion and editorial feeds from certain newspapers. Those in the analyst community will recognize this technique as a kin to using Google alerts. Using RSS is the better mousetrap and it doesn't clog you mailbox.

So what did I find from my pipes that I think is interesting, below are some examples:

175 in House Sign Westmoreland Pledge to Vote to Increase Oil Production
Oil prices put Plastic industry under pressure
Point/Counterpoint: Oil drilling
Oil drilling question looms as election issue
Calls for crack down on oil speculation increase
McCain's policies will not help oil crisis, curb climate change

With two candidates on opposite sides of the issue. Stories and opinion pages are pointing to the key arguments that each side will make. Two of the resident themes are drill now, and oil speculation. Being apolitical here, if there is one thing you can count on -- somewhere along the line-- both of these candidates are going to introduce some type of energy legislation as part of their platforms. The sentiment in the stories, on either side, is just too high for either campaign to ignore as a differentiating issue. With further enhancement, I can track these issues on a regular basis without having to subscribe to every newspaper RSS feed. And as a side benefit to the newspapers involved, to get the full story, I need to go their site - display advertising anyone?

Why Arrington is Wrong about Yahoo!-Google Deal

I was inspired by Fred Wilson's excellent piece on the subject to add my own two cents to Mike Arrington's rant about how Yahoo!'s deal with Google is bad for the industry. I wrote the following in Arrington's comment stream, and will reproduce it here:

Let me weigh in as well on why I don't think Google's dominance in search is going to cause the problems you imagine.

1. Search is only one way to find things. It's the most easily monetizable, so it gets the lion's share of the attention. But take a look at (and report on) what percentage of techcrunch's traffic comes from search. For the O'Reilly Radar blog, it's about 35%. Significant, sure, but hardly a sign of lack of competition. If Google absorbed both Yahoo! and Microsoft, the share of our visits coming from search would still be below 40%. (That tells you what a small share of our search traffic comes from the other guys today.) And that's just the web traffic. Count in RSS (which is much bigger than web for most blogs, including ours) and the Search share of traffic goes down to a much smaller amount. So there's not much worry about people not being able to find information.

2. You specifically raise the specter of Google taking a bigger share of the search dollar absent competition. I'd be interested to know if you have concrete case studies of better deals because of competitive pressure. Seems to me that if Google does this, they will undermine the virtuous circle that drives their success. Maybe they will do this, but if they do, attention and value will migrate elsewhere, as eventually happened with Microsoft.

At O'Reilly, we always say "Create more value than you capture." All successful companies do this. Once they start capturing more value than they create, their market position erodes, and someone displaces them. It may take a while but it happens eventually. If Google takes too much of the pie, it will be a great opening for a new competitor. Right now, because Google is creating the most value for the ecosystem, competitors continue to lose share. If they started taking a lot more of the revenue, Microsoft's share would go up, plus new startups would have an opening that they don't have now.

3. The real source of my argument for this position, which you linked to in your piece, but I'll point to again here, is that Web 2.0, the internet operating system we're building, is much bigger than search. Search is an incredibly powerful subsystem of that OS, but it is just a subsystem. There is lots of competition across the system as a whole, and we're a LONG way from the concentration of power that represents monopoly when we take that into consideration.

4. The landscape is changing so fast. To take only one axis, consider mobile. Google doesn't dominate mobile/local search. That's a whole new game.... Again, there's lots of competition.

MicroHoo: corporate penis envy?

After reading endless pieces about Microsoft's obsession with search, I am forced to offer the following theory: penis envy (from Wikipedia):

I worked with Freud in Vienna. We broke over the concept of penis envy. He thought it should be limited to women - Woody Allen in Zelig

While not the same kind of penis envy as that typically referred to in psychoanalysis, the phrase "penis envy" or "small penis syndrome" is also sometimes used to describe the envy of a male over another male's penis. Although this subconscious or conscious envy may solely be based on the idea that a larger penis is universally more satisfying and appealing to a sexual partner, other implications arise from the fact that a large penis has been seen in many cultures as a symbol of high masculinity, dominance and power.

Microsoft was once motivated by its own Big Hairy Audacious Goal: "a computer on every desk and in every home." They achieved that goal, and ever since, they've drifted. Now their only goal seems to be to stay on top of the heap. They need to stop focusing on eating other people's lunch and start thinking deeply about what kind of goals might stretch the company once again. "Organize all the world's information" is already taken, but there are a lot of other things that need doing, that Microsoft is uniquely capable of, and that would energize the creativity and passion of Microsoft's employees. What's more (as I'll get to later) there is a much bigger game afoot, and one that Microsoft would be far wiser to focus on.

Meanwhile, Yahoo! has let itself be defined by the same kind of penis envy. Here is a business that has beaten Google in area after area, that is unquestionably the #1 media company on the net, and yet has let itself be defined by the one area in which it is #2 -- and where it could be much more profitable and successful by partnering with #1 than by competing with them.

So, my advice to Yahoo!: continue with your plan to outsource search to Google, just like you did before 2002, and plow those increased profits and reduced costs into your own innovation, strengthening the areas where you are #1, exploring new ideas that will make YOUR users insanely happy, and generally focusing on what makes Yahoo! great, rather than on what doesn't. That is, unless Microsoft makes you so good a deal for your search assets that you just can't say no. But either way, let yourself be quit of the destructive competition and focus on adding real value for your users.

My advice to Microsoft: outsource your search to Google too!

Of course, I don't really expect you to do that. As Todd Bishop wrote in a piece called The Rest of the Motto back in 2004:

...people who have been following the company since the early days point out that the stated goal was actually, "A computer on every desk and in every home, running Microsoft software." In an October 1995 interview with Fortune magazine, for example, Bill Gates said: " ... I still believe in our vision -- a computer on every desk and in every home, all running Microsoft software."

Microsoft has long operated on the model of platform as lever for lock-in and competitive advantage, or as Tolkien put it, "One ring to rule them all." But as I've been saying in my advocacy about Web 2.0 from the beginning, there is another model, represented by both Linux and the Internet, that might be called small pieces loosely joined. (I don't use the term in quite the same way as in David Weinberger's eponymous book (linked just previously), but the phrase is just too right to ignore.) The Unix philosophy, laid out so brilliantly in The Unix Programming Environment, is of a network of cooperating tools, each doing one thing well. This philosophy took root on the internet as well, and has proven to be an enormous engine of innovation.

I believe that we're collectively working on an Internet Operating System, and that it will ultimately look more like Unix than it looks like Windows. That is, it will be an aggregate of best of breed tools produced by an army of independent actors, all playing by the same rules so that those tools work together to produce a whole greater than the sum of the parts.

Fighting over search is a bit like the Free Software Foundation re-implementing cat, ls, sort, and all the other Unix utilities that were already available in the Berkeley distributions of Unix. The real problem was solved by someone outside the FSF, when Linus Torvalds wrote a kernel, a missing piece that became the gravitational center of Linux, the center around which all of the other projects could coalesce, which made them more valuable not by competing with them but by completing them.

Don't get me wrong: I'm not saying that there isn't enormous room for competition, and that competition isn't good for the market. Compete where you have ideas that can really change the game, but don't play me-too.

So let's assume that Google has won at search, or close enough to make no difference. Is Microsoft better off trying to reimplement cat and ls, or trying to figure out what's still missing from the Internet Operating System? While they are locked in penis envy, all the really cute girls are going out with startups :-)

So think hard about the future internet OS: ubiquitous computing, with a computer not just on every desktop and in every home, or even every phone and every camera, but in everyday devices, clothing, shopping carts, cars, pens, toys, buildings, roads, the power grid, even human bodies--and yes, lots of server farms. An infrastructure of real-time data services across that "network of networks," with search (and search-based-advertising) only one of many such services. As David Stutz once wrote: Useful software written above the level of the single device will command high margins for a long time to come.

This is the real Web 2.0, the web as platform. Search and its advertising economy is only one subsystem of that platform.

I know there are lots of people at Microsoft Research already working on key parts of this vision. Get behind them. Pour resources into the future, not the past. Meanwhile, there will be a lot of client devices in our connected future. And the connected devices that are the most successful will be the ones that are most open, the ones that are best for using ALL the services that are popping up out there, not just the ones that are part of a single vendor stack. The internet operating system is still in its infancy, perhaps at the level of Windows 3.1 (a simple, flawed GUI skin over DOS) or the GNU system without the Linux kernel. Microsoft has an enormous opportunity to build client software and devices that are ideal citizens of the software cooperative that the internet is becoming.

(Aside: Apple's apparent success with an "own the stack, from the device to cloud" strategy is misleading. With both the iPod and the iPhone, a key element of success is precisely the device's openness to what Apple does not own. Imagine an iPod where you could only buy music from the Apple music store instead of ripping your own CDs (this is Amazon's mistake with the Kindle). Imagine an iphone without the Safari browser (opening a world of web apps to the phone) or the Google Maps application. Apple owns key elements of the stack, but it's a permeable stack, and getting more so.)

Learn from the best, partner with the best, fill in the gaps, and build for the future. Above all, remember that great companies have "big, hairy audacious goals." Energize Microsoft by pursuing a seemingly impossible goal that can change the world for the better.

MySpace's Data Availability is not Data Portability

Yesterday MySpace, Yahoo!, eBay, Photobucket (also owned by News Corp), and Twitter announced the Data Availability Initiative. While I could write at length about how this shows the big companies have already realized how to diminish the DataPortability group's brand by linking anything they do "data portability," that isn't the point of this post. The crux of the announcement yesterday was that shortly MySpace would begin allowing third-parties to embed MySpace profile information within their own services in the name of "data portability". Unfortunately, the details around this remain buzzword-laden at best.

Their press release yesterday stated:

Additionally, rather than updating information across the Web (e.g. default photo, favorite movies or music) for each site where a user spends time, now a user can update their profile in one place and dynamically share that information with the other sites they care about. MySpace will be rolling out a centralized location within the site that allows users to manage how their content and data is made available to third party sites they have chosen to engage with.

At first glance this seems like a great thing. MySpace is partnering with Yahoo!, eBay, Photobucket, and Twitter to solve a pain point on the web; the inability to keep parts of your profile in sync around the web where you'd like them to be. The announcement didn't however offer any insight into how this would work beyond that, "the MySpace Data Availability initiative uses OAUTH [sic] and Restful APIs as its core technology underpinnings." After this announcement I had the pleasure of speaking with a reporter who was on the briefing call. He explained that MySpace said that due to their terms of service the participating sites (e.g. Twitter) would not be allowed to cache or store any of the profile information. In my mind this led to the Data Availability API being structured in one of two ways: 1) on each page load Twitter makes a request to MySpace fetching the protected profile information via OAuth to then display on their site or 2) Twitter includes JavaScript which the browser then uses to fill in the corresponding profile information when it renders the page. Either case is not an example of data portability no matter how you define the term!

To make this worse one of the pieces of profile information made available is a list of a MySpace user's friends. Once again there are two reasonable ways to do this: 1) MySpace provides a user's friends as a list of hashed email addresses to Twitter or 2) MySpace provides a user's friends as a list of MySpace usernames. While the hashed email route would certainly be simpler and easier for sites like Twitter to match against their own user database, I highly doubt this will be the implementation due to concerns around undesired account linking. Rather I think MySpace will choose to provide a list of other MySpace usernames. What this means is that in order for Twitter to make use of the information they must encourage all of their users to fill in their MySpace account on Twitter so that they can map a MySpace username to a Twitter username. Obviously in the best interests of MySpace to have more of their profiles linked to from around the web thus increasing page rank, visitors, and thus ad revenue.

At the end of the day it seems that MySpace is trying to become a large centralized profile repository on the internet. One where information might be available but certainly not allowed to be actually moved outside the network's walls. A good try, but just as no one would like Microsoft own identity for the entire web with Passport I fail to see how others will let MySpace own all of the profiles.

Update: Just got off a plane from London and realized that I missed a link to Chris Saad, DataPortability's co-founder, explaining yesterday that they "hope to see the MySpace “Data Availability” initiative evolve toward becoming a compliant implementation of the DataPortability Best Practices." While MySpace did not say in their release that Data Availability is a form of data portability, it certainly seemed to be interpreted that way.

The battle for the cloud

Andy Kessler has a great op-ed in the Wall Street Journal, The War for the Web:

Microsoft was smart to walk away (for now) from its $44 billion bid for Yahoo. It's never good to overpay. But the software giant - whose stock has flatlined for eight years - was onto the right strategy in looking to the Web for growth....

With the Microsoft/Yahoo deal breakdown, everyone assumes Google walks away with the prize. Not so fast. This contest is just starting. For Microsoft or Google or anyone else to win, they need four key elements of an end-to-end strategy:

- The Cloud. The desktop computer isn't going away. But as bandwidth speeds increase, more and more computing can be done in the network of computers sitting in data centers - aka the "cloud."...

- The Edge. The cloud is nothing without devices, browsers and users to feed it....

- Speed. - Speed. Once you build the cloud, it's all about network operations....

- Platform. ...Having a fast cloud is nothing if you keep it closed. The trick is to open it up as a platform for every new business idea to run on, charging appropriate fees as necessary....

Andy's analysis is all in those ellipses. Succinct, on-point, and refreshingly insightful about the true drivers of Web 2.0. And I can't help pointing out that the Wall Street Journal has now noticed the fundamental premise of our Velocity conference: "Once you build the cloud, it's all about network operations."

If Velocity were a movie, don't you think that quote might be on the movie poster?

The Corporation's Two Bodies

The New York Times quotes Laura Martin of Soleil Securities, as saying "This is management putting its employees and its job security ahead of current Yahoo shareholders' interest." The sense of horror here--that management could actually put the interests of employees ahead of the interests of investors--is interesting, to say the least. It raises an important question that's really almost theological in nature. It is most certainly theological in, as Lawrence Ferlinghetti wrote, "the promised land where every coin is marked In God We Trust, but the dollar bills do not have it being gods unto themselves. ("Autobiography," A Coney Island of the Mind, 1958, New Directions)

(continue reading)

Building Better Silos

It's been good to watch the use of OpenID spread. It's great to see that ma.gnolia.com has dropped "traditional login" in favor of OpenID. And I was encouraged to read about Yahoo's support of OpenID. Granted, it took me a while to get around to trying it.
But when I got around to trying it, Yahoo!ID was a disappointment. The promise of OpenID is to return ownership of ID to the users, and to eliminate identity silos, in which the big sites compete to own your identity and your data. If that's the goal, Yahoo!ID may not be a step backwards, but it's certainly not much of a step forwards.

(continue reading)