Four short links: 12 November 2009

CRM on Rails, Data Mining on Hadoop, Disappointing Keynotes, The Teapot Effect

by Nat Torkington | @gnat | comments: 1

1. Fat Free CRM -- open source (Affero GPL) Ruby on Rails CRM system.
2. Bixo -- open source data mining toolkit that runs as a series of pipes on top of Hadoop. Built on Cascading workflow system for Hadoop that hides MapReduce. (via kdnuggets)
3. Andy Kessler's Keynote at Defrag Stank (Pete Warden) -- I'm sorry to hear it, because I loved Andy's book How We Got Here about the intersecting histories of economics, finance, and technology. Read the book instead of reading about the disappointing keynote.
4. The Teapot Effect -- the thing I love about geeks is how their passion causes them to explore, ruthlessly and quantitatively, the everyday phenomena that the rest of us take for granted. Such as dribbling teapots: “Previous studies have shown that dribbling is the result of flow separation where the layer of fluid closest to the boundary becomes detached from it. When that happens, the fluid flows smoothly over the lip. But as the flow rate decreases, the boundary layer re-attaches to the surface causing dribbling.” Read the post and the research it talks about to learn how to prevent Dribbling Teapot Syndrome ....

Four short links: 12 October 2009

DSL for NLP Task, Insider Tradespotting, Outsource Fail, Cloud Fail

by Nat Torkington | @gnat | comments: 3

1. Snowball -- a small string processing language designed for creating stemming algorithms for use in Information Retrieval. (via straup on delicious)
2. Insider Trades -- a Yahoo! Hack Day app that turned out to be worth continuing. Scans SEC systems every 30 seconds and alerts you if the stock you track has been traded by an insider. (via straup on delicious)
3. Air New Zealand Slams IBM -- central point of failure in the outsourced IT. "In my 30-year working career, I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client's customers is not the glowing endorsement you want.
4. Danger/Microsoft Loses Sidekick Customers' Data -- Regrettably, based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. This cloud had a brown lining.

Four short links: 19 August 2009

Survivor Bias, Algorithmic Trading, S3 Tools, DIY GSM

by Nat Torkington | @gnat | comments: 3

1. Business Advice Plagued by Survivor Bias -- "Burying the other evidence: [...] Doesn't most business advice suffer from this fallacy? Harvard Business School's famous case studies include only success stories. To paraphrase Peter, what if twenty other coffee shops had the same ideas, same product, and same dedication as Starbucks, but failed? How does that affect what we can learn from Starbucks's success? (via Hacker News)
2. A Bestiary of Algorithmic Trading Strategies -- insight into the algorithms used by quant traders. Statistical arbitrageurs are a sort of squishy area, similar to arbs, but distinct from them. They find “pieces” of securities which are theoretically equivalent. For example, they may notice a drift between prices of oil companies which should revert to a mean value. This mean reversion should happen if the drift doesn’t have anything to do with actual corporate differences, like one company’s wells catching on fire. What you’re doing here is buying and selling the idea of an oil company, or in other words, a sort of oil company market spread risk. You’re assuming these two companies are statistically the same, and so they’ll revert to some kind of mean when one of the prices move. (via Hacker News)
3. s3cmd -- commandline tool for moving files into and out of Amazon S3.
4. DIY GSM Network -- wow. How to build your own GSM network. Bit by bit, the telcos are getting pressured by the hobbyists. This barbarian is looking forward to the day when the walled gardens are sacked. (via Slashdot)

Four short links: 31 July 2009

NoSQL, Goldman Sachs, Yahoo! Developer Products and Bing, and Alternate Reality

by Nat Torkington | @gnat | comments: 3

On this day in history, Mt Fuji exploded (781), Daniel Defoe was put in the stocks for seditious libel but was pelted with flowers (1703), the first U.S. patent was issued (1790), and the radio show The Shadow aired for the first time (1930).

1. Tokyo Cabinet: Beyond Key-Value Store -- description of Tokyo Cabinet and code examples in Ruby. More on the nosql move to leave relational databases behind for certain modern problems (such as scaling).
2. The Great American Bubble Machine (Rolling Stone) -- I know it's old hat, but read it for the poetry if for nothing else. The first thing you need to know about Goldman Sachs is that it's everywhere. The world's most powerful investment bank is a great vampire squid wrapped around the face of humanity, relentlessly jamming its blood funnel into anything that smells like money.
3. Yahoo!'s Developer Program and Bing -- note from Yahoo! to developers, saying that YQL, YUI, and Pipes are safe. For SearchMonkey and BOSS they currently do not have anything concrete to tell you. I assume (and hope) that Delicious is a top-level product, not something under "search". (via Simon Willison)
4. Preparing Us for AR -- (Schulze & Webb) round up of some apps and toys that show what AR might be, unfettered by current day technological constraints.

Four short links: 15 July 2009

A collection inspired by Science Foo Camp attendees

by Nat Torkington | @gnat | comments: 1

1. Endogenous steroids and financial risk taking on a London trading floor (PNAS) -- We found that a trader's morning testosterone level predicts his day's profitability. We also found that a trader's cortisol rises with both the variance of his trading results and the volatility of the market. Our results suggest that higher testosterone may contribute to economic return, whereas cortisol is increased by risk. Our results point to a further possibility: testosterone and cortisol are known to have cognitive and behavioral effects, so if the acutely elevated steroids we observed were to persist or increase as volatility rises, they may shift risk preferences and even affect a trader's ability to engage in rational choice.
2. The Origin of Universal Scaling Laws in Biology -- eye-opening paper that blew my mind. Highlight of Sci Foo was meeting the author and shaking his hand. Relates metabolic rate, size, heart rate, and lifespan by applying physics to biology.
3. Ushahidi -- open source software for managing disasters. The Ushahidi Engine is a platform that allows anyone to gather distributed data via SMS, email or web and visualize it on a map or timeline. Our goal is to create the simplest way of aggregating information from the public for use in crisis response.
4. Dissecting the Canon: Visual Subject Co-Popularity Networks in Art Research -- In this paper we analyze a classic da- taset of art research, which collects ancient art and architecture and their Western Renaissance documentation since 1947. [T]here is clearly a long tail of monument popularity.

Four short links: 7 May 2009

iPhone Rocketry, Copywrongs, Econopocalypse, and Empire

by Nat Torkington | @gnat | comments: 1

1. How To Use An iPhone To Fly RC Airplanes and Helicopters -- So I had my basic idea down. iPhone joins the Linksys router network. It gets an IP address. Then, I open up my pilot program. The pilot program interfaces with the router via SSH (I couldn’t think of a better way that has redundancy, and speed, and was already buily by someone else). The pilot program interprets what the iphone is doing, and outputs data to one of the ethernet ports of which there are conveniently 4. Rudder, Ailerons, Throttle, Elevator.
2. Economist Debates: Copyrights and Wrongs -- The Economist live debate about copyright, with the moot "This house believes that existing copyright laws do more harm than good." Public comments, voting, and new informed opinions each day.
3. How I helped build the bomb that blew up Wall Street (NYMag) -- story of the software developer behind a lot of the mortgage repackaging software. Many good lines, e.g., But even then, I was wondering why I was making more than anyone in my family, maybe as much as all my siblings combined. Hey, I had higher SAT scores. I could do all the arithmetic in my head. I was very good at programming a computer. And that computer, with my software, touched billions of dollars of the firm’s money. Every week. That justified it. When you’re close to the money, you get the first cut. Oyster farmers eat lots of oysters, don’t they?
4. Yow -- words of wisdom from John Battelle on Google as the new Microsoft: If any lesson is to be drawn, perhaps prematurely, from all this, it's that no company - or two companies - can lead a culture for longer than half a generation. After that, the culture starts to distrust the companies' motives, regardless of whether they are pure or well intentioned.

Importance of Innovation in Finance & BarCampBank

by Jesse Robbins | @jesserobbins | comments: 2

“Progress is not the mere correction of evils. Progress is the constant replacing of the best there is with something still better.” -Edward Filene

Two years ago, when we were organizing the first BarCampBank in the US, many people found it hard to believe that banks & credit unions could a place for meaningful grassroots innovation. Even crazier was the idea of organizing an unconference to begin bringing open source, transparency, identity, and community into the very closed world of banking & finance.

Since then the BarCampBank idea has turned into a movement. There have been over 14 events all over the world, and many of the ideas generated are beginning to turn into action.

To me, the global financial system is a platform that exists to “create more value than it captures”. Tim explained this in his Work on Stuff that Matters post, saying:

“A bank that loans money to a small business sees that business grow, perhaps borrow more money, hire employees who make deposits and take out loans, and so on. The power of this cycle to lift people out of poverty has been demonstrated by microfinance institutions like the Grameen Bank. Grameen is clearly focused on creating more value than they capture; not so the like of Fannie Mae and Freddy Mac, or WaMu, or many of the other failed financial institutions involved in the current financial meltdown.”

There has never been a more important time to bring meaningful innovation into the financial system, and there has never been more opportunity for our community to make it happen.

The next event is occurring this weekend (April 25-26, 2009) on Treasure Island in San Francisco.

After that, the following events are planned:

• BarCampBankVegas is set for May 2, 2009.
• BarCampBankCharleston2 is set for June 13, 2009
• BarCampBankGermany is set for October 23-25, 2009

Thoughts on the Financial Crisis

by Tim O'Reilly | @timoreilly | comments: 24

The other day, we received a blistering email from a Radar reader complaining about our silence on the subject of the economic meltdown. I wrote back:

There are a lot of people bloviating about the financial crisis. It's outside of our area of expertise, so there didn't seem to be a lot of urgency to add to the hot air. Even professional economists and financial experts disagree on where this is going. I've been reading a lot, and sharing the best links via my twitter feed, but frankly, I'm feeling that we're in the middle of a wave that no one completely understands.

Meanwhile, I did in fact spend my NY Web Expo talk on the idea that "I sense a storm coming" (Rilke quote), and the idea that companies and individuals need robust strategies (ones that can work even in uncertain times), with one robust strategy being to "work on stuff that matters."

That Rilke quote, from the poem The Man Watching, translated by Robert Bly, has this great line about letting ourselves be washed over by events greater than ourselves:

I can tell by the way the trees beat, after
so many dull days, on my worried windowpanes
that a storm is coming...

What we choose to fight is so tiny!
What fights us is so great!
If only we would let ourselves be dominated
as things do by some immense storm,
we would become strong too, and not need names.

I've been feeling a lot like that. Watchful. Listening. Learning. Not rushing about fighting the small things of the moment but letting the storm wash in. It will change us. That can be good. And as the storm washes through, it will become clear what we have to do.

I've been quoting that poem in my talks since Why I Love Hackers at ETech in March. It ends with a ringing invocation to work on challenging problems, problems that stretch us, as the wrestlers of the Old Testament were challenged by wrestling with the angel. That seems to me to be the heart of what we need to do now.

I hadn't thought to do this earlier, but it occurs to me that I might also share here the message that I sent out to all O'Reilly employees by email a week or so ago (edited slightly to remove a few company-specific details):

Many of you have no doubt been alarmed by the developments of the last couple of weeks in financial markets, so I wanted to put a few thoughts out to all of you before disappearing on a combination of vacation and business travel for the next 3+ weeks.

...at the last company meeting, I talked about a theme that I've expanded on in public talks like the one I did at Ignite Boston and at Web 2.0 Expo in New York the week before last: the idea that robust strategies are ones you'd adopt in good times and in bad. And I argued that we probably end up with more robust strategies if we assume the worst rather than the best.

We could be in for a long, rough time in the economy. I'm not going to say otherwise.

But I also want to point out that rough times are often the best times for creativity, opportunity and change. We transformed ourselves from a technical writing consulting company into a book publishing company as a result of the huge economic downturn of the mid-80s. After the dotcom bust in 2001, we launched Safari Books Online, Missing Manuals, the Web 2.0 conferences, Foo Camp, Make: magazine, the O'Reilly School of Technology, and a host of other initiatives that have fueled our growth and that define the company today. We diversified and invested in new ideas.

It was a painful period but one that made us better and stronger as a company.

And if you look at history, you see that this has always and everywhere been true. It's not an accident that economist Joseph Schumpeter talked about the "creative destruction" inherent in capitalism. Great problems are also great opportunities for those who know how to solve them. And looking ahead, I can see great opportunities.

The energy crisis (both global warming and the oil price shock) is helping people to focus on how technology can transform the energy sector. The financial crisis has demonstrated just how out-of-whack an unregulated, proprietary, black-box approach can get. This will lead to an emphasis on regulation, but I hope, above all, on transparency. This is of course analogous to what happened with open source software. Meanwhile, the mobile revolution will continue, regardless of the state of the economy. If it can prosper in Africa, it can prosper even in an American downturn. And all the stuff we're exploring with Make: new materials, new approaches to manufacturing, and the "open source" approach applied to hardware, will take us in unexpected directions. And all of these areas can benefit from what we do best: capturing and spreading the knowledge of innovators.

We don't know yet how problems in the overall economy will affect our business. But what we can do now are the things we ought to be doing anyway:

• Work on stuff that matters: Assuming that the world does go to hell in a handbasket, what would we still want to be working on? What will people need to know? (Chances are good that they need to know these things in a world where we all continue to muddle along as well.)

• Exert visionary leadership in our markets. In tough times, people look for inspiration and vision. The big ideas we care about will still matter, perhaps even more when people are looking for a way forward. (Remember how Web 2.0 gave hope and a story line to an industry struggling its way out of the dotcom bust.)

• Be prudent in what we spend money on. Get rid of the "nice to do" things, and focus on the "must do" things to accelerate them.

  These are all things we should be doing every day anyway. Sometimes, though, a crisis can provide an unexpected gift, a reminder that nobody promised us tomorrow, so we need to make what we do today count.

This seems like good advice for Radar readers as well. I will try to write further on the theme of "work on stuff that matters" in the days ahead.

Tools for the Equity Research Toolbox

by Robert Passarella | comments: 4

When I was a kid, I would always remember commercials for a school called Apex Tech. One of their taglines was "look over the professional tools you get to keep when you finish your training". It's a lot like that today. Google News, along with Yahoo! Pipes are two tools that analysts, traders, and salespeople are discovering and using.

Today the NYTimes pointed to the slow growth of ad revenue as a disappointment to Google on it's news site see - "At Google, Slow Growth in News Site"

In digging through the article I saw this quote that really outlines what Google News is about for people like me, those of us that live in the news.

Google executives defend the news site, saying traffic is not a paramount goal. Google News, they say, helps the company produce better search results and helps users find news sources that they might not know about otherwise.

“For us, news is about search and helping people find information,” said Marissa Mayer, Google’s vice president for search and user experience, who oversees Google News.

This idea of finding "other" news sources is the key.

One of the paramount abilities of a good analyst is to spot trends early and realize their potential impact on a company or industry. What analysts are usually searching for is any hint of weakness or strength in competitive advantage. Sometimes the smallest trends start in the local newspapers. Google News makes locating those topics and stories much easier.

If you pair Google News with the enhanced filtering ability of Yahoo! Pipes, and your favorite feed reader; you can create some worthwhile tools that help your trend seeking abilities.

I am a big fan of some of the ideas behind Microtrends by Mark Penn. Mark is known for spotting the 'soccer moms' impact on the 1996 Clinton campaign. Basically his idea centers around identifying trends that begin when 1% of a population begins to adopt. So if 1% of the US population (3 million people) starts saying something you need to pay attention. The trick as an analyst is to identify topics that may be on their way to that benchmark.

My current interest is in the debate on energy policy as it relates to Oil exploration. What I'm really concerned with is the indirect results and unintended consequences of a change in the current energy policy. Any change from today's policy will cause all companies to respond. This will allow me to set up a group of scenarios which I can watch and be prepared for as a trend develops. A cardinal rule in investment research is to make sure you have as much relevant data as possible. You never want to be blindsided. You can discount information at your own discretion -- but being ignorant is hazardous to your portfolio.

Here is an example I've been working on as part of a wider range of investment ideas on Oil.

My first approach was to set up a search in Google News that highlighted anytime OIL was in the title of a story. You can do that with the 'allintitle' operator and since I wanted US based sources I added the 'location' operator with USA as the source- it looks like this in the Google search window.
http://news.google.com/news?hl=en&ned=us&q=allintitle:oil+location:USA&ie=UTF-8&scoring=n

To see more useful operators check out the Google Cheat Sheet

There are choices on the page to make this search into an RSS feed. Clicking a link on the page will create a feed url in either RSS 2.0 or Atom. You can then take that feed and do further refining in Yahoo! Pipes. I like to create a broad search from Google News and then apply a layer of filters in Pipes for key terms that I think are important. Once I have configured Pipes to my liking it becomes a feed for my RSS reader. I also created a pipe that looks at the opinion and editorial feeds from certain newspapers. Those in the analyst community will recognize this technique as a kin to using Google alerts. Using RSS is the better mousetrap and it doesn't clog you mailbox.

So what did I find from my pipes that I think is interesting, below are some examples:

175 in House Sign Westmoreland Pledge to Vote to Increase Oil Production
Oil prices put Plastic industry under pressure
Point/Counterpoint: Oil drilling
Oil drilling question looms as election issue
Calls for crack down on oil speculation increase
McCain's policies will not help oil crisis, curb climate change

With two candidates on opposite sides of the issue. Stories and opinion pages are pointing to the key arguments that each side will make. Two of the resident themes are drill now, and oil speculation. Being apolitical here, if there is one thing you can count on -- somewhere along the line-- both of these candidates are going to introduce some type of energy legislation as part of their platforms. The sentiment in the stories, on either side, is just too high for either campaign to ignore as a differentiating issue. With further enhancement, I can track these issues on a regular basis without having to subscribe to every newspaper RSS feed. And as a side benefit to the newspapers involved, to get the full story, I need to go their site - display advertising anyone?

Data on the Web: VGChartz vs. NPD

by Robert Passarella | comments: 58

After writing my first post, Equity Research in the Age of Web 2.0, I received a lot of comments asking really good questions. Most of the questions from readers asked about real world examples. So here's a taste.

In the world of securities research, in general, we talk about consensus sources. These are sources that everyone looks to as an arbiter of what may be happening in a market or industry. In the macro economic world, figures released by the US government, (measures like labor statistics or GDP) are used by the market as a whole to judge where the economy may be headed and where it has been. In certain industries, independent benchmarks are used to define the prospects of an industry or the individual performance of the firms involved. A great example of this is the NPD Group.

The NPD Group, provides pay services that collect information from various retailers and then in turn creates a measure of sales for many product categories. Usually the reports are done on a monthly basis coming out about 2 weeks after the previous months close. I am pretty familiar with their reports and most equity analysts and the market use them as the benchmark. Usually an analyst will write a report in which they predict sales for a particular company in relation to the data provided by NPD. The differences in an equity analyst's view of NPD and his own projection usually leads to an upside or downside surprise in a equity's price, once the NPD reports are published and received by the market.

For the video game sector, NPD produces the benchmark measures for North American game and console sales. Analysts like Michael Pachter of Wedbush Morgan, and Colin Sebastian of Lazard, create, on a regular basis their monthly forecasts of where NPD will come in at. Over the last couple of years a new entrant has entered the picture with different aims from NPD; enter VGChartz. By all accounts this newcomer is causing a headache for NPD and their for pay service.

VGchartz uses a similar methodology as NPD, in trending and forecasting game sales through representative samples.
Here is a excerpt from their About Us page.

"With a growing team of analysts, over ten years of experience and with over twenty years of historical data at our disposal, VG Chartz should be seen as a very powerful prediction tool for industry and casual user alike in looking ahead to the future market and making educated and informed predictions. Using our powerful and proprietary graphing and analysis tools, users can query the extensive sales database and compare the performance of hardware and software sales over time between different formats, genres, titles and manufacturers."

The main difference between NPD & VGChartz is that, VGChartz is a user-based community of video game enthusiasts/experts and NPD is a market research company. Their business models also differ, NPD is a for-pay service and VGChartz looks to be supported mostly by paid advertising. While VGChartz makes their North American data available on a weekly basis along with data from Japan and Europe. NPD is focused on North America, and their data is released monthly.

So if you're an investor interested in this sector or an analyst -- the first question you should ask yourself "is the data from VGChartz good?" My answer is, after looking at and using VGChartz data for the last two years, it's quality. But please in the Age of Web 2.0 form your own opinion.
Here are my reasons:

• The data is weekly as opposed to monthly -- time is a big advantage. (I saw Sony's changing future thanks to VGChartz much quicker than if I'd waited for monthly numbers)


• It is the only place to get as close as possible to worldwide figures on console & game sales.


• The best part of using or working with VGChartz data is that they give you their biases upfront along with a comparison of their numbers against NPD and key analysts like Michael Pachter.


• There is community that has arisen around the numbers; that is willing to have an open exchange about the short comings, and methodologies in VGChartz's multiple forums. You really don't get that with NPD.

It has to drive NPD nuts that a site like VGChartz can create a presence and provide high quality information with a completely different business model. I am sure in the NPD mindset, the thought of an outsider competing with them in this space was completely unanticipated. This was a theme that was echoed at our last Money:Tech. On one hand we had Reuters' Devin Wenig talking to Tim O'Reilly saying he didn't "... see direct competition from consumer media (including Google), arguing that professionals need richer, more curated information sources." And on the other hand, we had Sean Park of Six Paradigm talking about grabbing high quality useful data, as a buy sider, from consumer-oriented services like Yahoo! Finance and Google.

The one thing you can count on is this, smart professionals and investors will always look to high quality data sources that are leading indicators for consensus. If I can find a piece of data, or a source that is a good predictor for what a benchmark will do, I will use it no matter where it comes from. Time and proven results are the final judge. If it happens to be free, so much the better.

BarCampBank is spreading

by Jesse Robbins | @jesserobbins | comments: 1

When Ben Black and I organized the first BarCampBank in North America last year, we hoped that it would spread. According to William Azaroff's post on NetBanker, the movement is there and growing:

What's all this about BarCampBanks? From a North American premiere in Seattle almost a year ago, we've witnessed two more in the last few months, and eight more are either scheduled, or in the planning process.

Well, maybe not exactly “planned.” BarCampBanks emerge more than they are planned.

[...]It started as a technology summit, an un-conference where developers and technology geeks could share exploits, connect, and find like-minded companions to extol the virtue of open-source and emerging technologies over pizza and wine.

And then someone decided that this forum would be a perfect place to talk about banking and finance. Weird. And yet it works.

The next event will be BarCampBankDallas on June 21-22nd at the American Bank of Texas Building in Frisco, Texas. William has details on the other events this year on his blog, and a current list can always be found on the main BarCampBank wiki.

Equity Research in the Age of Web 2.0

by Robert Passarella | comments: 13

As a new blogger to Radar and the program co-chair for Money:Tech 2009, I thought I'd introduce myself along with a topic we plan to present at the conference.

After spending many years on Wall Street watching the buy side (investing institutions) and working with the sell side (investment banks); all I can say is that I am excited with what the future holds. The Internet is quickly becoming 'the' vast store house of data, research and commentary that I hoped it would. If you think the Internet is for message board jockeys, blogs for late nights in pajamas, and social media for dating; you are missing point. If you want to add to your company and industry analysis, as well as, your investment process, you need to be an interactive user of the Internet.

So why are we, "the professionals", turning to the Internet? Or more specifically; why are we turning to web sites, blogs and social media for investment research, data and commentary? Well I believe there are a few reasons.

Just about every public company has some type of presence on the Internet. In certain domains, enthusiasts and experts are better informed than most analysts on the actual products, goods or services a company may provide. And they are sharing this knowledge and information with each other in open forums. And if you decide to graduate from lurker to participant the gains are even greater. An informed investment professional, that knows his tickers and sectors, can quickly digest this information into his investment process, just like his phone calls with a Gerson Lehman or Coleman Research expert. This type of high quality information can be found if you know where and how to look.

The emergence of low cost publishing tools, cheap web hosting, and various ways to monetize knowledge is another trend that can put some people on the same level as a CNBC anchor and others on equal footing with a top ranked analyst. Just think of my well known blogger friends like Money:Tech Chair Paul Kedrosky, Barry Ritholtz, Nouriel Roubini, and Roger Ehrenberg. Their collective readership and influence is greater than most sell side analysts.

The economics and dwindling company coverage of most sell side research shops has forced the buy side to scale and employ a do-it-yourself ethic. Your typical buy side analyst needs a place to turn for leads and information, especially when switching sectors and industries. Those places are quickly becoming SeekingAlpha, Wikipedia, LinkedIn and Facebook.

And the biggest reason of all; the established sources of main stream financial news and data are ubiquitous. Everyone on the buy side and sell side has Bloomberg, Reuters, and Dow Jones. And if everyone knows everything at the same time what advantage is that?

So where are we headed?

We are headed to a place where the tools, practices and techniques are starting to emerge. It's a place where anyone with a need and drive can learn to scrape data, do their own automated channel checks, while benchmarking and creating reference data. Have an idea; model it yourself, grab the data from Amazon, benchmark it against Yahoo Finance, and set up your filters in Google Reader for stories or articles related to your themes.

The key is to meld the old with the new. Mixing your custom new found data with established sources from your vendors to provide leading edge mashups that help you complete or flesh out a company's position within the competing forces - think Michael Porter on steroids.

Researching during the age of Web 2.0 doesn't mean blindly following rumors on a web site -- that way leads to madness & ruin. You need to think more like Ben Graham in the early days of modern Wall Street when speculation was the name of the game. Graham was able to find high quality sources of information that others thought to be a waste of time, and he put them to good use.

The same can be said for the tools at your disposal in the Web 2.0 era. Put them to good use and you will be rewarded.

The Corporation's Two Bodies

by Mike Loukides | @mikeloukides | comments: 16

The New York Times quotes Laura Martin of Soleil Securities, as saying "This is management putting its employees and its job security ahead of current Yahoo shareholders' interest." The sense of horror here--that management could actually put the interests of employees ahead of the interests of investors--is interesting, to say the least. It raises an important question that's really almost theological in nature. It is most certainly theological in, as Lawrence Ferlinghetti wrote, "the promised land where every coin is marked In God We Trust, but the dollar bills do not have it being gods unto themselves. ("Autobiography," A Coney Island of the Mind, 1958, New Directions)

(continue reading)

New Release 2.0 on Money 2.0

by Jimmy Guterman | comments: 7

One year ago, we published an issue of Release 2.0 entitled "When Markets Collide" (download a PDF), in which we considered what Wall Street and Web 2.0 might have to teach one another. Quite a bit, it turned out: the key parallels we uncovered include latency (both have to do their jobs more or less instantly), connectivity (it's the liquidity of Web 2.0), sensors and actuators (and how to use them), and reputation (stockbrokers are no longer curators -- they're rated).

So it's a ripe time to consider the status of the relationship. What's new? What's changed? The amount of financial data available publicly is astonishing. That doesn't mean it's all useful. There's plenty of data out there, but it's plenty confusing. You can't extract alpha until you understand what you're looking at. As Michael Simonsen, president and CEO of Altos Research, puts it, "free data on the internet is a mess."

If anyone doubts that financial markets and technology markets are deeply intertwined, consider this: the same day that JPMorgan Chase revealed its "purchase" of Bear Stearns, a Gartner Group analyst released a report showing that "the financial services industry continued to lead all vertical markets in server revenue, as it accounted for 25.3 percent of worldwide server revenue in 2007." As goes one set of markets, so goes the other.

In this issue of Release 2.0, we consider the Wall Street/Web 2.0 mashup from a number of angles. We talk to Paul Kedrosky, chair of our Money:Tech conference and an influential blogger on the topic (as well as others), about why some on Wall Street hate Web 2.0 -- and what Web 2.0 can do to infiltrate Wall Street nonetheless. Entrepreneur Marc Hedlund, now chief product officer for OATV-funded personal finance startup Wesabe, examines what happens when hidden data gets surfaced. Cathleen Rittereiser talks to hedge fund managers to discover what they want from Web 2.0 -- and what they're actually getting. Longtime Radar contributor Nathan Torkington digs deep into prediction markets and spells out both how to manage them and what companies can gain from implementing them.

It's a truism that alpha lasts longest when it's hidden. That may have been true in the past, but the growing use of Web 2.0 tools means that less data will stay hidden, and what's hidden will stay hidden for a shorter period of time. As James Altucher of Stockpickr said at Money:Tech, "When it comes to data nowadays, closed source is a myth."

You can purchase the current issue of Release 2.0 or, even better, subscribe to the newsletter.

To be free, information has to be smart (comments on Chris Anderson's "Free!")

by Andy Oram | @praxagora | comments: 5

WIRED Magazine's editor in chief Chris Anderson, following up on the popularity of his Long Tail meme, theorizes in the March 2008 issue of WIRED about the modern tendency to put information online at no cost. I'll start this blog with the implications of offering free information in the computer field, and build from there to what I agree and disagree with in Anderson's article.

Anderson's taxonomy of "free" contains six models that justify giving the information away. The idea of "free as in freedom" (that is, open source information in the GPL or Creative Commons style) doesn't enter at all into his article. Is that important, given that the article is economic rationale for business? I think it's a crucial omission.

(continue reading)

Trendalyzer view of the banking crisis

by Jesse Robbins | @jesserobbins | comments: 3

The team at "And Still I Persist" has created their own version of Hans Rosling's "Trendalyzer" (see: Radar post) to visualize the current US banking crisis.

"First lets look at the top 8 banks and their mortgages that are 90+ days late. Below is a flash charting system, feel free to use the controls and experiment. We chart the total assets of the bank along the horizontal axis, the value of loans that go 90+ days late on the vertical, and the size of the circles represent the total loan portfolio for that bank. You can set the charts in motion by hitting the “Play” button and stop them at any time. Hovering over a circle will show you the value for that data point.

Our charts step forward in time for Q1-2002 one quarter at a time, reading directly from the bank’s own FDIC reports. "
Bank Portfolios - 90+ Days Late

See the original article for more about this visualization and the team that created it.

Update: Bruce Henderson invites anybody interested in working with a larger data set to take a look at the OSG Boomerang tool.

Understanding the undersea cable cuts... (updated: "fifth cable cut")

by Jesse Robbins | @jesserobbins | comments: 5

The Fiber Cuts in the Middle East are getting a lot of attention. The economic damage is real and the geopolitical issues are extremely complex (which is why I edited my earlier post).

From an operations perspective these kinds of outages are nothing new, and underscore why having "many eggs in few baskets" is such a problem. I believe we will see similar incidents when we have the first multi-datacenter failures where multiple providers lose significant parts of their infrastructure in a single geographic area. (Remember: location is a basket too!)

To really understand the current issue, I recommend Neal Stephenson's incredible (and lengthy) Wired article from 1996 entitled "Mother Earth Mother Board":

[...] It sometimes seems as though every force of nature, every flaw in the human character, and every biological organism on the planet is engaged in a competition to see which can sever the most cables. The Museum of Submarine Telegraphy in Porthcurno, England, has a display of wrecked cables bracketed to a slab of wood. Each is labeled with its cause of failure, some of which sound dramatic, some cryptic, some both: trawler maul, spewed core, intermittent disconnection, strained core, teredo worms, crab's nest, perished core, fish bite, even "spliced by Italians." The teredo worm is like a science fiction creature, a bivalve with a rasp-edged shell that it uses like a buzz saw to cut through wood - or through submarine cables. Cable companies learned the hard way, early on, that it likes to eat gutta-percha, and subsequent cables received a helical wrapping of copper tape to stop it.

[...] There is also the obvious threat of sabotage by a hostile government, but, surprisingly, this almost never happens. When cypherpunk Doug Barnes was researching his Caribbean project, he spent some time looking into this, because it was exactly the kind of threat he was worried about in the case of a data haven. Somewhat to his own surprise and relief, he concluded that it simply wasn't going to happen. "Cutting a submarine cable," Barnes says, "is like starting a nuclear war. It's easy to do, the results are devastating, and as soon as one country does it, all of the others will retaliate."

As the capacity of optical fibers climbs, so does the economic damage caused when the cable is severed. FLAG makes its money by selling capacity to long-distance carriers, who turn around and resell it to end users at rates that are increasingly determined by what the market will bear. If FLAG gets chopped, no calls get through. The carriers' phone calls get routed to FLAG's competitors (other cables or satellites), and FLAG loses the revenue represented by those calls until the cable is repaired. The amount of revenue it loses is a function of how many calls the cable is physically capable of carrying, how close to capacity the cable is running, and what prices the market will bear for calls on the broken cable segment. In other words, a break between Dubai and Bombay might cost FLAG more in revenue loss than a break between Korea and Japan if calls between Dubai and Bombay cost more.

The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is $30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark. [Link]

Update Feb-06 @ 08:52 GMT: I am aware of five cable segments that are experiencing problems, including one that was reported on January 23rd which had a repair already underway. I don't think this is a "fifth cut" as some people are starting to report, and I'll post an update if that changes.

A lot of needless confusion and worry could be avoided if FLAG Telecom and the other carriers involved would provide timely and useful updates on their website. It appears that they are doing a good job of restoring connectivity, but they are terrible job of telling an increasingly concerned public exactly what is going on. This kind of confusion resulted in false reports that "Iran was completely offline", which was corrected by the Renesys blog team after the story spread to influential blogs, Slashdot, Digg, and the mainstream media.

Web2Summit: Complexity + Tight Coupling = Catastrophe

by Jesse Robbins | @jesserobbins | comments: 4

Paul Kedrosky and Tim O'Reilly just talked about the "Quant Fund Meltdown" and how complex interactions between computer systems and people resulted in unprecedented hedge fund losses. I spend a lot of time thinking about risk and failure in complex systems, and I've found Charles Perrow's "Catastrophic Potential" model to be very useful. It's pretty straightforward...

strong>Update: Paul Kedrosky likes Charles Perrow too.

Technorati Tags: failurehappens, operations, web2.0, web20, web2summit

Financial Hacking -- Giving Creative Accounting a Good Name

by Andrew Savikas | @andrewsavikas | comments: 0

The subject of the next edition of Release 2.0, available next week, is the collision between Wall Street markets and Web 2.0 markets. The intersection between technology and finance is a busy one, and there might just be some hacker spirit hiding behind those suits.

I came across this article in Business Week last week about shared-equity mortgages, and it struck me as a bit of a "mortgage hack", a clever way of solving a very common problem (particularly in the still overheated -- if cooling -- housing market here in Boston).

Most people have a negative opinion of "creative accounting" in much the same way that most people have a negative opinion of "computer hacking", and I think there are some surprising parallels there. Yes, there's a lot of bad actors, like Enron's Andrew Fastow, using financial shell games to enrich themselves and conceal their actions (kind of a financial root kit). But there's also been -- and continues to be -- some amazing innovation, which perhaps is viewed as guilt by association or "too complex" for the general public to understand (again, sounds a lot like perceptions of computer hackers).

Just a few examples (among many) of this kind of innovation:

• Financial "derivatives" like futures and options contracts: These allow risk to flow smoothly from those who have more than they'd like (insurers with a lot of coastal customers) to others willing to assume that risk in exchange for substantial upside potential (speculators with models suggesting hurricane season will be mild).
• Exchange Traded Funds (ETFs): These bundles of stocks, bonds, or even futures traded together, offer tax and cost advantages to many investors compared with more traditional investments. They emerged themselves as alternatives to the standard Index Fund, another classic financial innovation, courtesy of John Bogle at Vanguard (his 2006 appearance on the Venture Voice podcast is well worth a listen).

More recently, with the aversion to IPOs seen in the wake of the dot-com bubble, there's been an increase in the hacking of IPOs themselves (arguably something Google did in their Dutch Auction) with things like "reverse mergers", which manage to separate the "going public" part of an IPO from the "raising capital" part. For smaller companies looking for funding, it's become a very attractive option: Looking at deals under $50 million, in 2001, there were 30 IPOs and 12 reverse mergers -- by 2004, that ratio had nearly inverted, with 64 IPOs but 117 reverse mergers.

Those numbers come from this PDF overview of reverse mergers, which includes the following text that sounds like it would be right at home in make:

Technically, reverse mergers are not the intended way for companies to go public.

Of course, just as with technology, there's plenty of opportunity for things to go badly -- the current subprime mortgage shakeout shows that it's clearly possible to get too creative with financing, especially when consumers are involved. But just as the latest news of data theft or site cracking should be viewed in the context of all the benefits that also accompany the technology that made it possible, we should keep in mind that most of the "creativity" that happens in the financial world is the good kind.