Four short links: 5 November 2009

Heat Maps in R, EC2 Blackhat Tricks, Snickersome Unicode, and Decoding Statistics

1. Heat Maps in R -- We used financial data here because it's easier to access than the airline data, but it's actually a pretty interesting way of looking at a financial time series. Weekend and holiday effects are a bit more obvious, and it's a bit like being able to see the daily, weekly, monthly and yearly closes all at once (by scanning your eye over the calendar in different directions). Includes source code. (via migurski on Delicious)
2. BlackHat and EC2 -- Theft of resources is the red-headed step-child of attack classes and doesn't get much attention, but on cloud platforms where resources are shared amongst many users these attacks can have a very real impact. With this in mind, we wanted to show how EC2 was vulnerable to a number of resource theft attacks and the videos below demonstrate three separate attacks against EC2 that permit an attacker to boot up massive numbers of machines, steal computing time/bandwidth from other users and steal paid-for AMIs. (via straup on Delicious)
3. Funny Characters in Unicode -- I never get tired of the wacky stuff in Unicode. I love the thought of a Unicode committee somewhere arguing passionately about the number of buttons on the snowman .... (via Hacker News)
4. Statistics to English Translation -- The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate. This matters. Bandying around numbers with misleading labels, or misinterpreting numbers that have a precise and defined meaning, does not further understanding. (Said 78.4% of statisticians, with a 20% confidence factor probability of false positives)

Four Short Links: 28 August 2009

The Future, Python Metrics, Distributed Version Control, and Stylish R

1. What The Future's All About (Webstock Words) -- Bruce Sterling on the future. We’re not going to get a future Cloud World as somehow opposed to a future Augmented Reality World. It can’t happen. The ideas can be clearly distinguished, but ideas about technology, labels for technology, predictions and suppositions about technology, they don’t map onto actual real-world technology. Human culture doesn’t work like a logical argument.
2. PyMetrics -- code analysis software that produces metrics for your code. (via the excellent 10 Ways To Let People Know You're a Bad Python Programmer by Noah Gift)
3. Prophet and SD 0.7 Are Now Available -- Prophet is a lightweight schemaless database designed for peer to peer replication and disconnected operation. Prophet keeps a full copy of your data and (history) on your laptop, desktop or server. Prophet syncs when you want it to, so you can use Prophet-backed applications whether or not you have network. SD (Simple Defects) is a peer-to-peer issue tracking system built on top of Prophet. In addition to being a full-fledged distributed bug tracker, SD can also bidirectionally sync with your RT, Hiveminder, Trac, GitHub or Google Code issue tracker.
4. Google's R Style Guide -- R is a high-level programming language used primarily for statistical computing and graphics. The goal of the R Programming Style Guide is to make our R code easier to read, share, and verify. The rules below were designed in collaboration with the entire R user community at Google. (via Bo Cowgill's blog)

Four short links: 20 August 2009

DIY SPY, Screencasting, Social Network Analysis, Term Extraction

1. DIY SPY - a homebrew 2.4GHz wi-fi spectrum analyzer -- As proof of concept (and a cool toy for anyone who has one of these lying around), I have implemented a working Wi-Fi spectrum analyzer on TI’s ez430-RF2500 development kit ($50), a 2-part USB dongle which consists essentially of a CC2500 radio strapped to an MSP430 low-power microcontroller (detachable bottom half) and a USB interface which enumerates as a virtual serial port (top half). The top half doubles as a standalone MSP430 programmer, so this kit is a great cheap way to get started playing with them. (via joshua on Delicious)
2. Screenr -- Instant screencasts for Twitter. Flash-based, uploads to their site and tweets the URL. The whole "for Twitter" thing is going a little too far: who records screencasts only for Twitter? It's like having a spellchecker only for three-letter words.
3. Social Network Analysis in R -- video and slides for talk on doing social network analysis with R.
4. We're Keeping the Term Extraction Service -- Yahoo!'s useful API gets a stay of execution. OK, we heard you. You’ve made it clear to us that shutting down the Term Extraction Service would be a mistake. So, we’ve changed our plans. We're leaving the service up and running indefinitely. (via Simon Willison)

Making Government Transparent Using R

Danese Cooper thinks it will be an important tool in Open Gov

With Open Source now considered an accepted part of the software industry, some people are starting to wonder if we can't bring the same degree of openness and innovation into government. Danese Cooper, who is actively involved in the open source community through her work with the Open Source Initiative and Apache, as well as working as an R wonk for Revolution Computing, would love to see the government become more open. Part of that openness is being able to access and interpret the mass of data that the government collects, something Cooper thinks R would be a great tool for. She'll be talking about R and Open Government at OSCON, the O'Reilly Open Source Convention.

James Turner: Why don't you start by describing where you came from, and you're involved in, and what your interests are?

Danese Cooper: Okay. I'm Danese Cooper. I serve on the board of the Open Source Initiative. I have been serving for the last eight years. And I'm also currently employed by Revolution Computing, which is a start-up focusing on an open source language called R, as in the letter R, that is very useful for analytics and statistical analysis. I'm also an Apache member. And I also serve on an advisory board for Mozilla.

James Turner: One of the two panels you're going to be speaking on at OSCON is on open source and open government. If you could talk a little bit about what interests you about open government and also what open government means to you.

Danese Cooper: Sure. Well, along with a lot of open source people, I got interested in the Obama campaign and in helping President Obama get elected. And part of why he was so compelling was that the vision of how Washington needed to change is pretty close to the way that we think about working collaboratively in open source. The night that he was elected, there was a great little clip on CNET of a Republican commentator actually explaining open source as exactly what I just said. It was a really brilliant little two-minute clip. He pointed at The Cathedral and the Bazaar, that canonical document about how open source works. And he said, "Microsoft is the cathedral. It's their way or the highway. And the bazaar is a bunch of people working together grassroots to collaboratively build the things that they need. And so Obama's basically asking for the government to become open source, and the problem is Washington isn't really like that right now."

So anyway, that's the transformation that has to happen in order for government to really be transparent. To me, open source government is transparent government. There's been an awful lot of shenanigans in recent political history, like the last decade has been pretty crazy in terms of things happening that couldn't be traced back to any source. Even just the way we vote and the way that voting is managed, and the fact that the software that runs the machines that we vote on is not open source so it can't be inspected. And nobody knows quite what it does. There are all of these stories of weird updates to the software that happened right before major elections in states where there are strange results. Transparency, in the same way that it helped the software industry transform, could really help the government transform. So that's what I'm talking about. There's a bunch of other people on that panel. My good friend, Brian Behlendorf, and I co-proposed it. And he's actually taken the next step. He helped found Apache. And he's run off to Washington to work on projects that are interesting to the Obama government to try to figure out how to help them to more open source solutions. And he'll be talking about his progress on that panel. So I think it's a pretty exciting panel.

(continue reading)

Four short links: 7 July 2009

Motivation, R, Games, and Open Source Medicine

1. Announcing your plans makes you less motivated to accomplish them -- Tests done since 1933 show that people who talk about their intentions are less likely to make them happen. Announcing your plans to others satisfies your self-identity just enough that you’re less motivated to do the hard work needed. I have noticed this myself. It must be balanced against the other finding that public commitment increases probability of followthrough, which might work in sales but seems to fail miserably in getting me to do anything productive. (via migurski on Delicious)
2. Rseek -- search engine for info on R. Necessary because of the non-unique project name. (via Benjamin Mako Hill)
3. Treasure World (Offworld) -- Nintendo DS game that turns wifi spots into collectible treasure. You have to explore the real world as you play the game, another of these games that mix the online and offline worlds. (via waxy)
4. 50 Successful Open Source Projects That Are Changing Medicine -- notice the large number of electronic health record (EHR) suites. What are the chances of any of them getting a slice of Obama's EHR money that the ex-RedHatters behind The Axial Project are going for? (via timoreilly on Twitter)

Big Data: SSD's, R, and Linked Data Streams

The Solid State Storage Revolution: If you haven't seen it, I recommend you watch Andy Bechtolsheim's keynote at the recent Mysqlconf. We covered SSD's in our just published report on Big Data management technologies. Since then, we've gotten additional signals from our network of alpha geeks and our interest in them remains high.

R and Linked Data Streams: I had a chance to visit with Dataspora founder and blogger Mike Driscoll, an enthusiastic advocate for the use of the open source statistical computing language, R. After founding and leading online retailer CustomInk.com, Mike went back to grad school and earned a doctorate in Bioinformatics. He has applied data analysis and programming in a variety of domains including retail, biotech, academia, and government projects.

Having been an avid user of S/S-Plus in the 1990's, I seamlessly switched over to R in the early 2000's. To this day, I consider the S/S-Plus user manuals to be the best reference and introductory books on the R programming language. (Mike wholeheartedly agrees.) R has been popular in the statistics community for many years, but I've been noticing that its visualization and analytic capabilities are attracting interest from developers. Moreover, recent efforts by the R community to improve its ability to scale large data sets (see brief update from Jay Emerson), will strengthen R's place in the Big Data stack.

(continue reading)