Four short links: 6 November 2009

Barcode Scanning, Downloadable Community Book, Gov Hack Day, Android Kludges

1. Red Laser -- "impossibly accurate barcode scanning". Uses Google Product Search to identify products that you scan using the camera on the phone. I remember Rael and I talking to Jeff Bezos about this years ago, before camphones had the resolution to decode barcodes. The future is here and it's $1.99 on the App Store ... (via Ed Corkery on Twitter)
2. The Art of Community For Free Download -- Jono Bacon's O'Reilly book on community management now available for free download (still available for purchase!).
3. Gov Hack -- Australian government ran a hack day with their open data, this is their writeup.
4. Android Mythbusters -- slides for talk by Matt Porter at Embedded Linux Conference Europe. A (long) catalogue of the kludges in Android.

Four short links: 3 November 2009

Electoral Cryptography, Dataless Airport Security, Visualising Transport Data, Mathematically Insecure Social Asymmetry

1. First Test for Election Cryptography (MIT Technology Review) -- The first government election to use a new cryptographic scheme that lets both voters and auditors check that votes were cast and recorded accurately will be held tomorrow in Takoma Park, MD. Founder of the company behind the technology is David Chaum, who ran the first electronic currency company in the 90s. That was ahead of its time (Internet faced a credibility problem, not a convenience problem), but his timing for this seems spot-on. (via timoreilly on Twitter)
2. Do I Have The Right To Refuse This Search? -- a former police officer questions the efficacy of TSA screenings and is doubly worried by by the lack of data collected. For years in policing, we relied on random patrols to curb crime. We relied upon this “strategy” until someone went out and captured some data, and did a study that demonstrated conclusively that random patrols do not work (Kansas City Study). As police have employed other types of “random” interventions, as in DWI checkpoints, they have had to develop policies, procedures and training to ensure that the “random” nature of these intrusions is truly random. Whether every car gets checked, or every tenth car, police must demonstrate that they have attempted to eliminate the effects of active and passive discrimination when using “random” strategies. No such accountability currently exists at TSA. Trend I see lately is a return to quantitative decision making, reality-based data-directed system interventions. (via BoingBoing)
3. Visualising Transport Data -- It can be hard to make meaningful information from huge amounts of data, a graph and a table doesn't always communicate all it should do. We have been working hard on technology to visualise big datasets into compelling stories that humans can understand. We were really pleased with what we came up with in just one and a half days. Like many places, the UK data.gov ran a dev camp to jumpstart people using their data. These appear to be successful, but I'm not aware of studies into the longterm effects nor the "value" of different types of developers.
4. Why Your Friends Have More Friends Than You Do -- there's a numerical optical illusion at work here: count your friends, then ask them to count their friends. If you average the friend counts of your peers, it'll probably be higher than your friend count. The reason for this is also why (on average!) your sexual partners seem to have had more sexual partners than you, and why previous generations seem more fecund than current generations. It's because connectors (with large numbers of friends) distort the average, so unless you're the connector (and if you're reading this, you might well be!) the average will be bigger than a normal person's friend count. Left unmentioned is what kind of person would count the number of friends they have, then ask their friends for their counts .... (via Hacker News)

The Hacker Ethic - Harming Developers?

On Monday Neil McAllister posed the question "is the hacker ethic harming American developers?" Slashdot picked it up and Tim forwarded it to the Radar list. As you might expect, it resulted in some spirited discussion.

James Turner kicked things off with this response (it has been slightly edited from its email form). After James lays out his argument I'll reply with my thoughts. Then we hope to hear from you. Let us know what you think.

I've worked in a lot of organizations that thought that the kind of rigid deforesting paradigms that Nayar is referring to were the magic bullet to keeping all three of the variable (dollars, time, features) under control. Without exception, all they did was get in the way and reduce the productivity of the most senior people to the level of the most junior. All of them exhibited some degree of failure, some catastrophic.

The India shops *love* methodologies like UML and the like, specifically because the problems have been reduced to a simplistic enough granularity that they can be doled out to junior-level staff, who may have only been onboard a few weeks because of the massive churn over there.

At least 3 times at 3 different companies, I've seen major pieces of work brought back in-house because the Bangalore team had fallen so far behind or proved so unable to get beyond the literal description of work that they were endangering the project. When you combine the time difference with a tendency to halt dead in their tracks as soon as they hit a stumbling block, it can be a recipe for disaster.

There are certainly some good Indian shops, and I know some outstanding Indian developers (most of whom have come to the States.) But I find Nayar's comments hilarious. It's akin to someone saying that American football players aren't employable in Jamaica because they aren't able to limbo well. Look at the most successful Web 2.0 companies today, most of them started as garage enterprises with a few talented developers, not a 60 person team of UML jockies following some Arthur Anderson project management program. Heck, look at Google Labs.

In huge projects, you obviously need some master planning and coordination to make sure the tracks meet at the right place to drive in the spike, but I don't see any effort being made these days to right-size the amount of project overhead to the needs of the projects. Instead we get a one-size-fits-all approach that smothers anything but the largest project in paperwork. Even some of the original authors of the Agile manifesto, when I've talked to them, point out that part of being Agile is picking and choosing the right components of project management that make sense for a given task.

Nayar's remarks are incredibly self-serving. "We're the best, because we can mindlessly follow some arbitrary and flawed development process." Or is he claiming that Indian projects do better QA? Not in my experience...

This entire debacle is representative of a problem I think is endemic in the industry these days, the inability or unwillingness to engage in rapid prototyping. Every successful project I've ever worked on (and I've worked on some fairly large enterprise-sized projects), we started by designing and coding a quick "throw-away" skeleton of the application, that let us look at how the thing worked, where the unseen warts were, and where the vendors had lied about their APIs, etc. This is the crucial and neglected stage in project design, one that most modern design paradigms ignore or actively discourage. Even Agile tries to jump in and start coding the finished application from the get go, although if project teams were willing to aggressively refactor (a tenant of Agile), early project work could be a rapid prototype (although the story model of scrum really doesn't fit well with this, unless you make the prototype a story...)

This is also something I've never seen an offshored team do particularly well...

Well, I'll be damned if I'm going to jump in and take the side that says hacking is bad for American programmers. First off, I don't need that kind of flame bait and second, I don't believe it. I think approachable programming is hugely important because that's how many people get into the field in the first place. However, my reaction to the article was very different than James' and I might as well try to explain.

I'm not going to take an opposing position, but it's not really an orthogonal position either. Maybe it has a power factor of about .7 or so. Here's my (also edited) response...

When I when I read McAllister's piece, at least some of it resonated with me. Before we were bought by a large firm, we were a small company that grew from nothing to 250 people, about 200 or more of them were programmers. So, a whole lot of my time over three years was spent hiring programmers and building cohesive teams that could deliver to our customers.

In our hiring we aggressively hired hackers into the mix. We wanted outside-our-industry thinking and we thought they brought in creativity. We called it "hiring weird, but not weird weird." Occasionally we pushed it to weird and a half. For our efforts we got creative problem solving and interesting (but frequently weird) dinnertime conversation when we travelled.

However, our pollyanna idea of "disciplined teams catalyzed with a bit of weird" didn't always work out.

That leads to the bit that resonated with me: the sense that hacker = distilled essence of American individualism combined with lots of ISTJ Myer's Brigg's Type Indicator. Individualism is a trait that I hold dearly, but it can make a cohesive team effort difficult if people are unwilling to suborn themselves to the goals of the team. Remember those tee shirts the football team always wore at your university? "There is no I in team?" I sometimes joked that I was going to make a batch that said "I'm the Me in team."

Maybe we were just growing fast and it was going to take more storming and norming than I had patience for, but at times it was a struggle to get everyone to see past their individual biases and focus on what we were trying to achieve, and we couldn't do what we were trying to do with teams of one.

But it really wasn't a hacker problem if hacker means self taught like McAllister implies. We had a lot of people with CS degrees and we used to talk a lot about whether and how their degrees had prepared them for their jobs.

Separate from the individualist approach to development, few of our recent graduates came to us prepared with the terminology and practices of any development approach (or engineering approaches like continuous integration etc.). They knew how to code, but not how teams coded.

At one point I gave a talk on agile software development to about 100 CS students at a university in Philadelphia and I asked them to raise their hands if they had ever done a team project with greater than two people on the team. I don't recall anyone raising a hand. Then I asked if they had ever covered development methodologies in their classes and a few acknowledged they had, but it had been abstract classroom stuff only. That part surprised me.

I'm not sure that the "sanctity of engineering" argument really makes that much difference. I have little faith in McAllister's scheme to do computer engineering instead of computer science coursework.

My undergraduate degree was in Mechanical Engineering and I can only imagine how useless I would have been to a firm that actually did engineering, and for mostly the same reasons. I knew how to take integrals and I still know the packing ratio of a hexagonal close packed material but I didn't know squat about how a complex machine actually got designed in a team setting. It's interesting to note that the Engineer in Training exam I took (a precursor to the professional engineer's exam) didn't probe my knowledge of team practices at all.

Maybe there just isn't time in an undergraduate degree to teach everything that an engineer needs to know. Plus, can you imagine the drop out rate in CS/CE if ITIL was a required course?

Since James mentioned Google I'll switch gears and muse about ecosystems for a moment... I guess I tend to bristle when I hear that everyone should just develop software the way Google does.

Google is to computing what LA was to Aerospace and Electronics in the early 60's. It's gravitational force attracts five sigma talent (probably a bunch of six too) in ways that the rest of us can only envy. More generally, Silicon Valley has had programmer talent flowing into it for the last twenty years the way Hollywood sucks pretty people out of the midwest.

Maybe it's not as obvious because you can't spot brains the way you can spot an oddly beautiful wait staff, but the valley has been the vortex of a talent-laden embarrassment of riches for a long time and, if you work there, you might not even notice it. However, I think that at some level this effects what kinds of processes work when you build software. I think it's at least a little part of the reason why an ERP system in a manufacturing town gets implemented differently than MapReduce (there are other reasons too having to do with software as product vs software as supporting infrastructure). Combine that with the very clear shared vision of "lets do something great and get rich together" thing that valley firms often have, and well, it's easy to see how smart people coalesce to build amazing stuff.

It's easy to denigrate Arthur Anderson's progeny or the offshore firms they compete with, but they do different work, with a different talent pool, for different ends, and with a very different set of personal and organizational incentives. Or, put another way, Kelly Johnson didn't build the SR-71 with General Motor's engineers, and General Motors didn't design the Chevy Cavalier with the Skunkworks' processes. However, even at the Skunkworks, Johnson's brilliant engineers did conform to a process and work together as a team toward a shared vision. And, conversely, I bet a lot of talent is left on the table at General Motors because of processes too restrictive in their attempt to remove all uncertainty.

So,... maybe it's possible that Google's (or the valley's in general) processes are appropriate to an ecosystem that, because of the intellectual environment and potential for riches, is rich in IQ and initiative. So it ends up feeling more "special forces" and less like "infantry regiment." And over there closer to the hump in the normal distribution curve, or in a different cultural environment outside of the valley, a different flavor of processes may be effective?

The counter argument to that, which I'll go ahead and provide, is that I once helped teach a team of engineers at midwestern defense contractor how to do agile development. The effect was amazing and immediate and their productivity and satisfaction went up tremendously; until their management freaked out and shut it down when they "perceived" that it created too much uncertainty in their processes.

Well, it's obvious that I don't know the answers here, so, with that, I'll stop thinking out loud.

What do you think?

Four short links: 4 May 2009

Maps, Africa, Protein, and Rockets

1. Old Japanese Maps on Google Earth Unveil Secrets -- Google criticised for putting up map layers showing the towns where a discriminated-against class came from, because that class is still discriminated against and Google didn't put any "cultural context" around it. Google and their maps didn't make the underclass, Japanese society did. Because they're sensitive about having the problem, they redirect their embarrassment into anger at Google. You could make a long and profitable career in IT consulting simply by charging to say "it's not a technical problem" and you'd be right more times than wrong.
2. See Africa Differently -- using the Internet to reframe a continent. Videos, essays, and more, all designed to get you seeing the majority of Africa, which isn't defined by conflict and famine. (via NY Times book review)
3. Fold.it - Solve Puzzles for Science -- science harnesses our "cognitive surplus" by inviting us to help solve the problem of protein folding, one of the hardest in biology. (via auckland_museum on Twitter)
4. Arduino Telemetry Payload in Class C Rocket (Jon Oxer) -- Because class-C rockets are so small and light they can't lift much of a payload and I had to keep the mass of the electronics as small as possible. You can get a sense of scale from this photo which shows a small white bundle in the bottom of the nosecone. Inside that bundle is an Arduino Pro Mini 5V/16Mhz, a 433Mhz transmitter module, and a Lilypad 3-axis accelerometer. PCBs ... in ... Spaaaaace!

Four short links: 31 Mar 2009

Web traffic, web design, hacker spaces, and feature spaces:

1. iPhone and Android Make Up 50% of Google's SmartPhone Traffic Worldwide -- Matt Gross found this interesting tidbit in a TechCrunchIT story.
2. Refining Data Tables -- Luke Wroblewski gives some seriously good tips for designing usable tables in web pages. After forms, data tables are likely the next most ubiquitous interface element designers create when constructing Web applications. Users often need to add, edit, delete, search for, and browse through lists of people, places, or things within Web applications. As a result, the design of tables plays a crucial role in such an application’s overall usefulness and usability. But just like the design of forms, there’s more than one way to design tabular data. (via migurski's delicious stream)
3. Hacker Spaces (Wired) -- "It's almost a Fight Club for nerds," says Nick Bilton of his hacker space, NYC Resistor in Brooklyn, New York. is the must-have quote, but the guts of the article is "In our society there's a real dearth of community," Altman says. "The internet is a way for people to key in to that need, but it's so inadequate. [At hacker spaces], people get a little taste of that community and they just want more."
4. Related Document Discovery Without Algebra -- latent semantic analysis has some scary math, but If the feature space (e.g. the terms/concepts associated to your documents) is small enough, and you make sure synonymy is not a problem, you can do without algebra. One such case is that of your blog postings and their tags. Includes Ruby code. (via joshua's delicious stream)

The Last HOPE

I made the trek to a steamy hot NYC this weekend to attend one day of the three day Last HOPE (Hackers on Planet Earth) conference at the Hotel Pennsylvania. There was too much going to adequately cover it here (or even take it all in), but a few things stood out.

Steve Rambam's eye opening talk on the death of privacy for example. For a solid three hours in front of a standing room only crowd he weaved back and forth between the Orwellian theme of how our privacy is being ripped from us by everyone from Google to Choicepoint and the theme that seemed even creepier to him, self contribution. Over and over he expressed disbelief at how willingly we post our personal details everywhere from Twitter to Facebook while thanking us all the while for making his job as a private investigator that much easier. What the marketers and government don't actively take, we actively give. Naturally I twittered the whole thing.

Cell phone tracking; artificial-intelligence-assisted reality mining; 3000 cameras per square mile in Manhattan; facial, activity, and even gait identification software; government outsourced investigative databases shielded from FOIA requests; UAV-based license plate scanners; beating anonymity by correlating multiple datasets; unanticipated database repurposing; and on and on... Finally I could twitter no more and left the venue hurriedly fashioning a tinfoil hat from a burger wrapper while consigning myself to doubling the dosage on my meds.

I will say this though, there was something deliciously ironic about standing in a room chock full of hackers all listening at rapt attention to a three hour chillingly dystopic harangue on privacy loss while nearly every single one of them was wearing an RFID tag around their necks. Even better, the tag was tracking their every move around the venue and was linked to a comprehensive self-contributed profile.

Moving beyond the privacy nightmare stuff, there was hardware hacking to be found everywhere at Last HOPE. Tables were covered with broken open electronic toys and electronic components and were surrounded by hackers with smoking soldering irons.

Of the completed projects on display, one of my favorites was a something of a hybrid that projected a 3D image onto carefully placed strings.

(continue reading)