Four short links: 4 November 2009

Electronics Hacking FAQs, Speech-To-Text Democracy, Open Source Column Database, Massive Online Analysis

by Nat Torkington | @gnat | comments: 1

1. ChipHacker -- collaborative FAQ site for electronics hacking. Based on the same StackExchange software as RedMonk's FOSS FAQ for open source software.
2. Democracy Live -- BBC launch searchable coverage of parliamentary discussion, using speech-to-text. One aspect we're particularly proud of is that we've managed to deliver good results for speech-to-text in Welsh, which, we're told, is unique. I think of this as the start of a They Work For You for video coverage. I'd love to be able to scale this to local government coverage, which is disappearing as local newspapers turn into delivery mechanisms for real estate advertisements.
3. InfiniDB: Open Source Column Database -- hooks into MySQL, uses MySQL for SQL parsing, security, etc. The commercial enterprise version has multi-server support (parallel scale-out). (via Brian Aker)
4. Massive Online Analysis -- MOA is a framework for data stream mining. Includes tools for evaluation and a collection of machine learning algorithms. Related to the WEKA project, also written in Java, while scaling to more demanding problems. . (via joshua on Delicious)

Four short links: 3 November 2009

Electoral Cryptography, Dataless Airport Security, Visualising Transport Data, Mathematically Insecure Social Asymmetry

by Nat Torkington | @gnat | comments: 0

1. First Test for Election Cryptography (MIT Technology Review) -- The first government election to use a new cryptographic scheme that lets both voters and auditors check that votes were cast and recorded accurately will be held tomorrow in Takoma Park, MD. Founder of the company behind the technology is David Chaum, who ran the first electronic currency company in the 90s. That was ahead of its time (Internet faced a credibility problem, not a convenience problem), but his timing for this seems spot-on. (via timoreilly on Twitter)
2. Do I Have The Right To Refuse This Search? -- a former police officer questions the efficacy of TSA screenings and is doubly worried by by the lack of data collected. For years in policing, we relied on random patrols to curb crime. We relied upon this “strategy” until someone went out and captured some data, and did a study that demonstrated conclusively that random patrols do not work (Kansas City Study). As police have employed other types of “random” interventions, as in DWI checkpoints, they have had to develop policies, procedures and training to ensure that the “random” nature of these intrusions is truly random. Whether every car gets checked, or every tenth car, police must demonstrate that they have attempted to eliminate the effects of active and passive discrimination when using “random” strategies. No such accountability currently exists at TSA. Trend I see lately is a return to quantitative decision making, reality-based data-directed system interventions. (via BoingBoing)
3. Visualising Transport Data -- It can be hard to make meaningful information from huge amounts of data, a graph and a table doesn't always communicate all it should do. We have been working hard on technology to visualise big datasets into compelling stories that humans can understand. We were really pleased with what we came up with in just one and a half days. Like many places, the UK data.gov ran a dev camp to jumpstart people using their data. These appear to be successful, but I'm not aware of studies into the longterm effects nor the "value" of different types of developers.
4. Why Your Friends Have More Friends Than You Do -- there's a numerical optical illusion at work here: count your friends, then ask them to count their friends. If you average the friend counts of your peers, it'll probably be higher than your friend count. The reason for this is also why (on average!) your sexual partners seem to have had more sexual partners than you, and why previous generations seem more fecund than current generations. It's because connectors (with large numbers of friends) distort the average, so unless you're the connector (and if you're reading this, you might well be!) the average will be bigger than a normal person's friend count. Left unmentioned is what kind of person would count the number of friends they have, then ask their friends for their counts .... (via Hacker News)

ICANN without restraints: the difficulties of coordinating stakeholders

by Andy Oram | @praxagora | comments: 2

People interested in coalitions and policy-making on a global scale--topics that are increasingly relevant in a world whose borders are irrelevant to carbon dioxide, flu viruses, and other critical entities--need to learn from other organizations that are dealing with these issues. This week brings particularly important news about the Internet Corporation for Assigned Names and Numbers (ICANN), which has been making policy for eleven years under a number of difficult premises:

• It was created hastily and arbitrarily without roots in the communities most interested in its mandate.
• Its concept of stakeholders is boundless, potentially involving anyone who uses the Internet or gets information that has passed at some point over the Internet.
• Its reach is global, and its decisions are affected by issues of language and culture.

(continue reading)

Computerization in Nilekani's Imagining India

by Andy Oram | @praxagora | comments: 0

Imagining India: The Idea of a Renewed Nation promises to occupy a central position in discussions about India as well as the world economy this year. The book was released last year in India, came out in the United States just this past March, and has racked up some prominent reviews recently. Particularly relevant to this blog are the book's observations on computers' role in the economy and society.

Author Nandan Nilekani can speak with quite a bit of authority on computers, having founded and led Infosys, an early success story in modern Indian commerce and a major player in the historic rise of outsourcing.

Imagining India is a huge book with many big agendas; it covers education, infrastructure, environmental challenges, government intervention, and the role of historical narrative, among other things. Biggest among its agenda--and the one that I wager will generate the most debate--is Nilekani's own version of a modern combination of neoliberalism and neoprogressivism that seems to be gaining ground. The general idea is that governments should take a leading role to promote social progress by creating an infrastructure that allows individuals to form their own destinies (good education, good health care, good physical infrastructure, a light-touch form of regulation that ensures quality, and occasional direct welfare payments) rather than preserving oases of protection and easily abused subsidies for particular interest groups, notably unions, small businesses, and disadvantaged castes.

But all that lies beyond the scope of the Radar blog and of my own powers of analysis. I'll just comment on the following points from the book, because they concern the role of computers and because they resonate with trends I see in the U.S. and elsewhere.

• Technophobia shouldn't be assumed
• Internet access goes along with transparency and egalitarianism
• Software leads innovation in other areas

Technophobia shouldn't be assumed

A lot of technologists glibly anticipate that computers and Internet access will be rejected by some group of people who are implicitly labeled ignorant or clueless: racial minorities, poor people, the elderly ("how can you get my grandmother to use this?"), etc. In every case, the key to adoption turns out to be access and sometimes the availability of useful applications. When presented with the opportunity, these populations always prove eager to take advantage.

Nilekani cites one instance after another of rural village dwellers, farmers, taxpayers, and others who quickly grasp what computers and Internet access can do for them. Whether it's the chance to learn English, check crop prices, or pay a utility bill, Indians at all levels come to depend on the computer once it's introduced. (The hard thing, as you might guess, is persuading agencies and local officials to install systems that undercut their power as gatekeepers.) And we've all heard of the Hole in the Wall Project, where Indian kids in slums come to enjoy and figure out how to use computers with little or no adult help.

Nilekani may be citing anecdotes selectively, but his observations echo other reports I've heard about disadvantaged or lagging communities. The problem is not the people, but other factors such as availability, cost, and usefulness.

Internet access goes along with transparency and egalitarianism

One reason the Indian population loves computers, according to Nilekani, is that it attacks favoritism and outright corruption. This advantage matches up with the promise of open government in the United States and other developed countries.

In some cases, Indians are burdened by extremely crude forms of corruption that crumble the instant computers are installed. One example in the book is the registration of changes in land records, which farmers are required to report to the government every year. Agency staff could easily steal land by deliberately filing wrong reports, or extract bribes by delaying the filing until the desperate farmer caves in. But a computerized system takes the staff person out of the process.

Bringing sunlight into government activities in most developed countries has somewhat subtler effects and becomes a more long-term project, but the essence is the same and depends on computerization to work. In the US, we have a lot more control over the stimulus package, thanks to Recovery.gov, than we have over expenditures in Iraq or the bail-out to the finance industry. Indians are similarly learning how to watch over their governments and raise their voices digitally, according to Nilekani.

The sunny role that people around the world are granting to the technologies of going online is not intrinsic to these technologies, because they also lie at the center of modern surveillance, warfare, and regimentation. The benign role is hard won, and represents a collective choice by the public that has adopted the technologies. As Nilekani puts it:

The idea of technology as something ominous and scary that is used by "Big Brother" to control our lives and eliminate jobs has given way to the idea that it empowers, liberates and gives us access to all the services that are due to us, as citizens and consumers.

Software leads innovation in other areas

The reason that the computer industry was the first to take off in 1990s India is that it required practically no infrastructure. Of course, it required a computer, which might require six to twelve months for an import license in those days. It also required electricity, which could be obtained in major cities and supplemented by private generators. (In areas of unauthorized urban growth, the slumlords strung the wires.) So in a regulatory environment that scrutinized and imposed conditions on every allocation of equipment, it was much easier for entrepreneurs to set up a computer firm than any business that had more physical manifestations.

As is well known, the relative independence of computing from physical infrastructure also made Indian companies lucrative in a world increasingly linked by the Internet. Nilekani says that this physical flexibility was also valuable internally, helping IT-savvy businesses cut across the logistical and political barriers that have always geographically segmented the Indian market.

Nilekani seems to believe that there's nothing about the computer industry that's uniquely suited to Indian talents and business acumen. Now that the computer industry set an example, the same advantages have been applied to many other industries. In the 1980s, economists doubted that India could succeed in any industry, and a few years ago they wondered whether India could succeed in any industry except computer services. The evidence is now strong that the country will become a leader in many areas.

Indian industry is just one example where computerization has shown light on a path that social change can take. A worldwide example is provided by the open source movement, which Nilekani mentions only in the most fleeting manner in his conclusion--unfortunately enough, because free software can be a compelling wild card in story of international development, especially as part of a trend I dubbed tech-splicing in another article.

The first open license was a software license (the GNU General Public License). When it was released, the phenomena of allowing unlimited changes and sharing these changes looked like a peculiar aspect of software. But many years later, these ideas seeped out into fields of innovation with a more physical basis, and research by Eric von Hippel showed they always had legs.

Software was also the inspiration for gene splicing and other aspects of synthetic biology, even to the extent that biologists share their innovations in repositories that look like software libraries (check out the BioBricks Foundation).

Finally, the popularity of scripting and other software hacking initiated--or revived, or perhaps just legitimized--a tradition of solving a problem through invention instead of settling for a standardized, commercial solution. The DIY movement found in many areas of the world--including the Indian practice of assembling local motor vehicles called jugaad--makes it more and more likely that products of many types will come out of small, even amateur workshops.

Products of creativity and pure thought embody a freedom that allows them to metamorphose and spread quickly. The added formality and clarity that software brings to these activities doubles the power of that freedom. So my guess is that software will often lead the way in social innovations by a decade or more.

Privacy and open government: conversations with EPIC and others about OpenID

by Andy Oram | @praxagora | comments: 2

A few days ago I proposed a way to offer more privacy to people visiting government web sites. This blog builds on that proposal, which was largely technical, by examining the policy and organizational issues that swirl around it.

My ideas are informed by a discussion I had with Lillie Coney, Associate Director of the Electronic Privacy Information Center. The blog is also inspired by two comments on the earlier blog and brief email I exchanged with one commenter, which intertwine with Coney's in intriguing ways.

As I said in the first blog, my proposal focused on a very narrow question driven by the Obama Administration's interest in revising a memorandum from 2000 concerning the use of cookies in web browsers. The proposal suggested a way to better approach anonymity, but didn't look at the related social and political issues:

• The kinds of privacy and the degree of privacy people want
• When it's appropriate to make visitors identify themselves, or at least to provide some persistent identity
• Whom people trust to maintain identity information

This blog offers a number of points about those issues. The sections are:

• Can the government be your friend?
• Anonymity, pseudonymity, and participation
• Who should run an OpenID server?
• Thought experiment: could federal agencies offer anonymous authentication for whistle-blowers?

(continue reading)

Personal Democracy Forum conference: initial themes

by Andy Oram | @praxagora | comments: 0

"So what's this conference you're going to?" asked my friends, not braced for an explanation that usually took me more than ten minutes. Ultimately, though, they all expressed excitement about the ideas driving Personal Democracy Forum.

These friends care about politics. They argue over all the issues, and at some level they take note of the processes that often matter more than any arguments. But although some know what an API was and a few even understood the concept of mash-ups, it's remarkable how completely they had been bypassed by the current movement toward open government, whose importance to the Obama administration was signaled by his release of a memorandum on transparency and open government on his first full day in office.

I hooked my friends through the idea of an irreversible political shift. Not a regulatory regime that could be dismantled like the agencies responsible for civil rights, or a mandate that could be defunded like federal housing initiatives--no, in this case a movement integrating the public into government functioning, and that therefore creates an external constituency that helps to perpetuate the system; an ecosystem of non-governmental organizations that will react precipitously and aggressively if the government tries to shut them out.

Digging for themes

PDF is appropriately held in New York City, a culturally open megalopolis that is ethnically and politically uncategorizeable. Free speech holds forth on the subways where the exhortations of the homeless prove that the great art of oratory is still alive.

A thousand people signed up for the conference (leading, of course, to more than a thousand Twitterers). At the gorgeous Jazz at Lincoln Center location, the Rose auditorium was totally filled, and the hallway was choked as attendees strove to reach pitifully undersized rooms for breakout sessions.

As a conference with a contemporary, tech-oriented bent, PDF ripples off into all kinds of online resources. At several points the keynotes were held against a real-time twitter feed, goading on the feeding frenzy by showing the accounts of the people who tweeted the most. This focus on immediate response--and on quantity of response--had a specific effect on the consciousness of the audience. The twitter feed reinforced through highlighting and repetition the most provocative sound bites and the statements most clearly relating to current issues at the top of attendees' minds

This is a useful function to play, but the provocative utterance and timely issue is only one superficial level of conference engagement. We all need to take away what we've experienced, sit with it a bit, and look for underlying themes that represent a significant trends that can guide us.

Give a few hours for reflection, I'll use this blog to synthesize three recurring themes I heard during the first day. I'm sure more ideas will settle out as I spend even more time thinking through these two days of meetings.

• The prerequisite: the power for change lies with the public
• The platform for democracy: infrastructure we all need
• Time to tune in: we can't tolerate static
• Miscellaneous insights from speakers and participants

The prerequisite: the power for change lies with the public

It's scary being a politician, let alone the an agency head. These people may seem indescribably powerful to the rest of us, but they live in fear of public pillory triggered by their own missteps.

Jeff Jarvis listed, as one of his four key elements of change, the ability for government to fail without risk of recrimination. David Weinberger approached the same theme from a different direction, talking about how all wisdom is provisional, emerging, and scattered. Vivek Kundra and Beth Noveck--who will be speaking tomorrow--have repeatedly made similar statements in the context of bringing the innovation culture of the Silicon Valley to the area around Foggy Bottom.

In my first ramp-up blog for PDF I talked about a four-part cycle for successful public/government collaboration. Perhaps we need to start the cycle earlier, or add some kind of parallel cycle, to recognize that the public has to make the commitment asked by Jarvis: the promise to show forbearance when the government fails and to grant it a mandate to do innovation.

The platform for democracy: infrastructure we all need

If one engages in some deep listening, you can hear beneath all the celebrations of transparency a recognition that success depends on several elements of infrastructure. Early experiments in open government may produce exemplary and even spectacular successes, but the culture won't take hold until this platform is in place.

Computing networking and computer technology are the most obvious requirement. Mark McKinnon, a Republican communications strategist, called for universal broadband during his keynote.

But as audience members pointed out, literacy is another requirement: basic literacy as well as media-savvy literacy and knowledge of the tools that let one participate.

Ethnologist dana boyd took the discussion to the next level by pointing out that even when people do go online and do use social media, they self-segregate by race, class, and educational status. Her case study for this claim was limited (the demographics of MySpace users versus Facebook users) but the statements she culled from young people showed that the digital divide is possibly even deeper online than these social divisions are offline.

I believe that a predilection for different forums and ways of interacting online doesn't have to prevent different races and classes from coming together on issues of common interest, such as health care. But boyd's point that people set up online barriers that make it harder for them communicate across these barriers is salient. She pointed out that we need to recognize that the sites we visit are not the same sites everyone visits, to spend time on the sites of people we want to influence or collaborate with, and to embrace different modes of interaction among different social groups.

Finally, open discussion requires a tolerant environment. Recent events in Iran, as well as the introduction of Internet filtering software in China, show that governments can choke off civil society online; the technology was described as a cat-and-mouse game where both the side of information dissemination and the side of repression learn how to increase their power.

Time to tune in: we can't tolerate static

The last theme I'll highlight from the first day is the sense that we can't stand still. Americans (and particularly young Americans) expect more and more that we can have a say, that we can move quickly and have choices, that we can contribute to decisions and their implementation. We've already seen how many businesses (not all, of course) that fail to keep pace with these expectations are shrinking. If governments don't meet the expectations, people won't be able to replace it the way they replace businesses, but there could be increased feelings of alienation and increased social dissatisfaction.

Miscellaneous insights from speakers and participants

ChallengePost announced today a site that brings together people with needs and problem-solvers, using a challenge model similar to the Netflix prize or the TopCoder software firm. In publishing a challenge, someone can offer money or just recruit people to offer thanks. Respondents may be motivated to solve the challenge by intangible rewards as well as money. ChallengePost offers advice on how formulate a good challenge and judge it expertly, but the form of each challenge is the prerogative of those who post it.

The Digital Literacy Contest tries to develop a generation of problem-solvers who can analyze the streams of government data coming online. They will run contests in high schools and colleges that start with test problems and then move to questions to which they do not have the answers. When several students converge on the same solution, it is published for the public benefit.

Morley Winograd of NDN briefly analyzed Ron Paul's failure in the presidential election despite his sophisticated use of social media. If I understood Winograd, the medium--which is well constituted for bringing groups together--contrasted too much with the message of individualistic libertarianism.

In a forum on participatory medicine, Esther Dyson said of the current health care debate, "We're focusing too much on health care and not enough on health, just as one might complain that the government focuses too much on laws and not enough on getting people to do good things." This was the start of a session that discussed ways patients and doctors could use information sharing to improve outcomes and lower costs.

New York Mayor Michael Bloomberg called in over Skype instead of coming to the conference. Over his call he announced an expansion of the famous 311 service and various initiatives to accept public complaints and provide public data online. I was glad Skype was available for the call, but I find it odd for the government to be using commercial services (Kundra moving staff to Google Docs, YouTube hosting White House videos, agencies going on Facebook, etc.). I can see why the government wants to use available social media for convenience, and it provides a familiar access method for constituents. But eventually governments should develop their own public-domain software, tailored to government needs and open to all.

Blair Levin, who is designing a national broadband plan at the FCC, started out buttering up the audience by making fun of incumbent telephone companies, then gave us a "homework assignment" of reviewing and making improvements to its presentation at the the July 2nd FCC meeting, material for a set of staff workshops in August, and plans to be make in the Fall to do research. A panel following Levin's presentation--matching up a much-applauded representative from Free Press with representatives from the cable and telco industries--looked at the issue of speed. Is it fair to set a single target for speeds? Will the FCC define broadband to more closely match more advanced countries?

Personal Democracy Forum ramp-up: adaptive legislation can respond to action in the agora

by Andy Oram | @praxagora | comments: 0

This article is the last in a series leading up to the Personal Democracy Forum. The first article was posted on June 16 and the second article on June 19.

---

Whole libraries could be filled with writings about the growth of executive power during United States history. The power of the executive branch is likely to increase with technology. But for open government, that growth may be a necessary transition to more public involvement.

As its name indicates, the executive branch is responsible for carrying out the law. The open government movement wants the public to have more say in its own governance, and envisions a more fine-grained implementation of government's role in everyday life. For instance, open government advocates want more citizen input into details such as the siting of physical facilities and the choice of projects for funding. Logically speaking, therefore, the public has more control over implementation if decision-making is shifted from the legislature to the agencies carrying out the law.

Congress should also crack open its hidden chambers; law-making itself could be much more open. It will be interesting to see what comes out of work on a collaborative law drafting project in health care, started by Congressman Anthony D. Weiner of New York. I don't harbor any fantasies, though, that much of his public input will survive the traditional Congressional horse-trading that will follow.

But even the most ideal legislative process ends up with a static document that tries cumbersomely to anticipate every use and abuse of its language. (That's why laws are filled with hedges such as "This passage shall not be construed to...") Legislation is like setting off over rough terrain in a tank. Although the tank can complete the journey, it does so only by flattening everything it encounters.

Some political scientists also think that the executive branch is inherently better suited to understanding and responding to public needs. Here is an intriguing quote from Jane E. Fountain's Building the Virtual State: Information Technology and Institutional Change, summarizing work by Alfred C. Stepan:

Intellectual activities and decisions of civil servants working for long periods on policy questions are arguably more powerful and influential than the sporadic attention of legislators to particular policies.

So I'll take a look at the future of the executive branch, and end this three-part series with speculation about how to build fewer legislative tanks and more Jeeps.

• The executive branch: power and potency
• The legislative branch: how to write laws for an engaged public
• Appendix: the top question asked on Change.gov

The executive branch: power and potency

There's little mystery concerning about why the power of the executive branch tends to grow. Of the three branches of government, it's the one that actually arrives on the scene. It makes decisions about real people and activities on a daily basis and takes responsibility for those decisions.

To act effectively, the executive branch tends to centralize. (Unfortunately, so have many legislative branches in recent decades. The Commonwealth of Massachusetts is run by three people, when they're not fighting indictments or running off to seek other positions: the Governor, the Speaker of the House, and the President of the Senate.)

Because knowledge is power, technology will cause the power of the executive branch to skyrocket over the next few decades. Civil liberties advocates already decry surveillance cameras, wiretapping, the subpoening of information collected by private firms, and the computer analysis that the government applies to all the resulting data. But the data currently available is miniscule compared to everything that will be collected by atmospheric sensors, electronic toll collectors, and various other technologies that are starting to be installed. If Microsoft can produce a game machine cheap enough for the consumer market with face recognition, voice recognition, and full-body motion sensing, what can the government do to track us?

So the power that the executive branch takes on in the political realm will be multipled by the potency it obtains from the data it collects and from ever more sophisticated tools for analyzing that data.

(Strangely, the strict constructionists and "original intent" scholars, who bar judges from interpreting the Constitution broadly, don't apply these restrictions to the ever-expanding executive branch.)

I don't know how to halt this expansion of power. We could open-source the Panopticon by demanding that the public have access to all data collected by public cameras and senors. That won't help, though, because the data will still prove useful mostly to large organizations with the time and expertise to analyze it. And do you want to encourage every budding computer hacker in the country to become a data-mining Nancy Drew?

We could call for strict laws to restrict the collection or sharing of data. You'll still suspect that somebody is collecting information on you. But you'll rest easier because the fear of prosecution will keep them from sharing the data with most of the people you are afraid to have know it.

Still, the reasoning in this article suggests that open government advocates should welcome the shift of initiative away from the legislative branch to the executive one. But only if that's a transitional stage to lodging decision-making more in public hands.

In fact, the other two branches of government and the public had better find ways to implement collective participation, because it may be the only alternative to a resurgent Government 1.0.

To make this shift a positive change, we'll need well-established government/public collaborations that run through the whole cycle I described in my first article. We'll need to make sure that everybody is online and has the training to participate in decisions at the level of their competence and interest. We'll also need to refine polls and discussions to give us confidence that the public's most important concerns and desires rise to the top of the forums.

And when all that's in place, we can start to experiment with adaptive legislation.

The legislative branch: how to write laws for an engaged public

I mentioned at the beginning of this article that legislatures could develop laws in a more transparent manner. But that's only a start. If they could rely on public participation during the implementation of the law, they could write laws that embrace such input.

Laws already include feedback mechanisms. Many call on an executive agency to collect information on the effects of the law, run hearings, and release a report after a fixed amount of time so that the legislature can evaluate whether the law is achieving their goals. This practice could be dramatically extended by involving the public in the implementation of the law at the start, though continuous forums. The feedback loop would be reduced from years to weeks.

Laws also include ways to delegate control. For instance. Community Block Grants are offered to municipalities to spend as they see fit. (My town manager spent several hundred thousand dollars of our block grant to improve a park next to Town Hall, which in my opinion showed dubious judgment during an affordable housing crisis.) The idea of delegation could also be extended to more and more facets of law. What if a virtual town hall debated the expenditure of the town's Community Block Grant?

Critics of government solutions to social problems--usually political conservatives--accuse the law of being too rigid. The legislative process has trouble evolving with the times and responding flexibly to new conditions. Well, with provisions for public comment and group decision-making, laws can be as flexible as we want.

Congress needs evidence, though, that public feedback reflects the diverse needs and values of the population. Public participation must be protected against the complementary evils of capture by special interests and tyranny of the majority, which I have termed the problem of stakeholders.

If the public can live with a law it debates and tweaks as well as it can live with a law designed by Congress, adaptive legislation is viable.

And we need this flexibility, because the really big problems we have to tackle are what computer scientists call "massively distributed." Problems of this type include climate change, health care cost control, a food crisis that leads to rampant obesity in some populations and rampant starvation in others, job creation in an era of reduced staffing needs, and more.

The presence of the term "Collaboration" in the Administration's open government initiative reflects their understanding that they cannot solve the problems by themselves. Nor can technology, the market, or educational efforts--they must all work together. The concept of Megacommunity perhaps reflects the size of the effort (I actually find the "mega" part of the term slightly redundant) but may not even be enough to capture the extent of cultural adaptation required. In any case, adaptive legislation could trigger related efforts and bolster their effectiveness.

Appendix: the top question asked on Change.gov

Although Obama's approach to data sharing is a welcome sea change from the previous administration, the most committed members of his constituency press him to show more transparency about things that particularly matter to them, such as the role the Administration is playing in the financial system and what it knows about torture.

When Change.gov opened a public forum for questions at the beginning of Obama's presidency, the first place was taken by a question about prosecuting US officials suspected of promoting torture. Progressives then cried foul when the Administration failed to answer.

But did Obama really fail to answer? On April 16 he released Bush Administration memos that showed irrefutably that highly placed officials had discarded legal safeguards to institute interrogation practices that were described by these memos in gory detail.

Yes, Obama has resisted investigations of torture before and after this moment. But I am convinced that by releasing the memos he launched a historical process that cannot be reversed. The memos were his answer to the question that the public forced on him in January.

He has made the kind of political calculation that is his hallmark, deciding not to confront Republicans directly with a torture investigation. But if decent citizens keep up the pressure, prosecution will ultimately reach any US officials responsible for human rights violations, just as it did Pinochet and Fujimori. Open government applications do not free activists from the responsibility to engage with every accessible locus of power.

What democracy advocates must remember is that open government is not just a discussion forum. It's a maelstrom of intersecting investigations and competing proposals just as complex as the current political process. In fact, open government can succeed only by integrating with a political process that has a twenty-five hundred year history, even though our goal over time is to transform that process.

Now that the Administration wants to dance, we must learn all the steps. Listen closely: the musicians have already struck up their first round.

Twenty-five hundred years of Government 2.0

by Andy Oram | @praxagora | comments: 7

This article is the second in a series leading up to the Personal Democracy Forum. The first article was posted on June 16.

---

There's been a lot of excitement lately about the term "Government 2.0." Strip away the RESTful interfaces and you see that the new practices in government transparency are just intensifications of things democracies have done for a long time: public comment periods, expert consultation, archiving deliberations, and so forth. So let's look back a bit at what democracy has brought to government so far.

Like any telescoped presentation of history, this one reduces the swirling forces that extend and retract their way through the centuries into a couple near-mythological categories. I do this in the service of evaluating the concepts we toss around when discussing government participation.

Government 1.0: empire

Last year, Boston residents and visitors got the chance to see an exhibit of sculptures preserved from the culture that earned a special role in history as the first major power to exert ruthless control over many peoples: the Assyrians. Other dynasties--Egyptian, Chinese, Babylonian, and Akkadian--were around before the Assyrian empire, but the Assyrians were the ones that set a new standard for cruelty. The fearful image assigned to them in biblical texts also assures them a special fascination for Westerners.

Most visitors to the Museum of Fine Arts were thrilled by the artistic quality of the wall reliefs, human figures, and everyday objects. Personally, I was depressed by the unrelenting scenes of war and cruelty.

Assyria refined a strategy of subjecting cities just outside their borders and using the resulting booty to raise soldiers and provisions to attack the next frontier. Any populations whose subjugation was in doubt would be uprooted and forced to move closer to the center of the empire, replaced in their old homelands by more compliant subjects.

When the court entertained local dignitaries or foreigners (the lobbyists of the day), they walked through "lobbies" adorned with the scenes of carnage that ended up last year at the center of the Boston exhibit. The depictions of chariots crushing helpless civilians and soldiers impaled on stakes gave visitors a clear message: submit or end up the same way. Thus the Assyrians promulgated a "shock and awe" doctrine four thousand years before US troops brought their own version to the same geography.

This went on, with interruptions, for 1,300 years, and established a practice that guided other empires for thousands of years to come.

Some empires were more humane, of course. Empires could provide their inhabitants with protection and stability through currencies, constables, and courts (remember Hammurabi's Code). But all these policies remained subject to the whim of the supreme ruler.

And that is the distinguishing trait of Government 1.0: unchecked power centered in one individual. The reason emperors could stay in power was that they exploited their hierarchies to delegate both power and wealth. As long as governors maintained loyalty to the emperor, they could exert broad powers in the regions under their control and use those powers to accumulate great amounts of money. They in turn delegated power to those beneath them, and so on down through the hierarchy.

What could be more successful than this carrot-and-stick methodology combining vast rewards with threats of terror?

Government 2.0: democracy

There must be something persnickety about the character of ancient Athens. They couldn't tolerate strong leaders. Almost anyone who ever pulled off a major military victory, proved to be a persuasive orator, or got a corner on political power eventually found himself executed or exiled. (The Athenians invented the idea of "ostracism"--a fiercely democratic institution in their implementation, ironically.) Socrates was just one of the later examples of the propensity Greeks showed for bringing down anyone who was widely admired.

So this seems to be a natural setting for a system that grants a voice to a wide range of citizens. The decisions they reach may not be the best, but they're decisions that the political body can follow through on, having been reached democratically. The losers (if they weren't powerful enough to scare the winners) can stick around and try again at the next gathering in the agora.

Greeks recognized from the beginning the problems of democracy with which we are so familiar today. They knew that many votes were bought outright, and that others could be pulled in by smooth-tongued sophists. They also knew their democracy rested precariously on the labor of the slaves and other disenfranchised residents. And that a democracy could become an oppressive empire, using behavior against people next door that it would never tolerate within the walls of its own city.

I like this disturbing contradiction. That's why my web site, identi.ca account, and Twitter account are named after Praxagora, a character in an ancient Greek play that shows both the flaws and the immense power of democratic systems. The name Praxagora combines "action" with "public forum."

Right or wrong, a democratically reached decision--which if properly done, comes into focus as an emergent property of the assembled masses rather than being imposed by one party or individual--has an irreproachable authority. Socrates didn't like democracy, but if we are to believe Plato (who also didn't like democracy), Socrates insisted on obeying the popular will, even at the cost of his life.

We shouldn't hang a halo around direct democracy. In fact, the trend in technology-driven government transparency is not Athenian direct democracy--despite its idealization by some activists--but a tighter agency/public partnership. Today's experiments in public participation go far beyond electing representatives. But even the traditional American political culture consists of more than bills and vote counts. For instance, the executive branch tends to consult regularly with the public, a topic I'll take up in the next article in this series.

As we don digital media and communications--those somewhat ungainly garments we try to mold to human forms--in order to improve on twenty-five hundred years of flawed Government 2.0, we can learn some lessons from those millennia:

• No individual can be allowed to gather too much power, but every individual needs to be heard and to be protected from arbitrary persecution.
• Those who are excluded from the benefits of society will eventually rise up to wreck it.
• The majority is often wrong, and any political system can be abused.
• Good decisions take time, and a willingness to subject the decisions to constant re-examination.
• We need to rise above rhetoric and pursue the ultimate (if ultimately elusive) truth.

Like any useful technology, digital media and communications can help us realize a vision. Government 2.0 is a very old vision. A recognition of what has been achieved and what still challenges us can guide the development of the proper technology.

For instance, we can learn from history to bring the technology of participation to every member of the population and give them the opportunity to learn it, to subject the results of electronic deliberation to review by authorities governed by outside checks and balances, to highlight experts' reputations so they can wield more influence, and to give participants on electronic forums a few cycles of decision-making to work out processes that make effective use of the technology.

Deploying Government 2.0 technology will teach us more about that technology, and about ourselves.

Next article (Wednesday, June 24): adaptive legislation can respond to action in the agora.

Personal Democracy Forum ramp-up: from vulnerability and overload to rage, mistrust, and fear

by Andy Oram | @praxagora | comments: 3

The Personal Democracy Forum will hold its sixth annual conference at the end of this month. The theme, "how technology and the Internet are changing politics, democracy, and society," has been central to O'Reilly's work over the past few years (and a theme on which we're holding a summit of our own in September). Over the next two weeks I'll write three blogs on the Radar site to get some of my current thoughts off of my chest, clearing some space so that when I get to the PDF conference itself, my blogs can focus on its events and statements made by its participants.

This blog covers:

• The government participation cycle: if you want to dance, sir
• Vulnerability: a reason to put brakes on outgoing information
• Overload: a reason to put brakes on incoming information
• Rage, mistrust, and fear: inhibitors of the government participation cycle

The government participation cycle: if you want to dance, sir

The grand vision for government/public collaboration is a set of feedback loops that intensify the influence of the collective will on government policy. A feedback loop might consist of a cycle like this:

1. An agency (or less likely, a legislature) posts data in a downloadable format through a flexible API and announces a call for applications.
2. Companies and public interest groups define goals and put programmers on the task.
3. The public uses the resulting applications to generate data and share it with the agency.
4. The agency sets policy or changes direction in response to the data.

At any step, a failure by any of the responsible actors to follow through will leave the process hanging and discourage future projects in open participation.

This doesn't mean every project needs to include all four steps. The public may benefit from government data without offering feedback, and programmers could put their work under an open source license or into the public domain for the benefit of the government or members of the public without asking them to share more data. Agencies can also use programs to improve internal coordination instead of working with the public. But the full four steps serve as a canonical model for government/public collaboration.

Successful examples already exist for each step. As I write this, Data.gov has 261 data sets and 30 tools; thousands more data sets are promised soon. Appeals for donations of code, such as Vivek Kundra's Apps for Democracy in Washington, DC and the Sunlight Foundation's Apps for America, show that coders will play their part, at least in the current atmosphere of enthusiasm for the new initiative. And the public has responded to requests for data.

But at the federal level, we need to dance a few rounds of the full cycle before feeling confident that open processes are fully entrenched. I'll return to this theme in the last section of this article. The cycles of public participation will teach lessons, of course, that feed into a still larger cycle of constant experimentation and improvement.

As public participation moves forward, it's worth remembering that resistance to the free flow of incoming and outgoing information is not irrational. The resistance spring from healthy coping mechanisms learned by individuals and organizations learned over their lifetimes. I have already published and solicited comments on a list of fundamental questions on government participation; in this article I describe two such issues that play a special role in resistance to information sharing.

Vulnerability: a reason to put brakes on outgoing information

A couple months ago, I read a stirring report from a federal agency manager trying to sound out the Administration concerning how much the agency ought to reveal. The manager was stunned and inspired by the response of Bev Godwin, a prominent director at the White House and General Services Administration, who advised talking about the bad things as well as the good and soliciting negative as well as positive feedback.

Vulnerability is the keystone of transparency and openness. Online forums, if they are run democratically and competently, encourage vulnerability through a combination of self-correcting mechanisms:

The right to respond

Anyone criticized in a forum has repeated chances to defend himself at length. If the forum includes a rating system, persuasive arguments and well-chosen facts will float above false accusations as well as flaccid excuses from the accused.

Support networks

Proponents of each side pile on to each debate, turning it into a community issue and diluting the personal biases brought by the people who began the debate. A bit of a mob scene can erupt at times, awakening the risk that the losing side will walk away in a huff while sensitive community members flee the fury. But as long as participants value the community over partisan agendas and prefer honesty to grandstanding, the community comes out stronger, more aware of its options, and ready to integrate what it has learned into further action.

Community memory

Forum members recognize when old debates are re-ignited, and can fill new members in on the history. They can also predict the way prominent participants will line up on an issue. Debates are thus tighter and more quickly resolved.

A propensity for truth

These traits all end up privileging accuracy and making it harder (although not impossible) for bad judgment to prevail through false claims, manipulative demagogy, appeals to group solidarity, and the other tricks used by insincere factionalists.

This list may present online forums in a bit too rosy a light. But they do permit social norms that protect vulnerable people, even if the norms don't function perfectly. The real problem comes when words leave these forums and end up in other environments not subject to the same rules.

Government staff have already witnessed too many negative experiences in traditional, non-virtual settings. They have seen what happens when a comment is taken out of context and bandied about in the broadcast or print media, introduced into court testimony, or used as ammunition in partisan debates. They know that comment posted on the Web can be fodder for the same opinion machine--and are in fact even more dangerous because the Web makes them more visible.

That's not fair. It's very hard for anyone outside an agency to judge why it came down on one side of a debate or what that decision's long-term effects will be. Most agency actions are a complex fermentation blending the data that was gathered, assessments of the data's accuracy, assessments of the possible trends indicated by the data, consultations with the public (yes, outsiders are routinely consulted), judgments about Congress's intent, judgments about the interests of the Administration, and more. But groups with a cause like to ascribe one-dimensional reasons for key agency decisions and mine public statements for corroborating evidence.

This doesn't mean that all agencies are honest and act in the public interest. Plenty of bad government decisions have been made under pressure from well-organized special interests or to pay off political donors. One role of civil society is to expose these influences--that's what open government and the Personal Democracy Forum are all about. So we want more of these online forums. But we also need to protect the agencies whom we expect to use the forums. To encourage the necessary vulnerability, we have to combat those who abuse the results.

Journalism is starting to incorporate its own feedback loops and open its pages. Elections and policy debates are also monitored by the blogosphere. So some forums are becoming friendlier to the cause of vulnerability (the court system is unlikely ever to change). But it will be a long time before it's safe to lay out one's thoughts in an open, self-policing community.

Overload: a reason to put brakes on incoming information

The previous section mentioned the possibility of a "mob scene," and if people putting out information must be able to tolerate being vulnerable, those requesting input from the public have to deal with a potentially low signal-to-noise ratio.

We need not look far for an example. Take last month's brainstorming session on open government, launched by the White House and the Office of Science and Technology. It drew over 1,000 submissions in a single week. (Even more are on the site now, but they arrived after the official close of the session.)

The thousand submissions offered quite a smorgasbord for a group led by the new Deputy Chief Technology Officer for Open Government, Beth Noveck, to spoon through. They ended up with many intriguing ideas. But the gathering of ideas was simply a suggestion box, not real crowdsourcing. The web site offered no tools for editing, combining, and culling entries (and there would be inadequate time to use such tools anyway). The only aspect reminiscent of group behavior was a casual and anonymous rating system, which played little role in the results.

And that's a relief. After all, how many Americans would be able to assess the Office of Open Government created by Florida Governor Charlie Crist, or the potential for Cooperative Research and Development Agreements to help convert government data and applications to open source? Both of these projects earned a place in the results, even though the Florida model got only 24 votes and the Cooperative Research and Development Agreements only 46. (Although they might have conceivably been mentioned in an earlier brainstorming session conducted among government workers, I couldn't find them in the publicly posted comments.)

In response to a question about the voting, Noveck wrote me, "We wanted to encourage the National Academy of Public Administration to try different voting techniques. They started out by allowing voting by unregistered users, and later restricted it to registered users. Given the change, we didn't want to disadvantage anyone who participated. Consequently, we viewed the voting as informative but not determinative. On our weblog, only registered users can vote on comments."

As her statement indicates, the second phase of this transparency project has already sprouted more of the checks and balances found in mature discussion forums. We can expect the Administration to wend its way toward systems that gather useful opinions from self-organized groups of qualified commentators, the model pioneered by Noveck in her Peer to Patent project.

But will the White House have the time and resources to establish a foothold for a solid and lasting open government program? That depends on public tolerance for the Administration as a whole.

Rage, mistrust, and fear: inhibitors of the government participation cycle

Everyone knows that productive collaboration can't take place under conditions of rage, mistrust, or fear. Americans unfortunately are suffering from all these feelings right now.

Their rage has been directed at the heads of the financial industry. No peasant at the time of the French Revolution felt more hatred for Marie Antoinette than some of the comments I've seen about AIG. In addition, the current conditions of recession and financial uncertainty breed mistrust toward all three branches of government, and fear toward anyone who could seem to wangle an extra advantage over other Americans.

I'm not going to factor in the recent murders of law enforcement officers, Dr. George Tiller, and others because I'm sure the hate crimes were caused by lots of diverse factors, and it's unclear whether they represent a widespread cultural movement. We have plenty to worry about just by considering problems that will undeniably have a broad impact on Americans.

Over the coming year, lots of homes will continue to be foreclosed (because Congress failed to put a system in place to stop them), a blight that hits many neighborhoods like a dry Katrina. This ongoing crisis will be joined by credit card crisis (because Congress's bill didn't do much to stop that either) and perhaps already a student debt crisis. The Administration has its own challenges, waging two untraditional wars that nobody knows how to win and tinkering with a global financial system that always cracks its casings.

Open government doesn't deserve to be at the mercy of current political controversies. It did not originate with the Obama administration, and it doesn't require a Democratic Party philosophy. The George W. Bush administration took some steps toward open government (often forgotten amongst all the complaints over their unsavory maneuvers and information withholding). The Bill Clinton administration took steps too. But Obama is making it a centerpiece.

This gives us more hope than ever for openness, but ties its fortunes to the larger sphere of activities by the Administration and federal government.

To establish a foothold, openness needs some early, impressive success stories. Federal CTO Vivek Kundra has said his initiatives will prove themselves by saving money, although that certainly isn't his sole aspiration. If the Administration can land a few universally recognized successes--budgetary or otherwise--and especially if it can run through the whole cycle I laid out at the beginning of this article, such efforts will be continued by future Administrations.

Next article (Friday, June 19): twenty-five hundred years of Government 2.0.

Four short links: 25 May 2009

by Nat Torkington | @gnat | comments: 6

1. China is Logging On -- blogging 5x more popular in China than in USA, email 1/3 again as popular in USA as China. These figures are per-capita of Internet users, and make eye-opening reading. (via Glyn Moody)
2. The Economics of Google (Wired) -- the money graf is Google even uses auctions for internal operations, like allocating servers among its various business units. Since moving a product's storage and computation to a new data center is disruptive, engineers often put it off. "I suggested we run an auction similar to what the airlines do when they oversell a flight. They keep offering bigger vouchers until enough customers give up their seats," Varian says. "In our case, we offer more machines in exchange for moving to new servers. One group might do it for 50 new ones, another for 100, and another won't move unless we give them 300. So we give them to the lowest bidder—they get their extra capacity, and we get computation shifted to the new data center."
3. Why Washington Doesn't Get New Media -- Things eventually improved, but despite the stunning advances in communications technology, most of federal Washington has still failed to grasp the meaning of Government 2.0. Indeed, much is mired in Government 1.5. Government 1.5? That’s a term of art for the vast virtual ecosystem taking root in Washington that has set up the trappings of 2.0 — the blogs, the Facebook pages, the Twitter accounts — but lacks any intellectual heartbeat. Too many aides in official Washington are setting up blogs and social media pages because they understand that is what they are supposed to do. All the while, many are sweating the possibility that they might actually have to say something substantive or engage the public directly. It is the nature of midlevel know-nothings to grinfuck any idea that would force them to substantially change their behaviour. We incentivize this when we talk about "you must have a blog" (ok, I'll get comms to write it), or "put up a wiki for this" (ok, but there'll be no moderation so it'll be ignorable chaos). Describe the behaviour you want and not a tool that might produce it. (via timoreilly on Twitter)
4. On the Information Armageddon (Mind Hacks) -- Vaughn points out that the much-linked-to New York Magazine article on attention is a crock. I didn't like it because it was wordy and self-indulgent, Vaughn because it didn't actually cite any studies other than one which was described incorrectly. History has taught us that we worry about widespread new technology and this is usually expressed in society in terms of its negative impact on our minds and social relationships. If you're really concerned about cognitive abilities, look after your cardiovascular health (eat well and exercise), cherish your relationships, stay mentally active and experience diverse and interesting things. All of which have been shown to maintain mental function, especially as we age.

Four short links: 30 Apr 2009

Youth, Government, Tween Arduino Hackers, and Table Slurpage

by Nat Torkington | @gnat | comments: 0

1. Ypulse Conference -- conference on marketing to youth with technology, from the very savvy Anastasia Goodstein who runs the interesting Ypulse blog on youth culture that I've raved about before. Register with the code RADAR for a 10% discount (thanks, Anastasia!).
2. Government in the Global Village -- departing post by the NZ CIO (and Kiwi Foo Camper) Laurence Millar. The principles here are applicable to almost every nation. We need to recognise the network effects of opening up government data in a form that means others can access it. Economic value is created by businesses building innovative new services using government data. Public value is created by enabling a richer and deeper understanding and dialogue among interested individuals about what the data tells us about our lives.[...] The legal, policy, and moral position is clear - New Zealanders own the data, having paid for its collection through taxes. These “problems” will all be solved by the community, and our role as government is to give priority to this. These efforts are stuff that matters. See also Google adds search to public data.
3. Children's Arduino Workshop (Makezine) -- video of three eleven-year old girls working on an Arduino project, and should be inspiration to anyone who has ever wanted to work on hardware projects with kids. Whoever did it succeeded in making it fun! (via followr on Twitter)
4. With YQL Execute, The Internet Becomes Your Database -- YQL is a query language for Yahoo! data sources, and now they've added a server-side Javascript way to import your own web page's tables into YQL. YQL and Pipes are turning into very interesting pieces of infrastructure (e.g., Museum Pipes blog). (via Simon Willison and straup on delicious)

Four short links: 16 Apr 2009

by Nat Torkington | @gnat | comments: 1

China, databases, storage, and git:

1. China's Complicated Internet Culture (Ethan Zuckerman) -- summary of Rebecca McKinnon's talk at the Berkman Internet Center. Democracy is complex and hard to transition to, online democracy doubly so. Rebecca questions the widespread but unjustified belief that the Great Firewall of China is all that separates Chinese citizens from the empowered liberty of the West, and lays out the tangled state of affairs in China's political Internet. Despite the rise of web video, “no one has managed to organized an opposition party on the web,” Rebecca points out. “There’s no Lech Walenza, no religious movement - Falun Gong has been squished pretty thoroughly.” (via cshirky's delicious stream)
2. Drop ACID and Think About Data -- Bob Ippolito's talk from PyCon about the things you can do easily when you foresake the promises of ACID. More in the ongoing reinvention of databases for the needs of modern web systems. (via cesther's Twitter stream)
3. The Pogoplug -- The Pogoplug connects your external hard drive to the Internet so you can easily share and access your files from anywhere. We're accumulating terabytes of storage at home, where it's very useful to all the computers in the home. This offers an easy way for non-technical civilians to make these drives useful outside the home as well. There are many possibilities for Interesting Things in the massive storage we're accumulating. (via joshua's delicious stream)
4. Gitorious -- open source (AGPLv3) clone of github. (via edd's delicious stream)

Four short links: 20 Feb 2009

by Nat Torkington | @gnat | comments: 6

Accessibility, trails, Pacman, and power today. Have a fun weekend!

1. Social Accessibility Project -- clever IBM approach to solving web accessibility problems: a sidebar for Firefox that lets people with assistive devices like screenreaders say "hey, I had this problem with this page", and a crowd will help fix it. (via Derek Featherstone's Webstock talk, notes here)
2. Why I Want a Million Quid (mySociety) -- Tom's onto something. I am hooked by this vision of "systems where each person who is helped to solve a problem leaves a trail of advice, contacts, insider information and new user-friendly web services behind them". We're used to the data people leave behind being discrete and implicit (another purchase for the recommendation engine) rather than longitudinal and explicit (people who looked at this item eventually went on to find their answer here).
3. The PacMan Dossier -- everything there is to know about Pacman, from designer Toru Iwatani's inspiration and design process, through to the logic errors behind bugs and why it's better to move the joystick before you reach the turn. (via Grand Text Auto)
4. Two Stanford Students Rethink the Light Switch -- a power switch with a network connection and tactile feedback: teh awesome.

New Zealand Goes Black

by Nat Torkington | @gnat | comments: 6

The previous government in New Zealand enacted an amendment to the Copyright Act that required ISPs to have a policy to disconnect users after repeated accusations of infringement, over the objections of technologists. While it's possible to have a policy that requires proof rather than accusation, APRA (the RIAA of New Zealand) strongly opposes any such attempts at reasonable interpretation of Section 92. The minor parties in the coalition government oppose the "three accusations and you're offline" section and want it repealed. This is the last week before that law is due to come into effect and the Creative Freedom Foundation, a group formed to represent artists and citizens who oppose the section, has a week of protest planned to convince the ruling National Party to repeal S92.

The first day's action was blacking out Twitter and Facebook avatars. I did it, as did Channel 3 Business News, a Creative Director at Saatchi and Saatchi, oh and Stephen Fry. Kudos to Juha Saarinen who first put out the call. This is building up to a full Internet blackout day on February 23rd. I'm delighted to say that the idea was formed at Kiwi Foo Camp, and the folks who were at Kiwi Foo have been running wild with it--building banners, releasing templates, spreading the word.