Robots.Txt and the .Gov TLD

by Carl Malamud | @CarlMalamud | comments: 5

I'm on the board of CommonCrawl.Org, a nonprofit corporation that is attempting to provide a web crawl for use by all. An interesting report just got sent to us about the use of robots.txt files within the .Gov Top Level Domain, a standard known as the Robots Exclusion Standard.

In examining about 32,000 subdomains in .gov, it turns at least 1,188 of these have a robots.txt file with a "global disallow," meaning robots are excluded from indexing this content. Even more curious, on 175 of these sites, while there is a global disallow, there is a specific bypass that allows the Googlebot to index the data. You can look at the raw data on Factual.

At Public.Resource.Org, we've always felt that the use of a robots.txt file by the government should only be used for purposes of security and integrity of the site, not because some webmaster arbitrarily decides they don't want to be indexed. Indeed, on several occasions we have deliberately ignored government imposed robots.txt files because we felt this was an arbitrary and illegal attempt to keep the public out.

And, needless to say, it doesn't make any sense at all to let in some webcrawlers and not let in others. If this is a reaction to a security/integrity issue, such as limited capacity, the proper thing to do is include in the robots.txt file a comment that can be used by other bots to explain what is going on. For example, it could be perfectly reasonable for a government group faced with limited capacity to ask a robot to limit crawls to a certain number of queries per second and only whitelist crawlers that agree to that condition.

Government webmasters should use the robots.txt file sparingly, and should do so in a non-discriminatory fashion.

Four short links: 19 November 2009

Chumby One, Gorgeous IE Debugger, Freer Than Free, and Phone-a-Friend for Government IT

by Nat Torkington | @gnat | comments: 0

1. Chumby One (Bunnie Huang) -- new Chumby product released. In addition to being about half the price of the original chumby, the new device added some features: it has an FM radio, and it has support for a rechargeable lithium ion battery (although it’s not included with the device, you have to buy one and install it yourself). There’s also a knob so you can easily/quickly adjust the volume. But I don’t think those are really the significant new features. What really gets me excited about this one is that it’s much more hackable.
2. Deep Tracing of Internet Explorer (John Resig) -- very sexy deep inspection of Internet Explorer. Finally, something IE does better than Firefox (other than exploits). dynaTrace Ajax works by sticking low-level instrumentation into Internet Explorer when it launches, capturing any activity that occurs - and I mean virtually any activity that you can imagine. (via Simon Willison)
3. Less Than Free -- begins by talking about Google giving away turn-by-turn directions on Android, and then analyses Google's "less than free" business model: Additionally, because Google has created an open source version of Android, carriers believe they have an “out” if they part ways with Google in the future. I then asked my friend, “so why would they ever use the Google (non open source) license version.” Here was the big punch line - because Google will give you ad splits on search if you use that version! That’s right; Google will pay you to use their mobile OS. I like to call this the “less than free” business model. This is a remarkable card to play. Because of its dominance in search, Google has ad rates that blow away the competition. To compete at an equally “less than free” price point, Symbian or windows mobile would need to subsidize. Double ouch!!
4. Expert Labs -- a new independent initiative to help policy makers in our government take advantage of the expertise of their fellow citizens. How does it work? Simple: 1. We ask policy makers what questions they need answered to make better decisions. 2. We help the technology community create the tools that will get those answers. 3. We prompt the scientific & research communities to provide the answers that will make our country run better. New non-profit from Anil Dash.

What Does Innovative Social Engagement Look Like For Businesses and Governments?

by Mark Drapeau | @cheeky_geeky | comments: 29

I've been thinking about the topic of Government 2.0 a lot lately. Part of this topic deals with the multi-directional engagement between government and citizens. This is what the White House and others have termed a more transparent, collaborative, and participatory government.

Unfortunately, the engagement for the most part is not very authentic nor meaningful. Boring "fan pages" on Facebook are one example I've written about, but there are many others. Often, engagement, when it does happen has so many rules associated with it, or such a high barrier to entry, or such a limited window as to be practically meaningless.

It seems to me that everyone can celebrate the fact that government entities merely have a YouTube channel here, a Twitter account there, or a Blogger profile some other place (the so-called "TGIF revolution"), or we can think a little harder about what the goals of citizen engagement really might be, and how to go about achieving them. But first, a personal example of responsiveness and engagement from the private sector.

On the evening of Nov 2nd, I tweeted from my phone about a local DC restaurant, Co Co Sala, just as I was leaving. We had a nice experience, but the hostess had been a little, shall we say, disinterested in helping us? So I commented as much.

Less than a week later, the co-owner of Co Co Sala sent me an email and cc'd his general manager. He apologized for the treatment I experienced, assured me it was not policy, introduced me to the manager, and said he'd talk to his staff. It was a four-paragraph email. I've never met him before, and furthermore, my personal email is discoverable but not the most easy thing to find.

This is what real social innovation looks like. This is what customer service looks like. This is what true engagement with stakeholders looks like. I want to give this great lounge Co Co Sala a hearty shout-out for not only having a great product, but also really caring about their customers.

Now, imagine we weren't talking about a restaurant here. Imagine we are talking about the Department of Motor Vehicles, or the Patent and Trademark Office, or your Congressman. If you tweeted, would they see it? Would they care? Would they react in any way? I think the answer in many cases is no. And when was the last time you gave the DMV a shout-out for a job well done?

Let's look at a sliver of data. According to TweetStats.com, the people behind the White House Twitter account reply to individuals less than 2% of the time, and seem to have never @ replied to any single more than once (i.e., they have never come close to a conversation). They re-tweet others' tweets about 6.5% of the time, but they only seem to re-tweet other government accounts and the New York Times. Granted, there are more people tweeting about White House issues than Co Co Sala, but does the above data represent any caring in any way, shape or form?

The terrific techPresident blog recently noted that actor Vin Diesel is the single most followed living person on Facebook - and that he recently passed up President Obama. Perhaps that's because Vin Diesel's Facebook fan page is awesome. He is engaged, his fans are engaged, and the tone is informal and fun. There are also many other high-profile people who have taken the plunge into innovative social engagement; my favorite at the moment is Alyssa Milano.

So when exactly did "serious and formal" become a substitute for "informative and meaningful" in government circles? And why is everyone scared of letting their guard down in public? People and entities that innovate and use new social networking tools to engage with stakeholders will be winners. The ones that don't will be losers in the long run. It's that simple.

If a goal of Government 2.0 is to provide citizens better services, and a strategy towards reaching that goal is to use social media tools to communicate better with citizens on multiple channels, it seems to me that listening and responding better to comments and complaints would be a great tactic.

The reason why people still cite the TSA's blog as a good example of citizen engagement is because few other outstanding examples of federal government social media engagement seem to have emerged in 2009. What does 2010 have in store?

It is somewhat outside the scope of this post, but my guess is that more and more local government responsiveness and engagement is happening. We heard some of those stories at the Gov 2.0 Expo Showcase in September. What are some new ones that the feds should hear about?

Four short links: 11 November 2009

Participation Tools, Open Data Requests, Go Programming Language, Why Open Source is Better

by Nat Torkington | @gnat | comments: 0

1. ParticipateDB -- database of online tools for public participation. Closed alpha now, with 32 tools and 15 projects in the database. (via Sara Winge)
2. DataTO -- like data.gov, but it's where users request data sets. (In this case, from the Toronto municipal government)
3. Go -- new language from Bell Labs and Unix central figures Rob Pike and Ken Thompson, who now work at Google. Bits of C, bits of Google, it compiles to native binaries and runs nearly as fast as C. Built with concurrency and memory management as central figures. Not used in production at Google yet, but grew from a 20% project to something worthy of public release.
4. On Commit Bits (Jacob Kaplan-Moss) -- that day-one-commit-bit is one of the starkest differences between the corporate and the open source development model. [...] Granted, Django’s very conservative when it comes to granting that commit bit, but I’m not aware of a single open source project under the sun that’d give out a commit bit on a contributor’s first day. I’ve seen developers who’ve been hired to work full time on open source work for months without commit access to the project they’re paid to develop! One of several posts that Jacob's made about why open source makes for (on average) better software.

Unlikely Group Working Happily Together To Solve Patent Problem

by Carl Malamud | @CarlMalamud | comments: 4

People following the issue of open sourcing the U.S. Patent Database might have been surprised to read an announcement in the official business opportunities web site of the U.S. Government: Synopsis for Public Data Dissemination Sole Source Contract to Google, Inc.

While the first reaction of many might be "OMG, WTF, how could they," this is actually good news, with an unlikely cast of characters working together including Google, Intellectual Ventures, and the Internet Archive.

In September, the Patent Office announced a rather strange "Request for Information" (RFI). Under this proposed scheme, the Patent Office would receive a substantial (upwards of $10 million!) donation of equipment from a vendor. In return, the vendor would get to be the official distributor of the patent database to the public, and would get to sell "value-added products." Among other things, the vendor would get access to the patents before the public does, allowing them to mine the database, and would be allowed to sell a variety of bulk products.

While the RFI makes a nod to public access, like all these Zero-Dollar deals the government cuts, there would be a lot of limits on what is "public" data as the vendor tries to recoup their investment by selling the so-called "value-added" products. Readers may remember a similar fiasco with the General Accountability Office where the Federal Legislative Histories were given away to Thomson West and now even the U.S. Congress has to pay to access this material.

The patent database is no ordinary database. This is the only database specifically called out in the U.S. Constitution as being the responsibility of the U.S. Executive Branch to run!  A lot of people think this Zero-Dollar deal the Patent Office is contemplating kind of stinks, and I'm really pleased to announce that a broad coalition has come together to make this data more broadly available immediately:

• Intellectual Ventures, the IP group founded by Nathan Myhrvold, is donating several terabytes of the back file to Public.Resource.Org, the Internet Archive, and a variety of other groups to make available to everybody.
• Google asked for permission to crawl the public application system (known as "PAIR"). The announcement by the Patent Office of a "sole source contract to Google" was the government's way of saying we have permission to crawl their system and bypass the CAPTCHAs. This is good news, because the PAIR system contains the "binders," which is all the material that supplements the basic applications and grants.
• The Internet Archive has set aside a boatload of disk drives to serve this data. In addition, Public.Resource.Org will provide the usual rsync and FTP, and we expect a variety of other groups to provide mirrors both for bulk access and end-user systems.

It goes without saying that Google, the Internet Archive, and Intellectual Ventures are 3 groups that don't often work together, and I think this illustrates the compelling public interest in making the patent database more broadly available. We announced this Section 8 Task Force in a letter to Congressman Mike Honda. And, we also sent in a FOIA request to the Patent Office, putting them on notice that we expect any responses to their RFI $0 boondoggle to be made available to the public, as required by law.

In the long-term, Patent Office just needs to fix their system instead of resorting to silly $0 deals. They have 600 staff in Information Technology and spend hundreds of millions of dollars. Surely, they can find a way to serve the public as part of that? Putting a lien on the Patent database in return for $10 million in hardware instead of fixing their 70's-era mainframes just doesn't make sense.

In the meantime, we should have the first 8 terabytes of data up pretty soon. Those interested in learning more about the issue are urged to consult the paper trail on our PTO page which includes letters to and from Congress, and pointers to the Patent Office procurement docs.

Four short links: 6 November 2009

Barcode Scanning, Downloadable Community Book, Gov Hack Day, Android Kludges

by Nat Torkington | @gnat | comments: 1

1. Red Laser -- "impossibly accurate barcode scanning". Uses Google Product Search to identify products that you scan using the camera on the phone. I remember Rael and I talking to Jeff Bezos about this years ago, before camphones had the resolution to decode barcodes. The future is here and it's $1.99 on the App Store ... (via Ed Corkery on Twitter)
2. The Art of Community For Free Download -- Jono Bacon's O'Reilly book on community management now available for free download (still available for purchase!).
3. Gov Hack -- Australian government ran a hack day with their open data, this is their writeup.
4. Android Mythbusters -- slides for talk by Matt Porter at Embedded Linux Conference Europe. A (long) catalogue of the kludges in Android.

Four short links: 4 November 2009

Electronics Hacking FAQs, Speech-To-Text Democracy, Open Source Column Database, Massive Online Analysis

by Nat Torkington | @gnat | comments: 1

1. ChipHacker -- collaborative FAQ site for electronics hacking. Based on the same StackExchange software as RedMonk's FOSS FAQ for open source software.
2. Democracy Live -- BBC launch searchable coverage of parliamentary discussion, using speech-to-text. One aspect we're particularly proud of is that we've managed to deliver good results for speech-to-text in Welsh, which, we're told, is unique. I think of this as the start of a They Work For You for video coverage. I'd love to be able to scale this to local government coverage, which is disappearing as local newspapers turn into delivery mechanisms for real estate advertisements.
3. InfiniDB: Open Source Column Database -- hooks into MySQL, uses MySQL for SQL parsing, security, etc. The commercial enterprise version has multi-server support (parallel scale-out). (via Brian Aker)
4. Massive Online Analysis -- MOA is a framework for data stream mining. Includes tools for evaluation and a collection of machine learning algorithms. Related to the WEKA project, also written in Java, while scaling to more demanding problems. . (via joshua on Delicious)

Thoughts on the Whitehouse.gov switch to Drupal

by Tim O'Reilly | @timoreilly | comments: 43

Yesterday, the new media team at the White House announced via the Associated Press that whitehouse.gov is now running on Drupal, the open source content management system. That Drupal implementation is in turn running on a Red Hat Linux system with Apache, MySQL and the rest of the LAMP stack. Apache Solr is the new White House search engine.

This move is obviously a big win for open source. As John Scott of Open Source for America (a group advocating open source adoption by government, to which I am an advisor) noted in an email to me: "This is great news not only for the use of open source software, but the validation of the open source development model. The White House's adoption of community-based software provides a great example for the rest of the government to follow."

John is right. While open source is already widespread throughout the government, its adoption by the White House will almost certainly give permission for much wider uptake.

Particularly telling are the reasons that the White House made the switch. According to the AP article:

White House officials described the change as similar to rebuilding the foundation of a building without changing the street-level appearance of the facade. It was expected to make the White House site more secure - and the same could be true for other administration sites in the future....

Having the public write code may seem like a security risk, but it's just the opposite, experts inside and outside the government argued. Because programmers collaborate to find errors or opportunities to exploit Web code, the final product is therefore more secure.

More than just security, though, the White House saw the opportunity to increase their flexibility. Drupal has a huge library of user-contributed modules that will provide functionality the White House can use to expand its social media capabilities, with everything from super-scalable live chats to multi-lingual support. In many ways, this is the complement to the Government as Platform mantra I've been chanting in Washington. When you build a vibrant, extensible platform, others add value to the foundation you establish; when you join such a platform, you get the benefit of all those features you didn't have to develop yourself.

Of course, it's easy to imagine that the use of open source software will slash the government's IT budget. After all, this software is freely downloadable. I have a feeling it's quite a bit more complicated than that.

First off, government has a huge number of special requirements (remember the flap over President Obama's blackberry?) Second, don't underestimate the difficulty of doing business in Washington. Procurement is done through a complex ballet understood by few open source companies. Third, a big IT deployment like this requires coordination between many companies, each providing a piece of the puzzle. According to techpresident.com, no fewer than five firms were involved in the switch: prime contractor General Dynamics Information Systems, Drupal specialists Phase 2 and Acquia, hosting provider Terremark, and CDN-supplier Akamai. (Disclosure: O'Reilly AlphaTech Ventures is an investor in Acquia.)

The special nature of the government marketplace is one of the reasons why I launched the Gov 2.0 Expo, which will be held in Washington DC next May. There are huge opportunities for open source, web 2.0, and new media companies in government, but there are also challenges reaching that market. One of my goals for the event is to increase the visibility of cutting edge technology firms not just to government agencies, but also to the prime contractors who are putting together these complex procurements.

The net-net is that I suspect that simply using open source software won't slash government IT budgets, at least not right away. What it will do is increase the amount of value we get for our money and the speed with which new technology can be adopted. Features that would have cost millions of dollars and years of development to add will now be rolled into the scope of current contracts.

It's also important to realize that using open source is very different from contributing to open source. Despite the exaggerated claims in the AP story, that "the programming language is written in public view, available for public use and able for people to edit", the White House has not yet released any of the modifications they made to Drupal or its operating environment back to the open source community. The source code for Drupal (and the rest of the LAMP stack) is indeed available, but the modifications that were made to meet government security, scalability, and hosting requirements have not yet been shared. In my conversations with the new media team at the White House, it is clear that they are exploring this option.

Giving modifications back to the Drupal community is the next breakthrough announcement that I'll be looking for.

Releasing code is more than just being a good open source community citizen, though. Code sharing is a major cost-saving opportunity for government. There are countless government agencies at the federal level, not to mention at the state and local level, that perform similar functions. Yet each of them does its own development, driving up costs. Federal CIO Vivek Kundra has made a great step forward in web services by creating data.gov. I'm eager to see an analogous code.gov portal for government agencies to share their open source software code.

Why Posterous Is a Smart Tool For Informal Government Blogging

by Mark Drapeau | @cheeky_geeky | comments: 11

For a few weeks, I've been testing a tool called Posterous, and I've come to like it a lot. You can see my account here. If you're not familiar with Posterous, it is essentially a very simple blogging platform. It may in fact be the most simple one; yet it is very feature-laden. And it has one relatively unique feature that could make it the most powerful tool for informal blogging by government employees.

That simple, amazing, singular feature is email as a primary interface. In other words, you can post blogs simply by emailing post@posterous.com or a similar address - you don't even need an "account" or a "login" or a "password." Even in the private sector, this is considered a cool feature. But for government employees, it could be a breath of life in an otherwise locked-down state of cybersecurity affairs.

You see, many government computer systems block domains like YouTube.com, Facebook.com, Twitter.com, and so forth. There's a current debate about the degree to which government employees can access such sites because of cybersecurity and other reasonable concerns - after all, there have been some very recent instances of bad things being passed through these social media tools and onto your computer. But when you can interact with a blogging platform through email - and in principle even through your official government email account accessed through a traditional program like Microsoft Outlook - you can get the functionality without the risk, and without needing permission from the IT shop.

As information is more decentralized and as more computing is done on mobile devices, quickly communicating information will be more commonplace - and more in demand by consumers of it. So to citizens, government content will still be king, but the speed at which it travels to them may be queen. And being able to blog on-the-go can increase that speed. Recently I've experimented with blogging while walking eight blocks to a date, blogging incredibly fast in reaction to breaking news, and blogging during a conference and posting my "journalism-style" article precisely at the end of a talk. There are innumerable other tactical applications of this tool.

Posterous has a lot of great features that I like. Perhaps most important among them is that links to the content you post can be instantly pushed to other social services like Twitter and Facebook - even if they're blocked in your office. Another great feature is that if you attach photos, videos, or documents to your email, Posterous automatically embeds them in your blog - and will also push them to services like Flickr, YouTube, and Scribd (which may also be blocked in your government office). Still another great feature is that multiple people from multiple email addresses can contribute to one Posterous page (say, for an office), and conversely one email can be associated with multiple Posterous pages (say, a formal public affairs page, and an informal tech thoughts page). In brief, you can be very powerful from your BlackBerry.

Posterous has been described by a Mashable writer as "unremarkable," but frankly, that's what a lot of government employees are interested in. The government has a lot of outstanding content, and their primary mission in many cases is to get it out; customizing the blog theme is definitely secondary. A standardized, simple blog platform controlled through email sounds like just what the doctor ordered, and it offers numerous advantages over something more complicated like WordPress; for example, it's easier to teach people how to use! Oh, and did I mention it's free?

Posterous would probably love it if people in the government wanted to jump on this bandwagon in a more official manner, too. If I understand the numbers correctly, Posterous currently only has about one million unique visitors a month - total. The U.S. Government has more employees than that. I'm not picking on Posterous - it's only been available since June 2008 and has some tough competition in the blog platform world - but my guess is that they'd be very willing to work with the General Services Administration and other appropriate people (as have companies like YouTube) to make Posterous work with official government interests and missions. And the same goes for local and state government employees too, who often deal with IT situations similar to those of their Fed counterparts.

Many agencies are working on social media policies and guidelines for employees, and education and training are no doubt part of successful use of tools like blogs by government employees. But assuming that people are trained and empowered to create online content, can you imagine if even 5% of Postal Service or FEMA or Army employees had a Posterous blog, and citizens and journalists could mine that information about what was happening in the country, or the world? It would be amazing.

So, for the 99% of government employees that can blog in their private lives and informally talk about their careers and more generally about their lives, I recommend getting a personal Posterous account. And because many of the things I said about the government also apply to large corporations, I think there's a huge opportunity there, too. Everyone's workplace has different rules about what you can and cannot use your computer and mobile devices for, and you shouldn't break them. But if you can interface with Posterous via email and help to achieve workplace goals by mobile live-blogging of conferences you attend, or posting photos of critical emergency situations, or provoking discussion over the issue-of-the-day, I say: Go for it.

(If you work in government or closely with it and use Posterous, I'd especially like to listen to your feedback as I help prepare content for the upcoming Gov 2.0 Expo in May 2010.)

Social Networking is the Means to Achieve Workplace Collaboration

by Mark Drapeau | @cheeky_geeky | comments: 9

Yesterday I live-blogged a bit from the terrific Government 2.0 event produced by FedScoop.com at the Newseum in Washington, DC. I wrote a post about how collaboration was not the means, but rather an end made possible by the means of social networking tools. You can read my original writing and some initial comments here. Below, I expand a bit on these ideas.

My post was initially inspired by one speaker's (WFED's Chris Dorobek) notion, shared by some others (Justin Houk commented that, "Taxpayers don't want to think about those in government sitting around on Twitter all day even thought that might be an effective way to collaborate."), that social networking tools come across as too social or "fun" and that being social is not what people are truly doing (in the government) when they use them - they're collaborating. Thus, when marketing Government 2.0 to wider audiences, he feels that a term like "collaboration tools" is more appropriate.

In my opinion, while this might sound better to a more traditionalist, untrained ear, I think it is factually wrong to say that things like Facebook or Intellipedia are collaboration tools. True, collaboration often happens with these tools. And perhaps one could argue that collaboration is mainly what people hope to accomplish with them in the workplace. Fair enough. But I think that collaboration is the end result of leveraging social networks, which is in actuality what the social networking tools are for.

In other words, social networks are a means by which to accomplish something. This something might very well be collaboration. It might also be putting together an office softball team, or a study group of employees all learning Arabic. Is arranging players on a softball team "collaboration"? I don't think so. Is it an important part of a coherent, productive workplace? Perhaps. There are many important things that happen in workplaces based around social networks that are not strictly collaboration on work projects.

One big thing I've been thinking about lately is "leveraging social networks to accomplish important stuff" and no one can deny that personal relationships can influence collaboration. How well you know someone, how much you identify with them, how much you trust them, their level of reliability or transparency - all of these are values derived from social networking that then, when leveraged, can influence collaboration. Collaboration is not an end in itself, of course - it is a means to accomplish some end (finishing a draft report, etc.). So, social networking is a means to collaboration, which is a means to achieving some work or personal goal.

I also reject the notion that there is something wrong with having some fun at work. The idea that having fun with social software shouldn't be allowed in serious workplaces is ridiculous. And of course, anyone who's ever passed around a joke-of-the-week email, celebrated a colleague's birthday with a cake in the break room, or ended work at 4pm for an informal happy hour with the office (i.e., effectively every government and corporate employee) would surely agree with me on this. Work can be fun and be productive, too. The director of the Office of Personnel Management recently visited Google for a reason.

So, briefly, I think social networking tools are not necessarily collaboration tools. They are social software that allows social networks to be leveraged to accomplish things you find important. That might be collaboration on a National Intelligence Estimate (protecting America, earning your paycheck), or arranging a carpool with people in your agency (getting to work on time, being more green), or finding a racquetball partner (staying healthy, living well, bonding) - all of which postitively influence the workplace, in government and in the private sector as well.

As Fred Wellman commented on my original post, "I can't help but wonder if Chris [Dorobek] is seeking a more politically correct or business sounding name of the same tools with the goal of breaking down barriers to implementation and usage as opposed to a lack of understanding of the power of social networking applications in the business of government." I think there's a lot of truth to that. But I also think that, as an academic, this is actually not what we are doing.

This may sound a bit esoteric, but from an academic standpoint I think pointing out that using social networks - online and off - is at the very core of what we are doing is an important thing to point out. When we are "collaborating," we are leveraging social networks to accomplish important stuff.

Four short links: 16 October 2009

Audio Geotagging, SF Open Data Stories, Wave Use Cases, Hadooped Genomes

by Nat Torkington | @gnat | comments: 0

1. Wiimote Audio Geotagging -- match audio with the map movement and annotations made with an IR pen and a Wiimote. Very cool! (and from New Zealand)
2. San Francisco: Open For Data -- Two months after it launched, the project is already reaping rewards from San Francisco's huge community of programmers. Applications using the data include Routesy, which offers directions based on real-time city transport feeds; and EcoFinder, which points you to the nearest recycling site for a given item.
3. Google Wave's Best Use Cases (Lifehacker) -- not cases where people are using Wave, but where they want to. Read this as "the Web has not provided all the tools to solve these problems". Something will solve them, and Wave is trying to. (via Jim Stogdill)
4. Analyzing Human Genomes with Hadoop -- case study from the Cloudera blog. Performs alignment and genotyping on the 100GB of data you get when you sequence a human's genome in about three hours for less than $100 using a 40-node, 320-core cluster rented from Amazon’s EC2. (via mndoci on Twitter)

Law.Gov: America's Operating System, Open Source

by Carl Malamud | @CarlMalamud | comments: 13

Public.Resource.Org is very pleased to announce that we're going to be working with a distinguished group of colleagues from across the country to create a solid business plan, technical specs, and enabling legislation for the federal government to create Law.Gov. We envision Law.Gov as a distributed, open source, authenticated registry and repository of all primary legal materials in the United States. More details on the effort are available on our Law.Gov page.

The process we're going through to create the case for Law.Gov is a series of workshops hosted by our co-conveners. At the end of the process, we're submitting a report to policy makers in Washington. The process will be an open one, so that in addition to the main report which I'll be authoring, anybody who wishes to submit their own materials may do so. There is no one answer as to how the raw materials of our democracy should be provided on the Internet, but we're hopeful we're going to be able to bring together a group from both the legal and the open source worlds to help crack this nut.

The idea for Law.Gov seems to be getting a good reception in Washington, D.C. Senator Lieberman, writing on behalf of the Senate Committee on Homeland Security and Governmental Affairs, the committee responsible for the E-Government Act, has already accepted our request to submit our report to the Committee. Additional formal requests to submit the completed report are outstanding.

Law.Gov is a big challenge for the legal world, and some of the best thinkers in that world have joined us as co-conveners. But, this is also a challenge for the open source world. We'd like to submit such a convincing set of technical specs that there is no doubt in anybody's mind that it is possible to do this. There are some technical challenges and missing pieces as well, such as the pressing need for an open source redaction toolkit to sit on top of OCR packages such as Tesseract. There are challenges for librarians as well, such as compiling a full listing of all materials that should be in the repository.

Law.Gov is an outgrowth of 3 years of work we've done at Public.Resource.Org along with our numerous colleagues in the open law movement across the country. There have been a series of piecemeal successes which have demonstrated that there is a demand and a need for more legal information to be more broadly available. I'm hopeful now that a truly national movement may have coalesced and that there is at least a chance we can bring this across the finish line and create a new function inside of government, the publication of America's operating system on an open source platform.

The factor that made this coalesce was the recent Government 2.0 Summit put on by Tim O'Reilly. I gave a talk at that summit about the need to put primary legal materials on-line, and it was gratifying to hear the Deputy CTO of the United States, in his closing keynote, highlight that as one of the issues which he thought the White House should help make real through their "moral authority and convening power." The Government 2.0 Summit was also an example of convening power, and I was very pleased that it was more than yet another conference about open government, it was a forum that brought together people interested in creating real change. Tim O'Reilly, as the Convener-in-Chief, should be congratulated, and I'm hoping that future Summits lead to even more concrete results.

My Conversation with Austan Goolsbee at Web 2.0 Summit

by Tim O'Reilly | @timoreilly | comments: 8

He introduces himself as "another tall, skinny guy with big ears and a funny name." Economics adviser to Barack Obama during the campaign, and now a member of the President's Council of Economic Advisers and chief economist of the Economic Recovery Advisory Board, Austan Goolsbee is a key figure in framing the economic thinking of the Obama administration. Perhaps most importantly for those of us in Silicon Valley, he's an economist clued in to the tech world. His economics papers cover such topics as the impact of taxes on technology diffusion, the impact of internet subsidies on public schools, and the economic impact of leisure time spent on the internet. He's worked closely with Richard Thaler and Cass Sunstein of Nudge fame, and thinks a lot about the power of default options to shape behavior, a topic that any web developer should also know by heart.

I'll be interviewing Austan Goolsbee on stage at the Web 2.0 Summit. In preparation for our conversation next week, I spent an hour with him yesterday morning. He's a fascinating guy. To give you a taste of the kinds of things we'll be talking about, here's a short transcript of his response to my question about the economic impact of the internet:

Somehow, in my economist heart always lies the revealed preference thing, which is: People are investing tons of their time, tons of their money, tons of their energy into the internet; they wouldn't be doing this for no reason. Regardless of whether we have the data, our presumption ought to be that it's a big productivity improver. But I also think that the evidence on big general-purpose technologies like that is usually that when they're first invented, the impact takes a while to show up, but when it does, boy is it a big time thing, outside just the industry itself, across the board.

If you go back ten years, which isn't that long, the social landscape and the technological landscape are almost unrecognizable. And just that impact, at this early stage, is sufficiently big that you've got to think that twenty years from now, the internet is going to have humongous productivity implications.

Take the health sector. People say "not only does the health sector need to enter the 21st century, it needs to enter the 20th century!" The technology is sufficiently backwards in terms of the information processing - everything's on paper! If you start envisioning healthcare, energy, the government itself -- major league shares of the GDP -- and what the potential is of marrying that to the newest technologies...! Economic potential, historically speaking, tends to be a bit like water. Water will always get to the lowest point. If there's big potential somewhere, it may take a bit of time, but we always find a way to unlock it.

We'll be talking about what Goolsbee would recommend doing differently if we had a "do-over" on the economic stimulus, the importance of innovation to any future economic recovery, education and income inequality, financial services oversight, and President Obama's desire for "iPod government" (which Goolsbee describes as "making [government] simple and easy to use, so that people like it, rather than giving people the third degree and a lot of red tape.")

If you had a chance to sit one-on-one with one of President Obama's economic advisers, what would you ask him? Help me prep for the interview by making suggestions in the comments. It will be tough to do as good a job as Jon Stewart, but hopefully we can come up with some questions that get Austan going!

Four short links: 14 October 2009

Multitouch Demo, Secrets Site Secrets, Hadoop Futures, Becoming Lucky

by Nat Torkington | @gnat | comments: 0

1. 10Gui Video -- demo of a new take on multitouch, a tablet and new GUI conventions. (via titine on Twitter)
2. Behind the Scenes at WhatDoTheyKnow -- numbers and stories from the MySociety project, which provides a public place for Official Information Act requests and responses. The fact information is subject to copyright and restrictions on re-use does not exempt it from disclosure under the Freedom of Information Act (though there is a closely related exemption relating to “commercial interest”). Occasionally public bodies will offer to reply to a request, but in order to deter wider dissemination of the material they will refuse to reply via WhatDoTheyKnow.com. Southampton University have released information in protected PDF documents and the House of Commons has refused to release information via WhatDoTheyKnow.com which it has said it would be prepared to send to an individual directly.
3. The View from HadoopWorld (RedMonk) -- fascinating glimpse into the Hadoop user and developer world. Hadoop can be used with a variety of languages, from Perl to Python to Ruby, but as Doug Cutting admitted today, they’re all second class citizens relative to Java. The plan, however, is for that to change. Which can’t happen soon enough, in my view. It’s not that there’s anything intrinsically wrong with Java, or its audience. The point, rather, is that there are lots and lots of dynamic language developers out there that would be far more productive working in their native tongue versus translating into Java.
4. Be Lucky, It's an Easy Skill to Learn (Telegraph) -- this one resonated with me, as it ties into some life hacking I've been doing lately. And so it is with luck - unlucky people miss chance opportunities because they are too focused on looking for something else. They go to parties intent on finding their perfect partner and so miss opportunities to make good friends. They look through newspapers determined to find certain types of job advertisements and as a result miss other types of jobs. Lucky people are more relaxed and open, and therefore see what is there rather than just what they are looking for. (via Hacker News)

Government Ambassadors For Citizen Engagement

by Mark Drapeau | @cheeky_geeky | comments: 6

To the average person, government is represented by an anonymous person on the other end of the phone, a pile of mandatory paperwork, and perhaps at best a friendly neighborhood postal carrier. If you ask the average American not living inside the Beltway to name a single individual who works in the federal government, how would they reply? My guess is that the broad majority of them couldn’t give you the first and last name of a federal government employee; In reality they would find it much easier to name their local pharmacist, garage owner, or supermarket manager. And from the perspective of the government, this is a shame. How might emerging social technologies help to bridge that gap, in combination with a modification in thinking about government public relations?

The ideal end state when a citizen is asked to name a government employee would be that a person working in a micro-niche of interest to them - finance, farming, foot-and-mouth disease - immediately comes to mind. Unfortunately though, interesting and talented people working at Treasury, USDA, NIH and other places are not well-known to the public, despite the great effects their work has on everyday life in America. Why is this? Partly, it is a vestige from the days when communications were controlled by professionally trained public relations staff and mainstream journalism teams. This was understandable - equipment was expensive, channels were few, and citizens trusted authenticated, official sources for their information. But this media structure that worked well for 40 years is now outdated.

In the Web 2.0 world, every individual is empowered to be not only a consumer of information, but a producer of it. Writing is searchable, discoverable, sharable, usable, and yes, even alterable. The proverbial “pajama mafia” of bloggers has morphed into a powerful society class of listeners, questioners, writers, editors, publishers, and distributors. And in some outlying examples from the federal government, such as the TSA’s blog, we see this same power being harnessed by individual employees (with their agency’s approval, naturally) - Individuals from the TSA not only blog, but interact with citizens who comment on the articles. But this form of government-citizen interaction is, honestly, a primitive version of how social technologies can empower citizen engagement with government.

The modern citizen is not a vessel waiting to receive press releases and government website updates. Even a sophisticated government website like the White House’s new blog can only expect to attract a subset of citizens a subset of the time. Why? Simply, there are simply too many avenues of information flowing towards these people formerly known as a captive audience. No matter how compelling your government information, they are not waiting to hear from you about it. Nor are they necessarily waiting to hear from the New York Times, MSNBC, or any other mainstream organization.

(continue reading)

Four short links: 13 October 2009

Open Source, Gov 2.0, Gaming, Education

by Nat Torkington | @gnat | comments: 0

1. Our Open Source School -- blog of Albany Senior High School in New Zealand, which only runs open source software.
2. Behind The Scenes at What Do They Know -- interesting post showing details behind the What Do They Know web site. In the last year there have been only seven significant cases where requests have been hidden from public view on the site due to concerns relating to potential libel and defamation. Three of those cases have involved groups of twenty or so requests made by the same one or two users. While actual number of requests we have had to hide is around 70 (0.4% of the total) even this small fraction overstates the situation due to the repetition of the same potentially libelous accusations comments in different requests. In all cases we have kept as much information up on the site as possible. Our policy with respect to all requests to remove information from the site is that we only take down information in exceptional circumstances; generally only when the law requires us to do so.
3. The Complete History of Lemmings -- a must-read for videogamers from the early 90s. Theres been much debate over the choice of colours as well, but the colours were selected, not because they were the easiest to choose, but because of the PC EGA palette. With the limited choice, it was decided the green hair was nicer than blue, and with that, the final Lemming was born. I was actually the next person to code up a demo on the Commodore 64, but I only got so far as having a single Lemming walking over the landscape before Dave put me onto another project.
4. Google Replaces TeleAtlas -- Tele Atlas confirms that Google has decided to stop using Tele Atlas map data for the U.S. Google will now use its own map data. Our relationship with Google for map coverage continues outside of the U.S. in dozens of geographies.

Larry Lessig and Naked Transparency

by Carl Malamud | @CarlMalamud | comments: 5

Larry Lessig had a dream. In this dream, he was standing on K Street, preaching in the dark. Suddenly, a naked posse on Segways went whizzing by, shining their flashlights in people's faces. Bystanders were all blinded by these random lights and lost their night vision. When Larry turned around, the naked posse was racing towards the White House for an open government rally, trailed by a screaming mob of marijuana-smoking birthers.

Larry Lessig wrote up his dream in a cover article for the New Republic entitled “Against Transparency: The perils of openness in government.” I suspect that this article will cause some angst inside the Beltway, where you're either with us or against us. But, before the posse turns into a lynch mob, it is important to give the article a careful read.

(continue reading)

How Long is Your City's Tail?

by John Geraci | @johngeraci | comments: 10

Here's something that everyone doing business on the web knows today, thanks to Chris Anderson: it's all about the long tail.

When the cost of each individual transaction falls to nearly zero, marginal and low-performing items, grouped together, can account for a lot more of the overall value of a company than the top-performing ones. Amazon.com makes more money from the aggregate of all of the books that sell one or two copies a month than from sales of best sellers. And Amazon is a much stronger, healthier, and richer company because of the extremely long tail of books it sells.

Everybody gets that.

What almost nobody realizes yet is that the same is true for cities - or can be.

Most cities right now are models of closed, rigid systems, systems that rely on a few, top-performing agents to get civic tasks done and keep quality of life high for residents. Most of these agents are departments of the city itself, though some are outsourced. Either way, cities rely on one agent per issue, no more. To use Amazon.com as an analogy, cities today are like an Amazon that only allows the #1 best-selling book from each category into its system.

A good number of cites are beginning to do deals with mega companies like Google and IBM, giving them access to city data so that they can build excellent tools for residents to use. This is a great thing. I love looking at Google Maps and seeing that bulging red line right next to my house indicating that traffic there is at a standstill so I should consider biking instead. Makes my life better.

Still, to use the Amazon analogy again, now these cities have allowed not just the #1 best-selling book from each category into the system, but best-seller #2, #3 and #4. It's still a closed system of cherry-picked agents with privileged access.

And that's about where a lot of people would choose to end the opening of cities' data - giving unrestricted access to the Googles, Microsofts, and IBMs of the world. The rest get limited access, or access contingent upon satisfying some governmental board or other. (New York's Mayor Bloomberg, who launched his Big Apps contest yesterday, is seemingly among this group.)

If we do that, of course, we're missing out on what is potentially the biggest piece of the pie - the tail. That's where a huge chunk of the value comes from.

So, imagine instead a city that has totally open, unrestricted access to data (say, San Francisco or DC in 2011). What does it look like? It has all of the familiar city-run departments providing all of the services and assistance they've always provided - that's not going away. Then it also has public services offered by the mega companies, the Google Traffic, IBM's Smarter Cities, and so forth. Those are huge added value to these open cities - they're used by a large percentage of residents and make life in those cities better. But THEN, it also has an insane long tail of services set up and run by anyone with an interest in doing so, just by hooking into city data, distributing it in a new way, improving on it, mashing it up, giving it back to the city, etc. These services each individually get used by a small minority of people, but collectively they get used by more than any other single source in the city.

That's the healthy, long tail city of the future in action: head, "meaty middle" and tail, all working together, all reinforcing each other, all driving each other forward.

And that's the future of cities.

So it might be time to ask yourself: how long is your city's tail shaping up to be? The answer may determine, to a large degree, how much your city is a thriving place to live in decades to come.

Questions (and Answers!) About the Federal Register

by Carl Malamud | @CarlMalamud | comments: 3

When the White House retweets Cory Doctorow, you know something unusual has happened. As many of you saw, the Office of the Federal Register announced that source code for the Federal Register is now available in bulk—for free—and has been converted to XML. Ed Felten's shop at Princeton created a site called fedthread.org to see what you can do with the data and Public.Resource.Org helped the Government Printing Office in testing early stages of the XML work.

All-in-all, a nice piece of public-private cooperation and an important step towards open source America's operating system, and I figured that was the end of that. So, imagine my surprise when I got a call from the White House saying they were making Raymond Mosley, Director of the Office of the Federal Register (OFR) and Michael L. Wash, the Chief Information Officer of the Government Printing Office (GPO) available just in case there were any technical questions from the net.

I gathered questions from a variety of sources, including on-line discussion groups and twitter, and have been doing email back and forth with both Ray and Mike. Hope this is useful (it certainly has been fun to do)!

(continue reading)

Four short links: 6 October 2009

Birdwatching Technology, Transportation Data, Multitouch in Python, and Face Detection on the iPhone

by Nat Torkington | @gnat | comments: 0

1. Bird-watching Turns To Technology (BBC) -- CCTV-esque automated bird watching. Sensor networks + computer vision for an ecological purpose. In a bid to track the guillemots behaviour, Dr Dickinson is refining established work that involves modelling the visual structure of an area around a nest. The computer system will be able to use this model to identify changing elements in the scene, and determine if they correspond to movement by a guillemot. "That is the typical way of doing surveillance," said Dr Dickinson, "work out what's moving, that gives you an idea about what is interesting in a scene."
2. The Case for Open MTA Data -- If you live in Portland, there are dozens of mobile applications that help fill gaps in transit information. You can check your phone to see when the next bus is supposed to come. You can plan a trip from one unfamiliar part of town to another. You can even have your mobile device buzz if you fall asleep before reaching your destination. For the basic stuff, there's no iPhone necessary (although that certainly helps for information luxuries). Anyone who has a plain old cell phone with text messaging can ride the train or the bus with greater ease thanks to these apps. (via Making Light)
3. PyMT -- a python module for developing multi-touch enabled media rich applications. Currently the aim is to allow for quick and easy interaction design and rapid prototype development. There is also a focus on logging tasks or sessions of user interaction to quantitative data and the analysis/visualization of such data.
4. Near Realtime Face Detection on the iPhone with OpenCV Port -- we're probably only one or two revisions of iPhone hardware away from being able to do some serious computer vision tasks on the handset. Proof of concept adds a tie to the face you're pointing the camera at.