Worldwide Lexicon: matching up technologies and culture to end the language barrier

I've reported before on the Worldwide Lexicon, the brainchild of my friend Brian McConnell. His most recent breakthrough, which I blogged about in August, was an impressive Firefox plugin that exploits both human and machine translations on the Web to provide pages you can read in your primary language.

As attractive as the Firefox plug-in can be, it's only the first stage in four that Brian plans toward a computing environment that encourages and leverages human translation. On the browser side, the next logical project is to reproduce the Firefox experience for IE users. Ultimately, he hopes the functionality becomes a standard part of every browser. Even better, he's working on a way to include the functionality on the server side so that it's browser-independent (although that technology would require support in the server software, of course).

And there's even more to come. He lays out his vision in an essay boldly titled The End Of The Language Barrier. The bottom of the article points to an equally important statement written for the World Economic Forum by Ethan Zuckerman, founder of the Global Voices site that extends the reach of weblogs to people in many countries who previously lacked access to such forums.

(continue reading)

World Wide Lexicon Toolbar changes the reading experience for the other 99% of web pages

Brian McConnell's latest coding effort, World Wide Lexicon Toolbar, meets my criterion for a piece of critical infrastructure: after two days with it I can't get along without it, and I plan to avoid any browser that doesn't have it installed.

Brian is a highly adaptive programmer. With roots in the telecom industry and several start-ups on his resume, he also wrote Beyond Contact: A Guide to SETI and Communicating with Alien Civilizations for O'Reilly. The World Wide Lexicon project he's been working on for the past several years is again something totally different.

Install the add-on (currently experimental) in Firefox 3.5 or higher and visit a page in some language other than your default. Before your eyes, headings and text change into your native language. You can get similar effects by submitting the page to a popular translator such as Google (which is one of the tools used behind the scenes by the WWL toolbar), but the instantaneous effect of the toolbar makes you feel closer to the people whose sites you visit around the world.

There are several languages that I know well enough to get the gist of a page, but where I miss some of the details and get frustrated by gaps in my vocabulary. Therefore, I set the WWL toolbar to "Bilingual view," so each block element of the original text is shown together with its translation. The bilingual view is considerably less attractive, because it swells the size of each block element, but I can tell already that it will improve my language skills quickly.

WWL is designed for volunteer translations. If it becomes more popular, people will submit translations that are much more accurate than the machine-generated ones the WWL must fall back on currently.

What's the process behind this new dimension to web browsing? McConnell let me in on some of the magic.

Volunteer translations

McConnell invented WWL several years ago with the core notion of encouraging people to translate web pages they thought should get a wider audience. When he first told me about the idea, I was skeptical that he would get many volunteers. But then I heard of other volunteer translation efforts. For instance, there's a whole subculture of people who write subtitles for popular Hollywood films. This runs afoul of copyright law, of course (and so do the copies of movies they're attached to, probably) but they show the lengths to which crowdsourcing has progressed in the translation area.

FLOSS Manuals, a project I do volunteer work for, also finds dozens of people willing to translate its open source documentation.

McConnell's first set of tools were designed to facilitate on-the-fly translations. Web designers could enhance their web sites by downloading from the WWL site some JavaScript that made each text element on the page editable. (I blogged about this in December 2007.) The paste-in displayed a little pencil icon, signaling to viewers that they could do instant translations. All they would have to do was click on an element, and a text box would pop up where they could enter their translation. The web site would then register the translation with the central WWL site.

World Wide Lexicon API

The WWL API covers the entire life cycle of a translation: registering a translation, rating translations for quality, searching for a translation of a particular page into a particular language, and retrieving a translation. Queries can specify a minimum rating.

Toolbar

The latest achievement of the WWL project is the toolbar officially released yesterday. It determines the user's native language through settings in the browser. When each page is visited, the toolbar uses the domain name and various tests on the text to make a guess about its language.

The toolbar then issues an API query to see whether any human translations exist. If so, it displays the translations with a light yellow or green background.

If no one has made a human translation (which is usually the case so far) the toolbar resorts to well-known machine translation services. It can make use of Google Translate, Apertium, and Moses, each of which offers an API, and will also query Babelfish when its API is ready. Machine translations are displayed with a light blue or grey background.

The progressive translation used by the toolbar is also interesting. It starts with the first 10 or 20 elements, then translates heading tags (<H1>, etc.), then the larger texts, and ultimately every element on a page. (I displayed one page that embedded a Google ad, and the translator recognized and translated that text too.) McConnell is working on making the various translations run in parallel. Because translation changes the sizes of elements, the toolbar makes various accommodations to display the page as attractively as it can.

In short, WWL is a cool combination of mash-ups, existing services, crowdsourcing, and Ajax. I'm sure that in a year's time I'll think back to its appearance today and be shocked at how primitive it was. But it will remain a transformative tool for me.

Completing the circle on journalists and public participation

Journalists, politicians, and foundations are all tinkering with forms of amateur input: inviting bloggers to major events, quoting popular online sites in newspapers, etc. But Capital News Connection has really jumped in full-tilt with Ask Your Lawmaker. A creative combination of public input and ratings with professionals who have their boots on the ground in the US Capitol building, Ask Your Lawmaker is a case study in progress concerning how to get experts and the public to work together.

I heard a talk from CNC founder and executive director Melinda Wittstock this evening at the Ethos Roundtable, a forum for non-profits in Eastern Massachusetts. CNC gets consulting input from Ethos Roundtable organizer Deborah Elizabeth Finn, and Wittstock came looking for volunteer help with such matters as developing a Facebook or iPhone application. As Wittstock said, Ask Your Lawmaker is still working on how to complete the circle of public input, feedback, and outreach.

Step one is the simple form (on the web site's "Ask A Question" tab) for submitting a question to any Congressman or Senator of your choice. Step two is the simple voting mechanism, reminiscent of the pre-inauguration Change.gov site.

At this point, the journalists working for CNC--who have years of experience at leading media sites--take over. They don't merely choose the highest-rated questions. Sometimes a question shouldn't have to wait around and gather votes because the topic is hot. The reporters use their judgment in combination with votes to pick timely and provocative questions, and sometimes direct a question to a more appropriate lawmaker (such as the sponsor of a bill or the head of a committee).

The next step invokes the power of professional journalism. CNC sends its reporters into the Capitol and congressional office buildings daily. Although they have regular routines with their typical journalists' questions, they throw in citizen questions where appropriate and tell the lawmaker how many people voted for each question. Wittstock mentioned that it's very hard for a congressperson to dismiss a question that came from a constituent, especially one that got a lot of votes.

Videos are very hard to make in the Capitol, unfortunately, because filming is severely restricted there by law and the lawmakers are understandably leery of allowing themselves to be filmed any place at any time.

The next step goes from real-time back to the web site, along with conventional radio stations. Questions and answers are taped and transcribed so they can be offered as both audio and text. CNC has contracts with a number of PBS stations who work public questions into regular news broadcasts.

Podcasts and texts are posted on the web site and served through an RSS feed, but you can also follow AskYourLawmaker on Twitter or search for hashtag #ayl. (Right now they're discussing the talk I attended.) This can bring the answers back to those who asked the questions.

Ask Your Lawmaker also offers a feed that visitors can add to their own web sites, and an iframe for each individual report, suitable for embedding.

Most powerful at all, citizens' questions can change policies. Lobbyists harangue lawmakers day after day, but sometimes they're more impressed by a simple question revealing a deep-seated need in their communities. They have been heard walking away from journalist interviews saying to their staff, "Brief me about that issue."

All very impressive for an effort that's so provisional, the journalists run the web site themselves. Several weak points remain before the circle is complete.

• Ask Your Lawmaker doesn't get enough publicity. It may or may not be mentioned on the radio station that reports its results. Hardly any listeners, I wager, realize that questions were generated by ordinary citizens, much less realize that anyone can ask a question.
• The site needs a way to accept questions through SMS. Attendees at this evening's talk speculated about the power of accepting questions for US lawmakers from victims of wars or globalization policies around the globe.
• The site doesn't exploit the potential for social networking to let questioners promote the site. Someone whose question is chosen should be informed when the answer is posted or broadcast on the radio, and should be encouraged to invite her friends and fellow workers to view the answer.

CNC is looking for ways to complete the circle--and will gladly accept volunteer help, as I mentioned--but they're doing a lot in the meantime to firm up their appeal and raise funds. They plan to allow cobranding and to let sites select the length and subject matter of the material they post, just as they now serve up very customized reports to the radio stations they serve. They may start accepting advertising, and they're looking for fun contests that will publicize their work.

Ask Your Lawmaker demonstrates a unique solution to a situation whered for amateur input can augment expert practice and expertise can augment what the public has to offer. In this regard, Ask Your Lawmaker is worth comparing to the landmark Peer-to-Patent project and to two commercial ventures I analyzed a few months ago, uTest and TopCoder. The opportunity for a virtuous cycle of public input, professional processing, and listener loyalty--especially in a field whose death has been predicted by many--puts Ask Your Lawmaker into an intriguing category of its own.