Four short links: 13 November 2009

Open Source Design, Interesting NoSQL Use, Copyright Documentary, Location Intelligence

by Nat Torkington | @gnat | comments: 1

1. Open Source Enters The World of Atoms -- an academic statistical analysis of open design. We indicated that, in open design communities, tangible objects can be developed in very similar fashion to software; one could even say that people treat a design as source code to a physical object and change the object via changing the source.
2. Why I Like Redis (Simon Willison) -- coherent explanation of why Simon likes and uses a particular nosql system. I can run a long running batch job in one Python interpreter (say loading a few million lines of CSV in to a Redis key/value lookup table) and run another interpreter to play with the data that’s already been collected, even as the first process is streaming data in. I can quit and restart my interpreters without losing any data. And because Redis semantics map closely to Python native data types, I don’t have to think for more than a few seconds about how I’m going to represent my data.
3. © kiwiright (Vimeo) -- short documentary about copyright, made to raise awareness of the issues in New Zealand. (just as applicable to the rest of the world)
4. Your Movements Speak For Themselves (Jeff Jonas) -- Mobile devices in America are generating something like 600 billion geo-spatially tagged transactions per day. Every call, text message, email and data transfer handled by your mobile device creates a transaction with your space-time coordinate (to roughly 60 meters accuracy if there are three cell towers in range), whether you have GPS or not. Got a Blackberry? Every few minutes, it sends a heartbeat, creating a transaction whether you are using the phone or not. If the device is GPS-enabled and you’re using a location-based service your location is accurate to somewhere between 10 and 30 meters. Using Wi-Fi? It is accurate below10 meters. A thought-provoking roundup of the information leakage with modern locative systems. (via TomC on Twitter)

Navigating the Future: Take Me to Bob

by Brady Forrest | @brady | comments: 13

Google has just announced a free turn-by-turn navigation app for Android 2.0 in the US (Radar post). Google Maps Navigation relies on Google's own mapping for routing you. As with many navigation devices you can search Business Listings. However, they are also including data not traditionally available to navigators. In the promo video Google demonstrates that you can ask to be taken to "The King Tut exhibit". GMN will determine that it's in Golden Gate Park and route you. This is "because it is connected to the internet it is using all of the latest information on the internet."

This is huge. To be able to request implicit destinations based off of realtime information is something that has never been available before. What new queries will be available to us because of this? Google has a lot of data. How much of it can be assigned a location? Lots. There are millions of KML files out on the internet. Here are some of the useful queries

"Take me to Bob Smith" - If Bob is your friend on Latitude then Google Maps Navigation can take you to him. If Bob moves then GMN could even re-route you. I wonder if they will enable the chase scenario.

"Drop me off in time for the #48 bus" - Google knows the public transit schedule. So not only can it drop you off at the nearest stop, it could drop you off at the stop that will ensure the shortest multi-modal trip.

"Show me homes under 500K in Capitol Hill" - Via Google Base, Google has real estate information (it has had neighborhood data for quite sometime).

"Take me to my next appointment" - If you use Google Calendar and you accurately fill out the location field then this is a snap.

"Take me to the nearest Winter Coat Sale" - Using Adsense for Google Maps, GMN can easily lead you to local sales.

"Take me to the bar my friends go to the most" - Using Social Graph API and the new, experimental Social Search to tap into Foursquare, GMN can determine where you friends go, aggregate their destinations and lead you to their favorite watering hole.

"Take me to the largest event" - Using a combination of Latitude and its new access to the Twitter Firehose (which will soon include location - Radar post), Google can determine where people are.

"Take me on a tour of the top 10 historical sites here" - Using Wikipedia Google can determine what the sites are and where you should be taken. Alternately, Google could take you on user-generated tour.

"Take me to the most picturesque place near here" - Several years ago Google bought Panoramio, a location-based photo site. Google can determine which place nearby has had the most photos of it taken.

"Take me on a tour of the site from Around the World in 80 Days" - Google already geoparses many of the books it scans (just see this map). This routing is quite possible.

"Take me to the EPA's protected sites" - Government data is becoming more available. This is just one possible governmental query. You could also ask to go on a tour of TARP fund recipients or Democratic donors.

Obviously not all of them will be enabled, but I bet that within a year some of them will be. What other scenarios can or should they implement?

A Conversation with Dr. Walter Scott of DigitalGlobe

by Joshua-Michéle Ross | @jmichele | comments: 0

Dr Walter Scott founded Digital Globe - a company you are likely not familiar with though you probably interact with their satellite imagery on a regular basis via Google Maps, Bing and others.


It is only recently that mapping technology and production has been driven by mainly commercial interests especially in the area of satellite imagery. With this commercialization corporations and media have access to information that was once considered closely guarded state property.

The potential for social good - from assessing and responding to natural disasters, to exposing political issues such as prisoner camps, to finding out where Richard Serra is keeping his massive sculptures… is enormous. In this discussion we cover DigitalGlobe's business, the state of commercial satellite imagery and the advantage of commercial vs. government ownership of GIS data.

Dr Scott will be delivering a HighOrder Bit at the upcoming Web 2.0 Summit.

Four short links: 16 October 2009

Audio Geotagging, SF Open Data Stories, Wave Use Cases, Hadooped Genomes

by Nat Torkington | @gnat | comments: 0

1. Wiimote Audio Geotagging -- match audio with the map movement and annotations made with an IR pen and a Wiimote. Very cool! (and from New Zealand)
2. San Francisco: Open For Data -- Two months after it launched, the project is already reaping rewards from San Francisco's huge community of programmers. Applications using the data include Routesy, which offers directions based on real-time city transport feeds; and EcoFinder, which points you to the nearest recycling site for a given item.
3. Google Wave's Best Use Cases (Lifehacker) -- not cases where people are using Wave, but where they want to. Read this as "the Web has not provided all the tools to solve these problems". Something will solve them, and Wave is trying to. (via Jim Stogdill)
4. Analyzing Human Genomes with Hadoop -- case study from the Cloudera blog. Performs alignment and genotyping on the 100GB of data you get when you sequence a human's genome in about three hours for less than $100 using a 40-node, 320-core cluster rented from Amazon’s EC2. (via mndoci on Twitter)

Four short links: 13 October 2009

Open Source, Gov 2.0, Gaming, Education

by Nat Torkington | @gnat | comments: 0

1. Our Open Source School -- blog of Albany Senior High School in New Zealand, which only runs open source software.
2. Behind The Scenes at What Do They Know -- interesting post showing details behind the What Do They Know web site. In the last year there have been only seven significant cases where requests have been hidden from public view on the site due to concerns relating to potential libel and defamation. Three of those cases have involved groups of twenty or so requests made by the same one or two users. While actual number of requests we have had to hide is around 70 (0.4% of the total) even this small fraction overstates the situation due to the repetition of the same potentially libelous accusations comments in different requests. In all cases we have kept as much information up on the site as possible. Our policy with respect to all requests to remove information from the site is that we only take down information in exceptional circumstances; generally only when the law requires us to do so.
3. The Complete History of Lemmings -- a must-read for videogamers from the early 90s. Theres been much debate over the choice of colours as well, but the colours were selected, not because they were the easiest to choose, but because of the PC EGA palette. With the limited choice, it was decided the green hair was nicer than blue, and with that, the final Lemming was born. I was actually the next person to code up a demo on the Commodore 64, but I only got so far as having a single Lemming walking over the landscape before Dave put me onto another project.
4. Google Replaces TeleAtlas -- Tele Atlas confirms that Google has decided to stop using Tele Atlas map data for the U.S. Google will now use its own map data. Our relationship with Google for map coverage continues outside of the U.S. in dozens of geographies.

Where 2.0 CFP Extended

by Brady Forrest | @brady | comments: 0

Every year Where 2.0 is a gathering of mapping companies, and geohackers. This year there will be a lot of discussion about mobile apps (iPhone vs, Android vs. Pre vs. Nokia), user-contributed geodata (like Waze and Google Building Maker), temporal mapping, government geodata (like Data SF), augmented reality (like Wikitude), and the geolocated web. We've extended the CFP entry deadline until October 20th. If you've got something to contribute then submit a talk for the CFP now.

I've included some of the topics that we are looking for after the jump

(continue reading)

Four short links: 2 September 2009

Happy Programmers, Usability Tool, Geo API, Zombie Math

by Nat Torkington | @gnat | comments: 0

1. The Programming Language With The Happiest Users (Dolores Labs) -- you'll be surprised. Age before beauty!
2. Judge It Now -- fast market opinions on design decisions. Compare to Optimal Sort. Usability tools hitting the mainstream web, so the time to learn what works shrinks and progress is faster.
3. BlockChalk API -- These new interfaces enable developers to do nearly everything that you can do at http://blockchalk.com. It’s now possible to build client applications, mash-ups, and other tools based on BlockChalk geolocation data and services. Also see the explanatory blog post. (via joshua on Delicious)
4. The Mathematics of Zombie Attacks (PDF) -- Zombies are a popular figure in pop culture/entertainment and they are usually portrayed as being brought about through an outbreak or epidemic. Consequently, we model a zombie attack, using biological assumptions based on popular zombie movies. We introduce a basic model for zombie infection, determine equilibria and their stability, and illustrate the outcome with numerical solutions. We then refine the model to introduce a latent period of zombification, whereby humans are infected, but not infectious, before becoming undead. We then modify the model to include the effects of possible quarantine or a cure. Finally, we examine the impact of regular, impulsive reductions in the number of zombies and derive conditions under which eradication can occur. We show that only quick, aggressive attacks can stave off the doomsday scenario: the collapse of society as zombies overtake us all. (via Doug McKenna)

Burning Man Gets an API (and a Whole Lot More)

by Brady Forrest | @brady | comments: 17

An API! SMS! Foursquare! An iPhone app! They are all coming to Burning Man this year. Will the festival be the same?

The annual tech-art festival in the Nevada desert, starts on Sunday. Normally the attendees leave their phones and laptop behind, but this year that may not be the case. As I ride from Seattle to Black Rock City, NV I am getting SMS from friends on the playa. In anticipation of wifi and possible data connections Foursquare has rolled out Black Rock City as a city (@sfslim is already the Mayor of The Man). If AT&T's service doesn't work then attendees may be able to take advantage of OpenBTS's local SMS project. Most of the attendees aren't there, but the tech is already making its presence known.

Burning Man is dismissed as a party by many people (attendees and non-attendees alike), but for many it is a unique opportunity to try out new software. Geohackers in particular find it to be a great playground. Black Rock City is a full city complete with a fire department, stores (where you can buy coffee, tea or ice bags), a Main drag and 40,000+ residents. However, since it is only around for a week each year (and is always in a new location) there is not time (or profit) for commercial companies to map it. The process falls to the community and they take advantage of the opportunity (and sites like Flickr use the resulting commercial-grade data).

This year the Burning Man organization is assisting with the launch of an API. With the API you get access to descriptions and locations of the Streets, Art, Camps and Events. When combined with a map this is everything you need for a local city guide. And that is exactly what the iPhone app does (it's not available in the app store; if you want it head to the Burning Man Earth Camp next to Media Mecca -- be nice). It maps all of those entities, will geolocate you and let you mark favorites (see the screenshot from my iPhone). You can learn more about the API project here. Burning Man still has its Virtual Playa project online.

There is also a move to take advantage of Flickr's machine tags. For example if you take a picture of Area 47 (with the online directory entry: http://earth.burningman.com/brc/2009/themecamp/2234/) then use burningman:camp=2234. The photo will appear on that locations page. We will see how many photos end up using these machine tags. I suspect that V2 of the iPhone app will add a camera that can apply those tags automatically and that we'll see more uptake then.

Burning Earth team member, Tom Longson, sent me the following.

Burning Man's theme this year is evolution which is fitting as Burning Man Earth launches an online directory, API, and a beta iPhone App. The group of artists, geo-wankers, and software developers are rapidly deploying systems, both off and on the Black Rock Desert playa to help participants find each other, schedule events, find theme camps, and artwork. It is a digital project aimed at providing better maps, and an online space to describe the community and art.

The open source webapp, named "Earth", builds upon Open Street Map, GeoDjango, and Pinax to create an easy to use, mapping interface for the event. Coupled with Jeffrey Johnson's prior work with aerial photography, and Andrew Johnstone's virtual playa 3D modeling, the platform is rapidly evolving to become an important part of the organization of the event.

Burning Man's API now opens the door for developers and artists alike to remix and reuse data about the event. For example, you could plot all the events in the next hour, build an Arduino belt that vibrated in the direction of the closest piece of artwork, or a web service for rating theme camps.

In addition, Mikel Maron is championing machine-tags to allow the project to couple Earth's database with other websites, such as Flickr. By integrating machine tags, people can say on Flickr what art installation their photo is of, and Earth will automatically pull up that photo. Likewise, Flickr will provide a link to the page describing the artwork itself.

Beyonds enabling mashups, the APIs are the foundation for the new beta iPhone app, which serves as both a directory and enhanced GPS designed for Burning Man. A small number of participants will get to try out the app, which will be in full production next year.

While it may sound like fun and games, the harsh conditions of the Black Rock Desert make the system a perfect testbed for mapping temporary places, people, and things. In this same way, these tools may just be the next best thing for helping disaster hit regions react and respond. Burning Man Earth is more than just an attempt at radical self-expression, self-reliance, and community building. It may just be a tool for tomorrow.

This is Burning Man at its best. Letting people create something just for the festival and its attendees. The question becomes how will the larger Burning Man community, expecting a cellphone free vacation, react to intrusions from the real-world?

BTW, If you are on the playa you may be able to find me at my group art project Steve the Robot H.E.A.i.D.

Where 2.0 2010 CFP is Now Open

by Brady Forrest | @brady | comments: 2

The CFP for the sixth annual Where 2.0 is now open. The three-day conference about location, mapping and geodata will be held from 3/30 to 4/1 in San Jose, CA. This year our focus will be on location-enabled platforms, mobile apps, user-generated geodata, sensors, and augmented reality. Here is more about the conference below,

Mobile: The iPhone, Android, and Symbian mobile OS’s are continually advancing the state of the art. By creating a wide-spread platform that allows for third-party development and geolocation they are bringing along the whole industry. The phone is going to become the primary I/O device for geodata in the near future. What new applications are you building for it? How are the social aapps effecting society and our notions of privacy?
Realtime Mapping: Mobile phones are being used to generate maps and other geodata. Sensors across the world are capturing more data every second. Reality mining systems are being used to release this data to users in realtime. Who is making the most of this deluge? How can they handle these new data sets?
Temporal Information: Realtime data requires the element of time to be added. This is uncharted design territory. How should time come to the Web?
Rich Analysis: Web mapping is moving past just allowing the display of data (aka red-dot fever). There are now many tools online that help people analyze data and could, in time, challenge traditional GIS systems. How is the Web different? Will end-users take up richer tools?
Geolocated Web: Every updated browser can now geolocate it’s user. Websites are now going to start using this information. What should they do with the information? What new services can be created?
Mobile Advertising vs. Services: Will people pay for their mobile apps directly or through ads? Which makes for a better product, a better user experience and a more stable revenue stream?
Augmented Reality: The combination of a camera, a GPS and a compass on a mobile phone is going to let us layer information on top of the world. What do you want to see? How will you edit the layers?
3D: Photosynth-like apps are becoming more commonplace. Google’s 3D Warehouse is filled with models. It’s safe to say that 3D is here. But do we need it? What are its limits?
Open Data: Governments are treasure troves of data. Increasingly they are releasing it online for free. How does open data effect the web? How can this data be widely available and yet maintain its creators? How is this critical information being put to use?
Crisis Mapping: The tools of neogeography are being used to spread the word of humanitarian and natural disasters. What are some of the best (and worst) examples?
Open-Source: The backbone of any independant mapping site is open source software. What are the newest tools that can be used to handle the location-enabled web?
Workshops
Where 2.0 will have a full day of workshops where participants can dig deep into a range of issues and leave the conference armed with new tools and skills. Workshops are one hour and fifteen minutes in length and will be held on Tuesday, March 30, 2010. Topics we’d like to explore include, but are not exclusive to:
Geo Support in Web Application Frameworks: As people design their own mapping applications, there has been a need for built-in geo support. We’re looking for workshops that teach about Mapstraction, Modest Maps, Open Layers, GeoDjango, GeoRuby, MapCruncher, and other tools.
GeoStack: As locations apps are brought in-house, companies need their own geostack. What are the best tools?
Mapping APIs: The location space would not have gotten as far as it has today without all of the innovation in the mapping API space. How can you test the limits of these free resources?
GeoTargeting: Knowing users’ locations has never been more important. Identifying it accurately can be difficult and expensive. What are the best methods?
Privacy Implications: As you are collecting user data, keeping track of your users, or collecting geodata, are you aware of the relevant laws? What would you teach others?
GeoBrowsers: Google Earth and NASA WorldWind are both amazing geobrowsers. How can you get the most out of them?
Data Management: Geo applications work with massive amounts of data. What are the tools, tips, and tricks that can be used to manage it?
Protocols and Formats: GeoRSS, GML, KML, EXIF, Microformats, Geo OpenSearch. Which formats are on the way in and which ones are on the way out? These are just some of the technologies and transformations we’ve noticed and represent just the starting point for the program. While we’d like you to tap into the theme as your inspiration in writing your proposal, feel free to wander. What are you working on that will change the world, or at least the world you’re in? What project is bringing you pleasure, or teasing your brain? Surprise and delight us; shake us out of our assumptions. We’re angling for shorter talks with longer breaks so you’ll have more time for one-on-one interactions.

IN adition to plenary talks and workshops we will also have opportunities for startups to launch, Ignitte talks and opportunities to experiment with RFIDs. The CFP closes 10/13/09. Submit a talk now.

Four short links: 21 August 2009

Moody Twitter, Future Geohistory, News Sucks, Whyless in Wonderland

by Nat Torkington | @gnat | comments: 3

1. TwitterMood -- using Twitter as a giant mood sensor for the world (see also temporal correlations, via kellan on delicious).
2. What Will Remain of Us -- The sea that brought trade to Dunwich was not entirely benevolent. The town was losing ground as early as 1086 when the Domesday Book, a survey of all holdings in England, was published; between 1066 and 1086 more than half of Dunwich’s taxable farmland had washed away. Major storms in 1287, 1328, 1347, and 1740 swallowed up more land. By 1844, only 237 people lived in Dunwich. Today, less than half as many reside there in a handful of ruins on dry land. (via blackbeltjones on Delicious)
3. The Three Key Parts of Stories You Don't Usually Get -- In reality, these longstanding facts provide the true foundation of journalism. But in practice, they play second-fiddle to the news, condensed beyond all meaning into a paragraph halfway down in a news story, tucked away in a remote corner of our news sites. Take a look at that WaPo page again. Currently, a link sits on the far right side of the page, a third of the way down, labeled “What you need to know.” Click on that link, and you’re taken here: a linkless, five-paragraph blog post from May. This basically captures our approach to providing the necessary background to follow the news.
4. Eulogy to _why -- a pseudonymous Ruby character, _why the Lucky Stiff, recently vanished from the net: all his sites and accounts were deleted. It's possible this is because someone tried to identify him, it's possible that his accounts were hacked. Either way, this is a touching tribute to him from John Resig. I for one would like to see more appreciation while the people are still around. Today, tell two good people that you enjoy what they do. You know you can.

Seeing the Future of Mapping in Crimespotting

by Brady Forrest | @brady | comments: 9

This week Stamen Design released San Francisco Crimespotting. It's a crime map and notification system that allows for time and crime trend analysis. SF Crimespotting has launched just over two years after the release Oakland Crimespotting (Radar post). Stamen had been waiting for crime data all this time and with the launch of DataSF they are able to use an official API for crime data.

SF Crimespotting is very similar to the initial release for Oakland. As I wrote in 2007:

Each type of crime is assigned a color-coded icon with an abbreviation. You can highlight all of a crime type's markers with a mouseover. You can also change the number of days for which crimes are shown. Each crime has a detail page and that crime can then be viewed in context with others. You can also slice the data by day, type and the intersection of the two. You can also subscribe to get email alerts and RSS feeds for a specific place in Oakland.

The latest releases of the Crimespotting platform reflect several important trends in online mapping:

1) Crowdsourced Maps - When Oakland Crimespotting launched it used Microsoft Live maps (which would now be called Bing). They have switched to Cloudmade maps which are based on Open Street Map data. The maps look amazing and at initial glance they appear to be the same as any other major provider's maps. Google's Mapmaker project (Radar posts) has also been seeing more attention and just this week expanded into Mexico (I wonder how long until they bring Mapmaker to the US). Waze (Radar post) is using user-generated traces to create their realtime maps.

2) Temporal Mapping - Time is being added to online maps and other visualizations. As data comes to use in realtime there are new conventions that need to be developed. Stamen, through this project and their work with Trulia Hindsight (Radar post) and MySociety (Radar post), are at the forefront of designing methods of dealing with varying scales and types of time data. In their post The Pie of Time Stamen details their thinking for how to represent hours, days and years in the project. The old Crimespotting did not allow you to navigate to archival data. With the new UI there are now permalinks to all crime reports The hours control is shown to the right. Only the crimes that occurred during the highlighted times will appear on the map. Stamen has included quick links to show specific times like "Commute" and "Nightlife".

The slider and dropdown used to navigate days, months and years are shown above. Each day of the slider shows the total amount of crime that day. The highlighted area dictates the crime shown on the map.

3) Government Data - The new federal administration has shown a renewed interest in releasing data (most of this will have some geocomponent). The 2010 Census is around the corner and that will add to the data flow. As more data is released you can expect an explosion of government mashups. You can also expect more civic minded companies (especially after this week's exit by Everyblock (Radar post)).

4) Geo-Analysis - GIS used within enterprises, governments and universities are designed to take massive geodata sets and simplify them so that decisions can be made. Crimespotting may look like a slick consumer app and that's because it is. However as you manipulate its many controls you'll realize that you can learn a lot about a city and how a time of day or section of the city impacts the likelihood of your being involved in a crime. You can determine if you're more likely to be mugged in the Castro on Thursdays vs. Tuesdays. The only problem is that you are limited to crime data. I'd love to have ability to add other layers like housing prices or average income. Crimespotting has a read API; I hope Stamen adds Write capabilities.

I'd ask the kind folks at Stamen (very nicely) to make a Crimespotting Seattle, but unfortunately we don't publicly release our crime data. Here's to hoping that we get a mayor in this Fall's election who will open the data coffers. Does your city share its data? If so include a link in the comments.

All of these trends are going to be big topics at this year's Where 2.0 (3/31-4/2 in San Jose). Submit your topic now!

Data Is Journalism: MSNBC.com Acquires Everyblock

by Brady Forrest | @brady | comments: 3

Everyblock, Adrian Holovaty's local data aggregator, has been acquired by MSNBC.com. Many are hailing it as local news acquisition. For 15 major US cities Everyblock aggregates crime data, restaurant reviews, health inspections, local news and more. This is data that is only of interest to people within a certain area. I care much less about crime ten blocks away than I do about crime two blocks away. Everyblock lets me know what is happening within three blocks of my home and filters everything out (on the web and iPhone). So Everyblock is a hyperlocalnews acquisition, but that is only half of the story (maybe less).

The future of news is data and Everyblock is the premier startup in this area. As Adrian phrased it on his site this past May in a post entitled The definitive, two-part answer to "is data journalism?":

It's a hot topic among journalists right now: Is data journalism? Is it journalism to publish a raw database? Here, at last, is the definitive, two-part answer:
1. Who cares?
2. I hope my competitors waste their time arguing about this as long as possible.

MSNBC.com stopped wasting time just in time.

There is a coming deluge of data from the new administration. Sites like Data.gov, USASpending.gov and Recovery.gov are hopefully just the beginning of new data sources. It's already too much for many organizations to make sense of. Without the proper tools many stories will never be covered. People will not get the info they need. Everyblock has proven that by taking free local government data sources and making them readily available to interested citizens you can create value. Now it's time to turn those tools and thinking onto a problem of a national scale. (If you'd like to learn more about the Obama administrations efforts to release data check out Anil Dash's latest piece The Most Interesting New Tech Startup of 2009.)

It's important to note that Everyblock recently open-sourced the code to their site and as Techcrunch pointed out their traffic is not that high. So MSNBC could have easily duplicated Everyblock and just turned their traffic hose at the new property. Instead MSNBC.com realized that they are facing a new problem and they needed a new team to tackle it head on. Enter Adrian and Everyblock.

Of course many people know Adrian as one of the co-creators of DJango. In his acquisition blog post he states that he will have more time to work on Django, that Everyblock will stay Python (and presumably continue to roll their own maps) and that this does not effect ebcode, the open-sourced version of Everyblock (Radar post).

Congrats Adrian it looks like you solved the dilemma (Radar site) of what to do once you've open-sourced your site; you tackle a bigger problem.

Post updated to reflect that it was MSNBC.com, not MSNBC, that bought Everyblock.

Locational Privacy: The EFF Weighs in on Safeguarding Your Location

by Brady Forrest | @brady | comments: 4

Increasingly our devices know where we are and are able to share that information. This is a trend that will enable many new services, but at the same time puts the consumer and the service provider at risk. The consumer is at risk of their "future self" forgetting that they are being tracked and then having their location being recorded unintentionally. The company is put at risk just by having this data stored. If they have user data then it is subject to subpoena or unintentional releases.

The EFF has weighed in on this trend with a timely whitepaper: On Location Privacy and How to Avoid Losing It Forever. The paper includes a number of scenarios with actionable solutions and a number of reason why companies should care. The scenarios are:

Anonymous payment and credentials - Many toll roads use electronic transponders to extract payment from drivers. These systems are not necessarily designed with the driver's privacy in mind. This Boston Globe article from 2007(!) talks of EZ-Pass records being subpoenaed in a case (there are many other articles on Google -- some going back as far as 1997). The EFF suggest letting users using an anonymous and encrypted form of electronic cash(ecash). This will still allow the service provider to track flow and estimate revenue on a realtime basis, while protecting their users. For those working on mobile payments or with sensors this is a scenario (and potential solution) to pay attention to. If you need to make sure that only certain people can gain access then you may need to use anonymous credentials to preserve locational privacy.

Location-based search - Often when a user does a search from their cellphone they are identified along with their location and their query. The user "needs" to be identified so that any personal information can be shared. The EFF correctly depicts this interaction as such:

"This is Frank's Nokia here. I see the following five WiFi networks with the following five signal strengths". The service replies "okay, that means you're at the corner of 5th and Main in Springfield". Then your device replies, "What burger joints are nearby? Are any of Frank's friends hanging out nearby?".

This is something that all of us with smartphones (and who use Loopt, Brightkite, Twitter, or use Find My iPhone to update Latitude) are doing multiple times day. An alternative method would be to have the phone send their location and query anonymously. The service can return that data along with a set of encryped data for that location. If any of it is aimed at the user they will be able to decrypt it. The EFF depicts this interaction as such:

"Hi, this is a mobile device here. Here is a cryptographic proof that I have an account on your service and I'm not a spammer. I see the following five wireless networks." The service replies "okay, that means you're at the corner of 5th and Main in Springfield. Here is a big list of encrypted information about things that are nearby". If any of that encrypted information is a note from one of Frank's friends, saying "hey, I'm here", then his Nokia will be able to read it. If he likes, he can also say "hey, here's an encrypted note to post for other people who are nearby". If any of them are his friends, they'll be able to read it.

The company still gets anonymized location data and the query, while delivering the same features. The problem with this scenario is that the web (and mobile in particular) favor speed. If a mobile service added several seconds to send down an encrypted payload of data that is much larger than needed then that service will lose users (or never gain them). The mobile handsets and networks that most of us are using now are too limited and to handle anything more than the bare minimum.

What's the value of locational privacy to a service?
If a service provider does not ever receive location data, the EFF points out that company potentially giving itself a competitive advantage. If you don't log it then you can't be subpoenaed to provide that user data and you (probably) won't ever inadvertently reveal someone's location incorrectly. The EFF is correct: not having to answer subpoenas can save costs for companies and not having a well-publicized privacy debacle is priceless.

The paper also points out that people are becomingly increasingly cognizant of privacy issues and that you can champion privacy as a selling point. I am not sure that I buy this argument completely. I think that quite often people don't realize their location and identity are being recorded. So though there may be increasing awareness it's not a selling point that will get a company much right now. Based on the adoption of social location services, I think that people are more concerned with how their location is shared with other people on and off the service than whether it is logged at all.

When considering these solutions, we need to take into account what the impact on the user experience will be. If it requires too much extra work or is not very transparent on the user's part then the solutions may end up killing the product before it starts. I fear that the encrypted payload used to anonymize local search would hamper any mobile service that tried it -- given our current set of handsets and networks (at least in the US).

Personally, I am a fan of sharing (and in some cases storing) my location data with a limited set of third-party services. However, the services that exist right now are lacking. They do not necessarily make it clear how long they will keep the data or how it will be shared with others. I often do not have the ability to delete my data from a service. I want to share my location (within bounds) and I pay attention to when I do so, but I do fear that my future self will forget--and I think that service providers have a responsibility to protect their users from themselves.

(Disclosure: I am a member of the EFF and Tim is a former Board Member)

A Crowd-Sourced National Communications Census

by Carl Malamud | @CarlMalamud | comments: 9

My last tour of duty in DC was Chief Technology Officer at the Center for American Progress. One of the fun things I got to do was figure out what everybody else did, including my fellow Senior Fellows, the folks that generated most of the policy work, many of whom are now occupying senior posts in the new administration.

One of the most fascinating was Mark Lloyd. An experienced Emmy-winning television producer, communications lawyer, and community activist, Mark is the author of a well-regarded book about communications and democracy and numerous columns. He's currently at the Leadership Conference for Civil Rights.

The project Mark Lloyd was working on was a National Broadband Map to show our true communications capabilities. And, he wanted to crowd-source the map from community groups, supplementing that with census and other data from several different places to create a big mash-up. This was in 2005, around the same time Adrian Holovaty was thinking about chicagocrime.org.

I think the time is now ripe for this project, and when the new folks at the FCC asked me what I thought they should look at I pitched Mark's idea (they're reaching out to lots and lots of folks, which is a great sign). I asked for posting privileges here at Radar so I could pitch the idea to the Internet as well since I'm taking your name in vain as the folks that would make this happen.

(continue reading)

Developers Create Unofficial Find My iPhone API

by Brady Forrest | @brady | comments: 7

The iPhone is correctly credited with bringing location services to the consumer. It started at launch with Google Maps. It kicked into hyper-drive with the launch of the App Store (there are now over 2800 location-enabled apps - via Skyhook). However, there is still a step to go, the iPhone needs the ability to share your location in the background to a third-party server. This has been done for them by a couple of hackers.

Apple already tracks the location of any Mobile Me user who has enabled the feature Find My iPhone. Tyler Hall has released Sosumi, Mobile Me scraper, to Github. Sosumi can access a user's location and send messages to the iPhone.

So what does this mean? For "normal" users absolutely nothing. Just like Google Latitude's nascent Location API this is really just a proof of concept application for developers. Before a Mobile Me Location API will get widespread use it will need OAuth, the ability to control the accuracy of my location data, and reminders and, most significantly, it will need Apple's official sanctioning.

Since it's launch the Find My iPhone has been used to track movers and to catch a thief. It's time for that data to be opened up properly as a service. Sites and apps like Loopt, Pelago, Fire Eagle or any of the other services that will take advantage of a user's location history to offer analytics, and alerts.

Recovery Mapping: ARRA Spending Across the US

by Brady Forrest | @brady | comments: 4

GIS is the killer app for data.gov

-- @mikehogan paraphrasing Spatial Sustain

To really understand economic and government data you need a map. This is especially important to remember right now with the American Recovery and Reinvestment Act (ARRA) spending. There's a lot of data out there and it's when you see can see the relative concentration of funds within a state that the spending priorities begin to become clear.

A number of states have put their spending plans online in map form. ESRI, the GIS toolmaker, has compiled a list of states and federal agencies that use their technology to put recovery data online. (ESRI is the dominant GIS provider to governments around the world so this a pretty good list). I applaud states and agencies that have taken the time to put their data online, I just wish that they had invested more in the UI (see Colorado's use of nurse heads to represent relative Health spending across the state for an example).

• Colorado - Economic Recovery Mapping
• Connecticut - Recovery Initiative
• Maryland - StateStat
• Montana - Reinvestment Act
• Oregon - Tracking the Dollars
• Utah - Stimulus Reporting Application
• Washington - Map of Recovery Funding
• U.S. Dept. of the Interior - ARRA Investment
• U.S. Dept. of Transportation - ARRA Projects and Funding

There are many states out there that are lacking recovery maps and there are resources for you to make your own. Here are the resources.

Recovery Map - MetaCarta has created a web portal that allows anyone to easily create their own personal map. You can filter the data and add on other layers such as news or photos. It's Stimulus Watch for geo data. Among the best are Chicago's Urban Beachfront, Schools Across the USA, Schools compete with "pouring concrete" for stimulus dollars, Income vs. Stimulus Dollars, and Layoffs Correlate in the News (featuring Slate Moneybox's Map of job losses).

Geocommons Maker - FortiusOne's Mapmaker has a number of ARRA data sets and they provide a very rich mapping interface. You can find data for Washington State and other ARRA data sets and then combine it with thousands of other data sets.

UMapper, Bing Maps, Google Maps, Platial - All of these online mapping sites provide the ability to make and share maps. None of them have specific resources or data sets designed for Recovery Mapping, but they are easy to use and provide a good starting point. For example this map, City & County of San Francisco ARRA Map, was made on uMapper.

Four short links: 6 July 2009

iPhone Maps, Tooth Milling, Scratch Updated, Newspapers for All

by Nat Torkington | @gnat | comments: 2

1. Offline Mapping App for iPhone -- carry Open Street Maps maps with you even when you're not in 3G/wifi range. (via Elisabeth)
2. My dentist used an in-office CAD & CNC mill to produce a new tooth for me today (Nat Friedman) -- hello, future!
3. New version of Scratch released -- Scratch is an excellent way to teach kids how to program (I've had success with lots of 7 and 8 year olds). The new version includes keyboard entry, webcams, and support for Lego WeDo. The user interface has also been changed to work on a Netbook's 800x600 screen. Kudos to the Scratch team! (via scratchteam on Twitter)
4. Newspaper Club - a Work in Progress -- blog for the Newspaper Club project. "We're building a service to help people make their own newspapers. This is the blog where we're alarmingly honest about where it's all going wrong." I can't figure out whether this is a brilliant decentralisation move that will disrupt the newspaper industry, or a paper form of steampunk. (via Simon Willison)

Everyblock's Code is Open-Sourced

by Brady Forrest | @brady | comments: 5

The code for Adrian Holovaty's Everyblock has been released. The open-sourcing of the site's system were apart of the Knight News Challenge Program. Everyblock is a very impressive site that aggregates and geocodes local data -- news, crime, fire, restaraunt inspections and reviews - and then lets users define their interests down to the block-level.

Adrian made the announcement on 6/30. Here's the list of newly open-sourced, GPL'd goodies found on Everyblock's new Code page:

The main package (probably the thing you're looking for) is the publishing system, known as ebpub.
Second, the packages ebdata and ebgeo contain Python modules for processing data and making maps.
Third, the packages ebinternal and everyblock round out the code that powers EveryBlock.com. They're internal tools and are likely not of general use, but we're including them to be complete.
Finally, ebblog and ebwiki are our blog and wiki software, respectively. Because, dammit, the world needs another Django-powered blogging tool.

Django fans, Python geohackers and anyone who wants to build a local data aggregator are going to be thrilled. Adrian was one of the co-creators of Django and was one of the first Google Maps Mashup creators.

Everyblock has only launched in major US cities. There's plenty of room in the market for locals to create their own version. Everyblock spends a lot of time curating the incoming data feeds so I doubt that anyone will be able to roll out new sites too quickly. One thing to note: the trademark Everyblock is not available. However, the Everyblock team would not mind being acknowledged if you use their code. Personally I get a lot of value of Everyblock in my city. I get a daily email with all the crime, news and errata near my house.

Everyblock is now going to move onto the second stage of its existence. About five months Adrian blogged about the dilemna they would be facing when they open-sourced their software. As he said at the time:

But now we've reached an interesting point in our project's growth: our grant ends on June 30, and, under the terms of our grant, we're open-sourcing the EveryBlock publishing system so that anybody will be able to take the code to create similar sites. That's a Good Thing, in that EveryBlock's philosophies and tools will have the opportunity to spread around the world much faster than we could have done on our own, but it puts the six of us EveryBlockers in an odd spot. How do we sustain our project if our code is free to the world?

At the time I suggested that they try to federate with new everyblocks. After yesterday's announcement I mailed Adrian to ask him for a hint about their future plans, but for now he's keeping mum.

Four short links: 24 June 2009

Open Source Kids, Crowdsourcing Lessons, Flickr Secrets, Hadoop Spatial Joins

by Nat Torkington | @gnat | comments: 0

1. The Digital Open -- The Digital Open is an online technology community and competition for youth around the world, age 17 and under. Building a community of young open source hackers.
2. Four Crowdsoucing Lessons from the Guardian's Spectacular Expenses Scandal Experiment -- Your workers are unpaid, so make it fun. How to lure them? By making it feel like a game. "Any time that you’re trying to get people to give you stuff, to do stuff for you, the most important thing is that people know that what they’re doing is having an effect," Willison said. "It’s kind of a fundamental tenet of social software. … If you’re not giving people the ‘I rock’ vibe, you’re not getting people to stick around." (via migurski on delicious)
3. 10+ Deploys/Day: Dev & Ops Cooperation at Flickr -- John Allspaw and Paul Hammond's talk from Velocity. You tell any mainstream company in the world "10 deploys/day" and you'll be met with disbelief.
4. Reproducing Spatial Joins using Hadoop and EC2 -- bit by bit the techniques for emulating important operations from trad databases are being discovered and shared in the new database scene. (via straup on delicious)

Before and After Shots of Google's Iran Maps

by Brady Forrest | @brady | comments: 6

There many places in the world where it is not possible for larger companies to map them. These can be for economic reasons as is the case for Black Rock City (the temporary 40,000 person home for Burning Man). Or for political reasons as is the case for Iran and countries such as China.
As I mentioned the other day Google greatly improved their map coverage of Iran via user contributions through their Mapmaker program. These user contributions were applied just a few weeks ago. Here are before and after screenshots of two Iranian cities. The before shot was taken on September 22, 2008; the after shots were taken on May 18, 2009.

Mashhad (Before and After)

Tabriz (Before and After)