Four short links: 20 November 2009

Social Network Search for Morons, Bulking Up Bio Data, Better E-Mail, Better Standards

by Nat Torkington | @gnat | comments: 1

1. Spokeo -- abysmal indictment of society, first prize in mankind's race to the bottom. Uncover personal photos, videos, and secrets ... GUARANTEED! Spokeo deep searches within 48 major social networks to find truly mouth-watering news about friends and coworkers. PS, anybody who gives their gmail username and password to a site that specializes in dishing dirt can only be described as a fucking idiot. (via Jim Stogdill, who was equally disappointed in our species)
2. Biologists rally to sequence 'neglected' microbes (Nature) -- The Genomic Encyclopedia of Bacteria and Archaea is project to sequence genomes from more branches of the evolutionary tree of life. Eisen's team selected and sequenced more than 100 'neglected' species that lacked close relatives among the 1,000 genomes already in GenBank. The researchers reported earlier this year at the JGI's Fourth Annual User Meeting that even mapping the first 56 of these microbes' genomes increased the rate of discovery of new gene and protein families with new biological properties. It also improved the researchers' ability to predict the role of genes with unknown functions in already sequenced organisms. (via Jonathan Eisen)
3. Mail Learning: The What and the How (Simon Cozens) -- a few things that a really good mail analysis tool needs to do. I hope that my mail client and server does these out of the box in the next five years.
4. Introducing the Open Web Foundation Agreement -- The Open Web Foundation Agreement itself establishes the copyright and patent rights for a specification, ensuring that downstream consumers may freely implement and reuse the licensed specification without seeking further permission. In addition to the agreement itself, we also created an easy-to-read "Deed" that provides a high level overview of the agreement. Applying the open source approach to better standards.

Four short links: 28 October 2009

Great Mail Feature, Speed Talks, Virtualisation History, Science Literacy

by Nat Torkington | @gnat | comments: 2

1. GMail Labs: Got The Wrong Bob? -- When's the last time you got an email from a stranger asking, "Are you sure you meant to send this to me?" and promptly realized that you didn't? Looks at the clusters of CCs you send and, if you normally send to Bob X but are trying to send it to Bob Y, asks you "did you mean Bob X?". This might be the best thing to happen to email since webmail and full-text search--it's ridiculous how little innovation is happening in email given how widely and heavily it is used.
2. Speedgeeks LA at Shopzilla -- eight talks about making websites faster. Latency Improvements for PicasaWeb - Gavin Doughtie (Google) - Great tips from a web guru about what makes PicasaWeb fast. Watch for when the slides to more talks become available.
3. 10 Years of Virtual Machine Performance Semi-Demystified -- fascinating history of virtualisation from someone who worked for VMware. Since 2005, VMware and Xen have gradually reduced the performance overheads of virtualization, aided by the Moore’s law doubling in transistor count, which inexorably shrinks overheads over time. AMD’s Rapid Virtualization Indexing (RVI - 2007) and Intel’s Extended Page Tables (EPT - 2009) substantially improved performance for a class of recalcitrant workloads by offloading the mapping of machine-level pages to Guest OS “physical” memory pages, from software to silicon. In the case of operations that stress the MMU—like an Apache compile with lots of short lived processes and intensive memory access—performance doubled with RVI/EPT. (Xen showed similar challenges prior to RVI/EPT on compilation benchmarks.)
4. Pew Research Science Quiz -- To test your knowledge of scientific concepts and recent scientific findings and events, we invite you to take this 12-question science knowledge quiz. Then see how you did in comparison with the 1,005 randomly sampled adults asked the same questions.

Four short links: 15 September 2009

Delegation, Journalism, Dating Numbers, Learn Git

by Nat Torkington | @gnat | comments: 1

1. Why You Shouldn't Do It All Yourself -- this resonated with where I am in a few projects. One of the hardest things to learn in management is how not to do it all yourself. People often call this a problem with "delegation". But the problem isn't with telling others what to do. The problem is learning how not to do it all yourself. (via br3nda)
2. The Story Behind The Story (The Atlantic) -- I would describe their approach as post-journalistic. It sees democracy, by definition, as perpetual political battle. The blogger’s role is to help his side. Distortions and inaccuracies, lapses of judgment, the absence of context, all of these things matter only a little, because they are committed by both sides, and tend to come out a wash. Nobody is actually right about anything, no matter how certain they pretend to be. The truth is something that emerges from the cauldron of debate. No, not the truth: victory, because winning is way more important than being right. Power is the highest achievement. There is nothing new about this. But we never used to mistake it for journalism. Today it is rapidly replacing journalism, leading us toward a world where all information is spun, and where all “news” is unapologetically propaganda.
3. OkTrends -- analytics from a dating site show what works in email. We analyzed over 500,000 first contacts on our dating site, OkCupid. Our program looked at keywords and phrases, how they affected reply rates, and what trends were statistically significant. The result: a set of rules for what you should and shouldn’t say when introducing yourself online. (read their note on how they protected privacy before freaking out)
4. Learn GitHub -- Here we have tried to compile the best online learning Git resource available. There are a number of articles and screencasts, written and arranged to try to make learning Git as quick and easy as possible.

Four short links: 7 September 2009

XMPP, Future of Web Frameworks, Infrastructure Stories, Better Email Client

by Nat Torkington | @gnat | comments: 0

1. App Engine Now Supports XMPP (Jabber) -- messaging servers, whether XMPP or PubSubHubBub, are becoming an increasingly important way to loosely join the small pieces. Google's incorporation of XMPP into GAE reflects this (and the fact that Wave is built on XMPP). (via StPeter on Twitter)
2. Snakes on the Web (Jacob Kaplan-Moss) -- The best way to predict the future of web development, I think, is to keep asking ourselves the question that led to all the past advances: what sucks, and how can we fix it? So: what sucks about web development? An excellent and thought-provoking talk about the possible directions for improvement in web framework design.
3. Ravelry (Tim Bray) -- We’ve got 430,000 registered users, in a month we’ll see 200,000 of those, about 135,000 in a week and about 70,000 in a day. We peak at 3.6 million pageviews per day. That’s registered users only (doesn’t include the very few pages that are Google accessible) and does not include the usual API calls, RSS feeds, AJAX. [...] We have 7 servers running Gentoo Linux and virtualized into a total of 13 virtual servers with Xen. [...]". Interesting technical and business discussion with an unexpected busy site.
4. So's Your Facet: Faceted Global Search for Mozilla Thunderbird -- email clients are LONG overdue for improvement. Encouraging to see an active and open research project to improve it from the folks at Mozilla Messaging.

Four short links: 23 June 2009

by Nat Torkington | @gnat | comments: 2

1. Easter Eggs for Real Life (Neil Gaiman) -- ok, I know easter eggs are already part of real life, but this is still cool. Gaiman recommends a restaurant run by a friend, and the friend has set up a special phrase that to mention to the server, at which point something good and special will happen for them to eat or drink. Think of it as a restaurant Easter Egg. I love language, I love Gaiman's books, I love surprises, and I love that here Gaiman's using the digital sense of Easter Egg (surprise hidden in a program) rather than the analog sense (because there's no searching involved).
2. ASCAP Wants To Be Paid When Your Phone Rings (EFF) -- what the title suggests. You are lost in a twisty maze of rights, all policed by vampires. From ASCAP's point of view, this is a legitimate claim. From anyone else's point of view, it's ridiculous.
3. Tooling Up The Body (MInd Hacks) -- using tools has lots of interesting effects on our perception is the general gist of an intriguing study that provides further evidence for the theory that the brain treats tools as temporary body parts. We talk about using the Internet as our "offsite brain", so it tickles me to learn that the brain treats tools as offsite body parts.
4. Email Patterns Can Predict Impending Doom (New Scientist) -- when Enron was about to collapse, email patterns changed: the number of active email cliques, defined as groups in which every member has had direct email contact with every other member, jumped from 100 to almost 800 around a month before the December 2001 collapse. Messages were also increasingly exchanged within these groups and not shared with other employees. Menezes thinks he and Collingsworth may have identified a characteristic change that occurs as stress builds within a company: employees start talking directly to people they feel comfortable with, and stop sharing information more widely. (via BoingBoing)

Four short links: 11 May 2009

Healthcare, Diagrams, Social Networking, and Email

by Nat Torkington | @gnat | comments: 4

1. OSCAR Canada -- open source healthcare (EMR) software, akin to VistA. See linuxmednews.com for more.
2. Instaviz -- iPhone app for mindmapping/any other blob-and-line diagram. I'm hypnotised by the correction of a fuzzy hand-drawn circle into a clean crisp algorithmic circle.
3. Buddypress -- open source software that turns a Wordpress installation into a social networking platform. Ok, so social networking software is now essentially free. What's the next big thing that will as hard and new as social networking was in 2003?
4. Getting Insight Into One's Own Email -- Thunderbird now shows interesting facts when there's no message to look at: recently read messages, messages most likely to be interesting, and a histogram of activity.

It’s at the Scene of the Crime, but it’s not the Criminal

by Linda Stone | comments: 7

People are saying technology is making us stupid. Technology is shattering our attention. Technology is ruining our children. Technology is making us busier than ever.

Taking that train of thought a step further: technology can fix the problem. I believe we can make smarter email and smarter phones - and we should. It just won’t fix the problem.

We can think of technology like cupcakes. The cupcake is at the scene of the crime, but it’s not the criminal. We can make smarter cupcakes -- sugar free, higher in fiber, but that doesn’t seem to be making any difference. The cupcake isn’t saying, “Eat five of me.” We make the choice. “I’ll have one and take a walk. I won’t have one.” Or, “I’ll have five.”

Why will it be different with technology? Technology is at the scene of the crime. The criminal is that voice inside of each of us that says, “Do it all. Have it all. Don’t stop to consider what you’re doing or why. Run fast and do as much as you can.”

Sharon, a former professor turned consultant, says it always seems easier to respond to emails than to work on the project files sitting right in front of her. Is she making this choice because picking up a project file requires focused attention and emailing requires less of a commitment? Or is there a buzz of completion and immediate gratification each time the send button is pressed in contrast to the delayed gratification from a meatier project?

The technology is at the scene of the crime - a weapon of mass communication turning productivity opportunities into an excuse for procrastination. How do the choices we make in each moment, about what we choose to do and what we choose to ignore, tell the story of what matters to us?

When a day begins and ends with a list of action items, it can lack a sense of purpose. Without a sense of purpose, we have no framework to guide our choices.
As we plan our day, while reviewing what we hope to do, we can ask ourselves: Why is each of these things on the list? What can I do to bring into focus what really matters to me? What can I exclude that would allow me closer alignment with my sense of purpose and my intentions?

Technology, just like cupcakes, is there -- for our pleasure. The crime only happens when we forget our sense of purpose and fail to make choices as to what we include or exclude.

RIP: Returned Every Email

by Linda Stone | comments: 10

I fell in love with email in 1983. I was a computer-savvy educator and children’s librarian teaching teachers about the new technologies 
available to them. Email came into my life, offering immediate gratification: no stamp, no trip to the post office, no phone tag, no long messages. Questions were answered quickly. Personal exchanges often felt as intimate as a written letter or a phone call, but were immediate and more frequent.

Years later, in 1990, I was working at Apple, and I missed a weekend call to my mother. She chided me: “Your tombstone isn’t going to say ‘Returned every email, returned every call.’ It could say, ‘Loving daughter of…” My mother was thinking about my tombstone and I was thinking about email.

Then, between 2000-2002, when I was working for Microsoft CEO Steve Ballmer, it wasn’t unusual for my inbox to have a thousand new emails a day. Everybody and their dog seemed to be on email. I filed, filtered, deleted, and delegated. And I called my mother on the weekends.

When I left Microsoft, my emails tapered off to 100-200 a day. In 2006, met Bruno, a mid-level manager in Silicon Valley. When I sent him an email, a message bounced back into my inbox:

“My email response time is 1-2 weeks. 

If you need immediate assistance, you can I.M. me between 9:30 a.m. and 6:30 pm PST or call me between 9:30 -11 a.m. PST.

For issues related to contracts, please contact…”

Bruno, GenY and twenty-something, named three communication tools: email, I.M., and the telephone. He spelled out his response habits. That got my attention.
Why don’t we all take a cue from Bruno? We could start a social movement. We can take back the inbox. I’ll call it eFree.

In the “signature” at the end of an email, people often include name, contact information, a quote, or a legal disclaimer. Let’s modify that. How about cutting and pasting the eFree signature below into your email signature? By adding it, you’re communicating your preferences, just like Bruno did. You’re letting the recipient know how to communicate with you.

eFree
1. Reply all is usually a bad idea. 

2. If you’re cc’d, there’s no need to reply.

3. A short, thoughtful email gets a quicker response. Long emails are read last.

4. If this issue cannot be resolved in 3 emails, consider scheduling a call or a meeting.

5. Thank you. Always lovely. Sometimes not necessary.

Are you ready to take back the inbox? Is there a funnier or more compelling way to say this? Radar readers have great suggestions, so thank you in advance!

(special thanks to Michael Tubach, an attorney with O’Melveny & Myers LLP, who helped craft the eFree principles)

This post originally appeared on BusinessWeek.com.

Phone in the Toilet?

by Linda Stone | comments: 8

My friend Sara sent me an email: "Linda, Sorry that I'm not able to call you back. My phone fell into the toilet."

We live in a world where phones can fall into toilets because our phones are following us everywhere. Untethered. Free. Free to fall into the toilet.

Last week, a high school sophomore told me that she brings her phone into the shower with her--in a Ziploc bag. She didn't want to miss an incoming text message. When I asked her if, in her sleep, she had missed life-altering messages, she looked at me blankly.

We are better at rationalizing what we do than being rational about what we're doing.
Untethered technology gives us the freedom to do nearly anything, anytime, anywhere. It can also enslave us when we feel compelled to use it wherever it is. Technology is neutral. How, when and where we use it is up to us.

When I recently visited an old high school friend in Ipswitch, Mass., I witnessed something unusual for most families today. Everything had a place. Cell phones were used at people's desks. Computers were used at desks. The kitchen was a place for meals and family fellowship. Family members were fully present for conversations--enjoying eye contact, listening and a meaningful exchange.

I mentioned this to a friend living in the Silicon Valley area, a former high-tech executive. She approved. "I moved the computer out of my kitchen. Now it's in the office. The office is an office, and the kitchen is a kitchen. I love it."

"Freedom" [free-d uhm] is the absence of or release from ties, obligations, etc. The promise of a phone that could go anywhere was and is the promise of freedom--freedom from being tethered to a place.

"Enslave" means to bring into servitude. Our phones have enslaved us even as they set us free.

How is this also true? Because we can, we do! Because we can, the phone accompanies us to the toilet, to the shower and to bed. Because it rings, we feel compelled to see who is calling and, often, to pick up. Because we can be accessible, we feel we must be accessible.

Is "freedom" just another word for nothing left to lose? Let the phone keep ringing the next time someone calls and you're in the midst of something else. When the caller later asks you why you didn't answer or where you were, you can smile and say: "I'm free. Free. I'm free to enjoy being in the moment."

And that's when you will become more powerful than any gadget on the planet.

This post originally appeared on Forbes.com.

Diagnosis: Email Apnea?

by Linda Stone | comments: 17

I've just opened my email and there's nothing out of the ordinary there. It's the usual daily flood of schedule, project, travel, information, and junk mail. Then I notice. I'm holding my breath.

As the email spills onto my screen, as my mind races with thoughts of what I'll answer first, what can wait, who I should call, what should have been done two days ago; I've stopped the steady breathing I was doing only moments earlier in a morning meditation and now, I'm holding my breath.

And here's the deal: You're probably holding your breath, too.

I wanted to know -- how widespread is email apnea*? I observed others on computers and BlackBerries: in their offices, their homes, at cafes. The vast majority of people held their breath, or breathed very shallowly, especially when responding to email. I watched people on cell phones, talking and walking, and noticed that most were mouth-breathing and hyperventilating. Consider also, that for many, posture while seated at a computer can contribute to restricted breathing.

Does it matter? How was holding my breath affecting me?

I called Dr. Margaret Chesney, at the National Institute of Health (NIH). Research conducted by Dr. Margaret Chesney and NIH research scientist Dr. David Anderson demonstrated that breath-holding contributes significantly to stress-related diseases. The body becomes acidic, the kidneys begin to re-absorb sodium, and as the oxygen (O2), carbon dioxide (CO2), and nitric oxide (NO) balance is undermined, our biochemistry is thrown off.

Breath holding and hyperventilating disturb our body's balance of oxygen, CO2, and NO. Nitric oxide, not to be confused with the nitrous oxide used in dental offices, plays an important role in our health. From a briefing document prepared for the Royal Society and Association of British Science Writers, Pearce Wright explains, "The immune system uses nitric oxide in fighting viral, bacterial and parasitic infections, and tumours. Nitric oxide transmits messages between nerve cells and is associated with the processes of learning, memory, sleeping, feeling pain, and, probably, depression. It is a mediator in inflammation and rheumatism."

As I researched the literature, and spoke with physicians and researchers about breath-holding, a relationship to the vagus nerve emerged. The vagus nerve is one of the major cranial nerves, and wanders from the head, to the neck, chest and abdomen. Its primary job is to mediate the autonomic nervous system, which includes the sympathetic ("fight or flight") and parasympathetic ("rest and digest") nervous systems.

The parasympathetic nervous system governs our sense of hunger and satiety, flow of saliva and digestive enzymes, the relaxation response, and many aspects of healthy organ function. Focusing on diaphragmatic breathing enables us to down regulate the sympathetic nervous system, which then causes the parasympathetic nervous system to become dominant. Shallow breathing, breath-holding and hyperventilating trigger the sympathetic nervous system, in a "fight or flight" response.

The activated sympathetic nervous system causes the liver to dump glucose and cholesterol into our blood, our heart rate to increase, our sense of satiety to be compromised, and our bodies to anticipate and resource for the physical activity that, historically, accompanied a physical fight or flight response. Meanwhile, when the only physical activity is sitting and responding to email, we're sort of "all dressed up with nowhere to go."

Some breathing patterns favor our body's move toward parasympathetic functions and other breathing patterns favor a sympathetic nervous system response. Diaphragmatic breathing, Buteyko breathing (developed by a Russian M.D., http://en.wikipedia.org/wiki/Buteyko_method), some of Andy Weil's breathing exercises, and certain martial arts and yoga breathing techniques, all have the potential to soothe us, and to help our bodies differentiate when fight or flight is really necessary and when we can rest and digest.

Now I want to know: Is it only the Big Mac that makes us fat? Or, are we more obese and diabetic because of a combination of holding our breath off and on all day and then failing to move when our bodies have prepared us to do so? Can 15 minutes of diaphragmatic breathing before a meal tune us in to when we're full? If, when we're doing sedentary work, and O2, CO2, and NO are optimally balanced, through healthy breathing, will we escape the ravages of an always-on sympathetic nervous system? Can daily breathing exercises contribute to helping reduce asthma, ADD, depression, obesity, and a host of other stress-related conditions?

I predict, within the next five-to-seven years, breathing exercises will be a significant part of every fitness regime. In the meantime, why not breathe while doing email? Awareness is the first step toward wiping out email apnea!

*Email apnea - a temporary absence or suspension of breathing, or shallow breathing, while doing email (Linda Stone, February 2008).

(originally published on The Huffington Post)

Stuffing Six Million Pages Down Google's Throat

by Tim O'Reilly | @timoreilly | comments: 23

I got two fascinating emails from Jason Hunter over the weekend, both concerning MarkMail, the open source mailing list search engine created by Ryan Grimm and Jason over at MarkLogic. I thought I'd share them, with Jason's permission.

The first was a fairly prosaic announcement:

In the last few weeks we loaded the PHP and PEAR mailing lists, a sum total of about 700,000 new messages. Contained within the new load is the php-general list, now statistically our largest list at 266,000 messages, passing by the old king tomcat-users with its 225,000 messages. Third place now goes to the main MySQL list.

It's great to know that you can now search these lists with the amazing MarkMail tools, which I wrote about recently on Radar. But what really caught my attention was Jason's next message, a bit of backchannel conversation that illuminates just how poorly the big search engines index small sites with large collections of data:

I thought you might find this interesting. One of the challenges we face with MarkMail is how to get Google to crawl all 6m (and eventually more) pages we have on our site. You often hear people talking about ways to increase their small site's PageRank, but you don't find many people talking about the challenge of stuffing Google full of (good) content.

Observations so far:

• Google is way ahead of the other search engines. Based on doing site:markmail.org queries: Google has indexed 760k pages, Yahoo 19k, and MSN 4k. It's not just the search algorithm that matters, it's also the crawl algorithm, and we have a clear winner here.
• Google started crawling right from the get-go. We peaked at 960k pages indexed about a week ago. The number goes up and down but the 1m line seems a tough one to crack.
• It is definitely possible to crack 1m. My old email archive system on Servlets.com stands at 1.3m pages indexed. Gmane has 2.5m indexed.
• This may be an impossible challenge. GMane has 2.5m messages indexed by Google but more than 50m messages archived.
• We theorize that Google judges a site on various factors and crawls accordingly. Based on the fast crawl speed in our earliest days, we think "momentum" is a factor besides just raw PageRank.
• We also theorize that faster page response times might help keep the crawl rate up. There's only 86,400 seconds in a day so to have 250k pages crawled means the Googlebot needs to hit your site three times a second. We expect the bot slows down if a site seems sluggish.

Attached is a picture of the pages crawled each day for the last 50 days (we don't have stats for the first few weeks). It's widely variable. Notice the happy increase this last week. That might be due to your blog entry giving us some extra whuffie. Or maybe it's due to the 30x speed optimization we made in response to your post to better handle the traffic. Or maybe it's just random. I wish I knew. [links added by me]

Jason's last point seems particularly insightful. There are only so many seconds in a day, and the larger the number of pages on a site, the harder a crawler would have to hit the site to index them all while keeping reasonably up to date copies. Small sites with lots of pages thus provide an impedance mismatch for crawling. Obviously, with Google showing 58.4 million pages in response to "site:myspace.com", and 73.3 million in response to "site:flickr.com", a high performance site justifies a high performance crawl, so the question is how Google makes the decision how deep to go. (Interestingly, "site:facebook.com" shows only 906K pages, suggesting that a huge proportion of Facebook's pages are still private.)

It does seem to me, though, that since most of the pages in mail archives are old pages, and not changing, some smarter algorithms that recognize the nature of these sites versus, say dynamic collections of potentially updated pages like flickr or myspace, might figure out how to do a purely incremental crawl, rather than re-crawling the pages over again. After all, archive sites like gmane and markmail have realized that they don't need to keep re-crawling archived mail messages. Why can't the big guys figure this out too, especially when someone else has done the work of collecting all the data?

Still, I'm reminded of a comment by Ben Bernanke reported in today's New York Times profile, The Education of Ben Bernanke, that he is "a believer in the laws of mathematics." I've become increasingly fascinated by the underlying math of Web 2.0 since reading Jeremy Liew's post about the economics of online advertising last year. Limits are, of course, made to be broken. But it's worth thinking about absolute (and temporary) limits to the growth of Web 2.0. What constraints do we take for granted? What constraints are invisible to us? Your thoughts welcome.

Reducing Email Volume

by Sarah Milstein | comments: 13

It's unfashionable to admit these days, but 14 years after getting my first account, I still like email. Of course, the volume is crushing, and so lately, I've been experimenting with email reduction. I'm getting good results with two key concepts that hardly anyone talks about and that focus on your sending habits rather than your inbox management.

First, send less email. These days, everyone knows not to pass along jokes and chain letters (in fact, I don't remember the last time I received either). But we're still struggling to figure out the form overall, and in the absence of social norms that work with the medium, it's endlessly tempting to treat email like a conversation. Which turns out to be its death knell.

Email just generates more email. So although sending less is surprisingly awkward at first, it not only cuts down on your volume, but it has positive network effects, to boot.

Second, worry less about what the people on the other end will think if you reply slowly or not at all. This is a biggie. If you're still treating email like in-person communication, then you're probably assuming that most people are waiting for swift responses to their messages. But since they're getting notes from dozens, if not hundreds, of other people, they're often aware of your particular thread only when it's popping up in front of them.

I don't mean to suggest you should blow off communication because you can, but rather that you probably have more time to respond than you feel you do. A productive way to use this extra time? By not sending email that says, "I'll look into that and get back to you." Just get back to them after you've done the looking. Even better, genuinely consider whether email is the most efficient way to get the issue resolved. If it's not, make the conscious choice to communicate in person, by IM or phone instead. This takes discipline, but it's effective.

Of course, the inbox is still ground zero and requires management. Read on for tips that work well for me.

(continue reading)

Could Open Email Work for You?

by Jimmy Guterman | comments: 14

Recently I had a case of walking pneumonia that quickly turned into flat-on-my-back pneumonia. The only good thing that came out of it was that I spent a few days away from email. When I returned, there were the better part of 1,000 missives waiting for me (and that's not counting what the spam filter held), but it was reassuring and instructive to realize that the world didn't end just because I wasn't an email hawk for a few days.

This got me thinking how people use and manage email nowadays. As I scrolled through the many messages, it was easy to see that plenty of people use email for the most ephemeral, not-worth-saving reasons: dozens and dozens of notes had single sentences or fragments such as "Done," "OK," Sure," and "Will do," sitting atop a long exchange. In some ways, it was faster reading all that email in one long, sub-Marissa Mayer session. I could isolate 15-piece email chains, delete or file the first 14, and focus on the most recent part. You can't do that in real time (I try; I employ GTD for email, but I'm not religious about it).

The most interesting way of managing email that I've read about lately is the open email regime employed by JP Rangaswami, CIO of BT Global Services. Stowe Boyd reports:

"JP has set up a stringent approach to filtering his email. He throws all email where he is CC'd directly into the trash. Basically, he only reads email directed to him, alone. Of course, for this to have any influence on people's behavior, he has to loudly and regularly let others know that he is doing this. More interestingly, he has opened access to his email to his staff. By treating his email as an open forum, he has found that his associates are more involved in his interactions with others. He has found that they can use this -- particularly his sent mail -- is a great learning opportunity."

So much of working online involves deciding what's public and what's private. Rangaswami has turned this around, at least for his work (I assume he still has a private email account he doesn't share). As fellow Radarite Brady Forrest noted when we discussed this recently, "Although this is analogous to making email like forums and wikis, the key difference is that you are using email as the entry point. It's not a separate wiki/forum site." And, since it's a tool that everyone uses already, it's more likely that the non-alphageeks you work with might be more likely to use it. What do you think: Could open email break down some walls in your organization?

Better Gmail

by Tim O'Reilly | @timoreilly | comments: 15

Paul Kedrosky pointed to Lifehacker's Better Gmail. I hope Paul won't mind if I just reprint his entire Better Gmail is Gmail 2.0 post here, since it says it all, elegantly and forcefully:

Gina at Lifehacker has created the #1 most important tool any Gmail user needs. It's an add-in, called Better Gmail, combining all the major Greasemonkey hacks for Firefox-based Gmail users, and it turns Gmail from indispensable into ... really indispensable.

I'm already using most of the pieces as discrete Greasemonkey hacks, but bundling them all like this is a work of genius. Just install it. Now.

Lifehacker also has a good tutorial on using the extension.

A really interesting side note: as Better Gmail is a firefox extension, it's not available for IE users. It's an interesting twist on the browser wars. In the old days, Microsoft and Netscape fought to lock in users with incompatible extensions. Here we see the same thing happening simply because that one platform is open and the other is not. The users themselves are evolving the browser. There's no intentional incompatibility (and it's not in the browser itself). It's just that one browser is getting more capable than the other as a result of its user community.

Another War We're Not Winning: Us vs Spam

by Dale Dougherty | @dalepd | comments: 29

Are we losing the war on spam? Is the war on spam a war we can win? Is there any reason for hope?

When I learned how much spam was hitting our servers at O'Reilly, I decided to ask several long-time Internet luminaries these questions. Was the situation as bad as I thought it might be? In short, the answer is yes, which only makes me wonder why more people aren't talking about it.

Let's first try to quantify the problem using O'Reilly's servers as an example. I'd like to see how we compare to other organizations.

All of our incoming email goes through one of two gateways, which route mail to servers that decide to accept or reject the message. This is, of course, before the message is delivered to an O'Reilly user, who may apply additional spam filters in their email program. The bottom line, according to Bob Amen, Director of Systems Engineering at O'Reilly, nearly 95% of ALL incoming messages are spam.

Here are the amazing stats that Bob shared with me. These numbers represent a one-week snapshot (last Monday 2/26 to Sunday 3/4.)

• 829,890 SMTP connections made to our two gateway mail servers
• 904060 attempted message deliveries
• 49194 messages accepted (I think this is actually a little high due to a configuration problem with our Zimbra server.)
• 94.6% of all messages were rejected

Here's the breakdown on how we reject messages:

• 282414 connections rejected due to bad SMTP HELO syntax
• 224722 connections rejected by IP address hits on black lists
• 31935 messages rejected due to invalid recipient
• 32402 messages rejected due to SpamAssassin score of 10 or greater
• 2788 viruses and other malware (most caught by ClamAV)

Individual users might not be seeing the increase because spam-blocking software mitigates the problem to a degree. Still, the solutions aren't adequate. A lot of spam makes its way past the filters. System administrators are having to spend more time on the problem and they need more servers and bandwidth to deal with the increasing flow of spam.

Of course, today, spam is not limited to email. Trackback and comment spam can hammer a blog server and they are a common reason why bloggers disable such features. I opened Skype today for the first time in a while and I had two XXX messages in an hour, giving a new meaning to "Call Girl." Nonetheless, let's stay focused on email.

I emailed a group of people who have been around the Internet a long time to see what they thought of the future of email. I asked them specifically if we're losing the war on spam.

Brad Templeton

Brad is Chairman of the Board, Electronic Frontier Foundation (EFF).

It seems to me that we're losing the battle. The spammers have won and there aren't any solutions in sight.

I wouldn't say that. There are a number of fairly decently working filtering systems, though a number of them have concerns about false positives. This doesn't rely on draconian blacklists, though some people use them.

There are a number of techniques not yet tried.

There are areas where we're losing, namely in the botnet department. As long as so many people run insecure systems, we are going to have botnets, and they will deliver spam that's hard to deal with except by filtering and challenge/response.

Paul Vixie

Paul is the author of several RFCs and founded MAPS (the mail abuse prevention system), known for its real-time blackhole list.

Is this a war we can win?

not with smtp.

Certainly, we're not winning it now.

right.

every potential smtp improvement or replacement that could do anything to actually stop spam, has been systematically patented. the crap that's left isn't going to do any good. we're headed for walled gardens.

Eric Allman

Eric is one of the authors of the SMTP RFC and the developer of Sendmail.

Are we losing the war on spam?

It depends on how you define "win". I still get junk phone calls, but the phone system is reasonably usable today. I think that spam can get to that level.

Is the war on spam winnable?

By the definition above, yes, but not without cost. As you probably know, I've been working on DKIM for cryptographic signatures on email. Assuming that DKIM is accepted and deployed, we'll be able to invert our way of thinking to make it more like the real world --- and more like IM.

In the real world I don't let anyone walk into my house. I look through the peephole to decide if they are someone I know or expect first. Right now we let just any old piece of email walk into our houses. Similarly, IM uses buddy lists, and it's not uncommon to only accept messages from buddies. Both of these cases are "filter in" vs "filter out". Right now we filter out messages that we consider to be spam, and everything left is treated as good mail. In the future I think we'll see a much more nuanced approach. Because of the nature of email it won't be pure "filter in", but rather something like this:

(1) Am I sure who sent the message (i.e., did the DKIM signature verify)?
NO: go to step (4)

(2) Do I know and trust the sender?
YES: accept the message

(3) Do my peers know and trust the sender?
TRUST SENDER: accept the message
KNOWN BAD GUY: refuse or drop the message
UNCLEAR: continue to step (5)

(4) Does the purported sender sign all messages?
YES: must be a forgery; refuse or drop the message

(5) Content scan the message --- is it probably spam?
ABSOLUTELY: refuse or drop the message
ABSOLUTELY NOT: accept the message
NOT SURE: quarantine the message

This is over simplistic, but essentially all we are doing today is step 5, and even then we usually err on the side of accepting the message (that is, if in doubt, accept it) in order to avoid false positives. But let's imagine a day when 80% of my incoming mail is signed. That means that less than 20% gets to step 5 (since step 4 also culls out some messages), and I can probably afford to turn up the sensitivity on my spam filters (if in doubt, don't accept it) without making my false positive rate work. This analysis is horribly over-simplified, of course, but the point is that we will be able to do a better job in the future than we can today.

Is there any reason for hope?

As described above, yes.

David Strom

David is a long-time writer on networking and email, including his own newsletter, The Web Informant at www.strominator.com.

I think the war is pretty much lost. Yes, the volume of total spam is vast compared to real message traffic, and won't be going down anytime soon.

You have to have spam blockers at various places on your network just to survive -- a gateway appliance, filters on each email inbox, and let's not forget about AV tool. It is a constant battle of wits, and an arms race as the blockers try to stay one step behind the spammers.

All in all, very depressing. I don't think the war is winnable unless we move towards sender authentication or secure email, which for the most part people are opposed to do.

Danny Goodman

Danny is an author of many books including SpamWars. On his spamwars.com site, he reports today that 96.3% of yesterday's email was "unwanted."

Danny and I spoke by phone. He said: "It's a lot like the war on terrorism. The hardest part is defining what the war is. The offenders are not clearly defined, the war is not clearly defined." He said the war seems like "a constant game of whack-a-mole."

In our conversation, we discussed that there were three approaches to combat spam: legislation, technical and user education.

"Legislation is weak, and in some examples, it almost legalizes some forms of spam. Enforcement is next to impossible. Plus, the amount of money to put a case together is incredible." He wasn't too optimistic that technological solutions would be acceptable. We discussed that the ability to use cryptographic signatures on email has been around (PGP) but it has not been widely adopted. "This is so obscure to most people," he said.

We discussed sender-verification in which an email server that receives an email contacts the sender to verify that its server sent the mail. When I discussed this option with Bob at O'Reilly, he used to do that but the volume of email (i.e., the volume of spam) makes that impractical. His already overburdened servers would have twice the workload. I am surprised that the technical community has not come up with a technological solution.

Danny mentioned a new form of spam that's been popping up: image spam. The content of the spammer's message is contained within an image to get by content filters. In response, spam-filtering companies are starting to use OCRs to detect words in images. And, in response to that response, spammers begin to distort images slightly so that they can't be read accurately by OCR software. The war escalates, perhaps making the point that spam-blocking solutions don't truly eliminate spam.

Danny discussed bots, which are installed on insecure computers around the globe, and do a bulk of the spam. I asked him if perhaps we should combat spam by writing programs that go out and remove bots. He said: "It's kind of happening but not in a good way. Rival gangs of bot developers are writing code to remove bots installed by other gangs and install their own."

Danny believes that user education might make a difference. He'd like to have Oprah do a show on spam to educate users so that they don't respond to spam and don't operate computers that are vulnerable to bots. I'm skeptical that you can educate people to not respond, especially when spammers are so skilled at deception. I don't know how to educate my own kids about what to do when I'm often confused by pop-ups that are disguised as messages from the operating system.

"In theory, the war could be winnable," said Danny. If nobody responded, spammers would go away. "That there is sufficient response as small as it may be manages to feed the spam economy. "

Danny's bottom line is this: "Each email recipient must be suspicious of every piece of email that arrives in his email box."

The Decline and Fall of Email?

Can email be saved? Are the days for SMTP numbered? Is this most basic Internet application so badly designed for the kind of online world we now inhabit that maybe we should think of leaving it behind? Is anybody working on the problem?

USENET was once a great Internet service. As it became more popular, more and more spam was flowing through it. Soon users moved on, finding other ways to communicate that didn't have the same problems, at least not yet. I do wonder if we could actually trace a migration from one set of services to the next, based on users leaving a sufficiently polluted space for greener territory.

Every Internet application that demonstrates the value of collective intelligence is eventually met with sophisticated attempts to dump garbage. Will any sufficiently open social network be met by ever more anti-social behavior until it eventually collapses?

The barbarians are at the gateway.