Four short links: 19 October 2009

YouTube Bandwidth, RFID Visualization, Social Software Arms Race, Google Voice to the Laptop

by Nat Torkington | @gnat | comments: 0

1. YouTube's Bandwidth Bill is Zero (Wired) -- they buy dark fibre and peer with the major ISPs.
2. Immaterials: The Ghost in the Text (Vimeo) -- visualising RFID fields. See also the blog post about the work by Timo Arnall from Touch and Jack Schulze from BERG.
3. The Commercial Speech Arms Race (Bruce Schneier) -- Whenever you build a security system that relies on detection and identification, you invite the bad guys to subvert the system so it detects and identifies someone else. Sometimes this is hard ­-- leaving someone else's fingerprints on a crime scene is hard, as is using a mask of someone else's face to fool a guard watching a security camera ­-- and sometimes it's easy. But when automated systems are involved, it's often very easy. It's not just hardened criminals that try to frame each other, it's mainstream commercial interests. Bad actors game systems, and social software is just another system to be gamed. It's very difficult to create a system with no incentive to misbehave or to accuse others of misbehaving.
4. A SIP of the Future (Tim Bray) -- he connected Google Voice with Gizmo5 so his Google Voice number forwards to his laptop. FTW.

Walking the Censorship Tightrope with Google's Marissa Mayer

by James Turner | comments: 4

Google sometimes finds itself at a difficult crossroad of wanting to make as much information available to as many people as possible, while still trying to obey the laws of the countries they operate in. I recently had a chance to talk to Marissa Mayer, who started at Google as their first female engineer, and has now risen to the ranks of vice president in charge of some of Google's most critical product areas, such as search, maps, and Chrome. We talked about some of Google's future product directions, and also about how Google makes the decision as to when information has to be withheld from the users. Marissa will be delivering a keynote address at the O'Reilly Velocity conference next week.

James Turner: As VP of Search Products and User Experience, you're responsible for a vast swath of the Google product line, from search to maps to Google Labs. You were also the first female engineer at Google. Can you talk a little about how you came to Google and what brought you to where you are today?

Marissa Mayer: Sure. My background is when I was at Stanford, I was doing a symbolic systems degree in artificial intelligence. And I was always somewhat interested in search. I ended up getting an email [from Google] towards the very end of my job interview process. I came to Google and did the interviews. And I came here because I really wanted to put my AI background to use. For about the first year or so, I did. I did a lot of work on categorization, some work on search quality. And then interestingly, we sort of had a void around how the site looked and felt and how it worked. And we tried very hard to hire someone in UI. We thought we needed someone to do UI like one day a week, and do systems engineering the rest of the time. After a few months of failing to hire such a person, Urs Hölzle, our VP of Engineering, pulled me in and said, "Marissa, we've looked through all of the resumes and you have this background in your undergrad on cognitive psychology and philosophy and things. And would you mind dedicating one day a week to UI?" So I did.

I pulled together a volunteer team to help out one day a week while we all still worked on our various AI and systems work. And then, of course, one day became two days which became three days or four days or five days. And I was also programming at that point. I switched over from a lot of the AI work I was doing to programming the front end for Google, working on the Google web server because it was nice for me to be able to not only make decisions about the UI but also to implement them.

And then because I was implementing the changes to the front end, I would go and meet with Larry and Sergey. And they would say, "What's happening on the site this week?" And I would say, "Well, I coded a change that looked like this. And I coded a change that looked like that. And translated this page and it'll go here. And there'll be a pull-down over there for the number of results." And without even realizing it, I did project management before Google had project management, by specifying how things were and looked and communicating to the rest of the company about how these changes would manifest. And then when we got to about 200 or 300 people, Larry and Sergey discovered that most companies have this function, product management, which I didn't know about; they didn't know about prior [to this]. And we decided we should have such a department. They realized there were a few of us around the company that were doing product management even though that wasn't our title. So they started the product management group and got it started earlier. And so I became a PM.

First, I was the PM on Google.com, really broadly across the whole site, because there were only three of us. One of us did the site, which was me. Salar Kamangar did the ads. And Susan [Wojcicki] did the partners. And then as our teams grew, I became the director of consumer web properties, where I still did all of the consumer facing work of the website including branching into Gmail and tool bars and some of these other areas. And then as we progressed, eventually, we restructured so we had search, ads, and apps, because Gmail and the related space of calendar and docs became large enough that it made sense to spin it out. So then I kept the search piece and the properties we have that are more related to search.

James Turner: There's always been other companies trying to take a piece of Google's dominant position in search; how does Google plan to stay a step ahead in search, especially in light of new players like Wolfram and Microsoft's new emphasis on Bing?

Marissa Mayer: Well, we are very focused on search. We have a large team here that's really focused on it and working hard on it. And we're constantly trying to forge in new directions. So we were really excited about the launch of our search options page because we think that allows us to try a lot of new ways to slice and dice and filter results. We were also really excited about Google Squared, which attempts to do automated text extraction from the web and present comparison tables for different entities in response to queries. They're both new ways to search. How do you generate a timeline from a web search? How do you generate a comparison table? And some of our competitors are also looking at those same issues. So I think on the whole, right now, our search is a very healthy ecosystem. There's a lot of interest. There's a lot of activity, and there's a lot of new ground being forged.

James Turner: Google users want the most useful results, but content providers want to get their pages seen, sometimes it seems at any cost. How will Google continue to provide the most useful results in the world of increasingly sophisticated SEO gaming?

Marissa Mayer: Well, we generally -- we really want to be fair in these issues as well as be good to users. We do think that spam is very detrimental to the user experience. We do have an incentive to find spam and remove it from our results. But we want to do something in a way that's very scalable. The web is scaling at an incredible rate. And we don't think it's really viable to try and fight spam in a manual way. So we're always looking for new algorithmic ways to understand new spam techniques, to be able to detect them in an automated way and remove them from our results. And the nice side benefit that scalability has is it's also reasonably objective and fair.

(continue reading)

Search for Developers

by Brady Forrest | @brady | comments: 0

Vanessa Fox just posted her slides from her talk Diagnosing Technical Issues With Search Engine Optimization. They are packed with handy SEO/SEM suggestions, checklists and resources. It's worth going through at least once.

Google Engineering Explains Microformat Support in Searches

by James Turner | comments: 8

Today, Google is releasing support for parsing and display of microformat data in their search results. While the initial launch will be limited to a specific set of partners (including LinkedIn, Yelp and CNet reviews), the intent is that very quickly, anyone who marks their pages up with the appropriate microformat data will be able to make their information understandable by Google. This technology would allow you to explicitly search, for example, for only printers that had an average customer review of 3 stars or higher. Initial support will include things such as:

• Review Ratings
• Product Prices
• Personal Details

We talked this morning with Othar Hansson and RV Guha, two of the Google engineers responsible for the new functionality, and you can listen to them discuss it in this exclusive O'Reilly interview.

JAMES TURNER: Why don't you guys start by introducing yourselves?

OTHAR HANSSON: Sure. I'm Othar Hansson: and I'm a tech lead on this project. And I'm in Google's Search UI Group.

RV GUHA: My name is Guha. I'm an engineer at Google and I do stuff across the board.

JT: So can you describe briefly, to start off, exactly what it is you're releasing today?

RVG: Okay. We are asking webmasters who have pieces of data like reviews or people profiles, and in an experimental form, things like information about organizations and products, to put the structure data representing the content on the webpage in a machine-understandable form on the webpage. Typically, what happens is that if you take a website and having created opinions, I can talk about the context of opinions. You would typically have a database in the back-end which has lots of information about products. People write reviews about them. And you get information such as the number of reviews, the average rating of the reviews, the price of the product, who sells it, et cetera, et cetera, et cetera. It's stored in a structured database in your back-end. You then use some scripts to format it into HTML as per the site's design. Now going from the structured data to the HTML is quite straight-forward. But going from the HTML back to the structured data in a fashion which works across sites is very, very, very hard. Now our search engine doesn't -- it's very difficult for a search engine to understand -- to sort of get back the structured data for all of the sites. Now if it were to understand that, if it were to understand that this is a review site where the product being reviewed is such and such and it has 30 reviews with an average rating of 3.2 and so on and so forth, we could do a better job of the search. In particular, we could do a better job of presenting the two or three lines of text that appeared as part of the search result so that the user has a better idea of what to expect on that page. And from our experiments, it seemed that giving the user a better idea of what to expect on the page increases the click-through rate on the search results. So if the webmasters do` this, it's really good for them. They get more traffic. It's good for users because they have a better idea of what to expect on the page. And, overall, it's good for the web.

JT: So in some ways, that's in the same way that right now for certain sites, you'll give the internal structure of the site as part of the search result or for shopping results, you'll give price ranges and things like this. This is just, again, enriching and providing more structured -- more than just a snippet, giving more of a structured display of the information on that page?

RVG: Yes. If we have a structured data, we can do lots of things. We're starting off by improving the snippets. It's an absolute no-brainer. It seems to be helping everybody. And, as you know us, we keep playing it on with different ideas and different things. As structured data becomes more prevalent, there's a ton of ideas, both inside Google and outside Google, on how you might improve search.

(continue reading)

Transforming the Relationship Between Citizens and Government: Making Content Findable Online

by Vanessa Fox | comments: 9

Thursday on this blog, Congressman Honda asked, "how can congress take advantage of web 2.0 technologies to transform the relationship between citizens and government?" He noted that "A dramatic shift in perspective is needed before that need can be met. Instead of databases becoming available as a result of Freedom Of Information Act requests, government officials should be required to justify why any public data should not be freely available to the taxpayers who paid for its creation." He asked for input on what web 2.0 features he should add to his website to take advantage of today's online world.

The most important feature government web sites can add isn't really feature at all. But it would absolutely transform the relationship between citizens and government and make an amazing array of public data available. What's this magic feature?

Make government web sites search engine friendly.

How we look for information
Search is the primary navigation point for the web. Often when citizens look for government information, they start at a major search engine. They don't think to themselves, I need some information on vitamins, so I'll just go on over to the Office of Dietary Supplements at http://dietary-supplements.info.nih.gov. And then I need to make sure I'm eating a balanced diet, so I'll just check out http://www.nutrition.gov from the National Agricultural Library. And before I head to the grocery store, I'll make sure I understand how nutrition labels work from the information provided by the Center For Food Safety and Applied Nutrition at http://www.cfsan.fda.gov. Mostly, they go to Google and type in [food labels]. And in some cases, this works perfectly and the information appears.

But when information from government web sites doesn't show up on the first page of results for those searches, the information may as well not exist at all. For instance, an amazing amount of data exists from the U.S. Census Bureau, but it's inaccessible from search engines because it's locked behind JavaScript forms and the content itself doesn't use language that searchers would use. If I search for [98116 census data], results from census.gov are nowhere to be found.

Obstacles to being found in search engines
One problem is that the U.S. Census Bureau pages don't use zip codes to denote regions. They use tract numbers. Even if the pages were written in plain language searchers might use, search engine crawlers couldn't get past the JavaScript forms to access the pages.

Try doing a search using the same terminology as the U.S. Census Bureau, and you start to see the problem with the site's findability. Take [census track 97.02]:

None of those results lead to these handy details:

In addition to being buried behind JavaScript and containing little language people would actually search for, it's hidden in a popup with a URL like this: http://factfinder.census.gov/servlet/IdentifyResultServlet?_mapX=281&_mapY=216&_latitude=&_longitude=&_pageX=442&_pageY=554&_dBy=100&_jsessionId=0001cv7n8rWxjslrmI9aRw5nr-V:134a7lbrs">http://factfinder.census.gov/servlet/IdentifyResultServlet?_mapX=281&_mapY=216&_latitude=&_longitude=&_pageX=442&_pageY=554&_dBy=100&_jsessionId=0001cv7n8rWxjslrmI9aRw5nr-V:134a7lbrs

The server appends a session ID to the end of the URL (the portion beginning with "jessionsId"), which is tied to an individual visitor session and times out after 60 minutes. If I share that URL on a social media site, email, or in this blog post, anyone who tries to visit it just gets a "session as expired" message. It goes without saying that this kind of URL can't be indexed by search engines no matter how sophisticated they become.

(continue reading)

Making Site Architecture Search-Friendly: Lessons From whitehouse.gov

by Vanessa Fox | comments: 11

Guest blogger Vanessa Fox is co-chair of the new O'Reilly conference Found: Search Acquisition and Architecture. Find more from Vanessa at ninebyblue.com and janeandrobot.com. Vanessa is also entrepreneur in residence at Ignition Partners, and Features Editor at Search Engine Land.

Yesterday, as President-elect Obama became president Obama, we geeky types filled the web with chatter about change. That change of change.gov becoming whitehouse.gov, that is. The new whitehouse.gov robots.txt file opens everything up to search engines while the previous one had 2400 lines! The site has a blog! The fonts are Mac-friendly! That Obama administration sure is online savvy.

Or is it?

An amazing amount of customer acquisition can come from search (a 2007 Jupiter research study found that 92% of online Americans search monthly and over half search daily). Whitehouse.gov likely doesn't need the kind of visibility that most sites need in search, but when people search for information about today's issues, such as the economy, the Obama administration surely wants the whitehouse.gov pages that explain their position to show up.

The site has a blog, which is awesome, but the title tag, the most important tag on the page, has only the text "blog". Nothing else. Which might help the page rank well for people doing a search for blog, but that's probably not what they're going for. This doesn't just hurt them in search of course. It's also what shows up in the browser tab and bookmarks.

The site runs on IIS 6.0. Does the site developer know about tricky configuration that makes the redirects search engine-friendly?

Search engines are text-based, so they can't read text hidden in images. Some whitehouse.gov pages get around this issue well, by making the text look image-like, but leaving it as text, such as below.

However, other pages have text in images and don't use ALT text to describe them. (This, of course, is an accessibility issue as well, as it keeps screen readers from being able to access the text in the images.) An example of this is the home page, which may be part of why whitehouse.gov doesn't show up on the first page in a search for President Obama.

There are all kinds of technical issues, big and small, that impact whether your site can be found in search results for what you want to be found for. (whitehouse.gov using underscores rather than dashes in URLs, the meta descriptions are the same on every page...) Probably the biggest issue in this case is the lack of 301 redirects between the old site and the new site. When you change domains and move content to the new domain, you don't want to have to rebuild the audience and links all over again. (Not that Obama or whitehouse.gov will have a problem with attracting and audience, but we all can't be president!) When you use a 301 redirect, both visitors and search engines know to replace the old page with the new one.

In the case of change.gov, it's unclear if they intend to maintain the old site. The home page asks people to join them at whitehouse.gov, but all the old pages still exist (even the old home page at http://change.gov/content/home).

And in many cases, the same content exists at both change.gov and whitehouse.org (see, for instance, http://change.gov/agenda/iraq_agenda/ and http://www.whitehouse.gov/agenda/iraq/).

As Matt Cutts, Googler extraordinaire pointed out, give them a few days to relax before worrying so much about SEO. And I certainly think the site is an excellent step towards better communication between the president and the American people. But not everyone has the luxury of having one of the most well-known names and sites in the world, so the technical details are more important for the rest of us.

If you want to know more about technical issues that can keep your site from being found in search and tips for making sure that you don't lose visibility in a site move, join us for the O'Reilly Found conference June 9-11 in Burlingame. And if you're in Mountain View tomorrow night (Thursday, January 22nd), stop by Ooyala from 6pm to 9pm for our webdev/seo meetup, and get all your search questions answered. Hope to see you there! (Macon Phillips and the whitehouse.gov webmasters are welcome, but my guess is that they're a little busy.)