Here are a few examples that point toward a world after candybar multitouch. Exactly how they can be used and integrated in the UI/UX remains to be seen.

Demo of Microsoft Surface with PixelSense from Samsung

I have written about Microsoft Surface before, which is large horizontal multitouch screen built as a table. In the new slimmer version of Surface, Microsoft together with Samsung have developed PixelSense touch sensing technology. In PixelSense every pixel in the screen is also an infrared sensor that detects warm fingers on the surface. Just imagine what a future development of this technology could do if Samsung manages to fit the three RBG color sensors in every pixel. The surface could double as a copying machine. You put a paper, coupon or picture facing down on the surface, and when you lift it up, the copied object is displayed on the screen.

A technology for high performance multitouch screens has been developed by the Swedish startup Flatfrog. Their multitouch is based on an optical in-glass solution (Planar Scatter Detection) that also can be used to create multitouch on curved glass surfaces.

Another Swedish startup is Tobii, which has developed a technology for tracking eye movements. Using cameras that track the position of the pupil it is possible to calculate exactly what the user is focusing on. The company’s initial markets have been expensive high end systems for paralyzed people, market researchers, and academic researchers in cognitive psychology. Tobii has now begun to target the mainstream market together with Lenovo which are integrating eye tracking in a prototype laptop.

Kinect is a technology that Microsoft developed for their gaming console Xbox. It is an add-on gadget for your gaming console or flatscreen with facial recognition, voice recogniton and the ability to track gestures such as arm and hand movements. With Kinect you can control a game or PC by talking and waving your arms. It can be used for controlling an action figure or for moving between windows such as browsing your music collection, zooming in and out of a photo, etc. Up to six users can be tracked at the same time.

Even more futuristic UI/UX modalities are BCI technologies (Brain Computer Interface) where brain waves directly control an UI or some machinery. BCI has been used in research labs for a long time with electrodes implanted in the skull. Newer products based on less invasive methods with the electrodes attached to the scalp are now hitting the market, often in the form of a headset. The precision and bandwidth of these methods are still very primitive. One of the few things that can be reliably measured with BCI are emotive states such as relaxation vs. concentration.

Most of these new innovations are early in their life cycle and it is still too early to tell if anyone of them has a strong disruptive potential. New technologies drive development of new form factors. It remains to be seen if and how this will create future killer hardware. There is also a shortage of apps that can take advantage of the new features and turn them into compelling user experiences.

There are several hurdles to overcome. Products such as Kinect, Tobii and Surface put significant demands on processor capacity and there is a learning curve for any new UI technology. Prices have to come down for the large mainstream market to accept them.

I am slightly skeptical about a technology that requires you to wave your arms. What’s fine when gaming in your own living room, lifting and waving your arms for an extended period of time is tiresome. This has already been shown by users’ resistance to large vertical PC touchscreens.

It is possible that these new technologies will find their way into the candybar smartphone/tablet. But I think it is more likely that the future smartphone will integrate these new UI technologies without residing in the handset. If most tables, office desks, and bars are made of hard glass with MS Surface technology perhaps the user could just place their smartphone on the glass and have all their apps, contacts and pictures displayed. The surface might even have built in eye tracking. Or maybe Corning’s vision of a world of glass will come true and the nearest wall will be able to display your smartphone home screen with built in eye tracking for navigation in the wall. Just make sure to control your eyeballs – you never know who might be looking over your shoulder.

Satio smartphone with camera

Unfortunately, I think they have made a serious design mistake in their choice of camera. For marketers that are used to the IT industry’s logic that “more is always better” it is a given that cramming as many pixels as possible into the smartphone is a desirable goal. Sony Ericsson’s motivation is that the Facebook generation wants an easy way to take high quality pictures. That is correct, but a 12 MP camera phone is not the answer.

The problem is that a sensor with a lot of megapixels diminishes the ability to take good pictures in low light which, for most users, is much more important than taking high resolution pictures with 4000 x 3000 pixels. Sony Ericsson (SEMC) will run into problems when disappointed buyers realize how limited the camera is in practical usage.

12 megapixels used in the tiny sensors in a camera phone is not the same as 12 MP used in larger DSLR cameras. The number of megapixels is not the only performance factor. How tightly each pixel is packed on the sensor (Mpix/cm²) is of equal importance. If too many pixels are crammed on a sensor that is less than a square centimeter (6 × 4.5 mm if Sony’s new sensor is used in the Satio) each pixel will be so tiny that the physical limitations of the number of photons that can hit this pixel will determine the capability to produce an image.

If the camera sensor is viewed as a football field filled with buckets, the photons can be viewed as a rain of billiard balls that fall down in the buckets. Each bucket is one pixel. Stronger light produces many billiard balls which increases the precision when the number of balls in each bucket is counted. Weak lighting conditions (few billiard balls falling down) might work if each bucket is large enough to at least catch a few balls. But if the same number of buckets is crammed into a handball field each bucket will only be the size of drinking glass and the errors (noise) will be much larger because the billiard balls are too few to fill up all the small glasses.

For pictures taken in direct sunlight in the middle of the day or with a strong flash, the lowest sensitivity (ISO 100) is sufficient. Under these lighting conditions a small sensor (the handball field) is almost on par with a large sensor (the football field). There are still differences in quality because the optics in a small sensor camera is always inferior to larger, high quality optics. Quality differences are also caused by the fact that the leading camera vendors (Nikon and Canon) have more experience with electronic image processing than new players such as SEMC.

But as soon as you take pictures in low light and have to increase sensor sensitivity to ISO 400, 800 (or even higher) the difference between a large and a small sensor with the same megapixel count becomes dramatic.

In web forums such as Nikonians and review sites like DPReview most pro photographers and photo nerds have been in agreement about this for a long time. They are generally skeptical about small point-and-shoot cameras since the image noise level becomes unacceptable indoors or with low lighting. They are also critical about the way the vendors try to compensate for mediocre image quality with exaggerated electronic post-processing, by saturating the colors and by increasing edge sharpness. The result is quite often pictures that look unnatural. The table below shows the differences in the sensor area and pixel density for a selection of digital cameras and camera phones.

Sensor format and pixel density

Notice in the table above that the value for pixel density (megapixel per square centimeter) should be as low as possible. That figure is a measure of how densely the pixels are packed on the sensor and it is of almost equal importance as the number of megapixels. With a sensor area of 27 mm² Satio will have a sensor in the same range as the smallest point-and-shoot cameras. This will give Satio a pixel density of 44 megapixels per square centimeter which is very high.

Another observation from the table is that there are large differences in the sensor area between the camera types. The sensor in most DSLRs is around 350 mm². Compare this with the sensor in compact cameras which is between 28 and 40 mm². The sensor in a DSLR has an area that is twelve times larger than the smallest compact digital camera, thus providing twelve times higher capacity to collect light.

The first vendor to use a small sensor with fewer megapixels which is optimized for maximum low light performance will capture an empty market segment. There are no products on the market today to fill this latent demand. A few years ago Fuji released the Finepix F31fd camera, a 6 Mpix compact camera using a larger sensor with very good low light performance. It was even possible to take decent night pictures without a flash at ISO 1600 which no other compact camera had ever managed. When Fuji discontinued the model the camera gained cult status and the used prices on eBay have sometimes been close to double the new price.

Of course a company should not design their products for the nerd market. However, it is worthwhile to listen to advanced users. In this case they are right, high sensitivity in low light is relevant for everyone. Taking pictures indoors of your friends without being forced to use flash is perfect for the Facebook generation.

That the mass market customer believes “the more megapixels a camera has the better it is” is natural given that almost all marketing from the vendors has focused on megapixels. However, sooner or later the well-founded criticism of the megapixel obsession (e.g. from the New York Times’ technology editor) will bring awareness to the general public.

The only major advantage with a high megapixel sensor in a camera phone is that you can zoom in electronically without a disastrous loss of resolution. The electronic zoom only provides magnification from the center of the sensor if the lens has no moving parts. However, the marketers’ obsession with large zoom range (12x for Satio) is also criticized by those who actually use these cameras. Even if a camera has image stabilization is not enough to compensate for the fact that light sensitivity and resolution deteriorate significantly with electronic zooming. The New York Times points out that a good wide angle range is more important (so you can take pictures of all your friends around the dinner table) than useless telezooms.

A manufacturer that launches a camera (or cam phone) with a smaller zoom range and superior low light performance but with fewer megapixels can easily communicate user value. A marketing message could be built on comparing test pictures or copy text such as: “No flash at the wedding”, “Say goodbye to washed out flash images”, “Take pictures without anyone noticing”, etc.

If Sony Ericsson wants to reduce the pixel density my suggestion is that they develop a much more light sensitive sensor of the same size with a resolution of 2400 × 1800 pixels. This resolution will deliver 4.3 MPix pictures and the pixel density will be 15.9 Mpix/cm² which is much better than 44. A resolution of 2400 × 1800 pixels should be comparable to the resolution of a typical PC screen. Most screens have a lower resolution of 1680 × 1050 or 1920 × 1080, which means that the picture’s size has to be reduced in order to fit on the screen. Even larger 24 inch and 27 inch screens don’t have a higher resolution than 1920 × 1200. (Since most digital pictures are only displayed on a screen and never printed on paper, the higher resolution requirements for printing are not as relevant here.)

Sony Ericsson’s Satio as a case study

For Sony Ericsson’s sake one has to hope that the Satio will be well received by the market and that all the other advanced features are enough to convince the customers, in spite of the problems with the camera, to buy the smartphone. SEMC’s choice of camera is of general interest for product strategists in the tech sector and I will use it as case for an analysis of how the process went wrong (from my perspective as an outsider) and give some free advice to SEMC:

• Do not allow internal politics within the Sony Ericsson alliance to govern product development. If Sony was given the responsibility of product development for the camera just because they have their own camera division it was mistake. As a camera maker Sony is weak brand. For instance, their DSLR division is built on Sony’s 2006 acquisition of two budget brands (Minolta/Konica) that were too weak to survive the competition from Nikon and Canon. Just because Sony developed a 12 Mpix sensor that they adapted for camera phones does not mean that SEMC should be forced to use that sensor.

• That SEMC did not pay attention to photographers’ discussions on forums and in product reviews is a sign of weak competitive intelligence and consumer insight. Criticism of the megapixel race and clumsy image post-processing is readily available on the Internet. Even though SEMC has no intention of developing cameras they need to be competent buyers of the camera module.

• SEMC (or another vendor) should be brave enough to go against the group think in the industry. Unfortunately, most consumer camera manufacturers have been technology driven in the wrong way, for example in their obsession with zoom range and megapixels. (My impression is that SEMC is stuck in the same pattern. A few years ago I spoke with one of the senior executives at a conference about the downsides of the megapixel race. My concerns were completely dismissed.) In addition, the vendors have also had a Microsoft-style ambition to fill their products with 100s of unnecessary, silly functions such as smile detection. At the same time they have neglected to implement basic functions such as lossless rotation of JPG pictures. (The last time I checked this in 2007, several compact cameras degraded image quality if they were rotated 90 degrees.) By breaking conventions and getting it right it is possible to gain a strong competitive advantage. Just look at the iPhone.

• SEMC should have procured technologies from leading vendors with strong brands. Nokia uses Zeiss optics in their high end camera phones, which improves the image quality somewhat. SEMC could have done the same or partnered with Leica. In the built-in camera module Sony sells as an add-on to their new 12 MP sensor the lens is made of plastic. Plastic!

• Using the same argument, SEMC should have bought the software for noise reduction, color balance etc. from Nikon or Phase One (though Nikon would probably refuse to sell their technology to a company affiliated with Sony).

• SEMC should already prepare for an immediate launch of the next model ”Satio II” this fall if Satio fails on the market. It should use a 4.3 Mpix sensor designed for low light and use a smaller zoom range (3x).

This article has previously been published on my Swedish blog.

The projects conducted by Mobile Life were somewhat disparate, however, they have worked quite a lot with expressions for emotions and the integration of sensors that can measure things such as body temperature and heart rate in new applications (e.g. the project Affective Health). The prototype FriendSense (see picture) is similar to a Twitter that uses images and colors to capture how the members of a group feel and whether they have cold or warm feelings for other people in the group. In ActDresses you can control the behavior of a robot (a doll) by dressing it in different clothing. The project Mobile 2.0 covers a number of more mainstream mobile applications with integration of maps, geo-tagged pictures, friends’ pictures, geo-tagged chat rooms, friend finders on the subway, etc.

FriendSense

SwarmCam is an application where users can upload streamed video from their mobiles to a mixer with editing capabilities. Envisioned applications are a DJ that can show videos from the dance floor on a large screen and real time citizen journalism from, for example, an accident site.

The project e-Adept which is a co-operation with the City of Stockholm develops services which enable handicapped pedestrians to get exact walking instructions. Geographical micro-data have been coded for objects such as park benches, crosswalks, stairs and lampposts. With the aid of a PDA with GPS and voice output, blind people can navigate in the city streets. Other projects focused on different aspects of Pervasive Games where the players move around in an urban environment aided by GPS and their mobile phones.

In her speech, the center director professor Kristina Höök addressed how mobile data facilitates a breakdown of the old closed telecom paradigm and that the mobile Glasnost is now entering a phase of mobile service revolution.

The keynote address was given by design professor William Gaver from Interaction Research Studio at the well-known Goldsmiths, University of London. I like the policy at Goldsmiths (Damien Hirst is a former student) which demands the highest academic standards from their creative departments. The projects Gaver’s research team have worked on display an impressive creative madness. How about the idea of placing a piece of furniture as the Double Deck Desk (see picture) in an office to study how people interact with it? It is a good sign that Mobile Life has succeeded in building networks with other leading institutions in the field such as Goldsmiths.

Double Deck Desk

As it is hard to objectively determine the difference between genial creativity and things that are just odd and pointless there is always the risk that resources are wasted on bad projects. The fact that design and creativity are subjective can unfortunately be an excuse for not setting the standards as high as they are in areas that can be quantified and measured. The way leading arts schools (such as the Royal Institute of Art in London or the Art Institute of Chicago) have solved this problem is to ensure that the creative leadership is in the hands of extremely talented people. These creative leaders are confident in their assessment abilities and use brutal honesty to uphold high quality standards by rejecting subpar project proposals. I assume that Goldsmiths has a similar system.

The reason for my elaborate digression is the failure of a similar Swedish strategic research project, The Interactive Institute. The story of its failure is a textbook example of how not to manage national strategic research.

The background can be traced to the former Persson government which became aware of the Internet around 1997. At that time, the MIT Media Lab was the center of media attention and the Swedish government asked the MIT Media Lab if it wanted to open a research center in Sweden. The MIT Media Lab knew the value of their brand and asked for $100 million (if I remember the figure correctly) just to begin working with the Swedish government. After that slap in face, the Swedish government decided to start on their own. The outcome was the Interactive Institute, with an annual budget of 100 million Swedish kronor, which made it one of the largest budgets in the world after the MIT Media Lab. The idea was that interaction designers, scientists and artists would work together and develop new creative concepts.

For politicians the populistic elements must have been irresistible: an aggressive investment in the future, flashing lights, bright colors and animation, kids skateboarding in the corridors doing cool things. What a joy to be connected to all this youthful vibrancy. Tony Blair had paved the way a few years earlier by introducing “Cool Britannia”. In addition, government ministers could bring foreign guests to the expensive office floors at Östermalm in central Stockholm and show something that was visually appealing and easy to understand.

The invested resources did not produce any lasting results. I believe the failure was due to the lack of professionalism and quality in recruitments and execution. Collaboration and developing networks with leading institutions were neglected. Direct political interference and decisions governed by regional policy is not the right way to build a world class research environment. When I visited their open house demos around 2000 I hardly found any interesting projects. In an international academic review in 2003 they received damning criticism.

The situation for Mobile Life is much better. They managed to secure funding in a highly competitive academic environment. The problem for Mobile Life is rather how they will manage to differentiate themselves from commercial product development.

Comparing with the MIT Media Lab in the 1990s might be unfair because they were in a much better position to make themselves interesting at that time. MIT had the resources to implement and test new interactive services before the technological infrastructure was deployed in the rest of society. They became a demonstrator for all the new cool applications that everybody could envision in theory but were unable to implement in 1996.

Today there is a huge industry with tons of start-ups, entrepreneurs, and major corporations that develop these services for the marketplace. This makes it much harder for the academic world to advance and produce innovative products.
In a world where iPhone, Twitter and Facebook already exist, academics will have to choose peripheral and sometimes unintuitive projects to avoid replicating commercial product development.

As usual, one gets updated through the grapevine at these events. The most interesting fact that I can write about is that the head of Ericsson Consumer & Enterprise lab, Henrik Pålsson is now stationed in India. He said that the reason for his move is that the Indian market is developing very quickly right now. It is notable that Ericsson has relocated its most senior user market expert from Lund, Sweden to India.