Mobile Blog: Video, Media, Augmented Reality, Rich Media, IVVR

photo credit: phauly

In desktop video conferencing, the video conferencing application is normally bundled to an Instant Messaging software that includes text chat capabilities. Users can appoint or prearrange a video conference using textual chat. In contrast, the current evolution of video telephony in UMTS (Universal Mobile Telecommunications System) networks based on the circuit-switched 3G-324M service does not seamlessly combine video conferencing with other communication channels. The notion of video telephony in the mobile environment is nearer to standard voice calling than in the stationary world. Therefore, it is more likely that somebody will place a video call without prior announcement. This leads to privacy and inconvenience concerns. The callee might not want to be seen during a conversation for a variety reasons: A video call “turns you ugly” (Harlow) because the build-in cameras are usually not placed just above the user’s line of sight but in the suboptimal position below the nose. Further, the video quality is meagre, and lightning conditions are poor. People might feel that exposing their face over a video call invades their privacy and, most times, callees do not want callers to see how he or she looks. Furthermore, the use of video telephony can depend on social factors. Societies in South East Asian countries—for example, Malaysia—are considered non-confrontational. This can be seen when people make decisions on which channel they use for communication. The author’s experiences in South East Asia revealed that most people prefer non-confrontational communication such as SMS (Short Message Service), instant messaging or e-mail, even in the business environment or with good friends. Voice calling is avoided as much as possible for a first or unexpected contact. It is obvious that P2P (Peer-to-peer) video calling is considered even more intrusive—and therefore unlikely to succeed in these societies.

According to an informal research of Sachendra Yadav (Yadav), opinion leaders and technology experts feel that video calling does not add much to a conversation compared to voice calling. In comparison to desktop video conferencing, which is mostly free nowadays, the cost-benefit analysis leads to resistance for using mobile video telephony.

For many reasons, 3G video telephony as a person-to-person conversational service is not as successful as projected. The existing technical foundation for video calling can be used to deliver IVVR (Interactive Voice & Video Response) services. A wide range of IVVR applications is imaginable, and some service providers and network operators already deploy them. Furthermore, special IVVR applications such as P2P Video Avatar can even compensate the drawbacks of classic P2P video telephony, making P2P-alike video telephony successful after all.

photo credit: Steven Fernandez

Media compression, error concealment measures, and the characteristics of wireless networks have side effects on the quality of 3G video telephony and IVVR (Interactive Voice & Video Response) applications.

3G-324M requires only the use of speech codecs. In contrast to audio codecs, speech codecs are designed for speech transmission within a narrow frequency range, making them inappropriate for transmission of music or a range of artificial sounds. This fact needs to be considered when designing IVVR applications—especially games, as most games utilize music and sound effects to create an immersive atmosphere.

H.263 and MPEG-4 Part II baseline were designed for images of natural scenes with predominately low-frequency components, meaning that the colour values of spatial and temporally adjacent pixels vary smoothly except in regions with sharp edges. In addition, human eyes can tolerate more distortion of high-frequency components than of the low-frequency components (Kwon and Driessen). In reference to the explanation of Kwon and Driessen, video codecs used for 3G-324M video telephony are great for natural scenes and talking-head scenarios. Depending on the type of IVVR application, these characteristics work against a good user experience.

Typical desktop or web applications have a monochromatic user-interface with boxes, buttons, and fonts that are clearly readable. Based on user interaction, the user interface can change its appearance frequently, perhaps only for some parts of the user interface or perhaps the whole screen. It is obvious that codecs used for 3G-324M video telephony are unsuitable for this kind of video transmission. Compressing such user interfaces with H.263 creates blurred fonts and tattered buttons and lines, leading to a user interfaces too distorted for a good user experience. The comparable high round-trip delays can make interaction tedious, with interfaces that require a high rate of user interaction and screen changes.

Depending on the type of game, the compression characteristics of video codecs used in 3G-324M can be advantageous. Contemporary 3D games such as first-person shooters or simulation games try to model the game environment as realistic as possible, creating natural-looking scenes and making them appropriate for compression using video codecs defined for 3G-324M.

However, more problematic are the delay requirements for mobile games that are essential for a good gameplay experience. 3GPP (3rd Generation Partnership Project) defines a delay variation of below 75ms for real-time games  and considers first-person shooters the most demanding ones with respect to delay requirements (3GPP). Other types of games, such as turn-based strategy games or visual novels, may tolerate a higher end-to-end delay and may require lower data rates.

3G systems were designed with the notion of enabling a single global standard to fulfil the needs of anywhere and anytime communication (Etoh). Compared to 2G (Second generation mobile networks, services and technologies) systems, 3G systems focus more on multimedia communication such as video conferencing and multimedia streaming. ITU (International Telecommunication Union) defined IMT-2000 as a global standard for 3G wireless communications and, within this framework, 3GPP (3rd Generation Partnership Project) developed UMTS (Universal Mobile Telecommunications System) as one of today’s 3G systems. W-CDMA (Wideband Code Division Multiple Access) is the main 3G air interface for UMTS (Holma and Toskala) implementing various person-to-person, circuit-switched services such as video telephony.

The high-level UMTS network architecture from 3GPP-R5 is described in documents from its Technical Specification Group in the figure below (Etoh).

High-level architecture of UMTS network

As shown in the figure above, the UMTS core network primarily consists of a CS (Circuit-Switched) and PS (Packet-Switched) domain. Typically, the PS domain is used for end-to-end packet data applications, such as mobile Internet browsing and e-mail. On the other hand, the CS domain is intended for real-time and conversational services, such as voice and video conferencing. Circuit-switched connections are most efficient for constant, continuous data streaming by definition (Etoh). In addition to the CS and PS domains, 3GPP-R5 also specifies the IMS (IP Multimedia Subsystem).

Using the PS domain, IMS is projected to provide IP multimedia services that also satisfy real-time requirements, including those that were previously possible only in the CS domain. In this thesis, I will discuss Mobile Games over 3G Video Calling based on 3G video telephony in the CS domain (see figure above, highlighted in dark green).

photo credit: iboy_daniel

In the following I describe how the characteristics of wireless networks affect the audiovisual quality of 3G video telephony and especially IVVR (Interactive Voice & Video Response) applications. The impact of these characteristics need to be considered when designing IVVR applications and services:

Wireless networks are inherently error prone. Bitrates in wireless systems tend to fluctuate more as compared with wired networks. In wired networks, phenomena such as fading, shadowing, or reflection are non-existent so that, for the most part, the same bandwidth and much higher bandwidths are present during transmission. Influences on signal propagation cause the constant changing bandwidths in wireless systems. Generally, the receiving power depends on the distance between sender and receiver. The receiving power p decreases proportionally to the square of the distance between sender and receiver:

p=1/d²

where d is the distance between sender and receiver (Schiller).

Receiving power is influenced further by frequency dependent fading, shadowing, reflection at large obstacles, refraction depending on the density of the medium, scattering at small objects, and diffraction at edges.

The effect of multipath propagation can cause jitter when the radio signal reaches the receiver by two or more paths at different times. Moreover, the mobility of the user adds another set of problems that results in fading of received power over time; the channel characteristics change over time and location. This exacerbates the effect of multipath propagation because signal path change will be increased as the user changes his or her location. Changes in the distance between sender and receiver cause different delay variations of different signal parts.

The phenomenon of “cell-breathing” is a special problem in CDM systems. In CDM systems, all terminals use the same frequency spectrum. Therefore, the more information that terminals are sending and receiving in a cell, the more noise that is produced. A higher noise level means that the noise level for far terminals will increase to the point that reception is impossible; ergo, the cell shrinks.

The UMTS (Universal Mobile Telecommunications System) or W-CDMA (Wideband Code Division Multiple Access) network counters but not eliminates these effects by implementing error detection, error correction, and error concealment measures. For example, in W-CDMA, cell-breathing is effectively prevented by implementing the wideband power-based load estimation to keep the cell coverage within the planned limits (Holma and Toskala). Nonetheless, these phenomena can still affect the audiovisual quality of 3G video calls and IVVR applications such as high delays, bit errors, or varying bitrates.

In 2007, CreaLog GmbH, located in Munich, Germany, has developed the example IVVR (Interactive Voice & Video Response) game Santa Claus Sleigh Ride. The aim of the game is to steer the reindeer sleigh to the North Pole by using speech commands or keystrokes. The game is constructed from a series of pre-recorded video clips showing Santa Clause steering either to the right or the left, played back depending on the caller’s commands. Although this game cannot be considered a real-time interactive game, is technically near to an IVR (Interactive Voice Response) supplement, perfectly illustrating how to cope with the various limitations of 3G video calling.

IVVR game “Santa Claus” from CreaLog GmbH

The picture above shows a number of screenshots from the game that makes clear that the game is adapted to the characteristics of IVVR technology. It copes with the high delays by simple reducing the interaction rate to a minimum. After a 3G video call is placed, video sequences start automatically, explaining how to play the game and how to steer the reindeer sleigh to the North Pole. The only caller interaction required is to say left or right, or to use numbers 4 or 6, as seen in the second screenshot. During my testing sessions, the speech recognition misinterpreted my voice commands in approximately one-third of the time. The response delay after a keystroke was about 1 second before the next video sequence was played back.

Media codecs used in 3G-324M are targeted for speech communication and talking head scenarios. CreaLog’s game handles this limitation by having no sound effects but rather a voiceover technique where somebody playing Santa gives hilarious comments about the player’s decisions. Furthermore, the game uses a 3D animated environment consisting of colourful gradients with a small amount of sharp edges that, unsurprisingly, compresses well using video codecs aimed for the compression of natural scenes, creating a decent visual quality of this mobile game.

More about CreaLog.