Introduction Today's Digital Media and Videogames Companies Work With
Total Page:16
File Type:pdf, Size:1020Kb
[Nullpointer] Plug−In : An arts coucil technology residency Introduction | Research | Project Stage 1 | Project Stage 2 | Project Stage 3 | Results &Links Introduction Today’s Digital Media and Videogames Companies work with very specialised new media tools to produce cutting edge products. In the forefront of exploration are creative tools used in the development of new generation game consoles, mobile phone entertainment and interactive TV applications. A three months residency will provide an artist to develop his/her own creative art practice by exploring such unique digital tools. Working alongside digital media industry professionals, the aim of the project is to investigate and produce innovative ideas on new digital platforms. The project focussed on developing an abstract series of visual toys for the symbian mobile platform (Nokia 3650,7650,nGage etc). The following stages are documented (also listed above). Certain sections may be overly technical, but some discussion of code and hardware in such a project is obviously necessary. Research: The process of identifying the appropriate mobile platform. Project Stage 1: Preparation for programming Project Stage 2: Development issues and details Project Stage 3: Final project outcome Results &Links: Conclusions and pointers to further information The inital project proposal is also detailed here, however the final project represents an inevitable development and deviation from this document. iFone acted as the industry partner for the project. Previous Work Nullpointer has produced several previous projects t hat contribute to the background in which the Plug−In project will be developed BitmapSequencer / Audiopool / AvSeq / PixelMap (follow the links for more information hosted on this site) In association with. 1 [Nullpointer] Plug−In : An arts coucil technology residency Introduction | Research | Project Stage 1 | Project Stage 2 | Project Stage 3 | Results &Links Research Which Platform? In the first stage of research I have investigated the different options available for developing applications on mobile devices. Although other systems are available I chose to focus on the two leading platforms of J2ME and SYMBIAN. A brief description of these systems is as follows. J2ME An interpreted language such as Java, which for mobile phones means J2ME (Java 2 Micro Edition). Javaenabled handsets are in wide and increasing deployment, and virtually all phone manufacturers (including Nokia) have committed to production of J2ME−enabled mobile phones. Although other interpreted languages are available on some handsets, J2ME is an open standard, easily learnable by programmers with Java experience, and more widely supported by handset manufacturers than the others. Symbian OS An operating system for powerful smartphones that runs applications written in the C++ language and compiled to machine code. Smartphones are less widely deployed than Java−enabled phones, but more fully featured and capable of running more sophisticated applications, and will experience further market penetration over time. As with J2ME, there are competing “smartphone” operating systems in deployment, but phone manufacturers representing more than 70% of the market (including Nokia, Sony−Ericsson, Siemens, and Panasonic) have committed to the use of Symbian OS for their smartphones, making it more widely supported than other operating systems. Additionally, Symbian OS is an open standard, available for license by any hardware manufacturer, and is specifically designed for use with wireless communication devices, thus providing superior functionality. The following chart provides a high−level comparison of the capabilities of the two technologies: 2 [Nullpointer] * Except for some older legacy phones. 1 However, Symbian OS applications are typically so large that OTA installation is impractical. 2 J2ME can display video in phones that support the Java Mobile Media API (such as the Nokia 3650). 3 Access to SMS from J2ME is possible using the Nokia SMS API (supported on the Nokia 3410s) or Wireless Messaging API (WMA, supported on the Nokia 3650 and newer Nokia 3410s). Chosen Platform: Symbian OS After examining the development environments for each system and looking through existing software I decided that the SYMBIAN platform would be more appropriate for the project. Some of the following factors lead to this decision. 1. Previous experience in C++. 2. Improved sound support and 12bit graphics. 3. Ability to produce 'cutting edge' audiovisuals on a mobile device. 4. Support for Nokia series 60 phones (some Sony/Ericcson, Siemens) 5. Symbian is a next generation mobile OS with increasing support (N−gage e.t.c.) More information on Symbian can be found at www.symbian.com and more details on nokias Symbian support at www.forum.nokia.com see also references. Conclusion Although further research would be beneficial it is obvious from the initial findings that the Symbian platform offers a familiar programming system and (for a mobile device) a powerful graphics engine. The issue with control would be a problem when writing for any mobile device (except one specifically designed to play games GBA for example) but a simple and elegant solution should be defined (with a minimal set of keys). In terms of sound the application may have to deal with unstable timing but there is 3 [Nullpointer] a possibility to use MIDI instruments in order to reduce filesize (only supported on the 3650). All these concerns will be taken into account when defining a technical &design document for the chosen project idea. Reference Points There are a number of artistic and commercial projects that act as inspiration to the Plug−In project. The intention is to learn from these existing products/methods to make the project a synthesis of new ideas and good practice. A small selection is detailed below. Toshio Iwai: Toshio Iwai has created many beautiful yet simple audiovisual games where the users can play in an open ended environment to create constantly changing patterns of sound and light. His work has been exhibited world−wide and used as performance tools by prominent composers. Ixi Software / Golan Levin / SoundToys / Processing These four sites provide free tools and documentation on small scale computer audiovisual composition toys and tools. The individual projects range from complex synthesis to simple atmospheric music. Mr Driller / Chu Chu Rocket / Puzzle Bobble These three games have little to do with audio, but they base their addictive and innovative gameplay on very simple rules that create emergent behaviour and wide variation (much in the vein of tetris). They also work in a scalable manner performing well on both handheld devices,consoles and PC. Spheres of Chaos / rRootage / Turux.org These two games and the website Turux.org use simple but effective algorithmic graphics techniques to generate abstract generative images. Often derived from the 'demo scene' such techniques are designed to work within the strict program parameters defined by most mobile/handheld devices. Next: Project Stage 1 >>> 4 [Nullpointer] Plug−In : An arts coucil technology residency Introduction | Research | Project Stage 1 | Project Stage 2 | Project Stage 3 | Links Initial Proposition Synesthesia : An audiovisual sequencing game 1. Outline 2. References 3. Techniques 4. Development 5. Conclusion 1. Outline Synesthesia is an audiovisual environment where users can compose music and visuals in a reflexive generative system. The possibilities offered by digital technology to synchronize the generation of audio and video processes are rarely explored in either sequencing software or games. In this project users would engage with a sequencing system that is also part of an abstract puzzle game. As the player progresses the environment evolves and reacts to the developing audio track which will likewise be effected by changes in the game world. This reflexive process between audio and video is guided and manipulated by the player allowing them to explore a vast range of different sonic environments. 2. References Some video games/art works use audio−to−video synchronisation techniques. Vib Ribbon (PS2) and Rez (PS2,DC) present abstract visons of such relationships whereas ‘Bemani’ games provide a more mainstream approach. It is rare that such projects venture far into the realm of creating a generative and holistic a/v system. Media artist Toshio Iwai has created art works that allow users to ‘play’ with sound and visuals but most ‘interacive’ av art only provides a very brief and shallow experience of this kind. 3. Techniques a) Realtime sequencing Using a pool of audio samplesselected specifically for a particular environment the player will be involved in the process of sequencing rhythms and melodies in realtime. The progress of the player through a level will relate to their interaction with these systems. This sequencing will be represented in a visual form integral to the game landscape. b) Digital Signal Processing (DSP) Realtime DSP effects and synthesis could be used to ensure that the audio generated is always unique. Audio−Art applications such as MAX/MSP, Nato and Jitter have shown how realtime processing can create a myriad of experimental and atmospheric effects. Sounds could be filtered according to player energy levels or modulated to represent certain game states. c) Image Synthesis Using audio to generate imagery is a common technique used in PC sound players (I−Tunes, WinAmp, Windows Media Player e.t.c.). A combination of FFT analysis and image quantising would allow the visuals of the game to relate more instrinsincly to the evolving audio track. This would result in a visual environment that would reflect the players progress and technique. 4. Development Synesthesia could be devloped in a number of ways and has the facility to be a scalable system. Obviously the most visually lush incarnations would be based on high end technology such as consoles and PCs. However handheld systems and mobile platforms also offer a good opportunity to create such an a/v sequencing game. There is also the ambition of creating networked versions of the software where players ‘jam’ with each other either in competition or collaboratively.