AMIS (Asynchronous Music Information Streams)
[1] INTRODUCTION:
AMIS is a software MIDI interface based on the ideas of Granular synthesis. It is not a Granular synthesizer by means of altering the physics of a sound rather it is a method of organizing MIDI information mainly based on the Asynchronous cloud streams approach of organizing Granular synthesis events. However, AMIS does not particularly follow any particular Granular implementation. AMIS is a software for performances therefore generates data in real-time. Consequently, some classic Asynchronous GS features such as the pre-defined cloud duration are abandoned here. Yet, other included features, such as parameter linkage (internal parameter dependencies / see Appendix 8), make this interface similar with other particle synthesis methods such as Grainlet synthesis. The general idea was to providing the user with plenty of control, but at the same time allowing creative musical interpretations to be taken by the AMIS algorithms.
AMIS provides an extra layer of control to any musical device being able to accept MIDI data in order to control a sound output. This device can either be an external MIDI module or even an internal software synthesizer (by using the MIDI Yoke or similar drivers). Therefore, AMIS is not constrained by internal wavetables, instead it provides control of any waveform available on the available MIDI system – either sampled or synthesized. This special functionality makes possible the introduction of Granular synthesis to any MIDI device connected with the AMIS interface and therefore extends the musical possibilities of the connected instrument. AMIS can easily produce in real-time, intense clouds of sound, combined of hundreds of elementary particles, which can vary in frequency and time according either to stochastic or probabilistic functions. The automated functions of AMIS facilitate the generations of these sound particles – the grains – and thus make it ideal for real-time performances of Granular synthesis. Furthermore, AMIS can be controlled by an external MIDI device that outputs MIDI data, in order for a user to discard the PC interface completely. In this configuration AMIS stands in the middle of the external control surface and the internal or external sound module (Appendix 1).
[2] OBJECTIVES & CONSIDERATIONS
The realization of the AMIS interface begins after reading about the Asynchronous Granular clouds approach by Curtis Roads. My initial idea was to create a Granular synthesizer, based on Asynchronous clouds, using internal wavetables. I found two main software environments necessary for the realization of the AMIS interface. The CPS environment, for the audio programming of the application, and Macromedia’s Director software necessary for the visual programming and the final Shockwave authoring of AMIS on the Internet.
Although I managed to construct a well sounded instrument with internal wavetables (http://www.sonicspace.org/asineShock.htm) I soon realized that I had an available palette of timbre constrained by the internal wavetables of this instrument. However, if I was able to connect the organisation methods of my first implementation in order to control wavetables coming from external sources, then I would have introduced an interface limited in timbre only by the wavetable provided by the external source. A basic sound engine such as General Midi would produce average result, but a powerful synthesizer would sound much more impressive. By separating the event process by the synthesis process it seems in a way that I am moving backwards in computer music, following the principles of the MAX N family. Latest music software implementations support a closer interaction between events and signals - however this is not always the case. MPEG-4 Structured Audio uses the SAOL language (Structured Audio Orchestra Language) [30] for describing synthesizers and sounds in order to assist the transmission of sound information through the Internet. Instead of transmitting the sound itself, MPEG-4 Structured Audio only transmits the necessary data to control any sound already resided on a local computer. AMIS is partly based on this idea in two ways. Firstly, it has been developed in a software environment that supports the MPEG-4 Structured Audio format and secondly it adapts this separation of control data from signal data. However, it is limited by the generation of only simple control events - MIDI information.
This separation offers two major advantages. Firstly, it constrains the minimum processing power AMIS required to run – ideal as AMIS runs on an Internet browser - and secondly it does not restrained the interface to a particular sound engine. Someone could use the processing power and features of the external sound engine (such as wavetables, filters, effects, etc) to alter the sound events AMIS is able to process. By doing that, AMIS would handle the organization of the sound events while the core alteration of the physics of sound would be performed by the external sound engine.
The decision to adapt MIDI as the communication protocol between AMIS and the external devices was very straightforward. MIDI is common to most new electronic sound modules and a MIDI to CV converter would also allow access to older analogue sound modules with the AMIS interface. The MIDI protocol in normal circumstances is also adequate enough for fast generation of grain events. MIDI uses a serial data connection running at 31250bps and 10 bits per byte. Therefore, it can send 3125 bytes per second. A typical MIDI note message is 3 bytes, so the fastest sequence of notes it is possible to generate is 1041 notes per second - about one per millisecond. AMIS generates two different MIDI streams, one for MIDI channel 1 and one for channel 2. This simplifies real-time control of the instrument and has been found efficient to produce solid clouds.
[3] CPS
CPS is a recently created real-time programming environment for audio and MIDI information. Patches in CPS build by placing objects and making wires between them. These objects allow a low-level access to a plethora of audio and MIDI commands. CPS is based on MPEG-4 Structured Audio (read more at http://sound.media.mit.edu/mpeg4) which is designed for real-time exchange of parameters through a network. In turns, MPEG-4 Structured Audio is based on C-sound the roots of which can be traced back in the MUSIC N family of music software. However, CPS has some significant differences in comparison with MPEG-4 Structured Audio, most noticeable the absence of separation between the ‘score’ and the ‘orchestra’ files that the latter uses. More about these differences can be found at the CPS website at http://www.bonneville.nl/cps/manual.
A CPS patch can run inside Director’s environment. This is possible with the CPS Xtra (a CPS plugin for Director). Therefore, you can save a CPS patch as Lingo code (the scripting language of Director) and then insert this code into a Director movie. In this way it is possible to process audio signals and MIDI data from within the Director movie. An extra set of Lingo commands is also introduced to control CPS from Lingo. These include functions to create an object, to create or delete a wire between objects or to send or receive parameters. A Director movie which contains the CPS lingo code is ready to be used on an Internet browser as a Shockwave movie. However, the browser must be prepared to handle the CPS engine, something that can be done by installing the CPS2Browser add-on.
[4] DIRECTOR
Macromedia Director started in 1987 as a software animation tool based on the Lingo scripting language. Since then it has been a leading multimedia creation software tool for building interactive content and applications for CDs, DVDs, kiosks, and the Internet. Lingo is a strong programming language that makes possible the development of powerful multimedia applications. However, the audio requirements for developing multimedia applications were far less than many up to recently. Present multimedia applications however, demands enhance audio output something that Director itself fails to address. MIDI for example is completely absence in Director. This along with other similar issues can be overcome by using Director Xtras (software plugins). These enhance the Lingo vocabulary and provide extra functions targeting audio or other appropriate section of Director. By combining the visual facilities of Lingo with the audio functions of the Xtras it is possible to develop complete applications aimed for musical purposes. One of these Xtras is the CPS plugin which I used to program the audio section of the AMIS interface.
AMIS IMPLEMENTATION
FEATURES:
Asynchronous and synchronous cloud generation.
Two modes of frequency calculation [Fc and Bandwidth].
Velocity to cloud density mode.
Density to frequency mode.
Frequency to grain duration mode.
Frequency, Density and Program Change automation.
MIDI In / MIDI Out
After reading Curtis Road’s papers about Asynchronous Granular synthesis and managing a good understanding about the subject, I started first with the implementation of the audio engine of AMIS in CPS. I had to address the following basic issues and find suitable solutions for the different sections of the software:
1. The real-time frequency control of the individual grains.
2. The real-time density control of a cloud in terms of asynchronous streams.
3. The duration control of the grains.
4. The Grain waveform.
5. The MIDI implementation.
6. The graphical interface of AMIS
It has to be mentioned that some of the following approaches and values used in the coming CPS patches have been found empirically through experimentation while others are based on the native CPS examples.
(1)
FREQUENCY in AMIS
The Asynchronous Granular approach specifies the fundamental frequency of the grains as a random value taken by a frequency bandwidth specified by the user. AMIS uses two different methods to calculate the fundamental frequency of the grains. A user can either select the output frequency to be the random result of certain bandwidth or be the result of the deviation of the centred frequency (Fc). In the first case, a value is taken randomly from a linearly distribution list of numbers at a rate that match the current density rate of the real-time cloud. This has been accomplished easily in CPS with the ilinrand object. This sets the lower and the upper bandwidth of a random output. In the second case, where the frequency output is calculated by the Fc deviation, a different approach has been deployed. A user is able to introduce a certain amount of frequency deviation of the centre frequency at a rate that match the current density rate.
This deviation has three modes. In standard mode [BOTH] the output frequency is the output frequency is calculated by a random value taken out of the summation and the subtraction of the fundamental frequency with the deviation amount. In [UP] mode the output frequency is calculated by a random value taken out of the summation of the fundamental frequency with the deviation amount. In [DOWN] mode the output frequency is calculated by a random value taken out of the subtraction of the fundamental frequency with the deviation amount. In all modes, when the output of the calculations results in a negative frequency, this frequency changes polarity and becomes positive. The frequency deviation has been accomplished with a subtraction module, a summation module and two klinrand CPS objects. These have been arranged as seen in figure 1b. The KPacthIn object on the top of figure 1b connects the main CPS patch (not seen in the picture) with the Frequency deviation sub-patch. This sub-patch outputs the following types of data in the left to right order: The fundamental frequency (Fc), the deviation value, a trigger message (On / Off) and a gate switch. As seen, the deviation value is summed or subtracted from the centre frequency before used as an argument for the klinrand object. Notice that because the first input of the klinrand object accepts the minimum value and the second input the maximum value, the output of the summation object connects into a different input of the klinrand object than the output of the subtraction object. The last gate object in the chain specifies the mode of the deviation. When in first position, the resulting frequency is the random selection of either the summation or the subtraction. Obviously, the output frequency would be the Fc itself when the deviation amount is zero.
Figure 1b: Frequency deviation sub-patch in CPS
The fundamental grain frequency can be itself automated. The user can specify an automated positive or negative frequency direction. The [AUTOMATION] section provides an area where the user can specify the direction (-/+), the frequency [TIME] and the steps [STEP] at which the automation process will follow. High [TIME] values results in slow frequency rising or falling while small [STEP] values results in more precise changes. Figure 2b demonstrates the CPS implementation for the automation.
2b. The CPS Automation sub-patch
An additional way to automate the Fc output is by setting active the [DENS TO FREQ] function. Then the output Fc is a scaled result of the current density of the cloud. This will produce low frequencies in sparse clouds and high frequencies in dense clouds.
(2)
DENSITY
To produce an asynchronous cloud of grains, control of the time intervals between successive grains required. Consequently, this controls the amount of grains per second. While small time periods between the grains will produce full sonic clouds, large periods will result to more isolated sounds. When the intervals are equal the resulting cloud is synchronous and the period between successive grains can be completely specified by the user. This result in rhythmic sonic clouds, made possible in AMIS by either selecting the [SYNCHRONOUS] option or by setting the [VAR] parameter to zero when in [ASYNCHRONOUS VAR] mode. However, the classic Asynchronous cloud approach defines density as the variation between the initial and the final density over the specific duration of a cloud. Because the AMIS cloud is constantly adjusted by the user this in only partly implemented. In [AUTOMATION] mode a user can specify the linear progression of the cloud’s density in time, by automating the [LOW DENS or VAR] parameter to change a certain value for every user defined time period (in milliseconds). This way the user can describe the variation of density within a real-time cloud for a certain time.
In normal non-automated mode the calculation of density derives either from a random value taken by a user specified density bandwidth or by a user defined random deviation of the target density value [LOW DENS or VAR]. In both cases, the CPS sub-patch is based on a noise generator, the klinrand object that returns a random value from a linearly-ramped distribution (see Appendix 10). In the first case, the user sets the minimum and the maximum value of the density spectrum and density values are taken randomly from within this spectrum. A user normally sets the minimum value to zero and controls the density of the cloud by only adjusting the maximum value ([LOWER DENSITY] slider). In this case, a sparse cloud is produced when the maximum value is high.In the second case, the density derives from a random value taken by the summation or subtraction or both between the selected density value [MEAN] and the specified variance value [VAR]. The CPS patch responsible for this operation is similar to the patch shown in figure 1b. This time however, this patch is used to control the density of the output events. Density can also be frequency dependant. When [FREQ TO DENS] is active the frequency is scaled to control the density of the cloud. High frequencies will result in a dense cloud, while low frequencies will produce a sparse cloud. Figure 3b shows how this has been implemented in CPS.
Figure 3b: Frequency to Density CPS sub-patch.
(3)
DURATION via MIDI
The grain duration considered to be very important for the timbre of the granular cloud. In audio terms the grain duration is usually results from an envelope curve which alters the progression of a sound in time. Finding a general way to apply this behaviour in a MIDI granular interface like AMIS proved challenging. In MIDI systems the triggering of a particular note and the release of the same note are considered as two separate events.
These events are the Note On and the Note Off MIDI messages. They consist of either a Note On or Off status byte, which also indicates the MIDI channel number, followed by two data types which specify the note number and velocity (figure 5b). When a Note On message is transmitted by AMIS, it triggers a particular frequency from the selected sound engine and keeps this frequency active until a Note Off message is generated. AMIS handles the duration of the grains is by delaying the Note Off Midi (128) message by a certain amount of milliseconds, thus keeping the Note On MIDI message (144) longer in time (figure 4b) .
Figure 4b: Delayed Note Off message.
Figure 5b : A Note Off message.
The midiMessage object creates a MIDI message (that may consists of 1 to 3 bytes), into a single value.
The duration of a grain event can also be controlled by the current fundamental frequency something that can be found in Grainlet synthesis. When the [FREQ TO DUR] is active the duration of the grains is proportional to the frequency. High frequency values will result in shorter grain durations while low frequency value will result in longer grain durations. This produces a very interesting spectrum comprise by grains of different durations according to their frequency. It also ensures that lower frequencies would have as equal as their bandwidth representation on the evolving granular cloud.
(4)
External grain WAVEFORM
The grain waveform totally characterized the granular cloud. AMIS does not use internal wavetables as waveforms for the grains. Instead, the sound module which is connected to the AMIS interface provides the necessary grain waveforms. The range of timbre of the AMIS cloud therefore, reflects the range of timbre of the connected sound module. The external grain waveforms can be of any audio type, as long as they can be triggered by MIDI messages. Hence, the grain waveform can vary from an analogue Voltage Control Oscillators (through a MIDI to CV converter) to software synthesizers or sampled sounds.
More specifically, the grain waveform is specified by a Program Change MIDI message, different for each AMIS MIDI channel. Following the MIDI specifications this has a range of 0 to 127 corresponding to different patches on the external sound module. AMIS is limited to two simultaneously output waveforms, each one representing a different MIDI channel (either 1 or 2). The grain waveform can also be determined by probabilistic means in which the occurrence of a specific waveform (MIDI Program Change Num) is the result of the weighted user inputs. Four different waveforms can be algorithmically automated. For each waveform a corresponding weight value determines how often this event (Program Change) will occur. A high weight value will result in less chances of occurrence, while keeping all four weighted values equal will result in a transchrome cloud. Figures 6b shows the CPS implementation.
6b: Grain waveform selection based on the Gaussian distribution.
In figure 6b, the kgaussrand object accepts the user specified target (mean) Program Change number value and the weighted value that serves as the variance of the target frequency. At time intervals taken by the current density of the cloud, the kgaussrand object sends a MIDI Program Change message to the equal to object, If the input value is the same with the original target value, then this specific patch – waveform is selected.
SPATIAL POSITION
The possibility to control the aural spatial location of the grains within the cloud is very important in the Asynchronous granular approach.
MIDI CONTROL
AMIS uses the MIDI protocol to communicate with external devices. It can send and receive an array MIDI messages in order to control in real-time external devices which are able to communicate through the MIDI protocol. More specifically, it is possible to control in different ways the frequency, the duration, the volume, the waveform and the time intervals between successive sound events, produced by the external sound device. The AMIS interface automatically identifies the available MIDI ports of a system and allows the user to select the appropriate MIDI In and MIDI Out ports in the [MIDIPORTS] menu. By selecting an appropriate MIDI Out port, AMIS can control an external device such as a synthesizer or an internal General Midi sound engine. Figure 7b shows the lower level MIDI Out configuration in CPS.
7b. The MIDI In CPS sub-patch.
Figure 7b demonstrates the most important outputs of the KPatchIn object. From left to right these are: The output frequency, the volume value, the MIDI channel number, the grain duration, the selected MIDI port and the selected MIDI Program Change number. The output frequency is connected to a midicps object that converts a cycles-per-second representation to a MIDI pitch output. The MIDI representation is finally sent to a notePlay object that triggers a MIDI Out event with a specific duration, volume and at the selected MIDI channel. The programChange object is actually a sub-patch as seen in figure 8b. The first input of the midiMessage object sets the status byte – the type of message which is 192 for Program change in this case – while the second input sets the specified value (The patch number of the connected device in this case).
8b: The MIDI Program Change message sub-patch.
By selecting the MIDI In port, some of the AMIS basic functions such as the grain frequency or the cloud density, can be controlled by an external MIDI controller. The [CTRLS] menu on the AMIS interface, provide an area at which the user can identify the MIDI controller number of a knob or a slider (transmitted by an external MIDI device) and assign it to a specific AMIS function. Figure 9b shows the higher-level organization of the CPS MIDI In sub-patch and figure 10b shows the sub-patch responsible for identifying the input MIDI controller number.
9b: The core MIDI In CPS sub-patch.
In figure 9b, the midiIn CPS object provides the basic communication between the external device and the AMIS interface. It fires the incoming MIDI message from the external interface to the voiceAllocator object and to the cpsmidi object. The voiceAllocator object translates the incoming MIDI message to a MIDI pitch and volume, while it transmits a Note Off message whenever appropriate. The MIDI note pitch representation is transformed by the cpsmidi object to a corresponding frequency value representation in order to be processed by the AMIS algorithms. As seen in the patch, a second connection between the midiIn and KPatchOut object is directly established for the identification of any possible control changes. (figure 10b)
10b
Figure 10b demonstrates the CPS part of how a user specified controller can be assigned onto an AMIS function. The extrControlChange object is responsible for capturing the selected controller (slider, knob or else) of an external device. It does that by analyzing the second and the third byte of an incoming MIDI message and forward these values to the switch object next. The switch object is basically a table that holds a specific number values in a serial order. When one of the enclosed values matches the first input of the switch object, then according to the position of this value in the table, the corresponding output will pass the second input of this object to the KPatchOut. The Lingo counterpart of this implementation allows a user to specify custom switch values, adapted to a specific external MIDI controller. The switch object data represents a single variable in Lingo, which is specified by the user (see Appendix 3).
GRAPHICAL INTERFACE
The basic graphical interface of AMIS conceived in the Director environment long before the final realization of the CPS programming of the application. It has been designed to be as simple and clear as possible but at the same time to return enough visual information about the status of the real-time granular cloud. All visual controls represent different CPS functions. Therefore it can be considered as an interface between the original CPS patch and the user.
Figure 11b; The AMIS interface.
The sliders on the left of figure 11b represent the most important functions of AMIS. These include controls for the grain frequency, the frequency deviation, the density of the cloud and the grain duration as explained previously. Many of these controls are also MIDI controllable by an external MIDI device. The right side of the AMIS interface offers control for the frequency, density and patch automation of the Granular cloud. The user enters the desirable parameter numbers and press the [ENTER] button to assign them into the AMIS engine. On the same side, someone can also select the desirable MIDI Program Change corresponding to the appropriate patch number of the external/ internal sound engine. In the lower right side, there is also control of the MIDI routing [MIDI ROUTE] (the channel that will be affected by duration, volume and panning changes) and control of the delay between channel one and two. The [MIDI ROUTE] control is basically introduced to reduce the number of similar function in AMIS and simplify the interface. Instead of providing two sliders for the duration of the grains in channel 1 and channel 2, with [MIDI ROUTE] the user can assign the channel at which the sliders will affect. The top side of the interface provides control of the MIDI volume and panning for each channel individually. Both of these are MIDI controllable depending on the selection of [MIDI ROUTE]. The bottom part of the AMIS interface contains different switches to trigger several parameter linkage functions. These include the frequency to grain automation and the frequency to density automation. In the same section there are also switches that trigger the frequency, density and patch automation on and off. When one of these controls is active, the description of the control will become red to indicate that this control is active. The menu on the top of the main panel provides information about the MIDI ports [MIDIPORTS], the MIDI controller numbers [CTRL], control of the cloud animation, the [MIDI IN] option and a very useful [help] option. When [MIDIPORTS] is selected, a new menu will appear within the main panel at which the user can select the MIDI In and MIDI Out ports. When [CTRL] is selected, a new menu will appear at which the user can identify an incoming MIDI control number and assign it to a specific AMIS function. When the animation is active [GRAPH], each output sound event generates a graphical line proportional in colour and size to the output frequency (see appendix 4). If [help] is pressed, it will change to a red colour. Then by moving the mouse upon the description of an AMIS function, an explanation of this function will appear on the main panel. Finally, the two vertical bars surrounding the panel represent in any given time the frequency and the density of the cloud (see Appendix 6).
FINAL THOUGHTS, FURTHER FUTURE IMPOVEMENTS
AND POSSIBILITIES
As discussed, Granular synthesis is a very powerful and creative method for generating and organizing sounds. The grain can be considered as the most basic element of larger audio events and the idea to control sound in a micro level is both very creative and creative at the same time. Through Granular synthesis we are able to understand other methods of synthesis, as different Granular synthesis approaches employ techniques found in many other synthesis methods. In a way, Granular synthesis can be seen as a form of additive synthesis or even subtractive synthesis (Pitch-synchronous GS). Asynchronous Granular synthesis is capable to produce very rich sounds and although is driven by a large amount of control data the user can control the output of the grains with very few parameters, like in the AMIS interface.
Concerning AMIS, is an ongoing software project. Although the current version (v 0.1) provides all the necessary controls to perform Asynchronous granular synthesis of MIDI data successfully, much could be further improved in the future. The core MIDI implementation of Granular synthesis in AMIS has already been conceived in CPS. Further changes could be made on the top of the current implementation without disturbing the initial AMIS algorithms.
The Shockwave visual interface could also be completely redesigned in the future to follow more abstracts representation of the AMIS algorithms. This is relative easy to be made as already all AMIS functions (CPS patch) have been connected with a visual representation in Director. The current interface although passes enough information to the user it can be improved. For example, the two vertical bars surrounding the main panel providing information about the current frequency and density of the cloud could also provide at the same time a visual representation of other AMIS functions. A switch could toggle between different representations. Moreover, the main animated representation of the cloud with lines could also be developed to something more visual attractive and more representative of the AMIS cloud. I have already conceived an interface as such but unfortunately I was not able to optimize it for use in this version. Probably the largest weakness of the AMIS interface relates to the way the Automation menu is arranged. Professional music software uses a graphical representation of the envelope of the automation. This makes possible for the user, instant changes of the automation envelope by introducing different break-points within the automation curve. AMIS does not use a graphical representation of the automation, but a numerical one. A break-point is introduced by the user whenever he/she interrupts the automation by changing a slider value. The automation itself could also be expanded to include more AMIS operations such as the grain duration or the volume. This along with other similar issues may be improved in future versions. A further drawback of the AMIS interface is that is limited to only two midi channels. In turns, this limits the real-time generation of the MIDI streams to only two. Furthermore, there is not total control of the second channel as it is dependant of the values of channel one. This has been made deliberately to simplify user control of the instrument. However, a future version might include complete control of more than two channels at the same time, thus expanding the waveform palette of the AMIS cloud. Finally, two major developments have been considered for the future.
Probably the larger limitation of the AMIS interface concerns its limited integration with major music environments (sequencers) through VST or other similar technology. This would have been made possible with the transportation of the AMIS algorithms into a different programming language like C++ or Borland Delhpi and would allow the function of AMIS as an audio plugin in a local software host. Through this environment, the exchange of music information through a network like the Internet would be also possible, something I am considering to implement in the current Shockwave version. This will permit the control of AMIS by a remote computer through a network and eventually the exchange of MIDI data between them. In this way, someone would be able to perform the AMIS algorithms on a MIDI device without necessarily having physical access of the connected device itself. This is relatively easy to implement by using the OSCparam CPS object which sends and receives parameters on a network through the OpenSoundControl (OSC) protocol. This future behaviour of AMIS will possibly appeal to musicians and composers with an interest to further explore Granular synthesis through real-time music performances based on a network.
Despite all considerations, the current final version of AMIS is a unique, solid, and musically supportive idea for composing granular music clouds in real-time from your Internet browser. The software can be found on the Internet at http://www.sonicspace.org/preamis.htm as a very small free Shockwave movie download (around 100Kb) and it is accessible to anyone wishes to experiment with Granular sonic clouds. I hope with my effort to have resonated some interest about the Granular representation of sound and help in a certain way for a further integration of GS in electronic music, so much need it as a counterpart to the unnatural smoothness and joyless predictability of many commercial music tools.
©sonicspace.org 2005