Interactive Shockwave

Click to execute a jig
Heap of Arteroids 1

Click to execute a jig
Heap of Arteroids 2

Click to execute a jig
A wee jig

Click to execute a dffierent jig
Green Heaps of Home

Click to launch dbcinema
dbcinema (visuals engine)

Other Interactive Audio Work

Click to launch Nio
Nio (2000)

Click to launch Oppen Do Down
Oppen Do Down (1999)

Click to launch War Pigs
War Pigs (in progress)

Click to launch Arteroids
Arteroids (2001-2004)

Click to launch Enigma n^2
Enigma n2 (2003)

Click to visit Vismu page
All published interactive audio works plus my essays on the subject

Click to visit links page
Links to works by others of interactive audio for the Web

CV

Click to view CV
Very hypertexty CV

Jig-Sound

by Jim Andrews

Technical Requirements

Video Introduction (5:30)

Pick one:

Overview

The Jig-Sound project will be a new type of album: an online jukebox of interactive Heaps of music. A jig-sound is sort of like a jig-saw only with sound. Heap is a term I've chosen to describe a combinatorium of sounds (usually loops) to be experienced via interactive sequencing and/or layering the sounds by the audience.

I am fascinated with the idea of Heap as musical form and with creating software for an online audience to experience and play heaps. Normally when we hear music, we hear a piece from beginning to end. We don't tinker with it except in our head and possibly we harmonize with it and drum the dashboard, fridge etc. The Jig-Sound project is an attempt to both build some software and create some music that helps us develop the notion of the Heap as musical piece/form for an online audience. As musical piece that we experience by exploring the Heap's compositional/jig possibilities. Also, the software that frames the Heap will allow the audience to 'play along' (via keyboard and mouse) with their own and others' compositions built from the Heap.

The challenge of creating heaps that possess strong and diverse compositional possibilities is something I'm trying to work on concerning the music. Concerning the software, it has to be unusually good in the synchronization it maintains of dynamically layered and sequenced rhythmic sounds,1 and it has to be fun, provoking, and easy to use by the audience. I invite you to check out both the music I'm making and play with the software to get a feel for the nature of it as musical experience and meditative, compositional activity.

The Audio Software

Jig-Sound will provide people with a new way of experiencing music online and on things like CDs and DVDs. There's a 'game mode' where you are timed on how fast you can put a song back together again, and your score is uploaded to the Internet. But there's also a 'play mode' where the goal is simply to sequence and layer loops of music in ways that you find interesting2—a bit like playing with sonic building blocks. And you can share your jigs/constructs with others either via uploading them or emailing them to friends.3 You can also save your jigs locally for later recall.

Jig-Sound is meant for a general audience from seven-year-olds to adults. It isn't only for musicians. It's a way to experience music, not primarily a way to compose music. Jig-Sound takes a kind of cubistic approach to the experience of music. Cubism offers multiple simultaneous perspectives. Jig-Sound gives you insight into the loop-based structure of music and encourages you to think 'what if' compositionally. 'What if I just move this part around and change the order here?' Jig-Sound also is a kind of instrument itself. You'll be able to play sounds using the mouse and keyboard, not simply put them together like building blocks and then play the structure you make.

Jig-Sound's audio programming is unusually good for a work of art. It synchronizes simultaneous sounds and makes consecutive sounds sound continuous. There are no audio gaps and the synchronization is tight. Usually you only see this sort of programming in commercially available audio production tools. Consequently, the artistic form of the rhythmic heap has been little explored in works of digital art, and the exploration Jig-Sound enables of rhythmic musical forms is part of its unique artistic and technical achievement.

There are many works of digital art in which interactive music is developed in all sorts of ways. But if the software does not permit synchronized layering and sequencing of heaps, then the territory it's exploring—and the types of music it can treat—are distinctly different from Jig-Sound. If you think of the operations you can perform on rhythmic music in audience-interactive music software you've encountered, you'll realize that either rhythm is not supported and the operations become rich, or rhythm is maintained and the interactive operations are usually either layering or sequencing but not both. Jig-Sound supports both.

You can also copy sounds or structures you've created and link them together. The constructive tools of Jig-Sound are quite powerful but very simple to use.

Also, the object-linking paradigm of its visual interface is a departure from the typical grid present in just about all programs where one may construct rhythmic music. The Jig-Sound paradigm is more like a flow chart than a temporal, channeled two-dimensional grid. Jig-Sound encourages a view of a piece of music as a lattice of loops linked in temporal sequence and in layers of intertwined rhythms. It also encourages you to see a piece of music as a network of sounds through which the music flows like blood through the body or like data through the Internet.

The Music

The Shockwave pieces at top left of this page give you a sense of the music that will form the Jig-Sound album. It's vocal music by me. The percussion is finger-snapping. And other tracks will be of songs by The Laughing Boot Quintet, a band I was the drummer and audio producer of.

Software art often risks disappearing into the technology. The Jig-Sound project is sophisticated technologically, but the music for the album is not particularly electronic. The Jig-Sound technology does not require high-tech music.

You can listen to another couple of heaps I've made, which will also be part of the album, in Nio and Oppen Do Down. Like my other heaps, these never were linear songs. They're specially composed heaps for being layered and sequenced.

How to Play

The best way to introduce you to the project is for you to visit vispo.com/jig and play around a bit. What follows in this section is how to play in the interface-in-progress.

Click to launch Jig-SoundYou see a bunch of sound icons like A. Each of these icons represents a sound. Click an icon to listen to its sound. This Heap of 17 sounds are ones I sang, recorded and mixed in Sonar, and smithied in Sound Forge. These sounds never were part of a linear song. They were specially made as a Heap. I am writing the software for this also. In Director.

When you mouse-over a sound icon, the cursor changes into a hand †or a pencil . When it's a hand, you can click and drag the icon around the window. When it's a pencil, click and drag to link sound icons. You can link sound icons in two different ways.

You can link icons with a green line when you click on the top or bottom of an icon and drag. Two icons linked by a green line play simultaneously. When you click an icon A that has any green links, that icon's sound plays and so do the sounds of icons green-linked to A. You can link as many icons together as you like with green-links. However, Director only has 8 stereo channels (each of the 8 is stereo); you can only play at most 5 sounds simultaneously in a jig-sound. Two of the remaining channels will be for keyboard playing of sounds (not written yet) and the final channel is reserved for stuff relating to sound synchronization. Basically one of the channels is always playing silence with a cuepoint in it so I can synchronize the channels.

You can also link icons with a blue line. You create a blue line when you click on the left or right side of an icon and drag. Two icons linked by a blue line play sequentially. In the graphic to the right, icon A plays first. Then icon B plays. Sound flows out of the right side of icons, through blue lines, and into the left side of blue-linked icons.

  • Icons whose sound is currently playing are orange. The most recently clicked icon (if it isn't playing) is blue-tinted. This is called the selected icon.
  • When the 'loop' button is depressed, which it is by default, then when a Jig-Sound reaches the end (if it has an end), it will loop back to the selected icon.
  • To delete a line you've created, click in the area where it joins an icon.
  • To select a bunch of sound icons, click the black background and drag a rectangle around the icons you want to select. You can also Shift+click an icon to add it to the group of selected icons.
  • To drag a bunch of selected icons, click and drag in the blue rectangle you've created (but don't click one of the selected icons).
  • To deselect a bunch of icons, click on the black background or on an icon. You can deselect a single icon from the group of selected icons by Alt+clicking the icon.
  • To copy a bunch of selected icons, click the 'copy' button.4
  • To delete a bunch of selected icons, click the 'del' button. Only icons of which there are more than one copy will be deleted.
  • The 'stop' button stops the audio. Click an icon to resume.
  • The volume control adjusts audio volume.
  • To swap the location and links of two sound icons, press and hold the Ctrl key (Command key on Macs). Then drag and drop one of the icons onto the other.
  • The icon that has ō as its symbol is the 'silent sound'. This icon behaves like a regular sound icon but is silent. It is useful when you want to blue-link to an icon A but the left blue connector of A already has a line attached to it. In this case, you just green-link A to the silent sound (or green-link the silent sound to any icon connected to A by a green link) and then blue-link to the silent sound.
  • To move between play mode and game mode, click the Jig-Sound logo at top left.

Game Mode: Clock that Jig

What we just looked at is part of 'play mode'. There will also be a 'game mode'. I'm still working on 'game mode'. I'm not sure how fun it is at this point. I think it needs work (so did another game I made, early on: Arteroids). But the basic idea of 'game mode' is to present something that is more like a jig-saw than what you experienced in 'play mode' but, unlike a jig-saw, you are timed on how fast you put the puzzle together, and scores will be uploaded to the Net (with the player's permission).

To try out the basic idea of 'game mode', click the Jig-Sound logo at top left of vispo.com/bc/a. After you click the Jig-Sound logo, you should see 22 sound icons that look a bit different than the previous ones. These ones just have two connectors, not four, as illustrated by the graphic in this paragraph. In 'game mode', you can't draw green lines. In other words, you can't layer sounds. You can only sequence them. Neither can you copy icons. I took the copy button out because it doesn't help you solve the puzzle.

The idea in this demo is to put Where Did Our Love Go? by Diana Ross and the Supremes back together again. I won't be using other peoples' music in the final version, but this gives you the idea of 'game mode'. The puzzle is completed when the 22 sound icons are linked together and all the lines between the sound icons are red, as shown below.

If two sounds are not consecutive in the original piece of music, then lines between the corresponding sound icons are blue. Whereas if two linked icons were consecutive in the original music, the line between them is red. This is how you know whether the 'puzzle piece' fits or not.

This puzzle is kind of hard without the cheat I've provided you. This should probably be level 3. Level 1 should probably contain about 6 sounds. The six sounds would be sufficiently longer than these so that the full song plays in the six sounds. Then level 2 would consist of 11 sounds. Level 3, as above, would consist of 22 sounds. Level 4 would consist of 44 sounds. Cut a good loop in two and you get two good loops (down to a certain level of time).

A Song in Play Mode (the Green Heaps of Home)

Here is another 'play mode' Heap: The Green Hills of Home. The music in this piece is by The Laughing Boot Quintet. I was the band's drummer and recording technician.

The sound icons are labelled in the order in which they occur in the original song; in other words, you can hear the original song by blue-linking all the icons together alphabetically.

Click to launch 'The Green Hills of Home'.The sound icons in this Heap are color coded to indicate which sets of icons can be layered amongst each other with some success. The particular group I'm using here (o-v) are pretty good layered together, in most instances. The music in the final project will consist of† Heaps by me and a few from music of The Laughing Boot Quintet.

Jig-Sound Visuals: Dbcinema

Click to launch dbcinemaDbcinema, a piece I've written in Director, will supply the visuals for the Jig-Sound project. Dbcinema fetches images from Google and Yahoo image searches.5 In the Jig-Sound project, the current Heap will feed dbcinema language that is relevant to the Heap so that images related to the music will be displayed in the background of Jig-Sound. You can see that dbcinema is already quite advanced in its speed and range of features.

However, it's currently more like dbslideshow, nonetheless. The way I will take it to dbcinema is via processing of the images. In other words, rather than simply displaying one image after another, as it does now, it will use the images as content for full-screen collage and abstract visual generation algorithms that can also be in rhythm with the music.

To get a sense of the generative abstract visuals I have in mind, have a look at some of the pieces at complexification.net . This is a site by Jared Tarbell. And imagine collaged images being revealed rather than what Tarbell reveals, which is simply colour. Tarbell's work is done in the language Processing. Mine will be done in Imaging Lingo (in Director).

Usually when generative computer visuals are successful, they have a strong abstract dimension. But usually they are weak representationally. dbcinema can be strong both in its representational dimension (it already is) and also in its abstract image processing dimension (still to be written).

I'm hoping this will be an interesting alternative to the music video, and to the sort of totally abstract visual accompaniment you see in Winamp or Windows Media Player etc. Typically the latter simply present abstract patterns in response to audio amplitude. Music videos, as a genre, somehow never quite blossomed into art, or at least not the commercial variety you see on Much Music or MTV, usually. Clearly there is lots of unexplored territory concerning the visual representation of music.

The audience will be able to use dbcinema as you can now, or they can choose to leave it alone and let the heap feed dbcinema language. The audience will also be able to select among the algorithms that will power the collage/generative visuals.

Some of those algorithms will work in rhythm to the music.

Luca Barbeni (Italy), in his curation of "Unlimited Cinema," said of dbcinema:

Artist, programmer and theorist Jim Andrews is one of the pioneers of online audio-visual experimentation. With dbcinema, launched in 2005 and still in progress, he is part of a line of research that, starting from Duchampís ready-made works, nowadays finds an interesting realization in works of art that use Google as a support and tool. Our favourite search engine is becoming the most popular media for retrieving images and sounds.

Part of the attraction of dbcinema, to me, is the way it presents a kind of collective impression of the concepts you type into it. The images range from professional photos to personal and idiosyncratic gestures to online menu graphics. The aesthetic of dbcinema is more from software art than art school. Also, it isn't simply about a look as an interactive process of visual generation. The audience interacts with dbcinema and dbcinema interacts with the collective humungous image databases of the Internet.

The language a particular heap feeds dbcinema will be selectable by the audience. For instance, if I were to use "Where Did Our Love Go?" by the Supremes (which I'm not going to do--the album will consist of music by me and the Laughing Boot Quintet), the audience might find the below:

They could then select among the search terms or let Jig-Sound feed this language to dbcinema at its own pace. Or the audience could type in their own choice of language to generate the visuals. The audience will also be able to click on images to open the web page from which the image was drawn. So the experience is not only one of interactive composition of the music, but selection or generation of the language used to generate the visuals, and possibly also browsing of web pages from which the images were drawn.

Still to be Done

As a form of music, there's some exploration to be done. Jig-Sounds can deal nicely with any song in 'game mode' where layering is not allowed. But 'play mode', since it involves layering, not only sequencing, is a more creative challenge to make audio for.

I have made/sung several Heaps in the past, two of which you can interact with in Nio and Oppen Do Down. I want to make four new Heaps of original music for this project. I also want to use songs by the Laughing Boot Quintet in this project. I will be working with the other members of the band on those songs and also using recordings.

In 'play mode', you will be able to save jigs you compose locally or upload them to the Net for others to download and listen to or edit. Saved jigs are just text files. Configuration information. Not sound files. So they are quite portable; they're relatively small text files rather than 4 Mb MP3 files6. Also, people will be able to email their friends a composition they're working on which the friends can listen to and/or edit. This functionality is a mixture of native Director coding plus PHP programming, which is server-side programming. Martin Kloss does the PHP in my work and provides me with a Lingo API to the PHP code. A piece in progress called War Pigs was Martin and my first go at combining Lingo and PHP. In 'play mode' of War Pigs, you can download compositions by others and upload your own. And game mode in War Pigs uploads scores to the server. So we have most of the Lingo+PHP worked out already. The email component needs to be done from scratch, though.

'Play mode' is more creative than game mode. You're not competing. The sounds you can use are not just cut-up songs but sounds composed for Heaps, usually.

Jig-Sound will be one integrated application. You'll be able to access multiple heaps, but that access will be within one application.

There will be a menu that lets you select whether you want 'game mode' or 'play mode' and you will have a choice of Heaps in either mode. Once you choose a Heap, it will be downloaded from the server using Deepdown, a downloader I've created in Director. You'll have access to sounds as they stream in. Currently you have to wait for the whole thing to download but that will change.


Partial Jig-Sound menu structure

You'll be able not only to engage in meditative compositional activity or game activity but also you'll be able to play the thing sort of like an instrument. The keyboard will be used to trigger sounds and you'll be able to adjust the size of the loops and other parameters of the sound (startTime, endTime, loopCount, volume, and synchronization).7

I want to make the visuals of the sound icons dramatic. Animated when playing. And the music should seem to flow through the construction, so there should be animation of the lines joining the sound icons to represent the music flowing through the construction.

We have seen the business of music changed by the Net via P2P. But we haven't heard or seen change in music itself via the Net's influence as dramatically, though this work is going on, not only by myself, but by others mostly working below the mass-media radar, for the moment. Jig-Sounds are about creating different ways of experiencing music online, and creating new forms of music that consist of Heaps. Over the last six years, I've developed the software required for this sort of experience. Now I need a year to put it all together and create more music. The result will be an album of music like none other and, hopefully, the start of a new form of interactive music.

Contact Information

Jim Andrews
2313 Esther Pl
Victoria BC
V9B 2E5
Canada

Phone: (250) 478-7727
Email: jim@vispo.com
CV: vispo.com/JimAndrews.htm
Site: vispo.com

Footnotes

[1] Jig-Sound's ability to synchronize multiple (green-linked) sounds on the fly and its ability to make sequenced sounds sound continuous is kind of special technically. You don't see much of that sort of thing in interactive audio art because it requires specially devious programming. An article I wrote in 2001 about audio programming is now part of the Adobe Director documentation, and Jig-Sound's abilities in this regard are way beyond the 2001 work. The main improvement concerning synchronization is that the files don't have to be the same length or even multiples of one another in length. I have been thinking about audio synchronization programming since 2000.

[2] Two of my other pieces, Arteroids and War Pigs, have both a 'game mode' and a 'play mode'. Mainly because I find that when I'm developing a piece, some of the features are appropriate to a competitive situation and some aren't. Also, having both provides the piece different cognitive and semantic levels of play and different but related activities. The classic example from the realm of computer games of 'play mode' is where players can create their own 'skins' of games.

[3] The way this will work is as follows. You'll send the email from the Jig-Sound interface. The friend receives an email containing your message plus a URL generated by Jig-Sound. Jig-Sound will have uploaded your jig to the Jig-Sound server when you sent the email. A jig is just textual configuration information, not sound files, so the upload is brief. When the friend clicks the URL in the email, that starts Jig-Sound and your jig is downloaded from the Jig-Sound server and displayed in Jig-Sound for your friend to play/edit. Six months after you sent the email, the jig is deleted from the Temp directory of the Jig-Sound server. But that's OK because it will have been saved to your hard drive and perhaps you also uploaded it to the Permanent directory of the Jig-Sound server. Also, it will have been saved to the friend's hard drive if the friend did not explicitly choose to delete it.

[4] The ability to copy sound icons relies on a piece of software I wrote and sell online called Windows for Shockwave (WFS). WFS is the only commercially available package that enables easy sprite creation/destruction in Director. Even with this feature, Director itself imposes a maximum of 1000 simultaneously instantiated sprites. So Jig-Sound is not an industrial-strength exe made out of C++. If Jig-Sound were to be developed later on as a tool, it would have to be ported to something like Visual Studio or Delphi and developed in C++ or Pascal or whatever as an offline app; Director is for art apps. I have no plans at the moment to turn Jig-Sound into a tool.

[5] I have generalized the programming of dbcinema so it won't be too hard to add other image sources such as Altavista and possibly Flickr. You'll also be able to point it at specific files and have it spider linked pages. And you'll be able to save particular searches you do where you choose which images are saved, since you usually don't want all the images dbcinema displays.

[6] Not part of this proposal but in-the-works-down-the-road will be the ability to jam/jig online with other people. This possibility arises from the fact that the only information that needs to be sent between people is configuration information, ie, textual configuration information, rather than sound files. Little bits of configuration information can be sent very quickly. This is how online games such as Doom and Counter Strike manage to make the multi-player real-time play as quick as it is.

[7] In the final version, when you right-click (Control+click on a Mac) an icon, you'll be able to open a window that lets you edit startTime, endTime, loopCount, volume, and synchronization concerning all sounds that play simultaneously with the sound icon you right-clicked. If you're familiar with the program Acid, the sort of window I have in mind here will be nothing especially new.

The differences between Jig-Sound and Acid are many. The main difference is that Acid is a general purpose audio production tool. You import your own music into Acid. Whereas Jig-Sound is not meant as something you import your WAV files into. Mainly Jig-Sound is meant as an interface that makes exploring the compositional possibilities of a heap easier than it is in Acid. And Jig-Sound will be an interactive album, a work of art, not a general purpose audio production tool.