NeoNio and Nio by Jim Andrews

NeoNio is a 2021 HTML5 upgrade of the 2000 Director interactive music piece mentioned by Matthew Mirapaul in the New York Times in his column on the best net art of 2001. The artistic aspiration of Nio is toward a new interactive, visual and combinatory form of music: the heap. A heap is a collection of sound icons. Each icon has a sound and an animation/video. The player can hook them together so that they play either simultaneously or one after another. In other words, the player interactively layers and sequences the sounds and animations.

At its best, a heap is not simply a song chopped into perfect loops of the same length. The sounds are created to be interactively hooked together in a range of sequences and layers. Not everything has to sound good with everything else. And, unlike Nio, not all the sounds have to be the same length.

And how do the accompanying visuals layer and sequence? These are all interesting issues toward making strong combinatory heaps. This interactive and visual form of music is for the very hands on, for those who enjoy interactive art and a creative approach to listening to music. Not everyone does, I realize. Some people, like Chance the gardener, prefer to watch. And that's OK. For those people, future heap-oriented software must work well in several modes: play mode; audience mode; and possibly also game mode (Jig-Sound, which I made after Nio, has a game mode).

The first step toward this dream, in 1999, was works such as Oppen Do Down—which has also been upgraded to HTML5. You can layer the sounds but you can't sequence them. And the visuals aren't much to speak of. Nio was the next step in that you could not only layer the sounds but also, in verse 2 of Nio, sequence them. Also, the visuals in Nio are more interesting and in synch with the sound. The third step in the evolution of the heap was Jig-Sound, which freed the icons from a grid.

I'm hoping to step beyond Jig-Sound, which doesn't have visuals, next. I want to create an HTML5 version of Jig-Sound that works well on mobile devices, has visuals, and the player's work is saveable and shareable. And you can import your own sounds and visuals. Sort of like you can import your own images into Aleph Null, another of my HTML5 net-art works. Actually, after I created Jig-Sound's audio functionality, I wanted to create visuals for it. So I created dbCinema. But that led me into generative visual art, and I never got back to Jig-Sound. Not only cuz I was captivated with generative visual art but because that was around the time Director died and the Web Audio API wasn't happening yet, whereas the HTML5 canvas was ready to go. But I still have this desire to bring the idea of the heap to fuller fruition via both software to play heaps and interesting musical and visual heaps themselves.

NeoNio works on pretty much all browsers on desktops, laptops, and also with touch on mobile devices for Android and iOS.

It's been fascinating to me to construct NeoNio. Natalie helped a lot with the animations. The original 16 were made with Flash, exported from Flash as SWF files, and imported into Director. But SWF no longer works on the web. I looked high and low for SWF-to-HTML5 converters and was pretty happy with SWF2js.js, a free bit of JavaScript software that did the conversion to HTML5. But it didn't work on iPhones or iPads. So Natalie took it upon herself to convert the SWF to SVG animations. She runs, among her several computers, an old Mac that has Flash 5.5 on it and a Flash plugin that converts FLA to SVG. Thanks, Nat.

NeoNio uses the Web Audio API. And it works really well on so many different devices, which is gratifying.

There are quite a few differences between the original Nio and NeoNio. Nio was of a fixed size; NeoNio's animations resize to fit the largest square that can be drawn on the screen. That's the biggest visual difference. The biggest sonic difference is that, in verse 2, in NeoNio, when you click a sound-icon, it will be in synch with what's playing, whereas in the original, you had to click at the right moment for it to be in synch. Also, verse 2 in NeoNio has a stop/start button. And when you click the Bonk button in NeoNio, you see the icons randomly relocate to different grid holes, whereas they just instantaneously changed position in Nio. Also, the bg colour slider in NeoNio works differently than in Nio; there's a random factor and it changes different bg areas. Also, NeoNio does not need the 'synch button' that appears in Nio cuz the sounds never get out of synch in NeoNio for more than one loop.

Another difference between Nio and NeoNio is that NeoNio has good keyboard interactivity in verse 1. Nio had no keyboard support. NeoNio's keyboard support is of two types. As you see in the below graphic, the letter keys can be used as a kind of instrument. Also, one may tab to the 16 sound icons and turn them on/off with the Enter key. The goal is to make verse 1 accessible to people who use screen readers. Verse 2 is tougher that way cuz of the drag and drop.

NeoNio verse 1

NeoNio verse 2 on Windows desktop browser

I've written some of the NeoNio code so that it's reusable by other people.

ZIndexMgr.js is an open source JavaScript object that's useful when you have DOM elements whose z-index value needs to change during the program. It's especially for managing the zIndex of the main layers of the app. But it's useful for other things too. In NeoNio, I create three instances of ZIndexMgr. One of them manages the main layers/windows (verse 0, verse 1, verse 2, and the logo, which is always topmost). Another instance manages the zIndex of sound icons in verse 2; when you drag a sound icon, it becomes topmost among sound icons. The third instance manages the zIndex of the 16 animations in NeoNio. When an animation becomes visible, it becomes topmost among the animations. I think the documentation is pretty good. Read it to use ZIndexMgr.js.

DragElement.js is an open source JavaScript object that's useful when you want an element to be draggable (mouse and touch). You can also specify a different element within which the dragged element remains, is caged, cannot escape. You can also deactivate and reactivate draggability. I use DragElement in NeoNio in verse 2 to make the sound icons draggable.

TwoStateButton.js is an open source JavaScript object that's useful when you want a button that works with mouse, touch and the keyboard (accessible). I use TwoStateButton extensively in NeoNio. The sound icons arranged in a circle in verse 1 use TwoStateButton. Also, some of the buttons in both verse 1 and 2 use TwoStateButton.

There are other NeoNio js files, such as Sounds.js, SimpleButton.js and TwoStateButton.js, that may be useful to others. I will add links here as they become more useable.

Nio is a terminally-ahead-of-its-time interactive audio/visual piece I created in 2000 with a commission from turbulence.org in New York. I wrote it in Director. It is now available as a 32-bit executable for Windows. For the real experience, download it, unzip it, and double click on the exe to start the interactive app. To exit, press the Esc key.

Here's a video about Nio and another interactive audio piece I did called Jig-Sound.

Here's some pics concerning the project.

Nio verse 1

Nio verse 2

Nio source code in Director MX 2004

Here is the old Nio home page with lots of links and images to explore, including the Director source code, the essay Nio and the Art of Interactive Audio for the Web, Song Shapes, and more.

Interactive-sound projects, which are programmed to produce music in response to a visitor's actions, are becoming more sophisticated. Doo-wop fans will adore Jim Andrews's ''Nio,'' at vispo.com.

Matthew Mirapaul in The New York Times article ARTS ONLINE; Driven by a Higher Calling, Not Dot-Com Dollars, Dec. 24, 2001.

Nio, like Jim Andrews's earlier visual poetry, explores the idea that poetic experience is not the exclusive territory of words and lines. The work casts the interactor in the position of composer, mixing and sequencing doo-wop voice audio samples and corresponding lettristic animations that swirl about in three dimensions. Without providing any form of explicit representation, the piece creates a playful environment for evocative synesthesia.

Electronic Literature Org Collection 1.

Jim Andrews’ work Nio presents the reader with a complex aesthetic experience that makes use of phonemes and letters but not of words. Andrews’s piece is a cross between a sound poem, kinetic visual art, and an interactive musical instrument. In two verses, Andrews provides the reader with two different ways of mixing clusters of letters, each of which have a musical voice track attached to them. In the first verse, those clusters of letters then do a kind of animated dance in the center of a circle as the voice loop they signify is sung. The loops are layered on top of each other, allowing the interactor to compose a shifting doo-wop melody/animation. In an accompanying essay, ‘Nio and the Art of Interactive Audio for the Web’, Andrews explains that he’s ‘trying to synthesize and transform image, sound, and text, not simply juxtapose them. I seek some sort of critical mass to fuse them’. He describes the work as a ‘synthesis of literacies’. In Nio and in much of his other work, including his visual poetry, Andrews attempts to rethink the relationship between poetry and language, creating interactive poetic experiences that utilize texts of various kinds that don’t rely on words to provoke a response from the reader. Letters in motion and the human voice alone, devoid of explicit denotation, can impart a great of emotional and semantic content. Nio is proof of the idea that poems needn’t be composed of words in order to be poetic and evocative.

Scott Rettberg, from
Dada Redux: Elements of Dadaist Practice in Contemporary Electronic Literature

Jim Andrews’ Nio (2001) is a digital “lettrist” poem that not only combines different medial processes, but also merges art with technology and technological applications. Here, as in much other digital poetry, the concept of play has pride of place as a bodily (re- )activity: Nio only materializes in a ‘ludic’ interaction with the reader/user. Displayed as a circle of icons issuing images and sounds, Nio’s design and appearance is to a certain extent dependent on my actions and interferences as a reader/player: the icons I bring to live participate in a dance of letters that change their shape with every new addition or deletion, the music changing only minimally in its repetitive gestures. If Wallace Stevens once claimed that «poetry is the subject of the poem», Nio performs this quite literally as the constant (re-)creation of lettrist shapes acting as the protagonists of the poem.

Kiene Brillenburg Wurth from
Multimediality, Intermediality, and Medially Complex Digital Poetry

Jim Andrews, who gives us the term "langu(im)age," works with individual letters which he can animate and overlay with sequences of sound loops in his Nio engine. He is not primarily, or at all, concerned with providing a reading experience. He says:

Much of my work is lettristic in the sense that rather than working with words and extended texts, I work with individual letters. Part of my attraction to working this way is philosophical and sonical... but part of it is also out of interest in treating literary objects/material, and individual letters are quite well suited to such treatment. Individual letters are graphically more interesting than whole words... [they] take up less memory, and are thereby manipulated more quickly. And they spin nicer than words do, for instance, because of their shapes. There is more variety in their shapes than there is in words. And they are quite mysterious to me. Geometry and basic architectures of language. (E-mail to Webartery list, February 10, 2001, on thread "re: teaser 2" [2001b])

Later, he says: "... it's really when you get down to the word and the letter, rather than the paragraph, that language cracks open and code spills out" [Webartery list, February 24, on thread "checkout counter"]. One feels the difference from the stenographic model, a model of accommodation rather than breaking and entering. But one also feels, and can sympathize with, an attraction to a different arena, the world of the purely sonic and visual, where compounds stay themselves and are thereby experienced more fluidly. However, oscillation does occur in Andrews's Nio, between the visual and the sonic elements, and this oscillation is elegant, playful, and deeply pleasing.

Stephanie Strickland from Moving Through Me as I Move