is a 2021 HTML5 upgrade of the 2000 Director interactive music piece mentioned by Matthew Mirapaul in the New York Times in
column on the best net art of 2001. The artistic aspiration of Nio is toward a new interactive, visual and combinatory form of music: the heap. A heap is a collection of sound icons. Each icon has
a sound and an animation/video. The player can hook them together so that they play either simultaneously or one after
another. In other words, the player interactively layers and sequences the sounds and animations.
At its best,
a heap is not simply a song chopped into perfect loops of the same length. The sounds are created to be
interactively hooked together in a range of sequences and layers. Not everything has to sound good with
everything else. And, unlike Nio, not all the sounds have to be the same length.
And how do the accompanying
visuals layer and sequence? These are all interesting issues toward making strong combinatory heaps. This
interactive and visual form of music is for the very hands on, for those who enjoy interactive art and a
creative approach to listening to music. Not everyone
does, I realize. Some people, like Chance the gardener, prefer to watch. And that's OK. For those
people, future heap-oriented software must work well in several modes: play mode; audience mode; and
possibly also game mode (Jig-Sound, which I made after Nio, has a game mode).
The first step toward this dream, in 1999,
was works such as Oppen Do Down—which
has also been upgraded to HTML5. You can layer the
sounds but you can't sequence them. And the visuals aren't much to speak of. Nio was the next step
in that you could not only layer the sounds but also, in verse 2 of Nio, sequence them. Also, the visuals
in Nio are more interesting and in synch with the sound.
The third step in the evolution of the heap was Jig-Sound, which
freed the icons from a grid.
I'm hoping to step beyond Jig-Sound, which doesn't have visuals, next. I want to
create an HTML5 version of Jig-Sound that works well on mobile devices, has visuals, and the player's work
is saveable and shareable. And you can import your own sounds and visuals. Sort of like you can import
your own images into Aleph Null,
another of my HTML5 net-art works. Actually, after I created Jig-Sound's audio functionality, I wanted to
create visuals for it. So I created dbCinema. But that led me into generative visual art, and I never got
back to Jig-Sound. Not only cuz I was captivated with generative visual art but because that was around
the time Director died and the Web Audio API wasn't happening yet, whereas the HTML5 canvas was ready
to go. But I still have this desire to bring the idea of the heap to fuller fruition via both software
to play heaps and interesting musical and visual heaps themselves.
works on pretty much all browsers on desktops, laptops, and also with touch on mobile devices for Android and iOS.
It's been fascinating to me to construct NeoNio. Natalie helped a lot with the animations. The original 16 were made with Flash,
exported from Flash as SWF files, and imported into Director. But SWF no longer works on the web. I looked high and low for
HTML5. But it didn't work on iPhones or iPads. So Natalie took it upon herself to convert the SWF to SVG animations. She
runs, among her several computers, an old Mac that has Flash 5.5 on it and a Flash plugin that converts FLA to SVG. Thanks, Nat.
NeoNio uses the Web Audio API. And it works really well on so many different devices, which is gratifying.
There are quite a few differences between the original Nio and NeoNio. Nio was of a fixed size; NeoNio's animations resize to fit
the largest square that can be drawn on the screen. That's the biggest visual difference. The biggest sonic difference is that, in
verse 2, in NeoNio, when you click a sound-icon, it will be in synch with what's playing, whereas in the original, you had to
click at the right moment for it to be in synch. Also, verse 2 in NeoNio has a stop/start button. And when you click the Bonk
button in NeoNio, you see the icons randomly relocate to different grid holes, whereas they just instantaneously changed position
in Nio. Also, the bg colour slider in NeoNio works differently than in Nio; there's a random factor and it changes different
bg areas. Also, NeoNio does not need the 'synch button' that appears in Nio cuz the sounds never get out of synch in NeoNio
for more than one loop.
Another difference between Nio and NeoNio is that NeoNio has good keyboard interactivity in verse 1. Nio had no keyboard
support. NeoNio's keyboard support is of two types. As you see in the below graphic, the letter keys can be used as a
kind of instrument. Also, one may tab to the 16 sound icons and turn them on/off with the Enter key. The goal is to make
verse 1 accessible to people who use screen readers. Verse 2 is tougher that way cuz of the drag and drop.
NeoNio Source Code
I've written some of the NeoNio code so that it's reusable by other people.
that's useful when you have DOM elements whose z-index value needs to change during the program. It's especially
for managing the zIndex of the main layers of the app. But it's useful for other things too. In
NeoNio, I create
three instances of ZIndexMgr. One of them manages the main layers/windows (verse 0, verse 1, verse 2, and the logo,
which is always topmost). Another instance manages the zIndex of sound icons in verse 2; when you drag a sound icon,
it becomes topmost among sound icons. The third instance manages the zIndex of the 16 animations in NeoNio. When an
animation becomes visible, it becomes topmost among the animations. I think the documentation is pretty good. Read
it to use ZIndexMgr.js.
that's useful when you want an element to be draggable (mouse and touch). You can also specify a different element
within which the dragged element remains, is caged, cannot escape. You can also deactivate and reactivate draggability.
I use DragElement in NeoNio in verse 2 to
make the sound icons draggable.
that's useful when you want a button that works with mouse, touch and the keyboard (accessible). I use TwoStateButton
extensively in NeoNio. The sound icons
arranged in a circle in verse 1 use TwoStateButton. Also, some of the
buttons in both verse 1 and 2 use TwoStateButton.
There are other NeoNio js files, such as Sounds.js, SimpleButton.js and TwoStateButton.js, that may be useful to
others. I will add links here as they become more useable.
End of NeoNio
Nio (2000) by Jim Andrews
Nio is a terminally-ahead-of-its-time interactive audio/visual piece I created in 2000 with a commission from turbulence.org in New York. I wrote it in Director.
It is now available as a 32-bit executable for Windows. For the real experience, download it, unzip it, and double click on the exe to start the interactive app. To exit, press the Esc key.
Here's a video about Nio and another interactive audio piece I did called Jig-Sound.
Nio, like Jim Andrews's earlier visual poetry, explores the idea that poetic experience is not the exclusive territory of words and lines. The work casts the interactor in the position of composer, mixing and sequencing doo-wop voice audio samples and corresponding lettristic animations that swirl about in three dimensions. Without providing any form of explicit representation, the piece creates a playful environment for evocative synesthesia.
Jim Andrews’ work Nio presents the reader with a complex aesthetic experience that
makes use of phonemes and letters but not of words. Andrews’s piece is a cross
between a sound poem, kinetic visual art, and an interactive musical
instrument. In two verses, Andrews provides the reader with two different ways
of mixing clusters of letters, each of which have a musical voice track
attached to them. In the first verse, those clusters of letters then do a kind
of animated dance in the center of a circle as the voice loop they signify is
sung. The loops are layered on top of each other, allowing the interactor to
compose a shifting doo-wop melody/animation. In an accompanying essay, ‘Nio
and the Art of Interactive Audio for the Web’, Andrews explains that he’s ‘trying
to synthesize and transform image, sound, and text, not simply juxtapose them.
I seek some sort of critical mass to fuse them’. He describes the work as a ‘synthesis
of literacies’. In Nio and in much of his other work, including his
visual poetry, Andrews attempts to rethink the relationship between poetry and
language, creating interactive poetic experiences that utilize texts of various
kinds that don’t rely on words to provoke a response from the reader. Letters
in motion and the human voice alone, devoid of explicit denotation, can impart
a great of emotional and semantic content. Nio is proof of the idea that
poems needn’t be composed of words in order to be poetic and evocative.
Jim Andrews’ Nio (2001) is a digital “lettrist” poem that not
only combines different medial processes, but also merges art with
technology and technological applications. Here, as in much other
digital poetry, the concept of play has pride of place as a bodily (re-
)activity: Nio only materializes in a ‘ludic’ interaction with the
reader/user. Displayed as a circle of icons issuing images and sounds,
Nio’s design and appearance is to a certain extent dependent on my
actions and interferences as a reader/player: the icons I bring to live
participate in a dance of letters that change their shape with every
new addition or deletion, the music changing only minimally in its
repetitive gestures. If Wallace Stevens once claimed that «poetry is the
subject of the poem», Nio performs this quite literally as the constant
(re-)creation of lettrist shapes acting as the protagonists of the poem.
Jim Andrews, who gives us the term "langu(im)age," works with individual letters which he can animate and
overlay with sequences of sound loops in his Nio engine. He is not primarily, or at all, concerned with providing a
reading experience. He says:
Much of my work is lettristic in the sense that rather than working with words and extended texts, I work
with individual letters. Part of my attraction to working this way is philosophical and sonical... but part of it is
also out of interest in treating literary objects/material, and individual letters are quite well suited to such
treatment. Individual letters are graphically more interesting than whole words... [they] take up less memory, and are
thereby manipulated more quickly. And they spin nicer than words do, for instance, because of their shapes. There is
more variety in their shapes than there is in words. And they are quite mysterious to me. Geometry and basic
architectures of language. (E-mail to Webartery list, February 10, 2001, on thread "re: teaser 2" [2001b])
Later, he says: "... it's really when you get down to the word and the letter, rather than the paragraph, that language
cracks open and code spills out" [Webartery list, February 24, on thread "checkout counter"]. One feels the difference
from the stenographic model, a model of accommodation rather than breaking and entering. But one also feels, and can
sympathize with, an attraction to a different arena, the world of the purely sonic and visual, where compounds stay
themselves and are thereby experienced more fluidly. However, oscillation does occur in Andrews's Nio, between the
visual and the sonic elements, and this oscillation is elegant, playful, and deeply pleasing.