BLUI™, a Body Language User

Interface for 3d gestural drawing

Bill Brody, Chris Hartman, University of Alaska Fairbanks,

Abstract

We are developing a system to implement gestural drawing in an immersive 3d environment. We present a virtual artist who draws expressive forms in virtual space. In the art world, the term 'gestural' commonly refers to mark making that derives from the richness of movement of the artist. This focus on the character of motion is much like a similar focus on follow-through in athletic activity. Accordingly, we base the appearance of the rendered image on the body language of the artist, hence the acronym BLUI: Body Language User Interface. BLUI is developed on the ImmersaDESK®, an immersive virtual reality environment where the artist wears head-tracking goggles and uses a wand. Information from video, wand, and head tracker is used to generate a virtual artist, whose brush tracks with the wand.

Keywords: Drawing, Body Language, Gesture, ImmersaDESK®

1. Introduction

Drawing is an act of imagination and creation. It is as symbolic as language. We are developing a new drawing instrument. We hope to use it to create novel works of art, perhaps defining a new art medium.

The physical body is the common ground to imagination, symbolic thinking and creativity. The mental activities listed mirror the effects produced in the brain by performing voluntary physical action. One cannot create new things without somehow activating the imagination and the powers of symbolic thought. Some artists speak of their work flowing unbidden from their fingers, dictated by the muses or by angels, for example, R. M. Rilke and his Sonnets to Orpheus. The artist's report is a subjective account of the immediacy of the imaginative experience, the immediacy and concurrence between the act of imagining, and the act of creation. Such creators are saying that to imagine something is subjectively the same as doing it. They are saying that their symbolic activity is grounded in physical experience. Drawing is symbolic and imaginative behavior as old as humanity. The roots of drawing lie in the realm of physical mark making, as well. We agree with these ideas. We have accordingly directed our attention to making drawings that reflect the body, unfettered by the limitations of physical surface. Our interface and its proof of concept, a 3d immersive drawing application is an attempt to address these issues.

The role that fine motor control (i.e. dexterity with fingers) plays in drawing is less significant than commonly supposed, except as it relates to the tactile interaction of drawing instrument and the support on which the drawing is made. Since we are drawing in a virtual space, devoid of tactile feedback, our implementation focuses on gross motor control. We examine broad gestures made with the entire body and correlate these with expressive line quality.

1.1. Drawing

There are mature examples of expressive line drawing and engraving over 30,000 years old. Artists use drawing as a tool to think about their world analogously to the way in which philosophers use written language and scientists use mathematics. Drawing is recognized as the fundamental skill for artists. Gestural drawing quality derives from the sensitivity to the physical and sensual act of drawing. We address the task of representing the body of the mark maker through the character of the mark she has made.

1.2. 3D Drawing

Drawing is fundamentally a three dimensional activity. One has but to witness the behavior of a very young child as they first draw. They move the drawing instrument around in space, only occasionally intersecting the tip of the tool with the drawing surface. It is some time before they learn that you can only get a mark when you physically interact with a surface. Observations of adult artists reinforce this observation. They, too, draw in space, but their off-surface motions are more like anticipation and follow-through in athletic activities. Many who are considered good at drawing seem to be ‘handling’ the object as they draw it. They appear to be touching and stroking the object. It is as if they feel it as they see it. We see an analysis and employment of body language as an essential part of understanding good drawing.

1.3. Body language

We speak of body language to explain how we understand non-verbal signals. Posture, and the sequence of postures through which the artist moves form a language of gestures. The scientific basis for our approach rests in an appreciation of visual perception, symbolic thinking1 and the relationship between imagination2, visualization and the neural components of physical and volitional behavior. People who draw well do so by reaching out with tools and making marks as if they were touching, stroking and manipulating objects. Kinesthetic and tactile perception is a component of imagining the object in space. The symbolic meaning of a drawing likewise derives its richness from an involvement with our sensual body.

2. BLUI

It is difficult to navigate or draw in 3d space using 2d tools. Virtually every step requires a minimum of two distance actions. BLUI is our attempt to make 3d drawing easier and more expressive. Implemented in C and C++ and using the CAVE libraries and OpenGL, it runs on an SGI Onyx2 driving an ImmersaDESK®, the 3d immersive virtual reality environment manufactured by Pyramid Systems. An artist using the ImmersaDESK stands in front of a high-resolution six by four foot screen, wearing a head tracker attached to LCD shutter glasses. The glasses alternately allow the left and right eyes to see the image presented on the screen, which is varied to give the illusion of depth. The head tracker allows the computer to know exactly where the artist is so that the image can be generated in correct perspective. For input, the artist uses a wand, a device (also tracked) with three buttons and an analog joystick (See Figure 1.)

We use the ImmersaDESK wand and head tracker for the information they provide about body position and orientation. This data is correlated with a silhouette from one or two video sources (see Figure 2.) Images from the video camera(s) are segmented into artist and environment. The segments representing the actor are then skeletonized (Figure 3), and correlated with the head and right hand positions and orientations from the trackers. This information is used to derive a 3d representation of the skeleton. The skeleton is then analyzed, viewing the relative position and orientation of the joints as time dependent signals. This analysis generates the parameters used to recognize gestures and to control the geometry and rendering of the 3d drawing. If an avatar is being rendered, its position is also dependent on the skeleton.

Figure 1. The authors at the ImmersaDESK® (Photo by L.J. Evens, ARSC staff)

2.1 Drawing with BLUI

An artist using BLUI draws by pressing a button on the wand and sweeping it through space. BLUI represents the wand as a 3d crosshair about 10 feet in front of the user. In real time, as the artist draws, the cursor leaves behind a partially transparent, colored tube. These tubes can then be edited, either by picking points along them and moving them, or by grabbing a whole tube and moving it with respect to other tubes. Moving a point within a tube gives a smooth deformation of the tube, by moving a region of adjacent points. The amount each point is moved is inversely proportional to its distance from the selected point. Changes in the drawing can be undone at any point, by the simple gesture of shaking the head "No."

In a separate mode, the computer can play back the actions performed by an artist. In this mode, viewers can watch, while a 3d virtual artist creates a 3d drawing. This mode is currently too slow to use in real time, due to the extensive image analysis required for the computer to determine where the (non-tracked) parts of the artist's body are.

The objects in an image can be saved in the .obj 3d format so that creations can be placed into a standard 3d modeling and animation environments. An example of this is the ray traced rendering of the 3d image of the script word, "Blui", that we placed into the MAYA environment for assignment of textures, transparency, refractive index, etc. (See Figure 4.) The object is clearly organic and expressive. At the same time, the rendering makes it clear that it is computer generated. It is therefore true to its origins in body language mediated through a computer application.

2.2 Line quality

As with a real brush or pencil, the tonal quality of a line varies with the stroke of the artist. BLUI implements this variation in the following way. With zero body language (that is, robot-like motion of the drawing arm), we draw constant cross-section tubes along the path of motion. When the angle of the wand varies relative to the line we are drawing, we taper the tubes, with cross-section varied at each sample point. More body involvement results in greater variation in line quality, so that with the elbow having angular motion in relation to the wand, we draw tapered tubes of varied radius. Significant angular motion of the shoulders relative to the torso results in a more complex taper, varied over a larger range, and significant rotation of the hips will draw yet more complex surfaces with yet more variance. The color of the tubes is varied as the speed with which the wand is moved while drawing. There is also a difference in line quality depending on whether the artist is pushing (with the tail of the wand following) or pulling (leading with the tail of the wand). Push strokes are represented by introducing a noisy factor into the radius, in contrast to the more smoothly tapered tubes produced by pulling. A similar procedure produces "noisy" coloring when the artist's motions are not smooth.

Figure 2. The silhouette from video, after skeletonizing.

2.2 Gestures

BLUI can also recognize certain gestures, given below. The current analysis depends highly on the tracking hardware. In the future, our goal is to minimize this dependence and use the image understanding components of BLUI to recognize many more gestures (marked below with "future"). We have found that the time dependent nature of gestures is extremely important. One cannot recognize gestures simply by recognizing static poses of the body, but must instead analyze changes in the body's position and orientation.

Yes (in response to a query) is signified by a nodding of the head.

I’m done is signified by dropping the arms to a relaxed pose and remaining there for more than one second, while doing nothing else. At this point, the user is queried "Do you want to quit?"

No (in response to a query) is signified by shaking the head.

Undo is the same gesture as No, but performed shortly after making some change in the drawing, not in response to a query.

Anticipation (used for line quality) is a measure comparing the direction the wand is pointing with the direction of motion.

Start to draw (future) will be signified by finding the Reach gesture followed by the index finger darting forward. (See Figure 5.)

Reach (future) is the arm moving in a smooth trajectory after a history of lower coherence. (See Figure 5.)

Pick (future) is similar to Start to draw, but characterized by one or more fingers and thumb spreading apart followed by pinching together in the vicinity of an object. (See Figure 5.)

Grab (future) is similar to Pick, but characterized by the fingers moving more as a unit followed by their closely surrounding an object or part of an object. (See Figure 5.)

Figure 3. Video to silhouette to skeleton to avatar.

Big Grab (future) will be signified by both arms closely surrounding a volume occupied by an object or part of an object.

3. Uses and plans for the future

We see BLUI used in multiple settings in both artistic and scientific realms. We have spoken with scientists at the Arctic Region Supercomputing Center and at the University of Missouri who are interested in using BLUI to annotate 3d scientific images and to enter visualization flight paths. In addition, the Digital Galaxy Project team at the Hayden Planetarium of the Museum of Natural History is interested in using BLUI as a tool to draw nebulae. Specifically, we have spoken with Jon Genetti of the San Diego Supercomputing Center who will be doing the volume rendering for this project and will be using BLUI to generate the 3d density data.

Figure 4. A rendered object of the word ‘Blui’ drawn in script using BLUI. The rendering was done in MAYA®.

We are also preparing to develop a plug-in for MAYA® so that BLUI will become part of the MAYA® environment, which will extend MAYA's inherent capabilities for 3d manipulation and drawing. Additionally, we will create art objects in the BLUI environment. These objects will be virtual forms, saved with data so that they can be edited and their creation can be played back. Additionally, we intend to extend BLUI so that these objects can be animated, through both direct manipulation and user scripts. We will be using BLUI to create objects that will then be rendered using ray tracing or other high quality techniques and then printed in large (or even panoramic) format.

4. Conclusions

We have demonstrated the potential for a gesture-based, immersive environment for the creation and manipulation of virtual objects. BLUI presents a solution to the need for a better set of creation and navigation tools by scientists and professionals interested in communicating scientific information about 3d. The concept presents the possibility of at least two novel art forms based on the older forms of drawing and CGI animation. Our instrument encourages the artist to dance his drawing into being. (See Figure 6.)

Figure 5. The Grab, Draw, and Reach gestures. Figure 6. An artist using BLUI to draw "Blui".

5. Equipment and Software

BLUI is written in C and C++ using the CAVE libraries on a Pyramid Systems ImmersaDESK driven by a 4 processor SGI Onyx2 rack with Infinite Reality Engine and 4GB RAM. Additionally, 2 hi 8 video camcorders are connected to the DIVO board on the Onyx2. We use Alias/Wavefront, MAYA®, Composer®, and various desktop publishing applications for rendering.

Acknowledgements

The Arctic Region Supercomputing Center (ARSC), is the environment in which we work. All of the equipment and software is maintained by ARSC staff. Both of us are on joint appointments with ARSC and our respective academic colleges.

The University of Alaska Fairbanks is an Academic Partner with Alias/Wavefront. Access to MAYA® and support for MAYA® has been significant to this project.

The authors are from widely different backgrounds, though both have undergraduate majors in mathematics. The opportunity to collaborate between disciplines as disparate as art and computer science has been tremendously rewarding.

 

 

 

References

1. S. Kosslyn, Image and Brain, MIT Press, 1996
2. T. Deacon, The Symbolic Species, Norton, 1997
3. I. A. Kapandji, The Physiology of the Joints, Volumes 1, 2 and 3, 5th edition, Churchill Livingstone 1988
4. E. Muybridge, The Human Figure in Motion, Dover Press, 1989