A) 3D Modeling and animation in preparation for supporting the Brahms Output

B) Development and and testing of a Brahms VE application: the Brahms Modeler's Side of the story

C) Overview of the BrahmsVE Core

Next we will look into a larger and more sophisticated 3D viewpoint object, an Avatar representing an agent in the scene:

Example avatar (Bill Clancey Avatar)

Steps and Tools used to create 3D objects (excluding world):

A whole series of steps and tools are needed to create and animate the 3D viewpoint objects.

1. Model is created in 3D Studio Max (3dsmax)
2. Model is then textured in 3dmax
3. Model is then exported to 3ds format
4. Model is then imported into Poser
5. Model then has bones applied
6. Other accessory objects that are needed are imported following above steps to be applied to the master object (ie: coffee cup for Bill's avatar).
7. Accessory objects are processed and then bones have to be applied as well
8. Next the animations are created frame by frame for each object (see Key Framing of Animations below).
9. Test animations to be sure they run aand then tweeked, that is triggering the cluster of animations set off by a Brahms output gesture
10. Once we are happy with the animations we then export to viewpoint
11. We then test in viewpoint
12. We then imported the viewpoint objects into Atmosphere and turn them into agents
13. We test in Atmosphere and correct any mistakes by going back to just when annimations are applied.

Key Framing of Animations

1. Load object (human figure) into the frame editor in Poser
2. Start with frame 1 (say have the avatar seated) Poser has a large library of typical positions but we cannot use them as they are too generic and often not detailed enough
3. Next move to the 50'th frame and then have the figure standing
4. Poser would fill in the blanks between frame 1 and 50
5. Go back and tweek in each frame to get it the way you want because Poser doesn't always do it correctly, this can be a time consuming process.

Timing: 8 hours to create a typical model and texture it and iteration for a week or so.

Reusability: Cutting and pasting of animations from one avatar to next. So far we have not created anything generic as we have been following a script.

Definition: Bones
Bones are pivot points in avatar or objects. skin (mesh) has to be attached to the bones in the right area. You have to attach multiple parts of the mesh to the bones. Textures are applied prior to the application of the bones. Once bones are completed then the animations can be applied.

B) Development and and testing of a Brahms VE application: the Brahms Modeler's Side of the story

The Brahms model must be developed in close cooperation with the VE 3D scenegraph modeling team. Brahms modelers must choreograph the action through the world, and help the 3D designers and JavaScript Core VE engine developer understand the sequencing, naming of areas and gestures and other animations, and engage in constant iterative testing.

1) The Brahms modeler's tasks

1. The Brahms model is developed, named gestures and areas in the 3D world are built right into the Brahms model itself.
2. The Brahms Modeler must specify the appearance and layout of the 3D world, pathways and named areas, the objects and avatars (agents) that must be in the world, starting clock, timing of actions and animations, any built in delays, and the textual reporting that will help the end user understand the model. Important from this process is the production of an inventory of objects, avatars, labels of areas in the world for motion of agents, and gestural sequences and how the mapping of a Brahms label for a gesture is satisfied by the cluster of local animations built into the 3D model.
3. The Brahms Oworld export must be generated along with a parallel human readable script for the 3D modeling team. Often this is accompanied by overhead storyboards, as showin in Figure 3 below.

Figure 3: Storyboard to aid the VE modeling team

Many iterations must take place between the Brahms team, 3D modeler and JS developer to achieve the desired results.

2) Sample Brahms Output

The following is the record format of the "Oworld" exported action output of Brahms from the start of the Planning Meeting (Scenario 1).

activity|primitive|0|57|projects.fmarsvre.RZ|AnnouncingStartOfPlanning|
activity|primitive|57|58|projects.fmarsvre.RZ|RequestingGeneratorStatus|
activity|primitive|58|114|projects.fmarsvre.RZ|OrderingToRefuelGenerator|
activity|primitive|114|171|projects.fmarsvre.RZ|RequestingWeatherReport|
activity|primitive|171|176|projects.fmarsvre.RZ|Getup|

The above exported actions can be placed into the VE model by cut and paste or by inclusion in a web interface (text buffer) which the JS for the VE has access to. The VE core then parses these commands. More is described in the section following on the BrahmsVE Core. In future these actions will be sent live via a connection (SOAP or another means). Figure 3 below shows our vision for the real time communications between Brahms and the VE.

Figure 4: the proposed real time communications between Brahms and the VE

C) Overview of the BrahmsVE Core

Figure 5 below shows the major components of the "VE Core" which is the Javascript library that is embedded into each Atmosphere scenegraph. Currently each VE Core contains both generic elements (threading, scheduling, triggering) and scripts that are custom built for the particular application (custom groupings of clusters of small animations to create one addressible gesture, such as "drink coffee").

Figure 5: the full vision for the BrahmsVE Core engine

There are a few ways we have used to get the data to the VE core. The first way, from around the time of the first coffee sequence, involved manually translating the input lines into function calls, one function call per line. Basically I was performing the role of the parser. Those function calls were tagged on at the end of the script and called after everything else was set up. You can probably recognize them at the bottom of the coffee sequence script.

2) VE Core Parsing and Execution Steps (by Rolu)

Once the Brahms output has been parsed a series of function calls are generated. The following lines below correspond to the function calls for the Planning Meeting Brahms output. These lines are executed when the Sequence button is pressed in the Controls interface:

this.ctrl.handleLine("activity|primitive|0|45|projects.fmarsvre.RZ|AnnouncingStartOfPlanning|");
this.ctrl.handleLine("activity|primitive|45|46|projects.fmarsvre.RZ|RequestingGeneratorStatus|");
this.ctrl.handleLine("activity|primitive|46|97|projects.fmarsvre.RZ|OrderingToRefuelGenerator|");
this.ctrl.handleLine("activity|primitive|97|153|projects.fmarsvre.RZ|RequestingWeatherReport|");
this.ctrl.handleLine("activity|primitive|153|156|projects.fmarsvre.RZ|Getup|");

However, as long as the lines are fed to the parser it doesn't matter where they come from. Having them hardcoded like in the above example is convenient, especially during testing and demoing (and therefore still in use), but not necessary. The third option is the webpage where you can paste the lines directly into a html form, after which they get parsed and executed by the script (running in the plugin). Since there is no preprocessing at all necessary this way it forms an almost direct link with Brahms. All you have to do is copy the lines from the generated text file and paste them in the web form.

The script that drives the Atmosphere scene contains three important parts. The first is the VE core. This contains all base functionality without regard to the simulation. It contains the parser, the generic code for agents and objects and everything else every simulation needs. The second part is the simulation-specific part, which contains the code for the used actions, agents and objects. The third part is the initialization part, it is where stuff like the above example function calls for this.ctrl.handleLine lines belong, and it initializes the whole simulation (the first two parts are basically just libraries, the initialization part connects them in the right way).

The script knows about all involved entities (agents, objects, places). Each of those entities is represented in the script (their specific code is in the simulation-specific part, which is based on generic code from the core, and they are initialized in the initialization part). The specific code for agents for example defines what aer and viewpoint objects are used for the agent, what animations it has access to, and details like how fast it's walking animation goes in order to correctly synchronize the walk animation with horizontal movement. The specific code for objects depends on what they can do - specific code for a door for example would contain open/close animations. Not all objects need specific code, the laptop as used in sc1 for example is completely inanimate and therefore does not appear in the script. Place entities, which form the mapping between something like projects.fmarsvre.GalleyArea and the position/orientation, do not have specific code since they are all very similar, those are all initialized in the initialization part and taken care of in the core directly.

The initialization part of the script contains, as said, the code that links up the libraries in the correct way. It starts with initializing the core - creating the controller object, for example. At this time it is a completely generic VE environment. Then it sets up the simulation-specific part. It creates and sets up objects for all entities involved. In case of the sc1 script for exmple it creates an object for every agent (there is some customized specific code for every person in the simulation-specific part of the script). It also sets up all the area's. There are no objects in use in sc1 at this time that require special code. Then it sends commands to the agent objects that make them be seated around the table. When that all is finished it's ready to accept Brahms output. It should be noted that every piece of the simulation-specific part of the initialization is now hardcoded, but can also (eventually) be triggered by sending the right commands to the parser. In fact part of the initialization is already done that way, and the rest can easily be adapted. This makes it possible to have a world with the VE core and a big set of simulation-specific libraries, so you can make it display any simulation that uses elements from the libraries, simply by placing some initialization commands in front of the Brahms output and sending all of it to the world.

3) Simulation Specific Code

As you probably understand by now, the capabilities of a script based on the VE core depend mostly on the simulation-specific code. There must be specific code for every used agent (you can have code around for agents that are not used in the simulation that you are currently running, that does not matter). There must be code for all actions the agents can perform (those are the "action scripts", they are more or less separate entities and can be shared by agents). There must be code for all objects. However, once the simulation-specific code is there you can combine it in pretty much any way you want. With a little improvisation you could model a game of musical chairs with the current sc1 script without touching one single line of it, simply by sending it a different Brahms output. All necessary scripting is there - getting up, sitting down and walking around.

Adding agents or actions however involves writing action scripts and modifying or creating the specific code for agents, possibly changing or adding objects, and creating the models and animations themselves. This requires specialistic knowledge on the areas of (among others) Atmosphere, Atmosphere javascripting, aer modelling, viewpoint objects and poser.

4) Journey of an action output

To get back to the original question - what happens when a line enters the script? In a nutshell, it first gets passed to the controller object, which runs it through the parser. This results in either an error message or a collection of information that's useful to the script. The agent object that will handle the line further is looked up, and the information is passed to it. The agent object will now queue the information to until it's time to play the action. When that time arrives the agent object looks up the corresponding action script and hands over control. The action script will do what's necessary and return control to the agent when it's done. Along the way there are several points where special actions can be performed, you could for example send a request for status information to one of the agents. This would be passed on to the agent, but not be queued but processed immediately.

5) Issues with the current approach

There was another question - on the problems with the current script. There are several, but basically they all come down to the "pioneer" character of the current work. Atmosphere is still a very new platform and as such there is a lack of experience on many area's - things like, what programming techniques work best? What is the best way to control several viewpoint objects at the same time? How can you interface with the outside world (that was basically impossible at the start of the project)? There are several programming techniques that work and are fairly standard in the typical small scale projects but don't scale very well to this level. To give an example, let's look at the standard way to perform simple animations in Atmosphere. In Atmosphere, every time before a frame is rendered the scripts in the world get a chance to update the world. The intervals between two of such events (called "timesteps") is directly depending on your framerate and can vary a lot. To make this easier the scripts get to know the total amount of seconds elapsed since the world was entered and the amount of seconds elapsed since the last timestep.

So, when your animation starts (say we are moving a box from A to B in 8 seconds) you remember the current time. You usually know how long your animation is going to take (8 seconds this time), so you can calculate at what time your animation should end - you remember that too. Finally you place your box at A. The next timestep you first check if the current time is past the time the animation should end - if so you place the cube at B and stop. If not, you check the time elapsed since the last timestep and use that to calculate where the box should be now, (elapsedtime/totaltime)th of the way between A and B.

We used this recipe in the VE scripts too. Most action scripts have a variation of it as their core. It worked fine at first. But later on it became more and more complicated. A standard animation like above needs a fixed duration to work. However, the actions we currently use have variable lengths, and in some cases no real length at all (since the end time is not known when the action starts). Furthermore we now have simulation time seperated from real time. However, the viewpoint animations still play in real time. They have to be sped up or slowed down according to a combination of the simulation time multiplier, a multiplier that results from stretching or shrinking the action and a multiplier that can be used to fix speed problems with the viewpoint animations (necessary in some cases). All of that can be done, however it gets overly complicated. Furthermore control over the agent object is in hands of the action scripts, which return it when they feel like. One bug in an action script can wreck havoc with the schedule. It's not possible to jump to a certain time in the simulation at once since most action scripts never return control in the same timestep as they got it. Finally, it's impossible to run the simulation backwards.

5) Future approaches

Instead we want to try an alternative approach where the concept of time "flowing" is abandoned, and the whole simulation is more stateless as far as time is concerned. Every timestep the controller object will determine the current time, depending on user input, the clock and Brahms output, and will simply tell every agent to evaluate themselves towards that time. Similarly, the agents will figure out what action script that time falls in (they will remember their whole queue since the start of the simulation) and will tell the action script to evaluate itself towards somewhere between 0 (start) and 1 (end). If the current action script is another one than the one used last timestep that one and all action scripts inbetween will have to be evaluated to 1 (if going forward) or 0 (if going backward) to keep everything consistent. This way you can go both forward and backward in time at any arbitrary speed (and hence jump to any time), while the agents will stay in control of themselves. Since viewpoint animations can be told to evaluate towards a certain time in the same way, handling is much easier for action scripts, and the rest of the code can stay cleaner (and therefore easier to build/maintain) too.