BrahmsVE: Proof of Concept for Human/Agent Intelligent Augmentation
02-2-H2.02-8957 ARC

Part 1 - TABLE OF CONTENTS

PART	DESCRIPTION	PAGE
1	Table of Contents	3
2	Identification and Significance of Innovation and Results of the Phase I Proposal     2.1. Identifying the Need     2.2. The Innovation     2.2. Results of Phase I Project	4
3	Technical Objectives and Work Plan     3.1. Technical Objectives     3.2. Work Plan         3.2.1. Technical Approach         3.2.2. Task Descriptions         3.2.3. Meeting the Technical Objectives         3.2.4. Task Labor Categories and Schedules	23
4	Company Information	31
5	Facilities and Equipment	32
6	Key Personnel and Bibliography of Directly Related Work     6.1. Management and technical staff members     6.2. Bibliography or directly related work     6.3 NASA and non-NASA advisors to project	32
7	Subcontracts and Consultants	37
8	Potential Applications     8.1. Potential NASA Applications     8.2. Potential Non-NASA Commercial Applications	38
9	Phase III Efforts, Commercialization and Business Planning 9.1. Market Feasibility and Competition 9.2. Strategic Relevance to the Offeror 9.3. Key Management, Technical Personnel and Organizational Structure 9.4. Production and Operations 9.5. Financial Planning 9.6. Intellectual Property	40
10	Capital Commitments Supporting Phase II and Phase III	44
11	Related R/R&D	44
	Proposal Budget

Part 2  Identification and Significance of the Innovation and Results of the Phase I Proposal

2.1 Identifying the Need

Space systems involving people working with machines are becoming increasing complex to design, test, train for and support during flight. This complexity is affecting all aspects of NASA’s programs and impacts mission viability in the critical areas of safety, cost, timeliness and effectiveness. We believe that collaborative software systems involving model-based agent architectures situated in realistic 3D visualization environments are vital new tools that will allow NASA and other government agencies and commercial enterprises to manage increasingly complex human-machine environments.

Two generations ago, NASA used specially equipped spacecraft doubles as simulators to allow crew and mission control to train and also to assist in problem solving during mission operations. The ground-based doubles of the Apollo XIII Command and Lunar Modules were instrumental for mission control to test power, life support and other system survival strategies and bring the crew safely back to Earth.

Today, NASA is building craft that are too large and complex to be able to have operational physical doubles on Earth. In addition, efficient and safe day-to-day operation of longer duration missions requires a deeper understanding of design for human work practice, psychology and teamwork. Add to the mix autonomous agents, and there is a real risk that the complexity of operational environments can overwhelm crew and mission control, leading to critical errors.

On August 16, 2002, the following news item appeared in the wires of the Associated Press [see bibliographic reference 1 in section 6.2]:

Whitson and the space station's veteran commander, Valery Korzun, got off to a late start installing the Russian cosmic-debris shields. They evidently forgot to open an oxygen valve in their spacesuits while getting dressed, and the air lock had to be repressurized so they could open their suits and fix the problem.

By the time the spacewalkers finally opened the hatch, 250 miles (400 kilometers) above the South Atlantic, almost two hours had been wasted…

Because of the late start, Russian flight controllers cut the spacewalk short at 4 1/2 hours. The retrieval of a collection tray for measuring jet residue was put off, as was wiping the area for signs of contamination.

The above story illustrates an event in the day in the life aboard the most complex of these vehicles to date, the International Space Station (ISS). In this case, the mistake was easily resolved without danger to the mission. However a similar error aboard a human mission in transit to Mars might prove fatal, especially if a failed micro-meteor shield needs to be replaced in an emergency.  For several years we have been working with teams at NASA and RIACS to provide a virtual environment software platform that will prevent this kind of error in the future.

Virtual environments assume a critical new role in satisfying the need to train for and to manage mission complexity

Virtual environments created in 3D VR technology on computer workstations or projected onto immersive environments such as CAVEs and head-mounted-displays have been used to good effect in mission training and planning for over two decades. Projects in this tradition include: early Ames tele-operations training with the Canadarm, the Virtual Shuttle, the modeling of the Mars Pathfinder surface environment, and the virtual training environment used by the crew of STS-61, the 1993 Hubble Space Telescope repair mission [2, 3, 4].

Project managers Loftin and Bowen [2] describe that in this project, approximately 100 members of the NASA HST flight team received over 200 hours of training using a virtual environment (VE). They note that in addition to replicating the physical structure of the HST and the interrelationships of many of its components, the VE also modeled the most critical constraints associated with all major maintenance and repair procedures. Figures 1-2 illustrate scenes from this project.

Figure 1: STS-61, the 1993 Hubble Space Telescope (HST) Repair mission team using VR training simulator: Astronaut Nicollier looks at a computer display of the Shuttle's robot arm movements as astronaut  Akers looks on (Image courtesy NASA archives)	Figure 2: Computer generated scene depicting the HST capture and EVA repair mission for mission planning (Image courtesy NASA archives)

Loftin and Bowen conclude by stating that for the first time, a VE has been integrated with a limited capability Intelligent Computer-Aided Training (ICAT) system and that (it performed to excellent effect).  They then issued the following challenge to future VE systems designers:

The results of this project serve to define the future role of VEs in training within NASA and to provide evidence that VEs can successfully support training in the performance of complex procedural tasks.

In the ensuing years, many new VE environments for training and design/test have been produced, including Transom Jack [6], Steve from USC [5], environments for submersibles and navy operations at the MOVES Institute at the Naval Postgraduate School [7,8,9] and significant projects within NASA including APEX from Ames [10,11] and VR interfaces for remote vehicle control [12].

We believe that no simulation and training environment thus far has represented more than a fraction of true underlying mission complexity especially when accounting for “humans in the loop”. For example, an ordinary EVA aboard the ISS not only has hundreds of individual variables including checklist items and utilized equipment, but also all personnel including mission controller, PI and engineering contractors back on Earth play a role in that EVA activity, working from geographically separated areas using often unreliable and distorting channels of electronic communications. In addition, traditional VEs, such as the one employed in the Hubble repair mission, used costly hardware and time consuming modeling processes.

Over the past decade, the increasing power of consumer personal computers, ubiquity of the Internet  and standardization of net-based software components such as languages (Java), web browsers and integrating protocols (XML), have permitted the creation of powerful new VE systems that are entirely based on commercial off the shelf (COTS) systems. Thus, there is a revolution about to take place in collaborative virtual environments for the management of complex human-machine environments and we are working to provide one of the major new tools of this revolution.

2.2 The Innovation: BrahmsVE

BrahmsVE is the result of three years of intense work among the Brahms teams at RIACS, NASA, and DigitalSpace beginning with an STTR in 2000 [13], continuing with work to model EVA and day to day operations aboard the FMARS/Haughton-Mars Project analogue habitats [14,15] and leading to a specification for the OWorld and Brahms interfaces [16,17].

The existing back-end architecture: Brahms

A virtual environment by itself is of little use without a powerful back-end architecture that can represent the complexity of human-machine systems.

For over a decade, teams at NYNEX, the Institute for Research on Learning and now, at Agent iSolutions, working with NASA Ames and RIACS, have been developing an intelligent multi-agent environment used for modeling, simulating and analyzing work practice. The environment is called Brahms [18]. Brahms is a data driven (forward chaining) discrete event environment usable for simulation purposes as well as for agent-based software solutions requiring the use of intelligent agents. Brahms and its applications are described in detail in bibliographic references [19,20,21,22,23]. From the Agent iSolutions Web site [18]:

Brahms allows us to model the work activities of each type of role, and each individual (or artifact) playing that role in an organization. The focus of a Brahms model is on the context of work, meaning, how does the work really happen. One of the essential requirements for Brahms is that we can model collaboration and coordination between people working on one task, as well as that people can work on more than one task at a time, and are interrupted and able to resume their activities where they left off.

Prior to the partnership with DigitalSpace, Brahms models could only be viewed in execution using a timeline bar chart-style interface. It was determined that Brahms could become a much more effective tool if it were to include interface that allowed realistic reconstruction and interaction with 3D scenes representing the real world people and systems being modeled. This new product platform is now under development  and its current implementation is depicted in figure 3 below and described next.

Figure 3: BrahmsVE architecture, Fall 2002

A prime directive for BrahmsVE was that it should enable highly realistic recreations of environments involving people interacting with systems. Figures 4 and 5 below illustrate some of the human figure, gesture representation and scenario reenactment of which the development version of BrahmsVE is capable. These are depictions of BrahmsVE simulations of crew activities aboard the FMARS/Haughton-Mars habitat during the 2001 field season.

Components of the innovation

We believe that BrahmsVE is a uniquely powerful new tool that will offer human-centered computing unique opportunities for advancement. BrahmsVE has the following unique properties:

Brahms Java-based PersonalAgent with compiler, virtual machine, builder, IDE and AgentViewer all of which are currently utilized in a number of NASA projects including mobile agents.

Brahms Virtual Environment runs on industry standard consumer grade computing platforms over ordinary Internet connections with no special hardware required.

BrahmsVE employs industry standard languages and protocols such as Java, JavaScript, SOAP and XML.

BrahmsVE’s virtual environment uses industry-leading technology in Adobe Atmosphere such as Havok physics, Viewpoint models and inverse kinematic animations all running in an open framework utilizing an open source community server.

For a full specification of the BrahmsVE environment as well as detailed technical documentation on the system, please see the final report filed for this SBIR and additional materials on the Web sites referenced in [14,24].

Additional 2003 BrahmsVE projects completed show promise of wider applications

In parallel to the work completed for the Phase I SBIR, BrahmsVE was used for two non-funded exploratory projects at NASA Ames. For one project DigitalSpace designed a model of the Mars Exploration Rover, which interacts with a virtual Mars surface utilizing the build-in physics engine. This test was successful and was presented as a concept piece to the Athena science team at JPL in January 2003 (see Figure 6). We believe that this shows that we will be able to apply BrahmsVE to mobile agents applications. The second project was to illustrate the Mars habitats on a virtual reconstruction of the entire Mars surface derived from Mars Global Surveyor and other data, a project commissioned by Geoff Briggs at Ames (see Figure 7).

2.3 Results of Phase I Project

Introduction – the solicitation’s challenge
The following statement from the topic for this SBIR called for proposals to produce the following tools:

Visualization tools combining "virtual reality" projection with actual objects in the environment, conveying information about object identity, part relationships, and assembly or operational procedures. "Cognitive prostheses" that qualitatively change the capabilities of human perception, pattern analysis, scientific domain modeling, reasoning, and collaborative activity. Such tools could incorporate any of a variety of modeling techniques such as knowledge-based systems and neural networks, and fit tool operations to ongoing human physical interaction, judgment, and collaborative activity.

This statement is a broad-brush challenge to the entire field of visualization, human-centered computing and cognitive science. While the scope of this challenge is large, we set out to achieve its central objectives by producing a proof of concept that is a good first step along the road to this total vision.

Objectives of Phase I

In Phase 1, we constructed a simple yet believable cognitive prosthesis in a virtual reality environment that implemented a canonical scenario in the study of human-centered computing: a semi-autonomous mobile agent interacting with a human subject. We referred to this mobile agent as the Personal Bot Assistant, or PBA.

Figure 8: Canonical example of a semi-autonomous agent interacting with a human astronaut

The objective of this project as stated in the Phase 1 proposal was therefore to produce, in a web-based 3D virtual environment, the canonical example of human/machine augmentation, that of a semi-autonomous agent assistant interacting with a human inhabitant of a space station or surface habitat. As the above figure (8) from the original proposal illustrates, that agent should be able to execute a minimum set of activities interacting both with its environment (the geometry and physical properties of a virtual space station or habitat) and with an astronaut agent. The astronaut agent in this example could be driven by a simulation engine or directly by a user at a workstation as an “avatar”. The virtual astronaut would in turn have a limited repertoire of commands, which can be directed to the PBA.

Our actual implementation of Phase I deliverables

Guidance from QSS, NASA and Brahms teams directed us to produce a simulated work practice utilization of the Personal Satellite Assistant (PSA), which is under development at NASA Ames Research Center, aboard a virtual analogue of the International Space Station (ISS). Design issues surrounding the PSA operating within the ISS environment provide an almost ideal challenge for the development of tools to aid in human/agent intelligent augmentation. Our colleagues who guided this project informed us that in a single simulation the following could be achieved:

1. Testing design concepts for the PSA in a virtual space prior to next generation implementation of hardware, in this case the addition of a laser pointing device on the PSA,
2. Modeling of the interior of the ISS, including obstructions, enabling agent and astronaut movement, tool mis-placement and comparison with the physical test fixture at Ames.
3. We were informed that this kind of simulation environment could evolve into a training simulator for astronauts who will be testing and then utilizing PSA aboard the ISS in the next several years supporting both pre-flight and during flight operations.
4. In the longer term, we were given the insight that coordinates and state transmitters aboard a flight PSA would be able to report its current position and activity to a 3D simulator which could be used by NASA JSC Space Station Mission Control to permit them to gain a PSA-eye view of the station and to determine the whereabouts and current activity of the PSA.

Empirical results from the Phase I implementation

We were therefore well informed as to the benefits of pursuing this simulation within an upgraded BrahmsVE architecture (the main beneficiary of this Phase 1 project). Over a six-week period we executed eleven versions of the simulation, each with more complexity and more interface and reporting affordances added. Each of these iterations is available for live execution within the 3D environment or viewing via Quicktime movies at the project Web site [24]. As a result of these iterations, the following empirical results emerged, answering the key questions posed to us by our collaborators:

1. Is a laser pointer a useful device for a PSA or similar semi-autonomous robotic agent? The simulation suggests it would be and that the laser pointer would be used not only to permit mission controllers to point to a spot (current suggested teamwork application) but also as an indicator to astronauts of what the PSA is doing (where its gaze is currently, its direction of travel). Of course the laser pointer would be used selectively as it has the ability to temporarily blind an astronaut given direct contact of the beam with the retina. It was suggested that a spotlight (not coherent) be substituted for simple indicator/tracking functions.
2. What other affordances does a mobile robot with human eye-gaze level awareness (microgravity operation) need? The simulation suggested that status indicator lights to tell astronauts what state the PSA is currently in (searching, waiting for cleared obstruction, powering up) would be needed to quickly and reliably inform human occupants of the station of the PSA’s intention or current state.
3. Can a PSA-like agent successfully avoid a collision with an astronaut while transiting the station, even in close spaces such as interface modules? Empirical tests in the virtual station seem to suggest that the ray-casting ability of the virtual PSA is able to determine when its path is blocked. Of course, only calibration with actual machine vision systems used on the actual PSA would be able to validate such a collision avoidance scheme.
4. Can such a mobile agent effectively locate a tool or move itself to a known location aboard the station? The use of the laser pointer suggests that plans to apply bar-codes to locales and object such as tools might greatly aid the PSA’s ability to locate itself in space and find objects.
5. Can the PSA successfully navigate and course correct given attitude adjustments of the station or passing astronauts and blower fans creating an airflow situation that the PSA would have to steer through? Only a single blower fan was implemented in our BrahmsVE implementation, which imparts a simple force vector on the passing PSA. More complete modeling of the ISS interior and orbital dynamics would have to be carried out to gain understanding of this issue.

The above heuristic results suggest directions for Phase II in which statistical reporting and the Havok physics engine will be added to gain a more quantitative set of results from hypotheses that will be tested with the next generation models.

Detailed description of the implementation

We will next discuss what was constructed to enable the ISS/PSA BrahmsVE simulation. We will begin with a tour, aided by visual screen captures of the simulation in action. We will then describe the extensions to the BrahmsVE architecture constructed during Phase I.

3D virtual world models constructed

Utilizing imagery and schematics, the DigitalSpace team modeled the interior of the current configuration of the International Space Station (ISS). Wherever possible, high-resolution imagery was used to enhance realism of the interiors, and we produced 3D models of significant equipment such as monitors, scientific equipment and hatches. We modeled two astronaut agents with head and body gestures representative of real astronauts aboard the ISS (floating forward, footrest positions, head tracking). Lastly we created a model of the Personal Satellite Assistant (PSA) complete with instruments and a laser pointing device which is used as an indicator of status, direction of travel, and ray-casting identification of a target object. The 3D visualization environments that resulted are pictured in figures 9-19 below.

Figure 9: BrahmsVE web-browser based components

Figure 9 above illustrates the new interface to the BrahmsVE environment produced in this Phase I project. The 3D window represents the viewpoint of a third participant within the virtual ISS model. In this case we, the viewer, are positioned behind the PSA agent, as it establishes a line of sight view with the astronaut agent, which has just repositioned across the module. The virtual PSA has presented us the following iconic views: the top icon representing that the astronaut has observed the PSA and the bottom icon representing that the PSA is in a wait state and able to respond to a command.

Above and below the 3D view are two text output windows. The top window reports all Brahms action commands and resultant actions and reports from the agents in the virtual environment. The text output buffer below the 3D window reports internal agent state (such as PSA remaining at station-keeping ).  To the right of the screen are two panels for controlling and reporting on the simulation, described below.

The above layout of the control panel for the ISS/PSA BrahmsVE simulation enables the operator to set up and then execute different scenarios, described next.

The scenarios implemented in this BrahmsVE application centered around the ability of the operator (astronaut agent or operator) to dispatch the virtual PSA to locate a missing tool (wrench, drill, flashlight) placed randomly aboard the station and to then return to the dispatching astronaut-agent and report the tool found (figures 11-19).

Later we added a second astronaut traveling randomly (figure 18) and a blower fan to cause the PSA to have to avoid collisions and to make course corrections through force vectors.

Figure 19: Report of successful location of drill tool back to astronaut agent

Status icons

Throughout the simulation, the status of the agents is reported in the text buffers and indicated by convenient “status icons” projected above the active agents (astronaut and PSA). The visual language for these status icons is described in tables 1 and 2 below.

Table 1: Task Icons (Top Icon)

This icon is an indicator of the Action the Agent is currently performing.

PSA & Astronauts
	The Agent is looking at, and tracking, the Target.
	The Agent is moving to a new location.
PSA Only
	The PSA is currently scanning its surroundings for its Target.
Power Lead	The PSA is currently recharging.
	The PSA is reporting the tool’ location.

Table 2: Target Icons (Botton Icon)

The Target icon represents what object the Agent is currently focused on.
This is either the Watching Settable or in the case of the PSA, the SearchFor Settable.

Icons	Target
	Drill, Flashlight or Wrench tools
	PSA is receiving commands (via laptop)
	Agent sees PSA or Astronaut
	PSA is affected by blower fan
	PSA has identified and is utilizing power station

Underlying technology enabling the above interaction

In the following section we will describe the underlying architecture that enables the above BrahmsVE implementation of the multi-agent simulation of the ISS with PSA, covering new additions to the architecture made possible by this Phase I support.

Use of industry standard platforms

A primary goal of the BrahmsVE is to utilize standard web components that run on ordinary consumer personal computers with no special hardware. Achieving this kind of ubiquity is a longtime dream of industrial design and simulation and has been made possible by the advent of the multi-gigahertz processor, high-speed net connectivity and low-cost, high performance 3D acceleration. BrahmsVE is therefore able to be used from virtually any net-connected PC in the world and has the added benefit that there is a multi-user option allowing distributed researchers to chat and be visible as “avatars” within a shared simulation. Thus, BrahmsVE is evolving into a collaborative virtual environment for distributed team use, a powerful and effective tool for 21^st Century work practice engineering.

Behind the scenes: Adobe Atmosphere, the OWorld engine and Brahms interfaces

Figures 20 and 21 depict the current state of the BrahmsVE architecture with the implementation of this Phase I project which is described next.

Figure 20: Current flowchart diagram describing interfaces between Brahms, the Web server and the entire BrahmsVE environment

Figure 21: Current flowchart diagram describing the OWorld engine operation with Atmosphere

Adobe Atmosphere

Adobe has partnered with DigitalSpace since 2001 to provide its Atmosphere 3D web plugin to the BrahmsVE effort. Adobe has contributed significantly to the platform (at no cost to DigitalSpace or NASA) based on our requests. Atmosphere is a stable, integral part of the platform and provides the 3D scenegraph, rendering engine (in software and hardware), scripting language, physics engine (from Havok, Inc.) and an object renderer and animator to represent objects and agents and their gestures (from Viewpoint Inc.).

The OWorld engine

The OWorld engine is the heart of the BrahmsVE platform and consists of the following major components (figures 20-21) which execute entirely within the framework of the Internet Explorer web browser and HTTP protocols:

1.       Interface layers that permit two-way communications with Brahms and Web services. This is implemented utilizing XML/HTTP. In this Phase I build, the full two way Brahms implementation is simulated by JavaScript. The Brahms team will be building special networking and command processing facilities into the Brahms server to permit the full two-way connection of our engine with Brahms.

2.       A command parser that requests (in JavaScript format) all Brahms commands (interface commands defining settables, detectables and reports).

3.       A series of multi-threaded message queues associated with each agent that interact with the entire OWorld framework and the settable/detectable actions.

4.       A path manager that reports on valid locations and traverses permitted by the world geometry.

5.       World geometry and viewpoint objects, which communicate valid path information and execute gestures required by actions.

Extensions to the OWorld engine

For this project the OWorld engine was extensively upgraded to provide multi-threaded support of multiple astronaut and robotic agents operating in parallel. A first generation path-finding module was added as well as the two-way XML dialogue layer for future network communications with Brahms (figures 20, 21). New Brahms commands will be described next.

Brahms commands and new commands implemented

Activity/Interface Commands

All Brahms commands are made up of multiple parts, separated by "pipe" characters ( | ). The first part of any command is what "type" it is. There are two recognized types in the OWorld library, "activity" and "interface". Which type it is will decide how many other parts the command will have.

Activity

The Activity type command will always have at least six parts. The second part (after the Activity type) is what "sub-type" it is. There are four recognized values, "move", "get", "put", and "primitive". The main used types are move and primitive.

The third and forth parts of the Activity command are the start and end time of the command. The command will not begin until the simulation time is equal to or greater then the start time (and all preceding commands have been completed). The execution of the "Action" (explained later) is accelerated or slowed to ensure the command is completed by the end time.

There are some special values and uses of the start and end times. A common "trick" is to specify zero (0) as the start time. Because the system time will always be greater than or equal to zero, the command will be executed as soon as all preceding commands are completed. This is useful when the length of time taken by a previous command is unknown.

Another "trick" is to specify negative one (-1) as the end time. This will cause the command to take as long as it needs to perform the "Action" without accelerating or slowing the command.

The fifth part of the Activity command line is the Agent. This is the full name of the Agent that will be performing this command.

The sixth part is the name of the "Action" the Agent will execute to fulfill this command. A common combination is the "move" sub-type and the "Walk" Action. In the case of a "primitive" subtype, this is usually an animation or gesture.

While every Activity command has to have at least six standard parts to be a valid Activity command, each of the sub-types requires different additional information. For each of the sub-types defined in the OWorld library, there is an additional part that specifies the "props" of the command. This is the name of the Prop (such as coffee mug or tool) that the Agent is currently using.

The "move" sub-type requires two additional parameters, from and to. These are the names of the pre-defined Areas that the Agent will be moved between. The OWorld system will produce a path for this movement when the command is parsed (not when it is executed).

The Action performed by the "move" sub-type command is responsible for things such as obstruction detection and path failure. The technique currently used in OWorld is that when the Action "Walk" detects a problem, it will define an Area where the Agent currently is, stop moving, and produce an Interface type Alert for Brahms, informing it of the failure. Currently the response to such an Alert is to produce a new "move" sub-type command, from the failure location to its previous destination. The "primitive" sub-type does not require any additional parameters (other then the props part). A primitive is an Action that is performed "as is". The "get" and "put" sub-types require two additional parameters, from and to, affecting Props.

Interface

All "interface" type commands require six parameters:

1. The first is the type "interface". The second is the sub-type, either "setDetectable" or "setSettable".
2. The third is time. Interface commands occur almost instantly, lasting for only one cycle (or frame). Therefore, they only require a start time. Like the "activity" type of command, they are executed when the simulation time is equal to or greater than the start time. In the same way, the "trick" of using zero (0) can be used to make the command occur as soon as all previous commands are complete.
3. The Interface command has an extra trick, which is using minus one (-1) as its time value. This is a specially checked condition, and will cause the command to be placed at the beginning of the queue, to be executed as soon as the current command (either Interface or Activity) has completed.
4. The fourth part of the Interface command is Owner. This may be either an Agent or an Interactive (an Interactive is an improvement on the previous concept of Props. They are essentially Agents with a limited set of available Actions).
5. The fifth part is the ID. This is the name of the Settable or Detectable being set.
6. The sixth portion is the Value the Settable or Detectable is being set to. In the case of a Settable, this may be any value (as long as it does not contain the pipe character). In the case of a Detectable, this may be "true", "active", or "false". "True" or "active" will enable the Detectable, causing it to report using an Interface Alert (explained later) when its condition becomes valid. A setting of "false" will disable it.

Interface Alerts

Interface Alerts are Brahms Commands that are sent from the VE to the Brahms server. They use the sub-type "detectable"; their time is when they were triggered. Owner and ID are the same as the other Interface commands. The Value portion depends on the Detectable that is sending the Alert. This is usually greater detail about the Alert, to be used by the Brahms server in deciding what will occur as a result.

A common example is the Idle Alert. While this is not actually a proper Detectable, as the Alert is sent by the Action itself whenever  an Agent starts or stops its "idle" Action, , its Value portion of the Alert signifies whether the Agent is starting to idle (Value is "true") or stopping (Value is "false").

Sample Brahms commands explained

The Brahms commands sent into and out of the environment are displayed in the text box at the top of the BrahmVE window (see figure 9). A > or < symbol indicates whether the command is being sent from the environment or from Brahms. Several of the commands have been extended to provide the capabilities required for this simulation. The most significant is the addition of the "interface" commands. These relate to “Settables” and “Detectables” (commands that set up the simulation and request reports or trigger actions), and are used in both directions. For example, an interface command will set the value of a Settable, or activate a Detectable, while a similar command in the other direction will notify Brahms of a Detectable being triggered.

Below is an example series of Brahms commands:

<interface|setSettable|0|projects.issvre.PSA|Watching|projects.issvre.flashlight1
activity|move|118.944999694824|-1|projects.issvre.PSA|Walk||projects.issvre.CentrifugeAccomodation.center|projects.issvre.USLab.center
interface|setSettable|0|projects.issvre.PSA|Watching|projects.issvre.Astro
>interface|detectable|108.944999694824|projects.issvre.PSA|Search|true

In this sequence, the virtual PSA agent has reported the triggering of its Detectable "Search". This resulted in the setting of the PSA Settable "Watching", making it watch the flashlight when Idle. It was also told that at the time 118... it should move from the Centrifuge Accommodation to the US Lab. The -1 informs it to take as long as it needs to (rather then taking a prescribed amount of time, resulting in the PSA moving very quickly). A complete documentation of the Brahms action command set and related documentation is available at the project website referenced in [24].

Implementation of a virtual environment inventory and geometry reporting system

The Brahms team requested the implementation of a BrahmsVE export command that would report the geometry and inventory of objects within the VE. A version of this was built and will be completed as part of Phase II. This will allow the Brahms modeler to “query” the VE and gain the starting set of definitions to then use in modeling exercises. This eliminates the “double work” of defining the geometry and objects both in Brahms and then again in the VE.

Current Limitations of the present BrahmsVE platform

The major current limitation with the current BrahmsVE platform is the lack of the return interface to communicate the results of Detectables and Interface Alerts to the Brahms environment running on its server. The implementation of new code in Brahms is being scheduled to handle the syntax of messages we are now generating. Phase II will see the full synchronous interface implemented and tested in several sample applications.

Limitations with line-of-sight testing

The primary technical limitation relates to the line-of-sight/ray intersection functions. These are a group of functions designed for detecting points of collision along a line (for example, finding out where a laser beam strikes an object, to place a reflection object/sprite). A limitation of these functions is related to the way Atmosphere handles loaded models. When performing a ray intersection test on the world geometry, one is required to use the ENTIRE geometry, including any models loaded by the script (be they Atmosphere objects or Viewpoint). As a result,  you cannot choose to "see through" objects that are dynamically placed, and you cannot tell the difference between a model and a wall. An example of this is when the PSA attempts to look for a tool. By performing a ray intersection (line-of-sight) test in the direction of the tool, it can determine if it can see the tool or if there is something in the way. However, you cannot tell what the object is. If the object that this test detects is supposed to be transparent (eg the PSA's laser beam), you cannot detect it.

A related problem is that the PSA and astronauts have to "look" from a point outside their own body models, otherwise the agent will see its own body, and think that something is in the way. This can lead to problems; for example,  if the agent is near a wall, the agent can often see through it (as the offset to avoid seeing itself places the point it is looking from on the other side).  A place where this is more of a hazard is in path finding. The path-finding routines perform these line-of-sight tests between pre-defined nodes (waypoints/Area) to determine which it is able to travel to, in an expanding pattern until it finds its destination. However, if something is blocking one of these waypoints (an Astronaut moving temporarily through that space, or even the model of the Agent that is attempting to find the path), these line-of-sight tests will fail, and it will think there is no way to reach the destination point. In the current implementation this is overcome by continuously retrying if a path isn’t found (hoping the astronaut will move). Often this can last for an extended period of time (especially since the line-of-sight tests are not 100% accurate when working with animated Viewpoints), causing the simulation to appear stalled.

Other implementation Issues

The current implementation does not use the Havok physics engine. This is due to a number of reasons, including inflexibility in the physics modeling (center of gravity is always center of geometry), physics modeling not being aware of changes in a Viewpoints geometry due to animation (at last test), manikin collisions (an astronaut’s entire body will bounce from the wall, rather than just his arm bending), and speed (particularly during collisions).

All of these limitations and implementation issues will be addressed in Phase II, as described in Part 3 below.

Part 3 Technical Objectives and Work Plan

3.1 Technical Objectives

After three years of work on the BrahmsVE platform and the successful completion of the goals set forth in this Phase 1 SBIR, we have arrived at a key point in the project. We now have a direct roadmap to completing and rolling out BrahmsVE as a production platform for extensive use within NASA, for other government agencies and for commercial users. This section will outline the technical objectives that will be met in Phase II that will bring BrahmsVE from its development phase to a 1.0 commercial release.

Figure 22: Future flowchart showing planned real time connection to Brahms and use of PHP to both synchronize virtual environments, archive changes in a SQL database.

Figure 23: Future flowchart showing planned upgrades to the OWorld engine and the employment of Havok physics, procedural animation and a more intelligent Path Manager

Roadmap to release

Figures 22 and 23 above illustrate the final architecture for the BrahmsVE 1.0 release. This is the completion of the vision of the architecture as specified in figures 20 and 21. In addition to addressing some of the limitations identified in Part 3 (line of sight issues and the need for the physics engine) we will endeavor to deliver the following new components in Phase II:

a) Database (for dynamic modular code loading) – shown in figure 22

Currently BrahmsVE is based on a static Javascript framework. However, we were always designing with the idea in mind of using a back-end database. OWorld is a modular design so that parts can be added and removed "on-the-fly". For example, for a given simulation, an agent might only be required to perform a small number of actions, so we would employ a database archive to deliver only the code for these actions rather then loading every available action.

An example would be having multiple astronauts in the ISS. While both astronauts would use the same base code (as both are Agents) and would each use some of the same code (moving, watching other Agents), each astronaut would only be provided the code relevant to what it is required to do. So the astronaut that is never required to "open can with fork" will not know HOW to "open can with fork". However, to give that astronaut that ability, the code would simply be dynamically loaded from the database/archive.

b) Database (for synchronization) – shown in figure 22

In BrahmsVE there is currently no synchronization between different instances of the environment, resulting in each visitor to the environment seeing something different on his or her workstation. Using a combination of PHP and a database (likely to be SQL), it will be possible to provide synchronization between all instances of the environment. This capability will be delivered in Phase II.

c) Normalization with Brahms Commands – shown in figure 22

Phase II will provide support for the final normalization of Brahms command types with the current “Brahms action” commands we are now processing. At the same time, we will develop a new class of Brahms commands that the AgentiSolutions team will implement. These commands permit interruptible actions, such as collisions, to be communicated to Brahms and for Brahms to respond (by switching action frames).

d) Brahms real-time synchronous interface – shown in figure 22

The key remaining largetechnical component to the BrahmsVE architecture is the implementation of a full synchronous interface between Brahms and the OWorld engine. Currently we are parsing Brahms actions generated by real Brahms models, but simulating Brahms interaction once the model begins to execute. Brahms team members are currently planning to implement the facilities to catch and generate real-time messages to the VE. We will be completing a two-way dialogue communications system utilizing XML/HTTP and PHP/SQL in Phase II to accomplish this real-time synchronous interface.

e) Building VE into Brahms developer environment (not shown in the figures)

The Brahms team has requested that the VE be visible inside the Brahms authoring environment, for a seamless means to test new Brahms models with the VE. We have determined that the Java Native Interface will be able to embed the Internet Explorer instance within the Brahms developer interface and this will be completed under support for Phase II.

f) Implementing voice synthesis and voice recognition to instruct agents
(will be provided as an interface in figure 22)

CommandTalk [25] and other environments built for military and civilian use have been proposed for use by John Dowding at Ames/RIACS to give BrahmsVE a voice recognition environment such that researchers can give voice commands to the agents within the virtual environment. As an additional project under Phase II support, we will add text to speech voice synthesis so that agents within the VE can be instructed to report their activities by voice. Voice synthesis was used to good effect as a training tool within the Steve environment [5].

g) Procedural (skeletal) animations – shown in figure 23

Procedural animation using a skeletal (skinning) model will drastically improve the quality and detail of model animations in BrahmsVE. It will allow for "rag-dolling", creating more fluid motion animation, as well as providing collision detection with a greater degree of granularity. This means that rather then detecting a collision with an object and reacting as an entire static object, portions of the model will be deformed (along bone structures), for example bending at the elbow when an arm brushes a wall. Animations will also become more dynamic. For example, rather than having a "put down" animation that shows the model placing an object always in the same location, the animation would be dynamically changed to show the model placing the object where it will actually go. It would also permit animations to allow for collisions. For example, if while drifting through the ISS an astronaut were to brush a wall, rather then either A) his arm going through the wall (non-solid object), or B) his entire body bouncing off the wall (solid object), he would be able to bend his arm out of the way. Ken Perlin of the New York University Media Research Laboratory has offered to provide procedural animation technology [26] for this project.

Maarten Sierhuis of the Brahms team has requested that Phase II deliver a closely coupled integration of an agent-based activity subsumption architecture for modeling human behavior with a low-level gesture generation capability to develop a high-level behavioral VE bot-language for easier development of the next generation VE applications. This is requested based on needs at NASA for uploadable just-in-time mission training software and the need to include models of human behavior and work practice for a "life-like" interaction between astronaut avatars and bots in a virtual world training application. Procedural animation will be able to deliver on this need.

h) Improved Path Finding – shown in figure 23

Bill Clancey of Ames has requested the building in of autonomous path finding capability into the VE agents. By combining increased geometric detail with advanced scripting, the path finding algorithms will be able to use all of 3D space, rather than the current lines-between-defined-nodes method. This will greatly increase the quality of the path finding, including things like curved paths, and moving around obstructions. Clancey’s request is that agents in BrahmsVE be able to “learn” new paths and report them back to the Brahms model. This will be an essential facility for the use of BrahmsVE in mobile agents and other semi-autonomous vehicle projects.

i) Implementation with Havok physics engine – shown in figure 23

With many of the above enhancements we will come to the point where we can activate the Havok physics engine built into the Atmosphere component of BrahmsVE. Adobe, provider of Atmosphere, has built in callback functions to permit us to use the physics engine for fine-grained operations and this will be fully implemented in Phase II.

j) BrahmsVE Test and Reporting Subsystem – part of database in figure 22

This new module will enable a log of reportable actions and statistical monitors that can provide an effective test and measurement of any BrahmsVE model and will be used in the test and validation phase described below. In addition, the VE object inventory and geometry reporting system will be expanded in this subsystem.

k) Test and validation of Phase II BrahmsVE platform

With the addition of the above components to the BrahmsVE architecture we will enter a test and validation phase of the BrahmsVE 1.0 platform. As with the prior years of work on BrahmsVE, testing and validation were carried out by building simulation scenarios in the platform and then testing them for both effectiveness and weakness. A significant benefit of this approach is that we obtain sample applications to be used both in the packaging of the BrahmsVE commercial release, including developer examples and market-opening examples for promoting the platform to industry. We propose the following sample applications for implementation in the test and validation of BrahmsVE during Phase II:

1. Expanded ISS with next revision of PSA, EVA activity :  Fully populate the virtual ISS with the full crew complement of three, plus shuttle crew visitors, add two models of PSA and test a number of simulation scenarios for teamwork with the ground (JSC Mission Control, see 3). This application will tie in with the recent work by the Brahms team to model work practices aboard ISS [28,29].

2. FMARS/MDRS Unified Habitat model :  Both floors of habitat, and exterior geography and vehicles, all operating in the same virtual environment. A number of models will be tried here including a complete “water tank filling” exercise which involves an EVA with vehicle, upper and lower floor coordination and feedback between crew and agents.

3. NASA JSC Mission Control operations or Science Backroom operations : BrahmsVE model of JSC Mission Control or a Science Backroom (Ames, JPL) to bring the ground support team into the full BrahmsVE modeling capacity. Tie model from mission control to actual activities aboard 1 or 2 above to create a unified ground/mission model.

4. Virtual Windfarm and Hydrogen Production Facility for DOD and DOE : Based on January 2003 presentation to Office of Secretary of Defense of initial windfarm model [27], extend model to factor in humans/vehicles for maintenance, build in hydrogen production facility. Use this model for opening important customers in DOD, DOE and energy producers.

5. Lights-out automated factory :Model of an existing all automated factory setting with occasional human participation in the form of maintenance personnel. Gain understanding of impact of human agents on uptime, from gaining an estimate of response time to fix errors. This sample application should help us present BrahmsVE to commercial customers in the manufacturing field.

Measuring applications on the virtual test fixture

In Phase I, we ran eleven simulation scenarios in the ISS/PSA BrahmsVE application and gathered empirical findings against the real experience of the QSS/PSA team who are employing a physical test fixture at Ames [30]. These empirical results were reported in section 2.3 above. In the Phase II virtual test fixture, we will use a new BrahmsVE Test and Reporting Subsystem to gain quantitative analysis of the performance of the platform and the applications. We plan to use this more formalized method to meet the goals from the solicitation set forth in Table 3. In addition, field science Ethnographic techniques described by Clancey [31] as well as techniques in collaborative decision-making and intelligent reasoning described by Wilkins, Jones, Bargar, Hayes, and Chernychenko in [32] could be applied to the evaluation of the effectiveness of the BrahmsVE platform.

Table 3: Goals from Solicitation

Goals set forth in solicitation	How BrahmsVE 1.0 test and validation will address this goal
On-board systems must aid people in diagnosis and repair and enhance safety	Use BrahmsVE to permit training in a virtual test fixture “just in time” prior to diagnosis and repair operations. BrahmsVE will be operable on standard computers by crew of ISS or other mission while on-orbit. In future, sensored agents can support a virtual world model tied to real mission support capability.
Interface design of mobile or built-in agent-assistants, human perceptual-motor coordination, cognitive operations, and group dynamics	Use BrahmsVE to train users to navigate the astronaut agent while communicating with agents. Permit several users to enter the scene simultaneously as “avatar astronauts” and interact with a single robotic agent or system, testing group dynamics in a kind of “learning game” environment.
Permit the testing of procedures where machine systems complement human activities	Utilize BrahmsVE in a training exercise where agents move in tandem with operator-astronauts to provide -camera sightlines to a second astronaut-agent (the viewer) or use a laser pointer for indications (teamwork with capcom).
Enable just in time training	BrahmsVE is being developed as a just in time training tool that can be run on standard personal computers on low bandwidth networks.
Understanding appropriate task separation between human and machine systems	Use BrahmsVE to test where mobile agent could play a role and take on human tasks where appropriate. Examine error rates of the agentin complex path navigation suggesting limits of a mobile agent (as in fan blowout on ISS model in Phase I).
Develop robust control systems and exploration tools that can be understood by people, easily learned, maintained, and directed	The later addition of the “tactile interfaces” on the 3D models within BrahmsVE as well as web-based interfaces (akin to a control panel) will permit experimentation with learning and operating a mobile agent or space suit. Need identified by Story Musgrave on HST repair mission.

l