DigitalSpace Commons
Data-Driven Virtual Environment Assembly and Operation
An extended abstract for the Virtual Ironbird Workshop
NASA Ames Research Center, Naval Postgraduate School
March 31, 2004, April 2, 2004
Bruce Damer, Stuart Gold, Dave Rasmussen, Merryn Neilson, Peter Newman, Ryan Norkus, Blair Berkelmans,
DigitalSpace Corporation (
William Clancey, Maarten Sierhuis, Ron Van Hoof, Mark Shirley, Tom Cochrane, NASA Ames & contractors

From the earliest work on 3D virtual environments for modeling, simulation and visualization, the field has been characterized by the predominance of hand-built content. As the need to represent large datasets in 3D in such applications as visualizing wind-tunnel aerodynamics or geospatial imagery, databases began to be employed to assemble 3D scene graphs. In the mid-1990s as Internet-based social and gaming multi-user 3D environments such as Alphaworld [1] began to propagate, the use of databases expanded to support the Lego-like placement of 3D objects as parts making up large scenes and databases were also employed in the managements of all user transactions. As with the increasingly database-driven World Wide Web, there was the beginning of a perceived convergence of IT and 3D [2]. Work in the early years of the VRML (now Web3D) Consortium on enabling databases to interact with 3D web-based content [3] led to proposals such as SQL3D [4]. Erick Von Schweber, SQL3D’s architect, wrote in 1998 of the sometimes-difficult boundary between the database and the 3D scene graph:

…with SQL3D the main point I was trying to make is that a clean an effective separation of concerns needs to be achieved, letting data management handle what it does best and the rendering engine do what it does best …we saw that the scene graph should become a local database, though a high performance one with very specific and optimized traversal paths, collaborating with other similar databases on other nodes on the network, forming a federation. [5]
1. DigitalSpace and early database-driven virtual environments

Figure 1: Virtual tradeshow space pieced together by database as part of the multi-user Avatars98 online event, November 1998.

DigitalSpace, working with the Contact Consortium, achieved a modicum of this separation of concerns with the 1998-2003 experimental annual projects to construct virtual tradeshows within the Active Worlds multi-user 3D environment for the yearly “Avatars” cyber-conferences. This evolving application connected a database, a web-server and a 3D virtual environment that permitted entrants to the cyber-conferences to design “virtual booths” assembled onto a navigate-able 3D tradeshow floor (figure 1 above) and to create “virtual discussion rooms” (VDRs) (see figure 2 below) hosting live talks with dynamic presentation materials.

Figure 2: virtual discussion room permitting users (forground) to chat with a speaker while witnessing a slide show operated by a database-driven “bot” (virtual headquarters project, March 1999).

To generate these spaces, participants simply selected choices on a 2D web form and created the resulting database record that was then used to assemble, present and operate these spaces during the live, interactive event. While databases had long been used to generate 3D spaces, the use of a database to also generate and operate live interactive elements was possibly a new development for online virtual environments. In the case of the VDRs, the database interacted with “bots” (server-based agents), which were instructed to monitor chat, place presentation materials on surfaces and follow other instructions such as text messaging to all users the entire extended event space.
2. NASA utilizes virtual environments in critical new roles satisfying training needs and managing mission complexity
Just in the past half-decade, NASA and its collaborators have made great progress toward the vision of effective human-machine augmentation. Computing architectures such as APEX [7,8] and Brahms [9-15] (see Figure 3 for a section of Brahms AgentViewer user interface below), in which support for multi-agent simulation with humans-in-the-loop has been developed.

Figure 3: Interface view from Brahms AgentViewer Figure 4: STS-61, the 1993 Hubble Space Telescope (HST) Repair mission team using VR training simulator Figure 5: Computer generated scene depicting the HST capture and EVA repair mission for mission planning

NASA began to use virtual environments and tele-operations in the 1970s culminating in the extensive use of VR in crew training for 1993 STS-61 Hubble Repair mission (figures 4,5) [15,16]. The JSC VR Laboratory, projects at other NASA centers [17] and the ARC/JPL use of a virtual environment for the 1997 Pathfinder Mars mission [18] suggested that synthetic environments would become key tools in all aspects of future missions.

Figure 6: Robonaut in tests at JSC, June 2003 Figure 7: PSA exterior instrumentation Figure 8: 3D exterior reconstruction from Intelligent Virtual Station

Recently the SimStation and Intelligent Virtual Station (figure 8) at ARC [19,20] and the use of VR and virtual worlds in projects to assist with the development of robotic agents including PSA (figure 7) at ARC [21,22,23], DART and Robonaut (figure 6) at the Dexterous Robotics Laboratory at JSC [24,25] has brought virtual environments to the forefront as a design tool to create flight-ready robotic hardware.

Elsewhere, industry, government and university laboratories have developed virtual environments for training and design/test, including Transom Jack [26], Steve from USC [27], environments for submersibles and navy operations at the MOVES Institute at the Naval Postgraduate School [28,29] and many more projects too numerous to list here.
3. DigitalSpace and NASA partnering to develop database-driven virtual environments
Starting in 1999, DigitalSpace began to use its earlier experience in developing database-assembled, agent-driven virtual environments to work with the Brahms team at NASA Ames Research Center. A series of funded projects beginning in 2000 allowed the combined team to develop the BrahmsVE (Brahms with Virtual Environment), which combines the discrete multi-agent Java-based Brahms [9-15] with DigitalSpace’s OWorld multi-threaded scene graph architecture and Adobe’s Atmosphere [30-32] 3D web-based browser plug-in. A complete synopsis of the BrahmsVE development is available in [36].

To provide insights into how a database can drive both the assembly and operation of a virtual environment, we will detail the architecture, experience and challenges surrounding the implementation of two BrahmsVE applications being developed in collaboration with NASA Ames: SimHab and SimStation Online. We hope that this initial experiential framework can, over time, evolve into a body of practice in this new field within the 3D modeling and simulation industry.
4. BrahmsVE Architecture
Before we examine the details of the SimHab and SimStation Online projects, let us present a brief overview of the BrahmsVE architecture.

A virtual environment by itself is of little use without a powerful back-end architecture that can represent the complexity of human-machine systems. For over a decade, teams at NYNEX, the Institute for Research on Learning and now, at Agent iSolutions, working with NASA Ames and RIACS, have been developing Brahms, an intelligent multi-agent environment used for modeling, simulating and analyzing work practice. Brahms is a data driven (forward chaining) discrete event environment usable for simulation purposes as well as for agent-based software solutions requiring the use of intelligent agents. From Brahms documentation:

Brahms allows us to model the work activities of each type of role, and each individual (or artifact) playing that role in an organization. The focus of a Brahms model is on the context of work, meaning, how does the work really happen. One of the essential requirements for Brahms is that we can model collaboration and coordination between people working on one task, as well as that people can work on more than one task at a time, and are interrupted and able to resume their activities where they left off.

Prior to the partnership with DigitalSpace, Brahms models could only be viewed in execution using a timeline bar chart-style interface (figure 3). It was determined that Brahms could become a much more effective tool if it were to include interface that allowed realistic reconstruction and interaction with 3D scenes representing the real world people and systems being modeled.

Three years of development described above have resulted in the following architecture connecting Brahms with DigitalSpace’s OWorld platform, the Adobe Atmosphere 3D web browser component and many additional subsystems (see figures 9,10). The combined environment allows researchers to assemble and operate networked simulations within a standard web browser on ordinary consumer personal computers with no special equipment. The entire BrahmsVE package is therefore easier and much lower cost to operate, develop models for and to introduce into geographically separated teams.

Figure 9: BrahmsVE client-server architecture

Figure 10: Current high level process architecture for BrahmsVE/OWorld platform

BrahmsVE is constructed out of lightweight web-deployable components. The Atmosphere 3D web-based player component is used in conjunction with OWorld, DigitalSpace’s platform for driving multi-threaded simulations in a 3D space supporting two-way communications with two servers:

1. Brahms, running server-side, is generating the initial setup of the 3D simulation and thereby executing agents with “beliefs” to take actions that are mirrored in the virtual environment and

2. The DigitalSpace BrahmsVE server, a MySQL/PHP based server which supplies web page delivery, configures the initial state of the simulation, builds the scene graph from existing 3D components and scripted animations, and then operates as the synchronization and reporting mechanism during server operations

The DigitalSpace BrahmsVE server has a key role in reporting information and exceptions back to Brahms such that Brahms agents receive “real world” input from the 3D environment. Some of this real world information includes collisions, which are reported when geometric volumes intersect in the 3D Atmosphere scene. Atmosphere uses the built-in Havok physics engine to report collisions to OWorld, which communicates these events via PHP to MySQL and the MySQL database then updates a table available to Brahms, thereby reporting this exception. Brahms may then alter its action frames or generate new frames to deal with the exception and then restart or continue the simulation. Throughout this entire process a system clock is employed to synchronize the activities within the scene graph and across multiple browser views.
A true marriage of the database and the scene graph
Figure 9 details the communications mechanism between clients and servers while figure 10 provides a sketch of the OWorld and Atmosphere client components. A number of databases are employed including MySQL on the server, Brahms’ own database which tracks agents, geography and activities, and a series of embedded simple databases are executing on the client side to manage “settables” and “detectables” (initialization and reporting instructions) and libraries of geometric data including animation scripts and wayfinding pathways.

Thus, BrahmsVE is a true marriage of databases and scene graph, with an evolved separation of concerns in the spirit enunciated by Von Schweber in the introduction to this paper.
5. BrahmsVE /SimHab and other early BrahmsVE applications
BrahmsVE is the result of several years of intense work among the Brahms teams at RIACS, NASA, and DigitalSpace beginning with an STTR in 2000 [30], continuing with work to model EVA and day to day operations aboard the FMARS/Haughton-Mars Project analog habitats (figures 11,12) [31] and leading to a specification for the OWorld and Brahms interfaces [32]. In 2002 DigitalSpace completed several “day in the life” scenarios aboard FMARS in the project called SimHab (figure 13) [33] as well as a concept project for Geoff Briggs at ARC and MER outreach (figure 14) for presentation at JPL [34]. Finally in 2003, BrahmsVE was developed further to support simulation of PSA aboard ISS (figure 15) [35] and an extensive water tank filling simulation for FMARS (figures 16-18).

2000-2001 (for STTR and RIACS)
Figure 11: Modeling of virtual environment for the HMP/FMARS habitat, rover and astronaut from first BrahmsVE feasibility project Figure 12: Scenario of the Bill Clancey agent astronaut enacting a complex animation sequence inside the virtual FMARS habitat
2002 (for RIACS)
Fig 13: Planning meeting simulation from 2002 BrahmsVE project to model a day in the life of the FMARS analogue Mars habitat (SimHab) Figure 14: MER rover modeled for JPL concept presentation
2003 (for SBIR and RIACS)
Figure 15: Modeling PSA inside ISS engaging in a tool location and reporting exercise, utilizing a laser pointer Figure 16: Figure from water tank scenario Figure 17: New exterior FMARS view Figure 18: Water pump station
Sources used to create SimHab and other early BrahmsVE applications
Many sources were employed to assemble and operate the 3D scene graphs for BrahmsVE/SimHab and other applications:

• Brahms Models developed in the Brahms development environment and AgentViewer

• Still images of FMARS and MDRS crew activities

• Documents detailing the layout of FMARS, MDRS, ISS/PSA and MER

• Video from FMARS showing astronauts in routine activities (this was our primary source for the “day in the life” series)

• Written notes detailing crew activities (logs)

• Rendered computer graphics for concepts of the PSA in flight aboard ISS

Future development of BrahmsVE/SimHab
Development of the SimHab application through FY2005 will concentrate on the delivery of a fully tested two-way synchronous interface between the scene graph and Brahms such that Brahms agents will be rendered with a real-time synchronized clock in the 3D scene graph. Therefore, a robot or human agent moving through steps in the Brahms model will have a synchronized view in the 3D scene. This 3D representation will be subject to exceptions, such as collisions, which will be reported back to Brahms in real-time, permitting a modification of the subsequent Brahms actions (such as an astronaut stepping around an obstruction). By the conclusion of the project, the full BrahmsVE environment will be packaged and delivered as an end-user product for use throughout NASA, other government agencies and commercial enterprises.
6. BrahmsVE as applied to the SimStation Online application
In our second experience case study, we will examine the recently undertaken SimStation Online project, being performed in collaboration with the SimStation team at Ames for customers at NASA’s Johnson Space Center.
Short term goals of the SimStation Online project
The immediate goals of the SimStation Online (SSO) project [37] are to supplement the current SimStation efforts at Ames and Johnson Space Center with an online rendition of the 3D component structure of current and future configurations of the International Space Station (ISS). The SSO application interface is presented within a web interface using the Adobe Atmosphere player, and DigitalSpace OWorld framework and can be shared and synchronized via SQL database with a group to permit collaboration. A key element of SSO is that by selecting a 3D component in the scene graph a database call is initiated that brings up documents, database queries, close-out photographs (photographs of components taken just before launch), video and notations.

Therefore, in a group setting, users can select a component, bring up common documents, associate new documents, images, and video notations and store these associations with a particularly 3D ISS configuration. Future goals for SSO include visualizations of the assembly sequences, EVA (extra vehicular activity) procedures and computational capabilities such as estimates of the length of cabling and inventories of parts.

Figure 19: Current view of SimStation Online showing 3D rendering and Constructor
Current state of SimStation Online development
SSO development is in its early phases with support of an SBIR Phase I contract which completes in June of 2004. Figure 19 above is a screen capture of the SSO application running within the Internet Explorer browser. The 3D window displays a rendition of a fully assembled ISS future configuration (15A) while the “Constructor” window shows a view into the graph database model for the assembly of that configuration. In this application 550 components including major modules and trusses, shuttle orbiter, re-supply vehicles, solar panels and small items such as handrails have been assembled by SQL queries driven by an XML-based schema and served by HTTP to the web-browser running the OWorld framework and Adobe Atmosphere player. At the time of the writing of this paper, the ability to attach documents, close-out photography, video or notations to each component was being implemented.
Sources used to create and operate the SimStation Online models
SimStation Online is a new project but employing similar modeling sources used for several years on the SimHab project, with a few additional inputs. The model shown in figure 19 was created from:

• Provided DirectX and VRML models of ISS exterior, shuttle, suits and parts (from Boeing and JSC)

• XML descriptions of ISS configurations

• Documents and web-site data detailing the currently deployed ISS and future configurations.

Future work on SSO will derive 3D modeling data and activities from the following sources:

• Video from ISS, NASA’s JSC Neutral Buoyancy Laboratory and other sources

• Still photos from NASA training, contractors, and close-out photography

• Written EVA checklists and procedural guides to operations

• Brahms models of human/systems activities surrounding ISS simulation scenarios.

Architecture of SimStation Online
SimStation Online is built on the BrahmsVE architecture described in the section on SimHab above. SSO has required us to make several important extensions to BrahmsVE including:

1. Full assembly of a 3D virtual environment component-by-component using a MySQL database driven by PHP commands from a web component. This web component, known as the OWorld Information Broker (OWIB) is able to assemble any configuration of a virtual ISS model drawing its sources from arbitrary we locations of locally from a file store or distributable storage medium such as CD or DVD. Figure 20 below illustrates the content assembly architecture for SSO. A variety of protocols are used to set up and operate layers of interfaces including a database-driven web site with various levels of user access, and web-embedded BrahmsVE applications for the virtual environment itself.

Figure 20: Content architecture for SimStation Online

2. Also utilizing the OWIB, we are employing an extended variety of communications protocols to enable communications with Brahms, Atmosphere, and Talkspace, a voice over IP technology developed by DigitalSpace. Talkspace will facilitate the triggered playback of voice loops commonly heard as part of EVA and ISS assembly operations. Figure 21 below details this communications architecture.

Figure 21: Communications architecture for SimStation Online
Future Goals for SSO
An example application of SSO for collaborative mission planning follows. This was paraphrased from SimStation team interviews at NASA’s Johnson Space Center carried out by Tom Cochrane:

The future addition of the Node 2 component to the ISS will be a complex operation, involving the lifting of Node 2 to orbit, docking of the shuttle, the lifting of Node 2 from the shuttle cargo bay and affixing to the US Laboratory module. After this, the shuttle must undock from Node 1, the adapter for Node 1 must be removed, then Node 2 must be detatched from the US Lab, moved and attached to Node 1. The final step is that the docking adapter must be attached to Node 2 to allow re-docking of the shuttle.

We hope that continued support for SSO development would permit us to provide a detailed 3D simulation of the above operation, which could be shared via the web with the engineering team and for use in astronaut refresher training. One key element of this operation is that the geometry of the modules and interface adapters must be well known as each step involves cabling (temporary and permanent) between modules. If at any step the cabling is of insufficient length, this could lead to serious problems during the operation. We would hope that SSO would augment SimStation, the Neutral Buoyancy Laboratory and the other visualization and training systems in use at JSC and NASA to permit safe completion of such complex on-orbit operations.

Other areas that will need to be addressed in the future for SSO include:

1. Connection of the platform with other efforts a NASA and elsewhere.

2. Assessment of the platform: training effectiveness evaluation, knowledge capture through sensored objects and activities, comparison of simulation versus on-orbit operations.
SimStation Online and databases
Experience in developing SimStation Online provides yet more justification why databases are integral to the creation of useful applications that employ 3D scene graphs. Effectively acting a 3D interface for access to and sharing of web-based and database-sourced documentation, SSO is an example of a 3D virtual environment acting as an exotic database search and updating interface.
In the case of both SimHab and SimStation Online, development is in a very early stage with focus on enabling and validating platform capabilities. The viability of both applications and their underlying architecture in supporting real mission goals has yet to be tested. Well-established methods from previous virtual environment simulation platforms have been used and close consultation with the end customers of the applications has been made a priority. Within a few years we hope that this synergy of databases, agents, 3D virtual environments, and collaborative web delivery will provide NASA and other end-users a capable platform to meet design, training and public outreach missions.

Current and future work on SimHab and SimStation Online is available at [31-37].
Work on BrahmsVE/SimHab would not have been possible without the long time support and guidance of Bill Clancey, Marten Sierhuis, and Ron Van Hoof, of RIACS and NASA Ames Research Center. We would also like to acknowledge the recent support and guidance for SimStation Online from Mark Shirley and Tom Cochrane of NASA Ames Research Center. Finally, we cannot conclude without thanking Michael Sims of NASA and Julian Gomez of RIACS for inspiring us to pursue the development of virtual environments for serious NASA applications.
[1] Alphaworld is described in the book by Damer, Bruce, Avatars, Building and Exploring Virtual Worlds on the Internet, (Peachpit Press, 1998), pp. 96-159.

[2] Gomez, Julian E., The Convergence of IT and 3D, Proc. Experiential E-commerce. 2001.

[3] Web3D (formerly VRML) Consortium and information about the Database Working Group, on the web at:

[4] Von Schweber, Erick, SQL3D documentation, on the web at:

[5] From a private communication with Erick Von Schweber, March 2004.

[6] Associated Press News Report dated 08/16/2002, on the web at:

[7] Michael Alan Freed. “APEX, Simulating Human Performance in Complex, Dynamic Environments”, A Dissertation Submitted to the Graduate School in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy, Field of Computer Science, Northwestern University, Evanston, Illinois, June 1998

[8] Michael A. Freed, Roger W. Remington (2000) Making Human-Machine System Simulation a Practical Engineering Tool: An Apex Overview. In Proceedings of the 2000 International Conference on Cognitive Modeling, Groningen, Holland;

[9] Clancey, W. J., Sachs, P., Sierhuis, M., and van Hoof, R.1998. Brahms: Simulating Practice for Work Systems Design. International Journal of Human-Computer Studies, 49, 831-865.

[10] Sierhuis, M. 2001. Modeling and Simulating Work Practice; Brahms: A multiagent modeling and simulationlanguage for work system analysis and design. Ph.D. thesis, Social Science and Informatics (SWI), University of Amsterdam, SIKS Dissertation Series No. 2001-10, Amsterdam, The Netherlands, ISBN 90-6464-849-2.

[11] Sierhuis, M.; Bradshaw, J.M.; Acquisti, A.; Hoof, R.v.; Jeffers, R.; and Uszok, A. Human-Agent Teamwork and Adjustable Autonomy in Practice, in Proceedings of The 7th International Symposium on Artificial Intelligence,Robotics and Automation in Space (i-SAIRAS), Nara, Japan, 2003.

[12] M. Sierhuis and W. J. Clancey, Modeling and Simulating Work Practice: A human-centered method for work systems design, IEEE Intelligent Systems, vol. Volume 17(5), 2002.

[13] M. Sierhuis, A. Acquisti, and W. J. Clancey, Multiagent Plan Execution and Work Practice: Modeling plans and practices onboard the ISS, presented at 3rd International NASA Workshop on Planning and Scheduling for Space, Houston, TX, 2002.

[14] A. Acquisti, M. Sierhuis, W. J. Clancey, J. M. Bradshaw, Agent Based Modeling of Collaboration and Work Practices Onboard the International Space Station. Proceedings of the 11th Conference on Computer-Generated Forces and Behavior Representation, Orlando, FL, May 2002.

[15] Loftin, R.B., and Kenney, P.J., "Training the Hubble Space Telescope Flight Team," IEEE Computer Graphics and Applications, vol. 15, no. 5, pp. 31-37, Sep, 1995.

[16] Engelberg, Mark[Ed] (September 11, 1994). Hubble Space Telescope Repair

Training System [WWW document]. URL

[17] Cater, J. P., and Huffman, S. D. Use of Remote Access Virtual Environment Network (RAVEN) for Coordinated IVA-EVA Astronaut Training and Evaluation. _Presence: Teleoperators and Virtual Environments_ vol. 4, no. 2 (Spring 1995), p. 103-109. (Training for Hubble Space Telescope repair.)

[18] Mars Pathfinder JPL site:

[19] Intelligent Virtual Station on the Web at:

[20] R. Papasin, B.J. Betts, R. Del Mundo, M. Guerrero, R.W. Mah, D.M. McIntosh, and E. Wilson, "Intelligent Virtual Station," Proceedings of the 7th International Symposium on Artificial Intelligence, Robotics and Automation in Space (i-SAIRAS 2003), Nara, Japan, May 2003.

[21] J. M. Bradshaw, M. Sierhuis, Y. Gawdiak, R. Jeffers, N. Suri, M. Greaves. (2001). Adjustable Autonomy and Teamwork for the Personal Satellite Assistant, in The IJCAI-01 Workshop on Autonomy, Delegation, and Control: Interacting with Autonomous Agents, Seattle, Washington, USA August 6, 2001. URL:

[22] Personal Satellite Assistant (PSA) Test Fixture (Greg Dorais, Yuri Gawdiak, Daniel Andrews, Brian Koss, Mike McIntyre) described on the web at:

[23] PSA Web site:

[24] Robonaut Web site:

[25] Rochlis, J, Clark, J.P. and Goza, M., "Space Station Telerobotics: Designing a Human-Robot Interface", AIAA Confeence on Space Station Utilization, Kennedy Space Center, October 2001.

[26] Transom Jack is described on the Web at:

[27] Jeff Rickel and W. Lewis Johnson. Task-oriented collaboration with embodied agents in virtual worlds. In J. Cassell, J. Sullivan, and S. Prevost, editors, Embodied Conversational Agents. MIT Press, Boston, 2000.

[28] MOVES Institute on the Web at:

[29] Blais, C., Brutzman, D., Horner, D., and Nicklaus, S., "Web-Based 3D Technology for Scenario Authoring and Visualization: The Savage Project", Proceedings of the 2001 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando, Florida, 2001.

[30] B. Damer, M. Sierhuis, R. van Hoof, B. Campbell, D. Rasmussen, M. Neilson, C. Kaskiris, S. Gold, G. Brandt (2001). Brahms VE: A Collaborative Virtual Environment for Mission Operations, Planning and Scheduling, Final Report for STTR Contract #NAS2-01019, October 8, 2001. URL:

[31] BrahmsVE/SimHab Project Home Page on the web at:

[32] Boris Brodsky et al, “TM00-0025 BRAHMS OWorld Event Specification Version 1.0 Draft”, August 14, 2002,NASA Ames Research Center.

[33] MER and Mars surface modeling projects on the Web at:

[34] BrahmsVE/ISS-PSA SBIR Phase I Project and reports Web Page:

[35] Brahms is described on the web at and in several papers at:

[36] DigitalSpace project pages and publications about SimHab and SimStation Online are available on the Web at:

[37] DigitalSpace SimStation Online project pages and publications are available on the Web at:

Return to DigitalSpace Documents