DigitalSpace Commons
FINAL TECHNICAL PROGRESS REPORT
NNA04AA32C SBIR I

BrahmsVE: Platform for Design and Test of Large Scale Multi-Agent Human-Centric
Mission Concepts

Report Date: July 19, 2004

Reporting Period: January 16, 2004 – July 19, 2004

PI/Contact: Bruce Damer, DigitalSpace Corporation

Table of Contents

PART DESCRIPTION

  Cover

1 Table of Contents

2 Identification and Significance of the Innovation

 
2.1. Identifying the Need
2.2. The Innovation: DigitalSpace's SimSpace

3 Technical Objectives
 
3.1. The Objective

4 Work Plan

 
4.1 Technical Approach
4.2 Task Descriptions
4.3 Meeting The Technical Objectives
4.4 Task Labor Categories and Schedules

5 Potential Applications

 
5.1 Potential NASA Applications
5.2 Potential Non-NASA Commercial Applications

6 Contacts

 
6.1 Key Contractor Participants
6.2 Key NASA Participants<
6.3 NASA and Non-NASA Advisors

7 Technical Activities

 
7.1 Cumulative Technical Activities
Future Technical Activities

8 Potential Customer and Commercialization Activities

 
8.1 Cumulative NASA Potential Customer Activities
8.2 Cumulative Non-NASA Potential Customer Activities
8.3 Other Cumulative Commercialization Activities
8.4 Future Potential Customer and Commercialization Activities

9 Resources Status

10 References

  Project Summary


Part 2 Identification and Significance of the Innovation

2.1. Identifying the Need

Space systems involving people and complex vehicles in long duration missions are becoming increasing challenging to design, test, train for, and safely operate during flight. With the Presidents new goals for long duration missions of exploration of the Moon, Mars and beyond [1], a whole generation of new software modeling and simulation tools needs to be brought on stream to enable design, acquisition and just-in-time training.
Simulation based acquisition: a new paradigm for space endeavors.
Successes in programs such as the Joint Strike Fighter (JSF) [2], the Boeing 777 [3] and other major vehicle acquisitions show that computer aided design (CAD) simulation of the entire physical vehicle in advance of any part being constructed reduces contractor error, advances time to flight, and enhances the final vehicle performance specifications. Similar CAD tools supporting a full “virtual vehicle full lifecycle environment” must now be employed ubiquitously within NASA and its contractor community for the new exploration mission to be possible.


Work practice design simulation and training key to safe operations


NASA’s vehicles and missions differ from military or commercial aviation in their need for careful human-in-the-loop work practice design. Manned space operations such as Shuttle and the International Space Station (ISS) have shown that complex vehicles where humans interact with a large number of subsystems and robotic agents are becoming increasingly difficult to manage. Mission control and astronauts alike often have difficulty visualizing the complexity of the flight environment and safety can be easily compromised by human error. Increasingly, synthetic environments are being integrated into training to permit astronauts to see procedures from new points of view, comprehend whole scenario complexity, and to aid planners in optimizing procedures. Some of the first extensive uses of virtual environments in mission training and operations was for the Hubble Telescope repair mission in 1993 [4-6] and the stereoscopically constructed 3D scene for Mars Pathfinder in 1997 [7].



2.2 The Innovation: DigitalSpace's SimSpace

DigitalSpace has received support from a variety of NASA centers and companies since 2000 to develop an open platform for cross-agency, multiple mission applications. This has led to what we believe will be an important simulation platform to support NASA’s new exploration mission: DigitalSpace SimSpace ™.

SimSpace was built to serve the multiple needs of systems within systems engineering and the spiral development cycles of the next generation of vehicles, a whole new iteration of design visualization and work practice simulators is being built.
Figure 1: DigitalSpace’s SimSpace Architecture

Figure 1 introduces the current components of the SimSpace architecture and illustrates that the open format of SimSpace supports (via Brahms Virtual Environment – BrahmsVE) both the Brahms and Mobile Agents projects of one ARC/JSC team (Clancey, Sierhuis et al [8-13]) while adopting the methodology a second ARC/JSC/Langley team (Shirley, Cochrane et al [14]). The open methodology of SimSpace enhances these two platforms, and any future platform with of the following unique and novel features:

·        Real-time 3D through the internet using a variety of 3D rendering technologies (commercial and open source engines).

·        Rapidly developed models and animation lower costs (employing a model repository which is the subject of a full Phase II development proposal).

·        Multi user interaction to permit shared presence in the simulations.

·        Collaborative database to associate existing vehicle databases with 3D scenarios.

·        Web-accessibility permitting universal access to the environments even employing computers in vehicles in flight.

·        Open scripting component architecture to permit physical and haptic device interfaces (planned).


The sum is greater than the parts

SimSpace combines 3rd party Foundation elements (Brahms, SimStation, shared models and any other open system) within a common Simulation layer (SimSpace including the BrahmsVE interface, Oworld Agent Information Broker (OWIB) and OWorld engine) and drives a flexible Presentation layer (commercial or open source 3D engines) enabling a much larger range of Applications than would be possible with a monolithic system. The next section covers some of these applications.
Applications built on the SimSpace platform

The approach has proven successful as SimSpace has begun to gain wider acceptance across NASA centers and within the contractor community. The following applications have been built on SimSpace and delivered to customers:

·        SimHab: FMARS and Mars Desert Research Station Analog Habitats (using Brahms from ARC - figures 2-4 below) [21].

·        SimRover: 3D scenarios of Mobile Agents work practice (current summer 2004) (Brahms from ARC and rovers from JSC). MER Rover surface operations reconstruction prototype in “Drive On Mars” (ARC VizOpps group, JPL surface data – figure 5 below) [22].

·        SimStation: Web-based collaborative rendering of SimStation application including exterior 3D ISS configurations with linked in photography and database documents (SimStation group at ARC, JSC, Langley –  figure 11 below) [14, 23].

·        SimEVA and SimIVA: EVA training prototype for Neutral Bouyancy Laboratory/Raytheon including CMG changeout for STS-114 and RPCM changeout on the ISS (SimStation group at ARC – see figure 10 below) [24]. Simulation of Personal Satellite Assistant in ISS IVA operations (ARC – figure 6 below) [15-17, 26]. Simulation of EVA on Mars for BrahmsVE application (ARC – figures 7-9 below).

·        SimMedical: VAST project for ISS crew medical training on CHeCS rack procedures (Lockheed Martin, ARC, JSC, Langley – figure 12 below). National Institutes of Health project for childhood autism, safety learning game (DoToLearn) [27].

·        SimVehicle and SimLunar: 3D simulations for Boeing of Lunar base and Crew Exploration Vehicle (CEV) concepts. Prototypes for Space Exploration Online (SEO), initiative for a massively multiplayer space oriented Lunar base reality game (Boeing CII project, ARC, JSC, MOVES Institute, Arsenal Interactive – figures 13-15 below) [28].

Pictorial descriptions of the past four years of projects employing SimSpace are presented below.


2000-2001

 
Figure 2: Modeling of virtual environment for the HMP/FMARS habitat, rover and astronaut from first BrahmsVE feasibility project (ARC) Figure 3: Scenario of the Bill Clancey agent astronaut enacting a complex animation sequence inside the virtual FMARS habitat (ARC)

2002

 
Figure 4: Planning meeting simulation from 2002 BrahmsVE project to model a day in the life of the FMARS analogue Mars habitat (ARC) Figure 5: MER rover modeled for JPL concept presentation (ARC)

2003

     
Figure 6: Modeling PSA inside ISS engaging in a tool location and reporting exercise, utilizing a laser pointer (ARC) Figure 7: Figure from water tank scenario (ARC) Figure 8: EVA exterior FMARS (ARC) Figure 9: EVA to water pump station (ARC)

2004

   
Figure 10: Simulated CMG changeout for STS-114 & RPCM changeout for ISS (Raytheon and NBL) Figure 11: SimStation Online database-driven construction of ISS exterior (ARC, JSC) Figure 12: VAST project: modeling emergency medical procedures aboard ISS (Lockheed Martin, JSC)
   
Figure 13: Virtual model of a concept Lunar lander (Boeing) Figure 14: Visualization of an early Lunar vehicle study (Boeing, from Eckart, Lunar Base Handbook) Figure 15: Real-time visualization of a Prometheus class interplanetary craft (Boeing)


We conclude with a description of the high level process architecture of the Simulation and Presentations layers is shown in figure 16 below. SimSpace is built using a number of web technologies including PHP, SQL, JavaScript, Adobe Atmosphere, the Viewpoint and Havok engines and common ActiveX components embedded in a standard Internet Explorer web browser. The internal architecture of the SimSpace engine Oworld consists of a state machine representing agents acting to enable behaviors of objects (astronauts, parts or systems) with a simulation scenario.

Figure 16: Current high level process architecture for the SimSpace platform


Commands initiate from any server-based environment including SQL databases, Brahms agents, or any third party simulation system. SimSpace communicates back to those server-based elements using web services protocols including SOAP or PHP. Geometry is loaded from a common repository in a number of formats and rendered together with animation within the Adobe Atmosphere 3D player or any other capable 3D engine.


Part 3 Technical Objectives


3.1. The Objective


In our original proposal we stated: It is clear that a major function of the ISS over the next decade will be as a test-bed and living laboratory for the development and long-duration certification of multi-agent, human-centric robotic and software systems. Indeed, the primary scientific and engineering output of the ISS may well be that of a body of practice and design experience in agent-based systems that can be applied to both future NASA missions as well as commercial applications back on Earth.

To help coalesce that body of practice, the islands of isolation that now exist between groups working on human-agent systems for ISS need to be bridged. It is proposed in this SBIR that we undertake a proof-of-concept project that creates a macro-scale simulation of a future ISS on which multiple agent systems are deployed. It is hoped that such a simulation, built through a process of consultation and utilizing BrahmsVE (and now SimSpace) can be made available online to researchers from several NASA centers and their collaborators and provide another mechanism for future collaboration across teams.


In the planning for this Phase I work it was determined that we would construct a complete external model of the ISS within the SimSpace platform and then add dynamic elements, an Extra Vehicular Activity (EVA) with segments of this ISS model. These two constructed components are documented in A) and B) below.

A) SimStation Online geometry and collaborative sharing component

As a chosen customer for this SBIR Phase I work, In February 2004 Mark Shirley and Tom Cochrane of NASA ARC tasked our team to produce a web-delivered prototype of their “SimStation” client application, which permits station engineers to view true fidelity CAD models of ISS configurations. SimStation had been successfully deployed at JSC and was in use by VIPER station engineers. It was felt that it would be of great value to have a collaborative version of SimStation allowing engineers to share configurations online and make annotations to views of ISS and associate close-out photography, video and database records.

Prior ISS modeling

In 2002 and 2003 the Brahms team completed a number of Brahms models to reconstruct activities aboard the ISS [13] which explored the use of the ISS model as part of an environment for teamwork between ISS crews and onboard software assistants and robotic systems and as a short term planning and scheduling tool for mission planners.

Other prior work on modeling the ISS includes the Intelligent Virtual Station at ARC [18, 19] as well as several efforts including telerobotics at JSC and in Moscow as part of the Russian space program [20].

SimStation Online

Based on the SimStation team’s guidance and with reference to other prior work, DigitalSpace constructed the SimStation Online application and delivered it for test in May of 2004. Figures 17-18 below show the elements of the SimStation Online (SSO) application, running in the Internet Explorer web browser.


Figure 17: SimStation Online running in Web browser, camera view selected inline to ISS orbit


Figure 18: SimStation Online showing selection of SO Truss in ISS model and different camera view from higher orbit

Explanation of the SimStation Online Application: User Experience

As figures 17 and 18 above illustrate, SSO is a web-based presentation layer on top of the SimSpace platform from DigitalSpace. In the central 3D viewing area a complete representation of ISS in configuration 12A is available for real-time rendering and exploration. Interfaces to predefined cameras are included with this model and allow the user to travel to preset viewpoints on the station geometry. The right hand menus show the current selected station component. In the first figure the user has selected the SM Core while in the second the SO Truss was selected. The selection of any part allows the user to add a textual notation, associated images (often close-out photography or on-orbit station video), station documents or direct database calls.

NASA domain experts informed us there are over 200 separate databases supporting ISS so this kind of application presenting web-based access to those databases could serve as an integration point for a great deal of valuable information for ISS assembly or maintenance planning.


Explanation of the SimStation Online Application: Architecture


SimStation Online is based on a combination of technologies:

  1. BrahmsVE, a platform based on a combination of Adobe Atmosphere and the OWorld engine developed by DigitalSpace. The Brahms discrete agent server is an optional component (and is planned for future use in SSO). BrahmsVE’s architecture is depicted in Figure 19.
  2. OWorld Animation Renderer (OWAR): a module developed by DigitalSpace which performs generated animations in the 3D scenegraph
  3. OWorld Agent Information Broker (OWIB): a MySQL application utilizing PHP to communicate with client installations of SSO. OWIB handles the brokering and assembly of all 3D objects, user log-ins and preferences, tours through the scene graph, association of URLs, documents, images and other records with elements of the scene graph, and all user interaction with the 3D geometry and associated web resources.


Figure 19: SimSpace and BrahmsVE architecture


Figure 19 shows the canonical SimSpace functional blocks which tie together a variety of technologies to produce a powerful and dynamic visualization and knowledge management system including:

Open extensions for the adding of the Brahms agent server, SimStation and any other NASA or contractor produced component SQL is the backbone of the SimStation Online system. All information used by SimStation Online is stored and organized using an SQL server powered by MySQL. Data comes in via multiple sources, including XML files and user interaction via the SimStation Online interface.

PHP is used to implement security. All sensitive operations (writing or removing information from the database) are first validated by PHP, then performed in a secure “server-side” environment. Information necessary for these actions is kept hidden from outside knowledge by the secure environment.

ActiveX is used to implement a high-quality interactive interface to the end user. It allows a higher degree of flexibility in the interface then available through other means, and improves responsiveness due to not having to notify the secure server of safe actions the user performs (e.g. viewing existing information).

HTTPS (Hypertext Transfer Protocol, Secure) is used to ensure that data transfer between the SimStation Online client and the server environment is secure. This ensures that information cannot be viewed, altered or replicated during transit. This “transfer level” security combined with our “information level” security ensures that information is added as it is intended, only by users with permission.


Figure 20: SSO Dataflow – Current

The dataflow model for SimSpace built to support SSO is shown in figure 20 above. All system architectural communications consists of a set of secure PHP calls to client side controls, the virtual environment (via ActiveX and the spigot interface to OWorld and Atmosphere) and then reciprocating communications to the SQL server. The above dataflow model illustrates the actions supporting a user logging into the SSO application and assembling an ISS CAD configuration.

Successes and Shortcomings

The SSO application met all the objectives of the Phase I SBIR project goals to model the exterior of the ISS. Our team was able to go beyond the original plan by permitting the selection and assignment of database records, documents, close-out photography and video to any component and to do this in a collaborative manner using a web-based SQL/PHP solution.

The sole identified shortcoming of the solution that would prohibit it from being deployed to VIPER team station engineers is the low rendering performance of the current Atmosphere 3D plug-in. This rendering performance is in the range of 4-8 frames per second when the standalone SimStation application developed at ARC is delivering 30+ frames per second. Adobe, provider of the Atmosphere renderer, recognized the shortcoming and is delivering to DigitalSpace a high performance version of the plug-in (going into test in July of this year).

It is suggested that as part of a Phase II proposal, that the SimStation Online platform be integrated with the new Adobe Atmosphere plug-in and deployed to JSC VIPER engineers for test.


Unique or novel features


The novel feature of this software is the web-database assembly of CAD models and presenting the complete 3D view along with interfaces to add records in a multi user setting. The advantages of this innovation is the ability to quickly create a model of a virtual vehicle and associate key object with parts of the vehicle including documents, photography, video or audio or database records. New problems arising from this innovation relate to the management of information in a more complex, web-based environment than normal CAD/CAM and planning tools are used to.


SimStation Online presented to wider NASA, commercial and DOD community


At the Virtual Iron Bird (VIB) workshop on April 2, 2004, the SSO application was presented to the NASA community from ARC and JSC as well as Naval Postgraduate School, DOD and commercial participants. This presentation led directly to the ISS Medical training application described in Part 5. Some of the conclusions from the SSO project and the VIB workshop as a whole are included below in the conference report [29]:

1. That there are islands of good work in the visualization field, even some agent-based and database-backed knowledge management systems underlying 3D scenegraph representations but that there is little in the way of cross-connectivity between systems or a standard methodology for connecting knowledge management assets to 3D information or interactivity.

2. That there was a big stress by some of the VIB presenters on open systems solutions which suggests that it is only open source or open script solutions that would allow for bridging between projects. Another factor is the scale of investment into different aspects. There seems to be a big investment in custom components coded in high level languages, such as C/C++ or Java but less investment, and more willingness to share, in scripted or database backed solution spaces (HTML, JavaScript, SQL, SOAP, XML (specifically X3D and XMSF from MOVES), PHP, Python).

3. It is therefore our conclusion upon talking to a number of VIB participants that for the field to grow it needs to establish more powerful application solution platforms by combining the best of the worlds of high level language front and back ends with the interoperability of open scripts, databases and open source components. In this manner, teams working on common problems could combine efforts to create single platforms that would service clients better.

4. As well as the business aspects of combining platforms, there is also the aspect of the high level, canonical definitions of what makes a capable platform supporting a variety of tasks. I counted the following in the application areas from the VIB workshop presenters and attendees:

·        visualization of data in the abstract

·        modeling, simulation, work practice studies of "real world" (habitat, station, vehicle) spaces

·        assembly planning for ISS

·        in-orbit and surface operation design visualization

·        astronaut or mission controller refresher training and

·        live as-it-happens 3D simulation to guide mission operations

The challenge is to create set of common platform interfaces and more importantly: common concepts that can underlay all of the above applications.



B) SimStation SimEVA component


As called for in the Phase I proposal, we committed to add dynamic elements (Extra or Intra Vehicular Activities) into the 3D ISS representation. We were requested by the ARC team to model two upcoming EVA procedures for the Raytheon contract managers at the Neutral Bouyancy Laboratory (NBL) at JSC. The ARC partners felt that it would be of more immediate value to model actual training procedures rather than future robotic assistants (PSA and Robonaut) as we had originally proposed.

The ARC and JSC partners supplied several hours of NBL video of two astronauts training for the upcoming STS-144 shuttle return to flight on which mission they will be replacing a failed Control Moment Gyro (CMG) aboard the ISS. DigitalSpace carried out the steps to model this training procedure using SimSpace by first making a computer graphics movie re-construction of the shuttle cargo bay sequence of the CMG changeout (figures 21-24 below) followed by a complete real-time simulation version (figures 25-29 below).


Figures 21-24, CMG changeout computer graphics re-construction of NBL training

Modeling EVA procedures from NBL training

Use of the movie-creation step allowed us to create a precise reconstruction of NBL video, from the perspective of divers in the NBL tank. We integrated voice from astronauts and NBL staff to correspond to the actual voice loop dialogue. This movie was shown to NBL and ARC team members who were satisfied by its realism vis a vis the actual procedures. We were then requested to construct an actual simulation of the sequence. This simulation was completed and the first version satisfied the goals set forth in section (2) above. We were then requested to extend the platform and model a second EVA in simulation the RPCM changeout procedure, this time without reference to NBL video (none existed) but using only a simple verbal description. A goal of this phase was to compare the realism of video-sourced modeling and procedure-sourced modeling.


Explanation of the SimStation Online SimEVA Application: User Experience


The following figures present the first SimEVA application, the model of the STS-114 CMG changeout procedure (shuttle cargo bay sequences). The second set of figures present a model, developed without NBL video, instead using generic descriptions and pre-existing models drawn from the growing library, of the RPCM changeout just completed by ISS crew.

Both simulations employ real station CAD geometry and are driven by the new SimSpace OWIB component.


STS-114 CMG changeout (shuttle cargo bay sequences) procedure simulation


Figures 25-29 below illustrate the simulation of the CMG changeout procedure as modeled from NBL training video via the above-described computer animation step. This application is executing in the Internet Explorer web browser using the Adobe Atmosphere and DigitalSpace SimSpace application. Note that the procedure steps and camera viewpoints in the HTML user interface on the right. The simulation may be run live at the web address at [24].



Figure 25: beginning of CMG changeout procedure (shuttle cargo bay and ISS Canadarm2) with two astronauts on EVA. One astronaut is awaiting delivery of faulty CMG from the station to affix to a ball stack in the aft section of the shuttle cargo bay



Figure 26: faulty CMG is attached to ball stack



Figure 27: astronaut moving along to replacement CMG to begin preparing it for transfer to station


Figure 28: both astronauts working to loosen bolts for release of replacement CMG



Figure 29: replacement CMG is now secured by astronaut who is attached by foot restraint to the station arm and is moving the CMG to the station


Subsequent procedure steps in this simulation reconstruct the faulty CMG being moved to stowage in the LMC for return flight and analysis.


RPCM changeout (shuttle cargo bay sequences) procedure simulation


The following sequences (figures 30, 31) show an EVA changeout scenario for a station remote power module (RPCM) with egress from the US portion of the station. The actual RPCM changeout was carried out from a Russian segment due to problems with the US suits. The simulation may be run live at the web address at [24].



Figure 30: Egress of both station crew from a US segment, carrying replacement RPCM.



Figure 31: Replacement of RPCM that powers 3rd CMG on Z1 Truss location (whole station not modeled)


Explanation of the SimStation Online Application: Architecture


The current environments operating under the SimEVA module utilize a database of OWorld Brahms Interpreter performing pre-written pseudo-Brahms statements. Figure 32 below illustrates how these components work together.


Figure 32: SimEVA current implementation

Core elements added to the SimSpace platform: OWorld Agent Information Broker

As originally proposed for Phase I a new control module the “OWorld Agent Information Broker” (OWIB - see figure 33 below) was built. The OWIB provides a means to provide communications and scheduling of all agent activities, reporting of exceptions and driving a unified user interface to all agent activities. A key part of the OWIB’s job is to interface between the SimStation Online CAD elements, the database records, and the SimEVA elements including the astronaut EVA procedure checklists.



Figure 33: schematic specification for the Agent Broker

Successes and Shortcomings


A visit to the NBL by DigitalSpace and ARC partners in April of 2004 obtained further interest and currently Raytheon is engaging DigitalSpace under contract to model the entire CMG changeout procedure for the NBL. It is anticipated that this simulation will be used iteratively by the NBL and the astronauts for scenario planning and refresher training for the STS-114 mission in early 2005.

A shortcoming of the process was identified by our difficulty in transferring the wide variety and complexity of CAD formats for ISS models into the format used by our 3D renderer, Adobe Atmosphere. The large rendering budget required for not only the models of ISS and shuttle hardware but for all the gestures and animations of the crew forced us to break the elements of the CMG changeout and STS-114 into separate simulations. It is hoped that a new version of Adobe Atmosphere, delivered to use for test in July 2004 will ameliorate these problems and permit “straight through” simulations.

A second shortcoming was the lack of a true agent architecture to drive the EVA simulations. In these examples, simple procedure checklists were tied to the animation sequences. In a Phase II proposal it will be suggested that Brahms, which is a fully capable agent architecture, be tied into the SimEVA capability. Brahms Agents could be embodied in the simulations to represent ISS crew, US and Russian mission controllers, instruments, power subsystems and even the presence of the sun/darkness cycle. Only a full agent architecture will enable a rich enough simulation environment for comprehensive crew training.

A key success of this work is that demonstration of the SimEVA capability of the SimSpace platform has led directly to the VAST application for Lockheed-Martin, described in Part 5.


Conclusions


The successful applications achieved in this Phase I demonstrated that a single open, extensible modeling and simulation platform can address the needs of several NASA centers and subcontractors in the collaborative visualization for ISS and its maintenance and crew training needs. It is hoped that the continued evolution of the platform will position it well to support the President’s new exploration initiatives and Code T Office of Exploration.


Possible Phase II value of SimEVA


It has been suggested that this second component of the Phase I project should be continued in a Phase II proposal. ARC partners in the project suggested that the ability to rapidly assemble training sequences from a pre-built repository of gestures and CAD parts and at low cost will provide a whole new modality of training in the future. It was suggested that this kind of portable, just-in-time training will be essential for crews on long duration missions in the Moon-Mars initiative, who cannot return to Earth (and the NBL and other facilities) for that training. If the SimEVA simulations can be operated by astronauts on the ISS laptops, this will prove the feasibility of delivering a class of training to where it is needed. The need for haptic interfaces for SimEVA was identified, especially in medical and repair operations. This development will be proposed in Phase II.


Continuation of this work in Phase II: Universal Model Repository and Applications


Through this work we determined that a productive goal for a Phase II proposal would be to build and deploy a large, open Universal Model Repository (UMR) of 3D geometry, gestures and actions tied to real procedure language and checklists to permit any NASA or contractor simulation designer the ability to create computer simulated training and other applications quickly and at low cost.


Relevance to new exploration initiative (Code T)


Five years of development of SimSpace has yielded a platform that has been applied in both the design-by-simulation and in the work practice mission planning and training roles. In addition, in preceding and parallel work, the BrahmsVE component of SimSpace is being utilized to simulate both the FMARS and MDRS analog manned Mars lander/habitats and is being extended to support current work on Mobile Agents being field tested with human participants at MDRS [21].  In April of 2004, the platform was used to model Lunar habitats for a Boeing workshop on Lunar return and the new exploration mission [28]. Through the modeling of a large and complex vehicle (ISS) as well as human activities with robotic and agent assistance (EVA with mission control, Station Canadarm2 and other components) we believe we have qualified the platform for use in the Office of Exploration’s upcoming spiral development cycles of CEV and return to the moon. It is felt that this work qualifies the platform for future use in designing and testing concepts in human/robotic Lunar and Mars surface operations.


Part 4 Work Plan


4.1  Technical Approach



The project began with extensive consultation with expert advisors at several NASA centers as well as contractors. NASA and subcontractors supplied an extensive library of CAD models for the ISS as well as XML schemas for assembly of those models. A review of the current SimStation application was carried out and goals for SimStation Online set forth. The first third of the project focused on implementing the basic rendering of the ISS in configuration 12A.

The second third of the project focused on the building of the Oworld Agent information broker software module (OWIB) and the 3D reconstructions of ISS with the interfaces for database records. In parallel, work began in studying the NBL training video for the CMG changeout procedure and parsing the procedure checklists.

The final third of the project saw final assembly and test of the SSO application and presentation to the ARC team. In parallel, the CMG changeout procedure was shown to NBL Raytheon program directors at JSC and the full CMG simulation commissioned by ARC partners. The final step was to produce the RPCM changeout simulated EVA.

All project materials were delivered by the web and are made available on the DigitalSpace site referenced at [30].


Part 5 Potential Applications


DigitalSpace has been engaged in the development of virtual world platforms for eight years and has successfully completed a number of major projects (see Part 8.1 below). The PI and members of DigitalSpace have contributed numerous publications to a variety of scientific and technical journals [32]. In addition to the work with USRA/RIACS and NASA described in Part 2 above, we have collaborated with numerous universities and companies.


5.1 Potential NASA Applications


The success of this phase I project has led directly to the following NASA applications, being carried out in the summer of 2004:

1. SimSpace/SimEVA is being contracted for use at JSC’s NBL under contract by Raytheon to create a complete STS-114/CMG changeout EVA simulation for use by crew and mission planners.


Figure 34: VAST medical simulation application

2. SimSpace/SimEVA is actively engaged in a project for Lockheed Martin at JSC in a project for space medicine trainers (see figure 34 above). This project, called VAST, is developing a simulation of ISS crew engaged a medical emergency procedure using the CHeCS rack.

Other NASA Applications include support for many aspects of the upcoming Code T Office of Exploration initiatives including the return to the Moon, and the Crew Exploration Vehicle (CEV)


5.2 Potential Non-NASA Commercial Applications


See part 8 below.


Part 6 Contacts


6.1 Key Contractor Participants


The following brief resumes introduce management/technical staff members for this phase I project.  DigitalSpace certifies that Bruce Damer, the Principal Investigator, has his primary employment at DigitalSpace at the time of award and during the conduct of the project.

Name: Bruce Damer (PI)

Years of Experience: 23

Position: CEO

Education: Bachelor of Science in Computer Science (University of Victoria, Canada, 1984); MSEE (University of Southern California, 1986)

Assignment: Mr. Damer will be the Principal Investigator for the SBIR Phase I effort. He will coordinate all interaction between DigitalSpace and its collaborators and NASA and other participants, be responsible for all staffing, technical design, reporting and documentation.

Experience: Mr. Damer is the world's recognized expert on avatars and shared online graphical virtual spaces having created much of the early literature, conferences and awareness of the medium. Mr. Damer is a visiting scholar at the University of Washington Human Interface Technology Lab and a member of the staff at the San Francisco State Multimedia Studies Program.

Name: Stuart Gold

Years of Experience: 29

Position: Chief Architect (communities platform)

Education: Royal Institute of British Architects

Assignment: Stuart Gold will serve as a consulting technical architect for the project and structure the technology components and architecture for the SimSpace platform.

Experience: Mr Gold is a pioneer of online systems, starting with his work on transaction processing for Prestel in the 1970s and concluding most recently with his leadership in the design and delivery of online virtual worlds including: TheU Virtual University Architecture Competition, International Health Insurance Virtual Headquarters, and Avatars98-2001 online events. Mr. Gold also is the chief architect of the DigitalSpace communities platform, implementing XML and JS based community tools for use by all DigitalSpace projects.
See http://www.digitalspace.com/papers for his recent writings.

Name: Galen Brandt (Content Development, Marketing)

Position: New business development, DigitalSpace

Experience: 26 years including creating market strategies for Dun and Bradstreet, SUNY Fashion Institute of Technology, DoToLearn and others.

Assignment: Content and Market development for Phase I and II. Mrs. Brandt has assisted in reviewing all aspects of proposals and documentation for the project, including the users guides. She has also assisted in terms of team communications and marketing concept and content development.

Name: Dave Rasmussen (PM)

Position: Co-director of the 3D Design Studio (DM3DS), DigitalSpace

Experience: 9 years experience in virtual world design, skills: 3DS Max, Java, ActiveWorlds, Adobe Atmosphere, PHP/MySQL database development

Assignment: Dave has served as PM for this phase, directing the team performing 3D modeling and animation, testing and delivery of the project.

Name: Merryn Neilson (Lead CD)

Position: Co-director of the 3D Design Studio (DM3DS), DigitalSpace

Experience: 9 years experience in virtual world design, skills: 3DS Max, Java, ActiveWorlds, Adobe Atmosphere.

Assignment: Web design on project, 3D worlds, avatar design, testing

Name: Peter Newman (SE & TE)

Position: Developer in C++, JS, PHP, HTML, 3D Design Studio, DigitalSpace

Assignment: Programmer of OWorld engine extensions.

Name: Ryan Norkus (CD & TG)

Position: Graphic artist, 3d modeler and animator, 3D Design Studio,DigitalSpace

Assignment: Focusing on the automation of animated sequences

Name: Lorenzo Hagerty (SE)

Position: Server support and voice system content support, Studio, DigitalSpace

Assignment: Focusing on the maintenance of DigitalSpace’s voice server and voice loop production for the simulations.

6.2 Key NASA Participants


Key NASA participants in this project have included: COTR Bill Clancey, NASA Civil Servants Mark Shirley, and various JSC personnel and others on the SimStation projects.


6.3 NASA and Non-NASA Advisors


The following individuals have provided technical and scientific advisement to this project.

Tom Cochrane, Raytheon, ARC

David Throop, Boeing, JSC

Dr. Maarten Sierhuis, ARC

Constance Adams, Lockheed Martin, JSC

Dr. Charles Neveu, NASA ARC/QSS, PSA Team

Dr. Mike Sims, ARC

Dr. Geoff Briggs, Scientific Director, Center for Mars Exploration, NASA ARC

Dr. Tom Furness III, HIT Lab University of Washington

Dr. Don Brutzman, Naval Postgraduate School, MOVES Institute

Dr Michael Zyda,Professor,NPS/MOVES Institute, Lead on America's Army massively multiplayer online role playing game, Founder, Arsenal Interactive

Captain Richard O’Neill, US Navy, Director, Highlands Group


Part 7 Technical Activities

7.1 Cumulative Technical Activities

Task 1: Construction of SimStation Online



During the first two months of the Phase I project DigitalSpace performed interviews of key advisors and participants including NASA ARC and JSC advisors, converted the 3D CAD models of the ISS and produced the first online rendering.


Task 2: Creation and test of OWIB component & commencement of SimEVA


During the following two months, the new OWIB control was created and tested in assembling the entire SSO application and providing collaborative access to databases. In this time period, a 3D computer graphics movie was created of the first NBL CMG changeout training video.


Task 3: Finalization and presentation of SSO and build-out of SimEVA applications


During the final two months of the project, the SSO application was finalized and presented to the ARC team for comment. The SimEVA applications for CMG changeout and RPCM changeout were completed and presented to ARC and JSC/NBL.


7.2 Future Technical Activities

Task 1: Complete upgrades to 3D renderer (Adobe Atmosphere)


The next two months will be dedicated to upgrading to the new Atmosphere 3D plug-in to address shortcomings discovered in the SSO application and utilizing one or more open source 3D engines to broaden the range of platforms supported.


Task 2: Complete Raytheon and Lockheed Martin projects using SimSpace/SimEVA


By September of 2004, both Raytheon CMG changeout and Lockheed Martin VAST space medicine projects should be completed.


Task 3: Integrate Brahms as an agent architecture for the SimEVA applications, implement Universal Model Repository


By Fall of 2004 we will integrate two-way real-tome Brahms interaction (under another contract) to support true agent representation in the SimEVA and SSO applications. As described previously, the implementation of a Universal Model Repository is seen as a strategic goal for a Phase II project.


Part 8 Potential Customer and Commercialization Activities In anticipation of Phase III commercialization and Phase II pre-launch marketing and customer activity, we have identified the following classes of valuable applications for SimSpace and the components it encompasses including BrahmsVE. Companies in these fields include Sense8, Virtualis, Planet9, RealSense, and Reality Capture.


8.1 Recent NASA Potential Customer Activities


MDRS and Mobile Agents


SimEVA and SimIVA



Recent applications for the Neutral Buoyancy Laboratory at JSC and the VAST project for medical emergency training aboard ISS suggests there are a number of immediate applications for SimSpace/SimEVA/IVA in training and mission planning. We anticipate building Haptic and other device interfaces through partnerships built for the VAST project.


Modeling and simulation for Mars Science Laboratory and Titan missons


Geoff Briggs and Michael Sims of Ames informs us that future Mars (Mars Science Laboratory ’09) and Titan missions will include the use of drills, rotorcraft and airplanes. While these are strictly robotic missions there will be “humans in the loop” in the form of significant science backrooms and mission control


Continued SimStation Projects


The SimStation project at ARC, JSC and Langley will likely be a continuing customer for the SimSpace especially with regard to SimEVA and the exterior ISS view.


New Exploration Initiative/Code T


A number of SimSpace-oriented RFI white papers and extramural Notice of Intent to Propose have or are being submitted to the Code T Office of Exploration initiative. SimSpace is being proposed along with a number of contractors (Boeing, Raytheon, Lockheed-Martin, SGI), universities (Stanford, Purdue, UCSC, USC) and smaller company partners.


NASA educational outreach and Space Camp


Immersive virtual worlds, virtual digital human (VDH), and 3D simulation modeling, have become a significant vehicle for NASA's effort to generate and communicate knowledge/understanding to K-12 and college/university students on topics such as the International Space Station and Space Shuttle/Space Transport System (STS) operations, Robotics, Intravehicular/Extravehicular activities, and the new exploration initiative. SimSpace has been proposed for use at NASA Space camp to Ed Buckbee, founder of Space Camp. In addition, the Space Exploration Online massively multiplayer online game concept has been proposed to NASA headquarters education/outreach as well as Code T.


8.2 Recent Non-NASA Potential Commercial Activities


K-12 and College, Education and Educational Web Services


The current set of SimSpace applications is being repurposed into educational course modules for schools, museums and other organizations. In discussions with the organizers of National Space Week, the Planetary Society, the Mars Society, and SPACE.COM, various modules of SimSpace projects have been made available and are now features in DVD, web and installation projects.


Public health applications


The National Institutes of Health, through the work of Dr. Dorothy Strickland and Do2Learn have contracted DigitalSpace to employ SimSpace to deliver safety games for children with autism. The first game was delivered for clinical trials at Emory University in Atlanta in July of 2004.


8.3 Other Recent Commercialization Activities


DOD and DOE – energy security design application


In January 2003 SimSpace was presented at a special workshop at the Arlington Institute held for the Office of Secretary of Defense. A prototype virtual wind farm [27] was presented using SimSpace and allowed us to show wind farm KW/Hour production scales for different configurations of turbines. Coupling this with a geographical information system provided by GeoFusion Inc. allowed us to show the DOD staffers and other energy experts how sites for wind farm power could be selected and then the production output modeled. The next phase of this work is in development for a follow-up meeting in December 2004.


8.4 Future Potential Customer and Commercialization Activities


Hotel design


Space architect Constance Adams has invited DigitalSpace to use SimSpace for a September 2004 presentation of new designs for hotel rooms which embody principles learned while designing optimal work and living spaces for ISS.


Robot games – educational and entertainment applications


Robot “wars” are one of the most popular forms of entertainment in the popular media and robot game competition are some of the finest learning events for K-12 and college engineering students and faculty. Ames sponsors such events with CMU students and high schools. We have communicated with the organizers of the Ames events and demonstrated them BrahmsVE. It is planned to partner with them and the local chapter of the Robotics Society of America to develop a kids’ robot design lab and competition space within the virtual spaces made possible by BrahmsVE. Massive multi-player online games are experiencing a large amount of investment and commercial interest. BrahmsVE is a competent platform for the creation of a successful multiplayer online game both as a learning tool and as a pay-per-play tournament environment.  We plan to seek support for a commercial, online robot games application. We have secured the trademark “digibots” for this project and are creating a business plan.


Defense design, training and operations applications


The military will be using semi and fully autonomous agents working closely to support troops and command in surveillance and combat missions throughout the 21st Century. Therefore we expect a great deal of interest surrounding a product in this space. We are already in contact with the Naval Postgraduate School MOVES Institute about cooperation on and adopting a new XML based standard in simulation communications.


Industrial design, training and operations applications


From factory floor automation to security systems, complex environments where humans work in tandem with mobile agents or other autonomous machine systems all need a comprehensive model-based environment with high fidelity 3D re-creation during both design, training and operations phases. Industrial training is a multi-billion dollar per year industry and BrahmsVE is uniquely suited to enter this market, running on industry standard platforms.


Consumer market research for personal wireless assistants


The emerging era of wireless, wearable personal assistants is picking up momentum with ever more sophisticated cell phones and other handheld devices. In a real sense, each of these devices represents the pairing of humans with machines, all which the BrahmsVE human/agent augmentation design environment can model for product design purposes.


Emergency first responder planning


Layout of cityscapes for planning first responder strategies in civil emergencies is an application well suited to SimSpace as it would employ the agent architecture of Brahms with the detailed simulation elements of SimMedical.


Part 9 Resources Status


Up to the report date 100% of the work has been completed.


Part 10 References


[1] President's Commission on Moon, Mars and beyond, report on the web at:

http://www.moontomars.org/

[2] DOD Joint Strike Fighter described on the web at: http://www.jsf.mil/

[3] “Computing & Design/Build Processes Help Develop the 777” from Boeing company site:

http://www.boeing.com/commercial/777family/compute/compute4.html



[4] Loftin, R.B., and Kenney, P.J., "Training the Hubble Space Telescope Flight Team," IEEE Computer Graphics and Applications, vol. 15, no. 5, pp. 31-37, Sep, 1995.

[5] Engelberg, Mark[Ed] (September 11, 1994). Hubble Space Telescope Repair

Training System [WWW document]. URL

http://www.jsc.nasa.gov/cssb/vr/Hubble/hubble.html

[6] Cater, J. P., and Huffman, S. D., Use of Remote Access Virtual Environment Network (RAVEN) for Coordinated IVA-EVA Astronaut Training and Evaluation. Presence: Teleoperators and Virtual Environments_ vol. 4, no. 2 (Spring 1995), p. 103-109. (Training for Hubble Space Telescope repair.)

[7] Mars Pathfinder JPL site: http://mars.jpl.nasa.gov/MPF/index1.html

[8] Clancey, W. J., Sachs, P., Sierhuis, M., and van Hoof, R.1998. Brahms: Simulating Practice for Work Systems Design. International Journal of Human-Computer Studies, 49, 831-865.

[9] Sierhuis, M. 2001. Modeling and Simulating Work Practice; Brahms: A multiagent modeling and simulation language for work system analysis and design. Ph.D. thesis, Social Science and Informatics (SWI), University of Amsterdam, SIKS Dissertation Series No. 2001-10, Amsterdam, The Netherlands, ISBN 90-6464-849-2.

[10] Sierhuis, M.; Bradshaw, J.M.; Acquisti, A.; Hoof, R.v.; Jeffers, R.; and Uszok, A. Human-Agent Teamwork and Adjustable Autonomy in Practice, in Proceedings of The 7th International Symposium on Artificial Intelligence,Robotics and Automation in Space (i-SAIRAS), Nara, Japan, 2003.

[11] M. Sierhuis and W. J. Clancey, Modeling and Simulating Work Practice: A human-centered method for work systems design, IEEE Intelligent Systems, vol. Volume 17(5), 2002.

[12] M. Sierhuis, A. Acquisti, and W. J. Clancey, Multiagent Plan Execution and Work Practice: Modeling plans and practices onboard the ISS, presented at 3rd International NASA Workshop on Planning and Scheduling for Space, Houston, TX, 2002.

[13] A. Acquisti, M. Sierhuis, W. J. Clancey, J. M. Bradshaw, Agent Based Modeling of Collaboration and Work Practices Onboard the International Space Station. Proceedings of the 11th Conference on Computer-Generated Forces and Behavior Representation, Orlando, FL, May 2002.

[14] M. Shirley, T. Cochrane, SimStation: A Knowledge-Integrating Virtual Vehicle, Virtual Iron Bird Workshop, NASA Ames Research Center, March 31, 2004. Available on the web at:

http://ic.arc.nasa.gov/vib/day1/papers/Shirley_Cochrane.pdf

[15] J. M. Bradshaw, M. Sierhuis, Y. Gawdiak, R. Jeffers, N. Suri, M. Greaves. (2001). Adjustable Autonomy and Teamwork for the Personal Satellite Assistant, in The IJCAI-01 Workshop on Autonomy, Delegation, and Control: Interacting with Autonomous Agents, Seattle, Washington, USA August 6, 2001. URL: http://csce.uark.edu/~hexmoor/AA01/IJCAI01-cfp.htm

[16] Personal Satellite Assistant (PSA) Test Fixture (Greg Dorais, Yuri Gawdiak, Daniel Andrews, Brian Koss, Mike McIntyre) described on the web at: http://ficworkproducts.arc.nasa.gov/psa_test_fixture/psa_test_fixture.html

[17] PSA Web site: http://ic.arc.nasa.gov/projects/psa/

[18] Intelligent Virtual Station on the Web at: http://ssrl.arc.nasa.gov/ivs.html

[19] R. Papasin, B.J. Betts, R. Del Mundo, M. Guerrero, R.W. Mah, D.M. McIntosh, and E. Wilson, "Intelligent Virtual Station," Proceedings of the 7th International Symposium on Artificial Intelligence, Robotics and Automation in Space (i-SAIRAS 2003), Nara, Japan, May 2003.

[20] Rochlis, J, Clark, J.P. and Goza, M., "Space Station Telerobotics: Designing a Human-Robot Interface", AIAA Confeence on Space Station Utilization, Kennedy Space Center, October 2001.

[21] BrahmsVE/FMARS Project Home Page on the web at: http://www.digitalspace.com/projects/fmars

[22] MER and Mars surface modeling projects on the Web at: http://www.driveonmars.com

[23] SimStation Online Examples:

http://www.digitalspace.com/projects/ssos/sso

[24] SimEVA CMG and RPCM Changeout

http://www.digitalspace.com/projects/eva-sims/

[26] BrahmsVE/ISS-PSA SBIR Phase I Project and reports Web Page: http://www.digitalspace.com/projects/iss_03

[27] DoToLearn projects for the National Institutes of Health on the Web at:

http://www.dotolearn.com

[28] SimVehicle and SimLunar work for Boeing on the Web at:

http://www.digitalspace.com/projects/lunarbase/index.html

[29] Virtual Iron Bird Paper:

http://www.digitalspace.com/papers/damer-vib-day2-paper/paper.html

[30] DigitalSpace Home Page and Resources:

http://www.digitalspace.com

Project Summary

Firm: DigitalSpace Corporation  Contract Number: NNA04AA32C

Project Title: Platform for Design & Test of Large Scale Multi-Agent Human-Centric Mission Concepts

Identification and Significance of Innovation:

The innovation produced in this phase of work on the BrahmsVE platform was to permit support of large scale multi-agent, human centric mission simulation. For this work, the following prototype project was selected: to model the International Space Station in a database-assembled 3D CAD representation, then model dynamic activity of astronauts (represented as agents) effecting repairs on the ISS. The innovation would permit both the exact representation of the geometry of flight hardware while modeling real EVA procedures derived directly from training carried out at NASA’s Neutral Buoyancy Laboratory in Houston.

Technical Objectives and Work Plan:
The work plan extended the BrahmsVE architecture to support a multi-agent human-centric simulation of future ISS configurations equipped with agents representing astronauts on EVA procedures backed up by space station arm and other facilities. The original plan was for modeling of the Personal Satellite Assistant and Robonaut. However, per request by NASA Ames, simulation of actual upcoming EVA procedures was performed successfully. The database agent broker was constructed first, followed by models of the ISS, astronauts and station arm with ORU parts, the scene was assembled and matched to captured NBL training video and voice loop was included.

Technical Accomplishments:
In order to permit assembly of the large number of CAD models representing the ISS, a database-backed agent broker module was constructed to assemble and operate exterior ISS 3D virtual world models, virtual human (astronaut-agents) and other subsystems. Input from expert advisors at several NASA centers was sought to create this macro-level simulation of work practice within two real training scenarios captured from actual training carried out at the Neutral Buoyancy Laboratory. This platform was delivered into prototype demonstration at NASA Ames and JSC's NBL with the ISS RPCM and CMG STS-114 changeout procedures modeled.

NASA Application(s):
For the upcoming new exploration initiative to the Moon, Mars and beyond, this platform will support concept design for long duration missions, Lunar and Mars surface operations as well as just-in-time crew in situ training and evaluation of safety and crew health. Prior to new missions, the platform is of relevance to safely completing the build-out of ISS. Mobile agents and Mars analog habitat studies will be empowered by the platform’s ability to represent human and machine agents working together.

Non-NASA Commercial Application(s):
This platform has been engaged in a project for the teaching of safety practices to children with Autism in a controlled study funded by the National Institutes of Health at Emory University in Atlanta. Other applications include the design of factory floors where people work in concert with robots, surgical theaters where physicians and staff need to optimize the utilization of space, time and equipment, and as an engine for battlefield training where agents represent combatants and geometry of vehicles and terrain is served out by the database and agent broker.  We believe that the successful completion of this Phase I project coupled with significant interest from NASA and non-NASA customers for the BrahmsVE & SimSpace platform including a new Universal Model Repository justifies Phase II continuation.