A Virtual Environment for Collaborative Assembly
Xiaowu Chen, Nan Xu, Ying Li
The Key Laboratory of Virtual Reality Technology, Ministry of EducationSchool of Computer Science and Engneering, Beihang University
Beijing 100083, P.R. China)
[email protected]
Abstract
To allow geographical dispersed engineers toperform an assembly task together, a VirtualEnvironment for Collaborative Assembly (VECA) hasbeen developed to build a typical collaborative virtualassembly system. This presents the key 留學生計算機dissertation定制parts of VECA,such as system architecture, HLA-based (High LevelArchitecture) communication and collaboration,motion guidance based on collision detection andassembly constraints recognition, data translationfrom CAD to virtual environment, reference resolutionin multimodal interaction.
1. Introduction
Virtual reality (VR) is a technology which is oftenregarded as a natural extension to 3D computergraphics with advanced input and output devices. NowVR has matured enough to warrant serious engineeringapplications [1]. As one of the important applicationdomains of VR, virtual assembly (VA) fulfills designflexibility by replacing physical objects with the virtualrepresentation of machinery parts and providing
advanced user interfaces for users to design andgenerate product prototypes [2]. VA can evaluate andanalyze product assembly and disassembly during theproduct design stage with the goal of reducingassembly costs, improving quality and shortening timeto market [3].With the distributed fashion of companies andresearch organizations, many design, assembly,manufacture, analysis works require the collaborationof geographical dispersed engineers. CollaborativeThis paper is supported by A.S.T. Fund (VEADAM ), NationalNatural Science Foundation of China (60503066), National ResearchFund (51404040305HK01015), China Education and Research Grid
(ChinaGrid) Program (CG2003-GA004), National 863 Program(2004AA104280) & (SIMBRIDGE).virtual environment (CVE) is a computer system thatallowsremote users to work together in a sharedvirtual reality space [4]. As an important category ofComputer-Supported Cooperative Work (CSCW) andVR, CVE systems have been applied to militarytraining, telepresence, collaborative design andengineering, distance training, entertainment, andmany other personal http://www.400190.com/Thesis_Writing/Computer_Science/andindustrial applications [5].Because of the advantages of CVE and the actualrequirement of industry, collaborative virtual assembly(CVA) is becoming one of research emphases of VA.
There have been many research efforts to developvirtual assembly systems [6][7][8]. A representativesystem is Virtual Assembly Design Environment(VADE) [9] which allows engineers to evaluate,analyze, and plan the assembly of mechanical systems.VADE simulates inter-part interaction for planar andaxisymmetric parts using constrained motions alongaxes/planes which are obtained from the parametricCAD system. In this system, direct interaction issupported through a CyberGlove.In the field of CVA, several systems have beencreated. For example, National Center forSupercomputing Applications (NCSA) and Germany’sNational Research Center for Information Technology(GMD) developed a collaborative virtual prototypingsystem over ATM network [10]. The system integratedreal-time video and audio transmissions let engineerssee other participants in a shared virtual environment ateach remote site’s viewpoint position and orientation.Jianzhong Mo et al. developed a virtual-reality-basedsoftware tool- Motive3D [11] that supportscollaborative assembly/disassembly over Internet andpresented systematic methodology for disassemblyrelation modeling, path/sequence automatic generationand evaluation independent of any commercial CADsystems. Shyamsundar et al. developed an internetbased#p#分頁標題#e#
collaborative product assembly design (cPAD)tool [12][13]. The architecture of cPAD adopts 3-tierclient/server mode. In this system, a new AssemblyProceedings of the Second International Conference on Embedded Software and Systems (ICESS’05)
0-7695-2512-1/05 $20.00 © 2005 IEEE
Authorized licensed use limited to: MACQUARIE UNIV. Downloaded on July 5, 2009 at 22:47 from IEEE Xplore. Restrictions apply.Representation (AREP) scheme was introduced toimprove the assembly modeling efficiency. The AREPmodel at the server side can be accessed by many
designers at different locations through client browsersimplemented using Java3D.But most CVA systems above are based on C/S orB/S architecture, and little effort has been put into theresearch of CVA based on distributed architecture.Sometimes there are some requirements on the desigofcertain industrial products, which expect adistributed virtual environment to support thecollaborative assembly, such as a Virtual Environmentfor Collaborative Assembly (VECA) presented in thispaper. VECA can build a collaborative virtualassembly system which allows geographical dispersedengineers to perform an assembly task together. VECAmainly includes HLA-based (High Level Architecture)communication and collaboration, motion guidancebased on collision detection and assembly constraintsrecognition, data translation from CAD to virtualenvironment, reference resolution in multimodalinteraction, and so on.
The rest of this paper is organized as follows.Section 2 presents the system architecture of VECA.The modules of communication and collaboration, data
translation, motion guidance, multimodal interactionare separately elaborated in section 3, 4, 5, and 6.Section 7 is the implementation and application ofVECA, and finally section 8 ends this paper withconclusions and future work.
2. System Architecture
The system architecture of VECA is illustrated inFig.1. Once an engineer has finished designingassemblies or subassemblies using a parametric CADsystem such as Pro/Engineer, he or she uses a plug-infor Pro/Engineer to translate CAD models to trianglemesh models (Multigen OpenFlight) includingassembly constraints and geometry featureinformation. Others download these models, and thenthey can assemble the product collaboratively in amultimodal shared VE to find the design defects or getthe feasible assembly sequence.Now there are five key parts in the system.
1. Communication and collaboration: connectsgeographical dispersed nodes to form a distributedcollaborative VE based on HLA.
2. Motion guidance: helps the user translate orrotate the parts in the VE freely and precisely usingcollision detection and assembly constraintsrecognition.
3. Data translation: this module translates modelsand extracts information from CAD (Pro/Engineer). Itis a plug-in for Pro/Engineer and is developed byPro/Toolkit and Multigen OpenFlight API.
4. Constraint manager: dynamically maintainsassembly constraints information. The design of thismodule and a constraint-based distributed virtualassembly model were already introduced in [14][15].#p#分頁標題#e#
5. Multimodal interaction: processes combinednatural input modes such as speech, hand gesture in acoordinated manner.
Data translationMultimodal interaction
Mouse, Keyboard, CyberGlove, Microphone, ... Display, 3D shutter glasses, ...Input/output deviceCollaborative virtual assembly
Motion guidance Constraint manager Comunication & collaborationVirtual environmentScenegraph management
CAD
Figure 1. System architecture of VECA
3. HLA-based Communication and
Collaboration
HLA standard [16] is a general architecture forsimulation reuse and interoperability developed by the
US Department of Defense. The conceptualization ofHLA led to the development of the Run-Time
Infrastructure (RTI). This software implements aninterface specification that represents one of thetangible products of the HLA. Some concepts and
terms in HLA are introduced as follows. “Federation”is defined as a group of “federates” forming acommunity. Federates may be simulation models, datacollectors, simulators, autonomous agents or passive
viewers. A simulation session, in which a number offederates participate, is called a “federation execution”.Simulated entities, such as tanks, aircrafts orsubassemblies, are referred to as “objects”. “Attribute”,referred to as object state, can be passed from oneobject to another. Objects interact with each other andwith the environment via “interactions”, which may beviewed as unique events, such as manipulation of the
object, or a collision between objects. Initially, anattribute is controlled by the federate that instantiated
the object. However, attribute ownership may changeduring the execution. This mechanism allows users toco-manipulate the same object, for example. Allpossible interactions among the federates of afederation are defined in a so-called “Federation Object
Model” (FOM). The capabilities of a federate aredefined in a so-called “Simulation Object Model”Proceedings of the Second International Conference on Embedded Software and Systems (ICESS’05)0-7695-2512-1/05 $20.00 © 2005 IEEEAuthorized licensed use limited to: MACQUARIE UNIV. Downloaded on July 5, 2009 at 22:47 from IEEE Xplore. Restrictions apply.(SOM). The SOM is introduced to encourage reuse of
simulation models [17].DVE_FM [18] is an application framework which isbased on the interface specification of HLA. It
provides the universal function of distributedinteractive simulation and insulates invoking of RTI.
DVE_FM reduces the complexity of development ofsimulation application based on HLA, thus the
developers can focus on domain application but notcomplex interoperation between simulation application
and RTI.The HLA-based simulation framework of VECA is
留學生dissertation網shown in Fig.2. Each of VA nodes joins federation as afederate. And the development of the federate is based#p#分頁標題#e#
on DVE_FM.TCP/IP
Run-Time Infrastructure (RTI)
DVE_FM
VM federate 1
DVE_FM
VM federate 2
DVE_FM
VM federate n
…...
Figure 2. HLA-based simulation framework
During a federation execution, each of federatesupdates the attributes of the objects (parts orsubassemblies) which belong to it. VECA has definedtwo types of interactions: CCooperationInteraction andCConstraintFeedbackInteraction.CCooperationInteraction represents federate
assembly/disassembly requirement. A federate that hasthe ownership of the parts or subassemblies mayreceive more than one CCooperationInteraction for thesame part (subassembly) at the same time, and it willselect one by a set of schedule rules to solve theproblem of competition and collaboration betweenmulti-users. If the requirement satisfies assemblyconstraints, the federate updates position or attitude ofparts (subassemblies), then informs other federatesthrough RTI. Otherwise the federate transferscurrent operation is not allowed.
4. Motion Guidance
In VA, due to the lack of physical constraintinformation and the limitation of location trackingprecision, it is difficult for the user to control themotion of the assembling parts precisely with currentvirtual reality I/O devices during VA [19]. Therefore, itis necessary to explore a technique to help the usermanipulate the target assembling part freely andprecisely in VA.
VECA adopts motion guidance based on collisiondetection and assembly constraints recognition [20] toaccomplish precise location of parts (subassemblies).Collision detection is used to solve assembling parts
interference problem. Assembly constraintsrecognition is used to assist in capturing the user’sintention.
Axis orientation constraint and face match
constraint are the two assembly constraints mainlyresearched into because other assembly constraints can
be converted to the combination of these two
constraints, and a pair of assembly units just need tosatisfy each of them for only once to form a newsubassembly. In order to simulate the actual assemblyprocedure, motion guidance first recognizes the axis
orientation constraint, and then recognizes the facematch constraint during assembly constraintsrecognition.The above discusses the logic sequence of a singleassembly from the angle of recognition satisfaction.Fig.3 describes the functions of motion guidance andprecise location from the angel of a part's(subassembly's) motion state transition during a singleassembly (disassembly).A single assembly process of a pair of assembly
units can be divided into three phases: free motionstate A, axial motion state, and free motion state B asshown in Fig.3. Free motion state A is the initial stateof assembly units, axial motion state means that theassembly motion unit can only move along or rotatearound the orientation axis under the axis orientationconstraint, and free motion state B means the assemblymotion unit and the assembly base unit compose a newsubassembly and return to the free motion state again.Note that the initial state of assembly is free motionstate A while that of disassembly is free motion state B#p#分頁標題#e#
because disassembly is the contrary course ofassembly. During assembly, the two assembly units arein the free motion state first, and collision detectionalgorithm guarantees that any parts (subassemblies)don't collide with each other, and similarly, it can alsoavoid penetration between users and that between usersand objects. Axis orientation constraints recognitionalgorithm detects whether axis orientation constraint issatisfied between the two assembly units, if satisfied,they will perform axis align operation via axial preciselocation, then they enter axial motion state in whichaxial collision detection algorithm guarantees axialmotion without interference and detects possible
design defects. Axial collision detection extends thegeneral collision detection which cannot distinguishbetween contact and collision by examining thenormals of the pieces where collision happens. Axialmotion solving algorithm maps the users' inputs to acertain length of translation along the orientation axis.
Proceedings of the Second International Conference on Embedded Software and Systems (ICESS’05)0-7695-2512-1/05 $20.00 © 2005 IEEE
Authorized licensed use limited to: MACQUARIE UNIV. Downloaded on July 5, 2009 at 22:47 from IEEE Xplore. Restrictions apply.
Axis orientation constraint relief algorithm is used to
detect user's intention of relieving axis orientation
constraint. Face match constraint recognition algorithm
is used to detect whether face match constraint is
satisfied between the two assembly units, if satisfied,they are assembled together successfully, and composea new subassembly, and then enter free motion state Bin which face match constraint relief algorithm detects
user's intention of relieving face match constraint(namely disassembly intention). Fig.4-Fig.6 show an
example of motion guidance.Axial motion
state
Free motion
state A
Free motion
state B
Satisfied axis
orientation
constraint
Relieved axis
orientation
constraint
Satisfied face
match constraint
Relieved face
match constraint
Motion guidance
• Collision detection
• Axis orientation
constraint
recognition
Motion guidance
• Axial collision detection
• Axial motion solving
• Axis orientation constraint relief
• Face match constraint recognition
Motion guidance
• Collision detection
• Face match constraint
relief
Precise location
• Axial precise location
Precise location
• Face match precise location
Figure 3. State transition of a part (subassembly) in
a single assembly
Figure 4. P1 collides with P2
Figure 5. Recognized the axis orientation constraint
Figure 6. Recognized the face match constraint
5. Data Translation
This module includes two functions: model#p#分頁標題#e#
translation and information extraction. The former
translates the parametric model created in Pro/Engineer
to triangle mesh model (Multigen OpenFlight) which
can be supported in VR. The latter extracts orientation
axis, match face and assembly constraints for motion
guidance. It is a plug-in for Pro/Engineer and
developed by Pro/Toolkit and Multigen OpenFlight
API.
The assembly hierarchy in Pro/Engineer is shown in
Fig.7. The assembly consists of a series of
subassemblies and parts, and a part consists of several
features. Surfaces and geometry items (such as axis,
dimension) constitute a feature. And the OpenFlight
also uses a multilevel hierarchy to define the
organization of the objects in the database (see Fig.8).
At the top level is the database node (Node is a generic
term for any record in the hierarchy). At the lowest
level are objects made up of polygons (fltFace), which
are, in turn, made up of vertices. Between these two
levels are a number of different types of organizational
nodes that are attached to each other to organize the
database. An assembly, subassembly, or part in
Pro/Engineer is represented by a DOF (degree of
freedom) node (fltDof) in OpenFlight. A feature
corresponds to a group node (fltGroup), and a surface
or an axis corresponds to an object node (fltObject).
The assembly constraints are stored in the
comments field of OpenFlight model.
…...
…...
…... …...
assembly
subassembly part ProCsys
part part ProFeature 1 ProFeature n ProCsys
ProFeature 1 ProFeature n ProCsys ProAxis 1 ProAxis n ProSurface 1 ProSurface n
ProCsys
Figure 7. Assembly hierarchy in Pro/Engineer
Proceedings of the Second International Conference on Embedded Software and Systems (ICESS’05)
0-7695-2512-1/05 $20.00 © 2005 IEEE
Authorized licensed use limited to: MACQUARIE UNIV. Downloaded on July 5, 2009 at 22:47 from IEEE Xplore. Restrictions apply.
…...
…...
…...
…...
…...
g2
fltDof
(assembly)
fltDof
(subassembly)
fltDof (part)
fltObject
(Csys)
fltDof (part) fltDof (part)
fltGroup 1
(feature)
fltGroup n
(feature)
fltObject
(Csys)
fltGroup 1
(feature)
fltGroup n
(feature)
fltObject
(Csys)
fltObject 1
(axis)
fltObject n
(axis)
fltObject 1
(surface)
fltObject n
(surface)
fltObject
(Csys)
db
g1
fltFace 1 fltFace n
Figure 8. Assembly hierarchy in OpenFlight
6. Multimodal Interaction
VECA allows users to interact with VE through
multiple modalities such as speech and gesture. The
multimodal interface model of VECA is shown in#p#分頁標題#e#
Fig.9. A key element in understanding user multimodal
inputs is known as reference resolution [21]. In VECA,
a reference resolution approach for virtual environment
(RRVE) [22] is employed. RRVE improves the
approach of Kehler [23], and adopts the method of
Senseshape [24][25] to disambiguate in object
selection of 3D interaction.
RRVE divides the objects in VE into four state
spaces: gestured objects which correspond to the status
point, selected object last time which correspond to the
status “in focus”, unselected but visible objects which
correspond to the status “activated”, unselected and
invisible objects which correspond to the status
“extinct”. The sequence of cognitive hierarchy is: point
> in focus > activated > extinct. In RRVE, resolution
reference is just as match operation between candidate
objects and reference constraints which are gained
from speech. There are two types of match operation:
“don’t conflict” and “meet entirely”. The former is the
necessary condition for the latter. For example, if an
object has attribute set: {name=”M4-1”, type=”bolt”,
feature=” hexagon”}, and the user input “select the red
bolt”, the result of “don’t conflict” is true and the result
of “meet entirely” is false.
In RRVE, Senseshape is a cone which is attached to
the finger of virtual hand (see Fig.10). It provides
valuable statistical rankings about gestured objects
through collision detection. For example, the time
ranking of an object is derived from the fraction of
time the object spends in the cone over a specified time
period: the more time the object is in the cone, the
higher the ranking. The point queue is a priority queue
of point objects and the priority is determined by
weighted average of these statistical rankings. The
activated queue is a priority queue of activated objects
and the priority is determined by distance between the
object and the current viewpoint. Extinct vector is a
vector of extinct objects but not considering the
sequence as above.
The algorithm of RRVE is as follows.
1. Get reference constraint from speech;
2. Get point object priority queue by method of
Senseshape;
3. If an point object doesn’t conflict with reference
constraints, choose that object;
4. Otherwise, if the focus object doesn’t conflict
with reference constraints, choose that object;
5. Otherwise, if an activated object doesn’t conflict
with reference constraints, choose that object;
6. Otherwise, if an extinct object meets all reference
constraints, choose that object.
This algorithm simply chooses the most salient#p#分頁標題#e#
entity that meets reference constraints, regardless of
the type of referential form. And we found that, when
users interacted with visible objects, the accuracy of
reference resolution was higher if the match condition
was looser. When users attempt to interact with extinct
object, most of cognitive resources fasten on speech
model. The information from the speech should be
considered sufficiently, so the match operation of
“meet entirely” is used.
Command type, constraint
Speech
parsing
Multimodal integration
Speech
recognition
Gesture
recognition
Gesture
parsing
Activated
queue
Voice command Sensor data
Focus
object
executed command or hint
Point gesture
Point queue
XML element, attribute
Extinct
vector
Figure 9. Multimodal interface model
Figure 10. Senseshape in VECA
Proceedings of the Second International Conference on Embedded Software and Systems (ICESS’05)
0-7695-2512-1/05 $20.00 © 2005 IEEE
Authorized licensed use limited to: MACQUARIE UNIV. Downloaded on July 5, 2009 at 22:47 from IEEE Xplore. Restrictions apply.
7. Experiments
The system of VECA is implemented in C++ using
Microsoft Visual C++ 6.0. The VR platform is
OpenSceneGraph 0.9.6-2, and the engine of speech
recognition is Microsoft Speech SDK 5.1. The
distributed interactive simulation platform is
DMSO_RTI 1.3v6 and DVE_FM v1.0. The CAD
software is Pro/Engineer 2001 and we implement
application development by Pro/Toolkit and Multigen
OpenFlight API. VECA is created to be capable of
supporting a variety of VR peripherals such as
CyberGlove, 3D shutter glasses.
The experimental environment is illustrated in
Fig.11. One PC is the Pro/E server and four PCs join
the simulation. Users can download triangle mesh
models which include corresponding information from
the Pro/E server. Then users utilize simulation
application (VM federate 1 to 3) to perform assembly
task collaboratively through CyberGlove, microphone,
mouse or keyboard. DVE_StealthAndPlayer can record
and replay the whole course of VA simulation. The
system runs over 100M Ethernet and a switch which
supports multicast. Experiment shows that VECA
provides a natural and effective simulation
environment, and HLA-based communication and
collaboration can effectively resolve the problem of
cooperation and competition between multi-users.
Fig.12-Fig.15 illustrates a single assembly course of a
part.
COLACTSTA-
1 2 3 4 5 6 7 8 9101112
HS1 HS2 OK1 OK2 PS
CONSOLE
100M ethernet network
Switch supported multicast
VM federate 1 VM federate 2 VM federate 3
Pro/E server VM federate 4 (DVE_StealthAndPlayer)#p#分頁標題#e#
Figure 11. Experimental environment
The experiment shows that VECA can effectively
solve the problem of competition and collaboration
between multi-users, and help users locate the part
(subassembly) freely and precisely. It has been
successfully applied to the design of certain type of
products.
Figure 12. Federate 1 and federate 2 strive for the
ownership of a part
Figure 13. Federate 2 gets the part, and the part
collides with the subassembly
Figure 14. Federate 2’s intention of assembly is
recognized
Figure 15. The assembly is finished
Proceedings of the Second International Conference on Embedded Software and Systems (ICESS’05)
0-7695-2512-1/05 $20.00 © 2005 IEEE
Authorized licensed use limited to: MACQUARIE UNIV. Downloaded on July 5, 2009 at 22:47 from IEEE Xplore. Restrictions apply.
8. Conclusions
This paper presents a Virtual Environment for
Collaborative Assembly, which can build a
collaborative virtual assembly system allowing
geographical dispersed engineers to perform an
assembly task together, since there are some
requirements on virtual assembly based on a
distributed virtual environment. VECA mainly
includes HLA-based (High Level Architecture)
communication and collaboration, motion guidance
based on collision detection and assembly constraints
recognition, data translation from CAD to virtual
environment, reference resolution in multimodal
interaction, and so on. Because VECA can only
translate the data from CAD to virtual environment,
it’s necessary to return the feedback to CAD in the
future work.
9. References
[1] Jayaram S., Connacher H., Lyons K., “Virtual Assembly
Using Virtual Reality Techniques”, Computer-Aided Design,
29(8), 1997, 575-584.
[2] Xiaobu Yuan, Simon X. Yang, “Virtual assembly
with biologically inspired intelligence” Systems, Man
and Cybernetics, Part C, IEEE Transactions, 33(2),
2003, 159-167.
[3] Jianzhong Mo, Chi Cheng, Prabhu B.S. and Gadh
R., "On the Creation of a Unified Modeling Language
Based Collaborative Virtual Assembly/Disassembly
System" ASME 2003 Design Engineering Technical
Conferences and Computers and Information in
Engineering Conference, Chicago, Illinois, 2–6
September 2003.
[4] Michael R. Maecdania and Michael J. Zyda, “A
taxonomy for networked virtual environments” IEEE
Multimedia, 4(1), January-March 1997, 48-56.
[5] Ling Chen, Gencai Chen, Hong Chen, Xiaolei Xu,
Chun Chen, “Study on Effects of Network
Characteristics on Cooperation Performance in a
Desktop CVE System” Computer Supported
Cooperative Work in Design, 2004, Volume 1, 26-28
May 2004, 121-126.
[6] Srinivasan H., Figueroa R., and Gadh R., “Selective#p#分頁標題#e#
disassembly for virtual prototyping as applied to
demanufacturing” Journal of Robotics and Computer
Integrated Manufacturing, 15(3), 1999, 231-245.
[7] Luis Marcelino, Norman Murray, Terrence
Fernando, “A Constraint Manager to Support Virtual
Maintainability” Computer & Graphics 2003, 27, 19-
26.
[8] Huagen Wan, Shuming Gao, Qunsheng Peng,
“VDVAS: An Integrated Virtual Design and Virtual
Assembly Environment” Journal of Image and
Graphics, 7A(1), 2002, 27-35.
[9] Jayaram S., Yong Wang, Jayaram, U., Lyons, K.,
Hart, P.,