Civil Engineering2

Collaborative Virtual Environment: An Application for Civil Engineering

By Mauricio Capra

Thesis submitted to the University of Nottingham for the degree of Doctor of Philosophy, April 2007

Index
COLLABORATIVE VIRTUAL ENVIRONMENT:.........................................................................I INDEX...................................................................................................................................................II LIST OF FIGURES..............................................................................................................................V

LIST OF TABLES..............................................................................................................................XI

LIST OF ACRONYMS.....................................................................................................................XV

ABSTRACT......................................................................................................................................XIX

ACKNOWLEDGEMENTS............................................................................................................XXI

1.1 COMMUNICATION, COLLABORATION, AND MUTUAL UNDERSTANDING......................................................25 1.2 POTENTIAL GAPS DURING THE CEN PHASES – COMMUNICATION, COLLABORATION AND MUTUAL UNDERSTANDING ....................................................................................................................................27 1.3 CIVIL ENGINEERING LIFE CYCLE........................................................................................................29 1.4 THESIS OBJECTIVES..........................................................................................................................32 1.5 PROPOSED SOLUTION........................................................................................................................33 1.6 STEPS FOLLOWED TO ACHIEVE THE OBJECTIVES...................................................................................34 1.7 THESIS STRUCTURE..........................................................................................................................35 2.1 SOLVING CEN CHALLENGES – DESIGN AND CONSTRUCTION PHASES......................................................44 2.1.1 Building Information Model.................................................................................................44 2.1.2 Researches addressing Communication, Collaboration and Mutual Understanding in CEn Projects .........................................................................................................................................48 2.2 IMPLEMENTING NEW TECHNOLOGIES IN A CEN ENVIRONMENT...............................................................55 2.3 IMPROVING CEN PROJECTS – COMMUNICATION, COLLABORATION AND MUTUAL UNDERSTANDING ............55 3.1 VIRTUAL REALITY............................................................................................................................59 3.2 AUGMENTED REALITY......................................................................................................................61 3.3 AUGMENTED VIRTUALITY..................................................................................................................62 3.4 MIXED REALITY..............................................................................................................................63 3.5 COLLABORATIVE VIRTUAL ENVIRONMENTS .........................................................................................64 3.6 RELATED PROJECTS .........................................................................................................................69 3.7 GLOBAL POSITIONING SYSTEM ........................................................................................................101 3.8 GPS FREQUENCIES AND DATA – BRIEF REVIEW................................................................................102 3.9 STAND-ALONE GPS POSITIONING ...................................................................................................103 3.10 DIFFERENTIAL GPS – DGPS.......................................................................................................106 3.11 REAL TIME KINEMATICS GPS - RTK-GPS...................................................................................108 3.12 AUGMENTATION SYSTEMS AND NETWORK SERVICES.........................................................................111 3.13 TESTING SINGLE FREQUENCY GPS, DGPS AND RTK-GPS............................................................115 3.14 OTHER POSITIONING SYSTEMS.......................................................................................................118 3.15 COMBINING AND INTEGRATING TECHNOLOGIES.................................................................................121 4.1 CONCEPTUALISATIONS OF USER GROUPS...........................................................................................124 4.2 DESIGN OVERVIEW.........................................................................................................................126 4.2.1 Office User Design.............................................................................................................128 4.2.2 Field User Design...............................................................................................................130 4.3 HARDWARE AND SOFTWARE............................................................................................................131 4.3.1 Hardware............................................................................................................................132 4.3.2 Software..............................................................................................................................141 4.4 FINAL ANALYSIS OVER THE DESIGN..................................................................................................145 5.1 FIRST PHASE – OFFICE USER AND CVE...........................................................................................149 5.1.1 Survey and Data Conversion..............................................................................................149 5.2 FIRST PHASE OF PRACTICAL WORK..................................................................................................154 5.3 SECOND PHASE OF THE PRACTICAL WORK.........................................................................................159 5.3.1 Integrating Software, Hardware, Field and Office Users..................................................163 5.3.2 Notes of How to Use the System.........................................................................................172 6.1 THREE EXERCISES FOR THE EVALUATION...........................................................................................175 6.2 EVALUATION.................................................................................................................................185 6.2.1 Pre-Test before the Final Evaluation – Final Adjustments................................................187 6.2.2 The Final Evaluation..........................................................................................................189 6.2.3 Outcome from the Exercises...............................................................................................192 6.3 FINAL POINTS OVER THE EVALUATION..............................................................................................207 7.1 SUMMARY AND REFLECTIONS...........................................................................................................218 7.2 THESIS CONTRIBUTION ...................................................................................................................223 7.3 DISSEMINATION..............................................................................................................................225 7.4 FUTURE WORK..............................................................................................................................226 7.4.1 Future on Hardware...........................................................................................................226 7.4.2 Future on Software.............................................................................................................227 7.4.3 Future Applications............................................................................................................227 7.4.4 Further Evaluation.............................................................................................................229 7.4.5 Concluding Remarks..........................................................................................................230 A.1 FEATURES FILE ............................................................................................................................233 A.2 SOURCE CODE REFERENCE ............................................................................................................235

iii

B.1 RTK-GPS CLASS........................................................................................................................242 GPSReveiver.java.......................................................................................................................242 GPSPublisher.java.....................................................................................................................243 B.2 TILT-SENSOR AND MAGNETIC COMPASS CLASS.................................................................................244 TCM2.java..................................................................................................................................244 TCM2Publisher.java..................................................................................................................245 B.3 READING FROM THE DATABASE.......................................................................................................246 Integrator.java............................................................................................................................246 B.4 KALMAN FILTER...........................................................................................................................248 Matriz.java..................................................................................................................................248 B.5 OFFICE USER KEY STROKE COMMAND............................................................................................250 ReadKeyStroke.java....................................................................................................................250 KeyStroke.java............................................................................................................................251 C.1 MERGING REAL AND DIGITAL ENVIRONMENT ...................................................................................253 C.2 USABILITY....................................................................................................................................255 C.3 HARDWARE..................................................................................................................................255 C.4 DIGITAL ENVIRONMENT..................................................................................................................256 C.5 OFFICE USER AND FIELD USER.......................................................................................................258 C.6 DIFFERENT/COMPLEMENTARY APPROACHES......................................................................................260

BIBLIOGRAPHY.............................................................................................................................264

iv

List of Figures
FIGURE 1.1 CEN LIFE-CYCLE AND INTERRELATIONSHIPS (NEIL S. GRIGG ET AL., 2001)........................................................................................................................................................30 FIGURE 1.2 GENERAL VIEW OF THE PROJECT.......................................................................34

FIGURE 1.3 THESIS STRUCTURE..................................................................................................36

FIGURE 2.4 REDUCTION OF COSTS AND TIME IN THE PROJECT EVALUATED ON (KHANZODE ET AL., 2008)...............................................................................................................48

FIGURE 3.5 THE “VIRTUALITY CONTINUUM” (MILGRAM AND KISHINO, 1994)..........60

FIGURE 3.6 POTENTIAL OIL FIELD – PETROBRAS.................................................................72

FIGURE 3.7 VRML MODEL OF THE SRI CAMPUS (SRI INTERNATIONAL, 2000A)..........74

v

FIGURE 3.8 VRML WIND VECTOR OVERLAY (SRI INTERNATIONAL, 2000B)................75

FIGURE 3.9 DOWNTOWN OF LA MODELLED BY UST TEAM (UST, 2003)..........................76

FIGURE 3.10 LAX ORIGINAL CONDITION (UST, 1998)............................................................77

FIGURE 3.11 LAX PROPOSED CONDITION (UST, 1998)...........................................................77

FIGURE 3.12 SIMCITY – VIEW FROM OLYMPIA (CITYSCAPE, 2004).................................79

FIGURE 3.13 VISUALISING DATA FROM A GIS (CAPRA, 2000).............................................80

FIGURE 3.14 TEXTURED 3-D VIEW OF MUNICH (ESRI, 2004B)............................................81

FIGURE 3.15 ARCGIS SCREENSHOT (ESRI, 2004A)...................................................................82

FIGURE 3.16 SHIODOME CYBERDOME STRUCTURE (MATSUSHITA ELECTRONIC WORKS LTD, 2003).............................................................................................................................84

FIGURE 3.17 SHIODOME CYBERDOME USER’S INTERFACE (MATSUSHITA ELECTRONIC WORKS LTD., 2003)................................................................................................85

FIGURE 3.18 ADDITIONAL INFORMATION OVERLAYING A HISTORIC BUILDING.....89

FIGURE 3.19 ADDITIONAL INFORMATION EXPLORED BY THE USERS (REITMAYR AND SCHMALSTIEG, 2004C)...........................................................................................................90

FIGURE 3.20 TINMITH WEARABLE SYSTEM (PIEKARSKI AND THOMAS, 2002)............93

FIGURE 3.21 AUGURSCOPE (BENFORD ET AL., 2003B)...........................................................95

FIGURE 3.22 AR SUB-SURFACE SYSTEM (ROBERTS ET AL., 2004).....................................97

FIGURE 3.23 VE PROJECTED OVER THE REAL ENVIRONMENT (ROBERTS ET AL., 2004)........................................................................................................................................................98

vi

FIGURE 3.24 ARMOBILE (WOODWARD ET AL., 2007B)........................................................100

FIGURE 3.25 ARWEBCAM (WOODWARD ET AL., 2007C).....................................................100

FIGURE 3.26 RANGE MEASUREMENT TIMING RELATIONSHIPS (KAPLAN, 1996)......104

FIGURE 3.27 CARRIER PHASE AND INTEGER AMBIGUITY (AQUINO, 1992) ...............................................................................................................................................................109

FIGURE 3.28 SINGLE FREQUENCY RECEIVER.......................................................................116

FIGURE 3.29 SINGLE FREQUENCY RECEIVER – A SINGLE COORDINATE...................117

FIGURE 3.30 DUAL FREQUENCY RECEIVER PERFORMING DGPS..................................117

FIGURE 3.31 DUAL FREQUENCY RECEIVER PERFORMING RTK-GPS...........................118

FIGURE 4.32 PROPOSED CARS ....................................................................................................125

FIGURE 4.33 DESIGN OVERVIEW OF THE PROJECT – FIELD AND OFFICE USERS....127

FIGURE 4.34 GPS RECEIVER, ANTENNAS, FLASH CARD, AND RADIO............................134

FIGURE 4.35 TCM2-50 CIRCUIT BOARD....................................................................................136

FIGURE 4.36 PLM-S700....................................................................................................................137

FIGURE 4.37 VFOV CALCULATION............................................................................................137

FIGURE 4.38 TWIDDLER2..............................................................................................................139

FIGURE 4.39 HELMET, PCGLASSTRON, LEICA BATTERY, SEE-THROUGH HMD AND CABLE.................................................................................................................................................141

vii

FIGURE 4.40 EQUIP PLATFORM (GREENHALGH, 2001F).....................................................144

FIGURE 5.41 STEPS FOR THE SYSTEM IMPLEMENTATION...............................................148

FIGURE 5.42 LANDSCAPE SURVEYED AROUND IESSG BUILDING..................................151

FIGURE 5.43 VIEW OF A COMPLETED VE ON VRML BROWSER......................................152

FIGURE 5.44 SYSTEM REQUIREMENT BETWEEN OFFICE USERS ONLY......................155

FIGURE 5.45 VIEW OF THE USING MASSIVE-3.......................................................................158

FIGURE 5.46 FRONT VIEW OF THE IESSG BUILDING..........................................................158

FIGURE 5.47 RED EMBODIMENT REPRESENTING A SECOND OFFICE USER..............159

FIGURE 5.48 SYSTEM REQUIREMENT BETWEEN OFFICE AND FIELD USERS............159

FIGURE 5.49 MASSIVE-3 CARTESIAN SYSTEM.......................................................................160

FIGURE 5.50 MAGNETIC COMPASS DATA...............................................................................171

FIGURE 5.51 PITCH DATA.............................................................................................................171

FIGURE 5.52 ROLL DATA...............................................................................................................171

FIGURE 6.53 BLUEPRINT OF PHOENIX TRAM STATION (CARILLION, 2000)................178

FIGURE 6.54 SIGN AND BIN...........................................................................................................179

FIGURE 6.55 SIGN AND LAMP POST...........................................................................................179

FIGURE 6.56 LAMP POST...............................................................................................................179

viii

FIGURE 6.57 TRAM SHELTER......................................................................................................179

FIGURE 6.58 PLATFORM...............................................................................................................179

FIGURE 6.59 TRAM STATION.......................................................................................................180

FIGURE 6.60 TRAM STATION – TOP VIEW...............................................................................180

FIGURE 6.61 TRAM STATION – CVE...........................................................................................180

FIGURE 6.62 VIRTUAL TRAIN APPROACHING THE STATION..........................................181

FIGURE 6.63 PLATFORM IN THE PHYSICAL WORLD (NOTTINGHAM TRAM CONSORTIUM, 2004)........................................................................................................................181

FIGURE 6.64 INTERNAL DIVISION (LUCENA, 1986)...............................................................182

FIGURE 6.65 RIGHT SIDE SCHEME (LUCENA, 1986)..............................................................182

FIGURE 6.66 COLOURS REPRESENTING DIFFERENT STAGES OF THE ENGINEERING PROJECT............................................................................................................................................183

FIGURE 6.67 THE COMPLETE CONSTRUCTION (DESIGN AND CONSTRUCTION PHASES)..............................................................................................................................................183

FIGURE 6.68 THE HOUSE AS IT LOOKS IN THE PHYSICAL WORLD...............................183

FIGURE 6.69 THE STRUCTURE OF THE HOUSE.....................................................................184

FIGURE 6.70 WINDOWS AND STAIRS.........................................................................................184

FIGURE 6.71 DIFFERENT COLOURS OF THE PIPES INDICATING DIFFERENT USES. 184

FIGURE 6.72 TOPICS COVERED BY THE EVALUATION......................................................191

ix

FIGURE 6.73 GROUPING POINTS OF INTEREST FOR THE COMPLEMENTARY OUTCOME..........................................................................................................................................208

FIGURE B.74KALMAN FILTER DATAFLOW............................................................................249

x

List of Tables
TABLE 3.1 DIFFERENT POSITIONING SYSTEMS...................................................................121 TABLE 4.2 LLQ – LEICA LOCAL POSITION AND QUALITY................................................133

TABLE 4.3 PIN FUNCTION OF TCM2-50.....................................................................................134

TABLE 4.4 TCM2-50 MESSAGE ....................................................................................................135

TABLE 5.5 AN EXAMPLE OF .CVE FILE....................................................................................153

TABLE A.6 FEATURES’ FILE STRUCTURE...............................................................................233

TABLE A.7 VRML HEADER’S MODIFIED..................................................................................234

xi

TABLE A.8 GLOBAL VARIABLE..................................................................................................235

TABLE A.9 FUNCTION MAIN........................................................................................................236

TABLE A.10 FUNCTION OPEN_ARQUIVO.................................................................................236

TABLE A.11 FUNCTION TAKE_INFORMATION......................................................................236

TABLE A.12 FUNCTION VRML_GENERATOR.........................................................................236

TABLE A.13 FUNCTION INPUT_FEATURE................................................................................237

TABLE A.14 FUNCTION BUILD_CVE..........................................................................................237

TABLE A.15 FUNCTION TABELA_POLIGONO.........................................................................237

TABLE A.16 FUNCTION FBGUPNPOLY......................................................................................237

TABLE A.17 FUNCTION INPUT_ROAD.......................................................................................238

TABLE A.18 HIGH LEVEL ALGORITHM...................................................................................238

TABLE B.19 GPSRECEIVER.JAVA ALGORITHM....................................................................242

TABLE B.20 GPSPUBLISHER.JAVA ALGORITHM..................................................................243

TABLE B.21 TCM2.JAVA ALGORITHM......................................................................................244

TABLE B.22 TCM2PUBLISHER.JAVA ALGORITHM...............................................................245

TABLE B.23 INTEGRATOR.JAVA ALGORITHM......................................................................247

TABLE B.24 READKEYSTROKE.JAVA ALGORITHM.............................................................250

xii

TABLE B.25 KEYSTROKE.JAVA ALGORITHM........................................................................251

xiii

List of Equations
EQUATION 3.1...................................................................................................................................105 EQUATION 3.2...................................................................................................................................105

EQUATION 3.3...................................................................................................................................105

EQUATION 3.4...................................................................................................................................105

EQUATION 4.5...................................................................................................................................138

xiv

List of Acronyms
American Standard Code for Information Interchange Application Program Interface Architecture, Engineering and Construction Architect1 Augmented Reality Augmented Virtuality Building Information Model centimetre Central Processor Unit Chemical Environmental and Mining Engineering Civil Engineering Closed Circuit Television Coarse/Acquisition Collaborative Augmented Reality Systems Collaborative Environments Collaborative Virtual Environments Communication Port Computational Fluid Dynamics Computer Aided Design Computer Automatic Virtual Environment Computer Scientist1 Computer Scientist2 Constructive Solid Geometry (ASCII) (API) (AEC) (A1) (AR) (AV) (BIM) (cm) (CPUs) (SChEME) (CEn) (CCTV) (C/A code) (CARS) (CE) (CVE) (COM1) (CFD) (CAD) (CAVE) (C1) (C2) (CSG)

xv

Dave’s Graphic Library Differential Global Positioning System Digital Terrain Modelling Disk Operational System encrypted P Engineer1 Engineer2 Engineer3 Engineer4 Engineering and Physical Sciences Research Council European Geostationary Navigation Overlay Service European Space Agency Federal Aviation Administration Field of View Frequency Modulation Full Operational Capability General Packet Radio Service Geographical Information Systems Global Orbiting Navigation Satellite System Global Positioning System Global System for Mobile Communication Graphical Unit Interface Head Mounted Display Heating, Ventilating and Air Conditioning hertz Identifications Inertial Navigation System kilo hertz kilobytes Knowledge Base System Leica Local Positioning and Quality Level of Detail Local Area Network Los Angeles Los Angeles International Airport Mega Hertz Metre Mixed Reality Mixed Reality Laboratory Mobile Augmented Reality System

(DGL) (DGPS) (DTM) (DOS) P(Y) (E1) (E2) (E3) (E4) (EPSRC) (EGNOS) (ESA) (FAA) (FoV) (FM) (FOC) (GPRS) (GIS) (GLONASS) (GPS) (GSM) (GUI) (HMD) (HVAC) (Hz) (IDs) (INS) (kHz) (k) (KBS) (LLQ) (LoD) (LAN) (LA) (LAX) (MHz) (M) (MR) (MRL) (MARS)

xvi

Multi-Functional Satellite-based Augmentation System Multi-Functional Transport Satellite Multiple User Dimensions/Dungeons National Marines Electronics Association Nottingham Express Transit Object Oriented On The Fly Ordinary User1 Ordnance Survey Packed Internet Gopher Personal Computer Personal Digital Assistant Precise Precise Positioning System Pseudo Random Noise Quasi-Zenith Satellite System Radio Data System Random Access Memory Real Environment Real Time Kinematic Global Positioning System Standard Positioning System Stanford Research Institute Super Video Graphics Array Swedish Network of Permanent GPS Reference Station Technical Research Centre of Finland Tile Set Manager Application Program Interface Uniform Resource Locator United States Department of Defence United States Geological Survey Universal Time Coordinated University of California Urban Simulation Team Video Graphics Array Virtual Environment Virtual Design Construction Virtual Reality Virtual Reality Modelling Language Visual Simulations WAAS Master Station Watts Wide Area Augmentation System

(MSAS) (MTSAT) (MUD) (NMEA) (NET) (OO) (OTF) (O1) (OS) (PING) (PC) (PDA) (P) (PPS) (PRN) (QZSS) (RDS) (RAM) (RE) (RTK-GPS) (SPS) (SRI) (SVGA) (SWEPOS) (VTT) (tsmApi) (URL) (USA-DoD) (USGS) (UTC) (UCLA) (UST) (VGA) (VE) (VDC) (VR) (VRML) (VizSims) (WMS) (W) (WAAS)

xvii

Wide Area DGPS Wide Area Reference Stations World Geodetic System 1984 World Wide Web/File Transfer Protocol

(WADGPS) (WRS) (WGS84) (WWW/FTP)

xviii

Abstract
The present thesis designs, implements and evaluates a collaborative augmented reality system for civil engineering purposes oriented to design and construction phases. The work starts with a survey of the potential gaps and solutions presented so far regarding to communication, collaboration and mutual understanding. Such gaps can occur on multidisciplinary teams of architects, engineers and constructors when working in the same civil engineering project. In the context of this thesis, outdoor users are equipped with a real-time kinematic global positioning system receiver, a notebook, a head-mounted display, a tilt sensor and a compass. A virtual environment representing parts of a civil engineering work is projected before their eyes. Outdoor users share this collaborative virtual environment with indoor users. They can talk and see each other through an embodiment. Indoor users can be at any place where internet is available. The goal of the thesis is to show that a networked solution of at least two users (between indoor and outdoor users) is a requirement when outdoor users are performing complex tasks at the same time that they are experiencing an immersive augmented reality application. Indoor users interact with outdoor users when manipulating the virtual environment, working as a guide through the scene, making clear obscure points of understanding, etc. The thesis evaluates how users interact with the system using a formative approach. Users are introduced to the system and motivated to “talk loudly” what they are experiencing during the tests. All xix

users are video-recorded while performing the exercises and interviewed straight afterwards. The evaluation reveals that users end experiencing a too immersive system, narrowing their “attentional spotlight” to the virtual environment and not experiencing an augmented reality system. The evaluation also makes clear that the design of the virtual environment is eventually more important for users than the system itself and it is completely dependent of the kind of application that it is being used to and who the users are.

xx

Acknowledgements
The author would like to thank the Engineering and Physical Sciences Research Council (EPSRC) for the scholarship awarded to this project. To Professors Steve Benford, Alan Dodson, Tom Rodden and Chris Greenhalgh, who always believed in my potential, thank you very much for your guidance, help, support and encouragement. To Dr. Marcio Aquino (and his family) who from the very beginning (1999, in Salvador, Brazil) was the person to always show me the right path, thank you very much for your friendship. To Dr. Boriana Koleva for the help and patience in the early stages. To Dr. Norman Teferle and his family, Marcos Andreotti and his family, and Stefan Egglestone, thank you for your friendship. To my wife Laiz Chen Capra, thank you for all your love, dedication and specially your patience.

xxi

To my parents Mauro Paseto Capra and Sonia Erdmann Capra and my brother Marcos Capra, who showed me from the beginning the need to stand up, proud of ourselves and confident, even in the worst scenarios. To my parents-in-law Chen Wen Yii and Maria Sylvia Rubinger Chen for the support and love. To Leif Oppermann, Matt Watkins, Rachel Jacobs and Robin Shackford for the really good moments together. To Sensei Andreas Santschi, Sensei Masaki Negishi, Dominique Schmid, Nadia Probst, Andre Borges, Simone Bueno, Fred Goede, Anneke Boom. You are amazing people that help me a lot at the right time. Thank you very much! To all my friends and family, who I did not name here but who motivated me to do this work.

xxii

This thesis is dedicated to the memory of the best fighter I ever saw. She only fell in her very last battle, the one that everyone succumbs to. My mom Sonia Erdmann Capra

xxiii

Chapter 1
Introduction
Civil Engineering (CEn) as many other professions relies over a complex network of knowledge and information among the parts involved. As mention in (Ugwu O. O. et al., 2001) in this collaborative design environment the real challenge is to find a common representation of the problem-solving domain from the various experts. The difference in terminologies can guide the communication between parts to a dead end. Professionals can use the different terms to same concepts or same terms to different concepts. The online dictionary (Merrian-Webster, 2008) translate “ontology” as “a branch of metaphysics concerned with the nature and relations of being”. Augenbroe (Augenbroe et al., 2002) mentions the difficulties of communication and collaboration between beings. He exemplifies the intensiveness and paper work-flow in design and construction business. Any change of request could take weeks to be approved between contractors, engineers and architects. Heitz (Heintz, 2002) mentions in his paper the “fail to capture the collaborative nature of the design process and do not support the efficient and effective management of the dependencies between designs tasks carried out by different actors”. The problem of communication and collaboration is also explored by (Wilkinson, 2001). Her paper describes the resistance of acceptance of project managers and their work by different actors in the CEn environment.

24

Although there are several other points that could also create some influence for the relation between beings do not work as smooth as it is desired, see (Heintz, 2002) for some other examples, the citations illustrates where collaboration, communication and mutual understanding between those involved in the numerous steps of CEn cycle do not work. This relation (ontology) between parts or actors creates the motivation for the present work. The present work aims to explore and evaluate a tool created for users to enhance communication, collaboration, and mutual understanding between parts involved in a CEn project. Such tool is based on a Augmented Reality (AR) system concept.

1.1

Communication, Collaboration, and Mutual Understanding

The three key worlds for this thesis are communication, collaboration, and mutual understanding. Along the chapters this thesis explores its objectives as a motivation, based on the three key worlds, aiming to reduce some of the gaps identified on CEn environments. A general definition of communication can be found in Wikipedia (Public Domain / Wikipedia). Communication is a process to transfer information from “A” (also know as sender) to “B” (also know as receiver) through the use of a medium. It is possible to exemplify ways to perform communication as: • • Auditory means: speaking, tone the voice, etc. Physical means: body language, eye contact, writing, etc.

Communication is defined as a process by which we assign and convey meaning in an attempt to create shared understanding.”…“It is through communication that collaboration and cooperation occur. (Office of Superintendent of Public Instruction)

The online dictionary Merrian-Webster communication as:
1: an act or instance of transmitting

(Merrian-Webster,

2008)

translates

25

2 a: information communicated b: a verbal or written message 3 a: a process by which information is exchanged between individuals through a common system of symbols, signs, or behaviour <the function of pheromones in insect communication>; also: exchange of information b: personal rapport <a lack of communication between old and young persons> 4 plural a: a system (as of telephones) for communicating b: a system of routes for moving troops, supplies, and vehicles c: personnel engaged in communicating 5 plural but sing or plural in constr a: a technique for expressing ideas effectively (as in speech) b: the technology of the transmission of information (as by print or telecommunication)

The world collaboration comes almost as a consequence of communication. In Wikipedia (Public Domain / Wikipedia) collaboration means:
Collaboration is a recursive process where two or more people work together toward an intersection of common goals — for example, an intellectual endeavour (MerrianWebster, 2007, Encyclopedia Brittanica Online, 2007) that is creative in nature ((Eds.) Simpson J. A. and Weiner E. S. C., 1989) — by sharing knowledge, learning and building consensus.

From Merrian-Webster collaboration can be understood as:
1: to work jointly with others or together especially in an intellectual endeavour 2: to cooperate with or willingly assist an enemy of one's country and especially an occupying force 3: to cooperate with an agency or instrumentality with which one is not immediately connected

Godfried (Godfried Augenbroe et al., 2002) cites Gray (Gray, 1989) when defining collaboration in his paper:
…finding common ground for multiparty problems which relates to collaboration as a process through which parties who see different aspects of a problem can constructively explore their differences and search for solutions that go beyond their own limited vision of what is possible.

Mutual understanding embraces communication and collaboration. Thesaurus (Thesaurus, 2008) resumes mutual understanding as:

26

Noun 1. mutual understanding – sympathy of each person for the other mutual affection sympathy – a relation of affinity or harmony between people; whatever affects one correspondingly affects the other; "the two of them were in close sympathy"

Merrian-Webster (Merrian-Webster, 2008) has the following translation:
1: a mental grasp : comprehension 2 a: the power of comprehending; especially : the capacity to apprehend general relations of particulars b: the power to make experience intelligible by applying concepts and categories 3 a: friendly or harmonious relationship b: an agreement of opinion or feeling : adjustment of differences c: a mutual agreement not formally entered into but in some degree binding on each side 4: explanation, interpretation 5: sympathy 3a

The citations above have shown a strong link between these three key worlds. With good communication it is possible to achieve collaboration between two or more people. It is the element that must be present if it is aimed collaboration (Godfried Augenbroe et al., 2002). Collaboration just happens with mutual understanding. Parts/actors in CEn environment (as well as in any other relations between beings) must to find a balance to enhance these three key worlds in order to achieve the goals of the project.

1.2

Potential Gaps during the CEn Phases – Communication, Collaboration

and Mutual Understanding The challenges exemplified by Kamara (Kamara J. M. et al., 2000a) are a clear example of the complexity to attend the requirements of a client when ordering a new CEn project. Kamara’s paper describes the development of methodologies to enhance the understanding of client requirements (Kamara J. M. et al., 2000c). To achieve full comprehension of what the client was requesting the information should facilitate the concurrent work of the number of disciplines involved in the project; the early

27

consideration of all life-cycle issues; the integration of all professional; and the traceability of the design decisions. The paper highlights:
In the context of the multidisciplinary teams, this implies that the presentation of the requirements has to be neutral enough to allow different professionals to understand them in the same way (i.e. in the perspectives and priorities of the client).

The communication across disciplines regarding design constraints and trade-offs, where it is also possible to translate as different group of people with different expertises, is mentioned on Fiksel and Haynes (Kamara J. M. et al., 2000b, Fiksel and Hayes-Roth, 1993) cited in Kamara. As mentioned in the beginning of this chapter Ugwu (Ugwu O. O. et al., 2001) does reference to “ontology” for the difficulties faced by members of different groups to work together. For him, a tool that can enable to enhance collaboration would help to reduce the problem:
One potential solution to this communication problem is the development of an ontology of collaborative design concepts and terms along with their unambiguous definitions. The existence of such an ontology in a problem domain would enhance the ability of the participating agents (both human and artificial) to inter-operate without misunderstanding, and consequently share or re-use data, information and knowledge.

The study of ontology is very wide and it is not limited to the issues in communication and interoperability between beings. An extended review about the subject can be found in (Uschold and Gruninger, 1996). The difficult faced by those involved in a CEn project is also highlighted by Mokhtar (Mokhtar, 2002). Although construction teams work over what is called the plans of constructions errors still being made. The lack of coordination of design information leads to an incompatibility between the plans. Mokhtar completes his view about the issue with:
The errors lead to change orders, contractual disputes, cost overruns, time delays, compromise to quality, frustration and client dissatisfaction. Common types of errors are: o Inconsistency in design information. For example, the location of a specific column is not identical when comparing the architectural and the structural drawings.

28

o

Mismatch between connected components. For example, Heating, Ventilating and Air Conditioning (HVAC) ducts dimensions, which are given in the mechanical drawings, do not match the dimensions of the related pass-holes in the structural beams, which are given in the structural drawings. o Components malfunction. For example, electric supply in a room is designed to serve a classroom activity while architectural drawings indicate that the same room has been re-designed as a computer lab.

Tiley (Tilley and Barton, 1997) cited by Mokhtar complements the view of the problem with the following citation:
Unfortunately, such errors have become very common in the construction industry, especially with escalating complexity in buildings, the increasing specialization in design teams, and the tightening of financial and time resources available to designers.

The increased demand for collaboration especially when construction becomes more and more complex is also covered by Heintz (Heintz, 2002):
When it comes to collaboration, effective planning and conduct of the project occurs informally and not through either formal planning procedures or contractually specified relations and products. It would therefore seem essential, in a time when the design of buildings is an increasingly collaborative effort, to find planning methods that can support collaboration.

Although the subject was not covered to exhaustion the citations above give a solid foundation for the motivation of this thesis: there are lacks of communication, collaboration and mutual understanding between those involved in a CEn project. Clearly it will depend of the size and complexity of the project. However, such gaps can arise at critical moment during the CEn life-cycle and often when in its very beginning – the planning phase.

1.3

Civil Engineering Life Cycle

The tool created and evaluated by the present thesis covers few of phases of what is call CEn life cycle. The life cycle of a CEn project is not resumed to the construction site. Neil (Neil S. Grigg et al., 2001) demonstrates the number of steps related to CEn cycle (Figure 1.1) and their interrelationships.

29

Figure 1.1 CEn Life-Cycle and Interrelationships (Neil S. Grigg et al., 2001)

Considering that the process starts in “planning” and runs clockwise until “replaced/decommission” there are eight major steps for the CEn cycle. Those steps are not restricted and can vary to different authors. For each of the phases there are sub-phases that can be explored. Phases can move backwards (from preliminary design to repair/rehabilitate). At the centre Figure 1.1 shows the society. Those are the ones involved directly or indirectly by the CEn project. The following bullet points were extracted from Neil (Neil S. Grigg et al., 2001) work and they are a briefly review of the phases showed in Figure 1.0 • Planning – This is the very early stage of a CEn project. This phase is assigned by the discussions and exchanges of concepts take place in the realm of planning before the formal process begins. It is where the conceptual design relies, it is also where the needs of the project are identified, and the first set of options is developed. The planning phase breaks up to: o Reconnaissance – identifies projects that meet the goals established in the overall planning and development process. o Feasibility – financial, technological, environmental and political. o Defined project plans – plans, specifications and operating agreements.

30

•

Design – The preliminary design can lead the project to a different direction considering the outcome of the investigations for the feasibility of the project. This phase uses a systematic methodology for analyzing options for the project selecting the one that best fits the requirements (Kamara J. M. et al., 2000c). The option selected should even have public involvement. The final design phase extends the analysis done in the preliminary design, like some of the calculations, to a more detailed level. This sub phase includes the “specification of the material, dimensions of the projects components, and details of member connections”(Neil S. Grigg et al., 2001). Finally the design review sub phase confirms that the design in optimal for the cost.

•

Construction – This phase starts with the “contractor planning all aspects of work for finances, resources, permissions, environmental awareness, safety, etc”(Askew, 2007). As the name suggests, it is on this phase that the project becomes a reality in a construction site. The project management is performed by a contracted company, an individual, or the same company hired to develop the project. that manages the project in name of the client (Wilkinson, 2001). Construction should reach the final goal as much close of the design phase as possible including the budget. Any change in the construction will force in a change or at least in a design review. “All construction changes must be evaluated to ensure that the design integrity of the project is not compromised”(Neil S. Grigg et al., 2001). This phase also host the processes of quality control and quality assurance leading to the final inspection and acceptance activity (Neil S. Grigg et al., 2001). Another important aspect of this phase is the project management. Although this position can offer a valuable help to make a CEn project to achieve its final goal managers can face very strong opposition as presented in (Wilkinson, 2001). This phase is finished with the approved completion of the construction contract – commissioning (Askew, 2007).

•

Operation – The facility is monitored by one or more engineers. It is their duty to make operational decisions like quality control for example. Greene (Greene, 1984) cited by Neil spread operation management in the following disciplines: production management, facilities management, maintenance management, and information management.

31

•

Maintenance – As mentioned in (Neil S. Grigg et al., 2001) “preventive maintenance heads off problems and is the most common maintenance operation in organisations. Corrective maintenance has minor and major aspects, depending of the extent of the correction needed. Corrective maintenance involves repair, replacement, and rehabilitation of facilities”.

•

Rehabilitation/Replacement – CEn projects should be already designed to support rehabilitation/replacement during its live-cycle. This has as key issue the impact created when a major infrastructure must to pass through a rehabilitation/replacement process (Neil S. Grigg et al., 2001). Hartshorn (Hartshorn et al., 2005) gives successful examples of sustainability in CEn projects.

•

Demolition/Decommissioning – Neil (Neil S. Grigg et al., 2001) signs with an example of the importance of demolition/decommissioning: “consider the engineering challenges in decommissioning a nuclear power plant or a hazardous waste processing facility”. Again it is also part of the design process to prepare the CEn project for its very last step in the life-cycle.

1.4

Thesis Objectives

The objective of this thesis is to investigate, create and evaluate a Collaborative Augmented Reality System (CARS) able to explore communication, collaboration and mutual understanding on CEn environments. The CARS created can be easily adapted for any of the phases of a CEn life-cycle. However, this prototype had its focus directed to design, construction (coordination and supervision sub-phases) phases. It can also be used for both people with technical skills (engineers, architects, contractors, etc.) as well as any other person that is being affected by a CEn project (society in general terms). The selection of only two between eight phases on CEn life-cycle (Figure 1.1) is not a singular opinion. Shin and Dunston (Shin and Dunston, 2008) describes in their paper an wide research about where AR projects best fits on CEn environments.

32

Although they sub categorise CEn life-cycle they could identify that: layout, excavation, positioning, inspection, coordination, supervision, commenting and strategising are the best matches for applications based on AR in CEn projects.

1.5

Proposed Solution

Once it was possible to identify that there are potential gaps of communication, collaboration and mutual understanding along of different phases of a CEn life-cycle this thesis proposes a solution to reduce them through the development of a CARS. The diagram on Figure 1.2 is a wide view of the CARS that was developed by this thesis. In the left side of the diagram it is introduced the indoors engineers. They are equipped with a computer where they can explore a Virtual Environment (VE) with the help of a standard mouse. Their work is to assist those that are exploring the VE outdoors. The VE explored shows parts of a CEn project that are relevant for the moment. They can navigate freely in this VE, walking though it, fly over it and even move the objects. Indoor engineers can communicate using the audio link or through the movements of their representation inside of the VE – an avatar. They can see where outdoors engineers are in the VE also through their avatars. Engineers when indoors do not need to stay closed to the CEn site. They can be virtually anywhere around the globe as much as there are Internet connection available.

33

Figure 1.2 General View of the Project

Outdoor engineers are introduced in the right side of the diagram (see Figure 1.2). As mentioned before it could be any person with any professional qualifications 1. The outdoor engineers must carry all the equipment with them once they are mobile in the Real Environment (RE). In a backpack they have a notebook with wave-Local Area Network (LAN) enable. In this computer they have the same VE running as indoor engineers. An external set of equipment is connected to the notebook: high accuracy Global Positioning System (GPS) receiver, an inertial sensor, and a see-through Head Mounted Display (HMD), etc. They can communicate with other users through the audio link or through a limit number of body movements. Outdoor engineers can explore in the RE the VE displayed before their eyes thanks to a see-through HMD. The images of the VE are rendered in real time following the body movements (head and body) of the users. They can, for example, walk inside of a house at same time that they talk and share points of view with other users of the system. The system processes and transmits any action inside of the VE (for both indoors and outdoors engineers) to other participants when sharing the same VE. A waveLAN router is installed nearby the area where the outdoors engineers are. The data takes whatever way of transmission available (wireless or not) from the router to indoors engineers and vice versa.

1.6

Steps Followed to Achieve the Objectives

Several steps were performed to achieve the final objective of this project, the investigation, creation and evaluation of CARS. The investigation process starts with the identification of the state of art tools created to enhance communication, collaboration and mutual understanding in CEn projects.
1

The “engineer” word was chosen for the matter of clarification and to exemplify the use of the tool.

34

The next step is the investigation of the resources available to develop the project itself. This phase gives preliminary directions of how far the development of the project can achieve. The understanding of technologies like high accuracy GPS is also a requirement for this project. Questions like: • • • • • Why this technology is the best fit for the project? How does this technology work? Why there are differences between methodologies? How to set up the equipment (reference point and rover)? What kind of data can be extracted from the GPS receiver?

There are similar questions for the inertial sensor too. The mapping of the areas where several preliminary experiments of the project were performed is also part of the steps towards the final objectives. For the second half of the research the steps are: • • • • • • • Initial implementation of the system (integration of hardware and software). Testing of the system in single user mode. Further implementation of the system exploring more a collaborative environment. Testing of the system in collaborative mode. Final integration of the components (hardware and software). Final test of system behaviour. Evaluation of the project with users with different expertises.

1.7

Thesis Structure

The structure of the thesis follows the diagram presented on Figure 1.3. 35

Figure 1.3 Thesis Structure

Chapter 2 – Review of CEn Issues Some of the challenges over CEn environment are introduced in Chapter 2. The research looks for potential gaps on communication, collaboration and mutual understanding when actors from Architecture, Engineering and Construction (AEC) are together working in a project. The idea of collaboration over AEC is not new but just now information technology (hardware and software) is becoming good enough to attend the requirements for such complex integration/merging of disciplines. There are already in the market solutions provided by companies trying to create a culture and a standard of what format and platform to use. However this is a not a finished issue and there are ongoing researches looking for what AEC professionals really want to know/have in hands as information at the right time. The way that information is being retrieving from previous construction scenarios, how data can be filtered for a specific user, how to exchange formats are just few examples. It is a fact that technology helped to create a dynamic environment over AEC but it is also a fact that if such information does not became easily and broadly available people will still keep it as “owners of knowledge”.

36

In any case, the solutions so far suggested by companies and researchers are already proving that the investments made buy AEC societies worth. There are reductions of costs and time, reductions of injury workers and even a reduction of time to document a project between many other advantages. Chapter 2 shows the advances in technology for AEC but also illustrate that information can be better explored (using visual sense). Such gap is a point for the proposed thesis. The data can be available but the right output for some of the cases can be better explored. Depending of the application an AR system can make the difference for builders, managers, engineers, architects, customers and the society affected by a new construction. Chapter 3 – Review of MR and GPS The concepts of Mixed Reality (MR) are introduced in the beginning of Chapter 3. This chapter reviews not only all the concepts involved on Milgram and Kishino (Milgram and Kishino, 1994) diagram but also gives special attention to collaborative environments. The reason for it is the creation of a base of what this project is about – a collaborative environment created over the communication and mutual understanding between users the users. Chapter 3 reviews projects with some relation of what is being proposed here. The survey covers projects with the following aspects: • The way that information is being explored and shared by users – the survey reviews projects where users are working in a stand alone mode or in collaboration with other users. • The way that information is being display to users – users can visualise information in computer’s screen, large screens with stereoscopic images, handheld’s screen and see-through HMD. • • Where information is available for users – users can explore information in stable environments like an office, in a theatre, or outdoors. The position system used to explore information – users can explore information in a click of mouse devices or using complex indoor tracking

37

systems. The research also survey outdoors AR systems with or without GPS to track users in the field. The chapter covers a review over GPS as well. This positioning system is the one choose for the kind of application that is being proposed in this thesis. The text goes over basic knowledge about how the system works and why sometimes it does not provide the expected accuracy. Three main methodologies of GPS are explained – single frequency GPS, Differential GPS (DGPS) and Real Time Kinematic GPS RTK-GPS. It is also parte of this chapter a demonstration of how different results can be when using different methodologies to survey a coordinate in the field. Such results are useful to demonstrate why RTK-GPS is the technology used to track outdoor users. Chapter 3 finishes with a review of augmented systems created to improve GPS signal or to overtake the necessity of a second receiver as base station. They are also called augmentation systems and they are becoming more popular with wide areas being covered by the availability of the service. The advantage is the reduction of the costs for high accuracy GPS and a system producing corrections twenty four hours day seven days a week. Chapter 4 – Design Overview Chapter 4 introduces how the system is implemented. The chapter starts with an overview about its design, who the expected users for the system are and how the system is design for interaction. The overview shows the complexity of how the system works and what is necessary to make it run. The introduction of the design requires a close look on Chapter 5. Several process mentioned in Chapter 4 are fully explained in the next chapter. Diagrams help to understand the connections between processes and the order that they run when a key is stroke in the keyboard or when a new GPS coordinate arrives in the Communication Port (COM1) for example. The hardware is then reviewed together with the messages that are expected from their input – like RTK-GPS receiver, magnetic compass and tilt sensor. The hardware integration together with the platform used to manage the VE completes this chapter – Equip and MASSIVE-3. 38

Chapter 5 – First Phase of Implementation In the practical sense during this part of thesis some knowledge about how to survey parts of RE and convert it to a VE becomes a requirement. The survey of an area involves techniques about setting up receivers and tripods, collecting points, posprocessing, interpolation and conversion of coordinates to a common format. The next step is the design of the scene based on the coordinates of terrain and the features mapped during the survey. Such procedures are detailed in the beginning of Chapter 5 and the results are demonstrated together with the first tests using the VE. Another key point of Chapter 5 is the description of the process involved when the system is operational. Such documentation is important to make future users aware about how the technology works and what is necessary to do to make it run. Parameters for the processes are described and complement the information started on Chapter 4. Collaboration starts to work when two users share the same VE over a network. This is the first step towards the final objective – to have an outdoor user exploring a VE assisted by an indoor user enhancing the three keywords. At the end of the chapter it is described the results achieved with the implementation of the Kalman filter for the magnetic compass and tilt. The graphs indicate a reduction of jerking caused by the disturbance of magnetic field around the sensors. This reduction improves the comfort for the user when exploring the VE with the see-through HMD. Chapter 6 – Second Phase of Implementation and Final Evaluation The second phase of the implementation is marked by the combination of indoor and outdoor users sharing the same Collaborative Virtual Environment (CVE). The system is tested for several times until become reliable for a final evaluation. Three exercises are also created for the final evaluation. Two of the VE created are based on real constructions and the third one is created to demonstrate some extra potentialities of the system.

39

Before the final evaluation a pre-test is performed in order to learn if there are any final adjustments to be done. The results show that outdoors users cannot perform complex tasks using the handheld keyboard. All interaction over the VE must be performed by the indoor user. This final requirement makes the system easier to be explored by outdoors users. Users now have their hands free and can put more attention in exploring the AR system than to control it. In total eight people are invited for the final evaluation. It is used a formative approach to extract the results from the tests. As mentioned in (Vicente, 1999) “formative approaches focus on the way things could be by identifying novel possibilities for productive work”. Users are chosen by their knowledge about subjects like: Information Technology, CEn, MR, new technologies, etc. Each user has his test recorded by a cameraman, together with notes taken by the author (performing the indoor user), and an interview is also recorded in video. Both tests and interviews do not follow a strictly format. However they are run mostly in the same way with users doing the similar tasks during the exercises and answering similar questions along of the interviews. The evaluation process shows that the composition of the VE is a very important issue. Users become more responsive regarding to “what they are seeing” than “how they are seeing”. There is a lack of quality in the see-through HMD that cannot be avoided. The device is not designed for outdoors AR tasks. The evaluation highlights that the system proposed is well accepted when used in the right application. Chapter 7 – Conclusion and Future Works A final conclusion is provided on Chapter 7. This overview highlights the key points of the thesis as well as the goals achieved. The points that could not be achieved are reviewed as well. Together with them there are the expectations if they were completed and the implications for the thesis for not have them complete. Along the development of the thesis several new ideas started to be elaborated as a complement of the work. Due to the time frame these ideas are mentioned in future works. These ideas are perhaps just a starter point for even more elaborate ideas that

40

can become new researches in a near future. The future works also demonstrate the potentiality of the tool here suggested and the way that it can be adapted. Appendixes The appendixes complement the research produced by this thesis. They are created to clear the reading of the text as well as keep more technical information concentrated in a single point. Appendix A reviews the program created to convert the surveyed coordinates and features of the RE in a Virtual Reality Modelling Language (VRML) (W3C, 1995) format. It shows the parameters and a high level algorithm based on C language (Huss, 1997). Appendix B is an implementation reference for all other Java classes (Schildt, 2001) created to link external devices to the controller of incoming data to the data base. There is also an explanation and a diagram of how Kalman filter (Welch and Bishop, 2001) is implemented. Appendix C is the transcription of the interviews of all participants during the evaluation process. Chapter 6 – Evaluation – contains exceptions of Appendix C. Compact Disk This thesis is followed by a compact disk that resumes the research. It is recommended the download of the video to a hard drive for better visualisation.

41

Chapter 2
Civil Engineering Challenges
This thesis argues that collaboration, communication and mutual understanding are vital to better handle complex information in CEn projects and there are issues where these key worlds are not fulfilled. Regarding to the project proposed in this thesis the users do not suppose to explore the VE alone. They are expected to share their views, to ask questions and to talk about what they have understood about the CEn plans. From this interaction between users, new conclusions can be reach, adding value to the information that is being explored. The concept of an “augmented architect” is not new. However, it was just during the past few years that technology has improved sufficiently for the creation of the first prototypes. Engelbart cited by Brate (Engelbart, 1962) introduced views about collaboration in CEn or architecture using VE:
In 60’s Douglas Engelbart imagined the “augmented architect” situated at a workstation with a three-foot display screen. The user would communicate with the machine by means of a keyboard or similar device. The computer would be a “clerk”, an assistant to the human problem solver. The clerk would compute the math and display information, both textually and graphically. Several people could work together to model, for example, the interior of a building. Each architect would be presented with the same information and collaborate in the same environment. The computer would manipulate and display information in real time, and people would have an opportunity to simulate

42

various solutions to a problem. In this way, the technology fosters intuition and creativity as well as logic and reason. This, Engelbart said, is human augmentation.

The possibility to use computers and electronic devices as tools that support the manipulation of huge volumes of data in a matter of seconds is nowadays feasible. Engelbart mentioned the augmented architect as a clear example that several people should work together in CEn projects. This means collaboration. The human augmentation is the combination of human beings and computers. Computers are able to process several megabytes of data allowing human beings to go even further to collaborate with each other and to resolve any unclear points faster. In the Augmenting Human Intellect manifesto mentioned in (Engelbart, 1962), the collaborative environment is exemplified as:
Three people working together in this augmented mode seem to be more than three times as effective in solving a complex problem as is one augmented person working alone – and perhaps ten times as effective as three similar men working together without this computer-based augmentation.

The challenges however remain and they are the motivation for this work as mentioned in the previous chapter. It is not always that information flows smoothly to all parts of a CEn project. Haymaker (Haymaker et al., 2004) presents some complementary challenges in a modern CEn project: • Multidisciplinary – engineers and architects from different companies must to work together. Therefore companies have different priorities and projects and there are always clash of interest representing different engineering criteria. • • • • Constructive – a creation is usually over the views of other creators. The information should flows towards one direction only. Iterative – during the project modifications in the project could happen. Such modification must to pass through others engineers and architects. Unique – projects are unique in several aspects. Error-prone, time-consuming and difficult – “manually constructing and integrating dependent views from source views causes many problems on AEC projects today”.

43

This chapter introduces what researchers and companies are creating to mitigate such issues.

2.1

Solving CEn Challenges – Design and Construction Phases

The following subsections were created to explore the current state of art on CEn. The reviews are focused on design and construction phases (where this thesis is centred) and where the possibilities are for further research. The projects that are introduced bellow come to answer some of the difficulties introduced by Kathleen (Liston et al., 2001). Actors in a CEn project faced major difficulties to deal with information when using conventional 2D plans. They need to flip over plans, schedules, contract documents, etc., to find answers as quick and precise as possible. If the answer is not there, actors need to leave the meeting rooms to bring an answer. The environment becomes unproductive with actors spending more time trying to understand their own information than giving proper answers, than sharing their views about a subject. 2.1.1 Building Information Model

Salazar (Salazar et al., 2006) mentions in his paper that the disparities between AEC are being reduced since tools pursing collaboration and coordination started to become a reality in CEn. Also know as Building Information Model (BIM), these tools help AEC teams to deliver CEn projects with “higher quality in shorter times and at lower costs”. BIM models are very extensive and cover several areas of knowledge/information like: geometry, spatial relationships, geographic information, quantities and properties of building components (Public Domain / Wikipedia, 2008). Part of the BIM framework is a database to store all these information. Once such information is digitalised and shared by AEC any handover of a project has a greatly reduction of information loss. This is specially significant when exchanging data between different software (White, 2006).

44

BIM can greatly decrease errors made by design team members and the construction team (Contractors and Subcontractors) by allowing the use of conflict detection where the computer actually informs team members about parts of the building in conflict or clashing, and through detailed computer visualization of each part in relation to the total building. As computers and software become more capable of handling more building information, this will become even more pronounced than it is in current design and construction projects. This error reduction is a great part of cost savings realized by all members of a project. Reduction in time required to complete construction directly contributes to the cost savings numbers as well (Public Domain / Wikipedia, 2008).

ArchiCAD from Graphisoft (Graphisoft R&D Zrt., 2007) is an example of tool with BIM proprieties. The company claims that the software can make users do better: focus on design; manage change; evaluate design alternative; collaborate; coordinate. It was created to give more “control over design, while maintaining accuracy and efficiency in documentation”. A central database stores and shares all information available for the project reducing the risks of replications and working over the same issue again. Any change is automatically spread in the database keeping the integrity of the project. The software also allows 3D interactive design sessions with customers: edit, navigate and interact with the session under design. Another example of BIM came from AEC Design Group (AEC Design Group, 2008) with ArTrA BIM. Their website informs that the models created can be handled by the design team, to contractors and subcontractors, and them to the owner of the project. AEC Design Group claims that the main advantage is the freedom for the users to choose their own Computer Aided Design (CAD), saving money and time of training. Companies like Autodesk (Autodesk Inc., 2008a) are also aware of the potentiality of BIM concept and are guiding its products towards this direction. On their website there are several examples of how BIM can improve the productivity of companies, the management of projects (small and big ones) reducing time to produce paperwork. Product like Revit allows designers to interact with customers in real time, changing plans, showing the modifications in a 3D environment. Bentley (Bentley Systems Incorporated, 2008) puts the challenge for BIM as:

45

Unite the project lifecycle, bringing together design and documentation information and workflows for the design, construction, and operation of all types of buildings and facilities around the world, from the conventional to the most inspiring projects of our time.

Their products are aiming to bring together the life-cycle of CEn projects considering (design/building/operation) with information about the building and its life-cycle. “The definition and simulation of the construction, its delivery, and operation using integrated tools”. The study of the advantages achieved in a CEn project (in this case Mechanical, Electrical and Plumbing – MEP) using BIM can be found in (Khanzode et al., 2008). In this particular study, they evaluate the construction of a medical office building with an initial contract of $96.9 millions USA dollars, 250,000 square foot, three level building, and a car park. All these constructed in 27 months – from January 2005 to April 2007. Khanzode highlights the first impressions of CEn projects difficulties:
Lack of ability to identify conflicts due to the 2D representation of the designs Delays in construction process due to conflict being identified in the field Lack of trust in the fabrication offsite due to the fear of system not fitting leading to a lot of on the site fabrication o Rework to fix the conflict issues not identified during design and coordination o Increased site supervision required to avoid conflicts between trade contractors o Increased administrative burden of more request for information and change orders due to identification of conflicts in the field after budgets are approved o “Install first” mentality amongst trade contractors so as to avoid having to move their systems in case conflicts arise leading o Overall reduced productivity for everyone involved in the process o o o

The team leaders decided to go for a BIM and Virtual Design Construction (VDC) – specifically 3D/4D and automated clash detection tool – even though this being their first experience with these technologies. A the end of the project the team members created a guideline which forms – in their point of view – the best practices for doing MEP coordination using BIM/VDC tools: o Clarifying the role of the general contractor and specialty contractors in the
coordination process.

46

o o o o

Increased site supervision required to avoid conflicts between trade contractors. Developing the levels of detail in the architectural, structural, and MEP models. Setting up the coordination process – with different sub-levels. Managing the coordination of the installation process.

The paper describes how teams worked in small groups, organising themselves, orchestrating their tasks over the construction.
The project team should collaboratively determine how the design will be broken down into small enough batch sizes that allow detailers to coordinate and complete an area so that fabrication can begin. This is an iterative process between the design and construction teams.

Between the many benefits of the use of BIM/VDC tools that are reported in the papers it is highlighted: • • • • • • • The drastic reduction of time spent on request of information or change of orders due to field conflicts – reliving architects and engineers. There was no change of orders related to field conflicts. Information about the building is easier to recovery. Savings of $6 millions USA dollars and 6 months. There was a drastic reduction on time spend by general contractor – from typical 2 to 3 hours day to 10 to 15 hours in 8 months. The number of accidents on site was also well reduced. The building parts were pre-fabricated off site. It reduces the risk of injuries (lower number of machinery) and makes the site cleaner. The delivery method was “just in time” reducing the amount of stocked material too. • • All trades have finished their work ahead of schedule. “The subs could use lower-skilled labour for the field work compared to other projects where higher-skilled field labour is necessary for installation as the labour force typically needs to interpret 2D drawings, etc”. • The 3D design should address not only the structure and the displacement of the rooms of the building but also the furniture (especially in a medical centre). • Sub contractors spend around 2 days per week with architects and engineers to issues being addressed. This is more than what they predicted initially. 47

Figure 2.1 (Khanzode et al., 2008) shows the savings of time and money along the CEn project. The difference between the estimation (red line) using standard methods of design and building and the final achievement (green line) is significant thanks to the tools (BIM/VDC) used to help to manage the project and the collaboration between teams. The image (Figure 2.1) is a courtesy of DPR construction, Inc., CA, USA and extracted from (Khanzode et al., 2008).

Figure 2.4 Reduction of Costs and Time in the Project Evaluated on (Khanzode et al., 2008)

There are several other companies producing BIM products in the market. Some of the examples are but not limited to: (Building Integration Software Company Ltd., 2008, BulidingExplorer.com, 2007, Tekla, 2008, AceCad, 2004, Nemetschek North America, 2008). More information about BIM can also be found in (White, 2006, Maldovan, 2007).

2.1.2

Researches addressing Communication, Collaboration and Mutual

Understanding in CEn Projects The projects presented in this sub-section are part of researches aiming to reduce the problems on communication, collaboration and mutual understanding in CEn projects. They are more focus in one point than others making them not a “total solution” as claimed in some commercial products. 48

Kamara (Kamara J. M. et al., 2000a) describes the difficulties in establishing and processing client requirements on concurrent CEn. Kamara quoting Winner (Winner et al., 1998) explains that the concurrent approaches considers all the elements of the product life-cycle including quality, costs schedule and customer requirements. The paper is focused in the development of methodologies to enhance the understanding of client requirements (Kamara J. M. et al., 2000c). With this methodology client’s requirements will be better comprehended by multidisciplinary teams to work collaboratively. The requirements are processed and prioritised at the time that they are also cross correlated against several other requirements that normally arise in a CEn project like: site requirements, environment requirements, regulatory requirements, design requirements, and construction requirements. The output of the system “serves as the basis for adopting a particular procurement/contract strategy, such as design and build. This approach focuses on clarifying the ‘problem’ using methods other than design, as is usually the case in current briefing practice”. Ugwu (Ugwu O. O. et al., 2001) work describes that “multiple-agent systems paradigm allows distributed organizations and domain experts to be modelled as sets of collaborating (but not necessarily co-operative) agents that interact in a problemsolving space to achieve set goals”. The goal mentioned by Ugwu is:
…often the agreement on a product configuration that is mutually satisfactory or satisfying to the design team. The design team includes: Architect, Structural Engineer, Steel Fabricator, Building Services Engineer, Quantity Surveyor, Health and Safety Personnel, and other specialist trades that are required in a given project.

The paper describes the early stages of a website creation – or a portal to facilitate a collaborative design – helping to spread knowledge between the multiple-agents. Data and information processing improves the decision of the agents in a collaborative design area. An important point in the paper is about the understanding of the concepts of collaboration and ontology as well as the functionality of the system by the participants. Even in an environment like this agents/participants can act like owners of the knowledge not using the potentiality of the collaboration between parts. The challenges in these systems are in translating what experts know as owner of knowledge, capture, translate, and represent it using digital forms. The

49

complexity to create systems like this is part of the research presented by AmmarKhodja (Ammar-Khodja et al., 2008). In this paper the research has focus on make actors from different parts work collaboratively aiming knowledge be transferable between actors using an automated system. The paper reviews the concepts of Knowledge Base System (KBS) and its complexity to be used it. More refined products systems like the Knowledge Base Engineering complements what can be produced by CAD systems, automating design activities and creating a structure of knowledge for the product. Tuzmen (Tuzmen, 2002) presents a conceptual model of a distribute process management environment in the establishment of a collaborative design environment. The tool supports remote access where users can filter information to see only what is relevant for their work – like the activities that they need perform for example. Users can create a new process definition or reuse an existing one modifying it when necessary. The process definitions are composed by activities and the relationship between each other and are linked to each phase of CEn life-cycle. A process enacting service interprets the execution of the tasks completed identifying the next one. Users are advised by the system each new update. Similar project can also be found in (Mokhtar, 2002, Augenbroe et al., 2002, Heintz, 2002). Sarshar (Sarshar et al., 2002) describes seven themes that projects like the ones mentioned in this sub section that should address: • Model driven as opposed to document driven information management on projects – information is not centralised/unique to everyone but shaped for the requirements of each case. • Life cycle thinking and seamless transition of information and processes between life cycle phases – communication and knowledge should be shared between phases of CEn life-cycle. • Use of past knowledge (information) in new developments – reuse of past knowledge in present and future projects.

50

• •

Dramatic changes in procurement philosophies, as a result of the Internet – new business opportunities should arise with the advance of technology. Improved communications in all life cycle phases, through visualization – taking advantage of visualisation to improve communication between parts making it more effective and user friendly.

•

Increased opportunities for simulation and what if analysis – simulation can improve the understanding that helps to predict feasibility, planning and scheduling.

•

Increased capabilities for change management and process improvement – it is not only the technology that should improve. As mentioned before the structure of the organisation must also understand the concepts involved, what technology has to offer and restructured itself for this new era.

Another approach to integrate AEC project models is presented by Haymaker (Haymaker et al., 2004). This research “investigates the conceptual simplicity of formalizing a project model as a directed acyclic graph of views and their dependencies, allowing engineers to manually manage the cycles”. Their work introduces an approach to “formalise methods to construct views for other views and control the integration of a directed acyclic graph of views and dependencies as the project progresses”. The Perspective tool logs information like: “sources, status, and nature of its dependencies” of a CEn projects. Engineers can compose and reuse information from past projects creating then new “perspectors”. The tool automates or assists engineers to add more information analysing if all part of the project are integrated and making sure that all requirements of the project were satisfied. The prototype created allows users to see and interact with perspectors and analyse the geometry of the construction in 3D using VRML. Shelbourn (Shelbourn et al., 2006) claims that the challenges do not relies “only” on “capturing, storing, sharing, and re-using” information. The paper describes the “creation of tools that enable the development of organisational practices in the construction sector to promote knowledge creation, prior to sharing and re-use, along with the tools to support such a process”. They work over the knowledge

51

management and rise reasons for the limitation on creation and dissemination of the knowledge between actors in CEn projects. Some of the reasons are: • • • Knowledge residing in the head of single individuals. People can move to Decisions are not often recorded in documents. Information is only recorded at the end of the project instead of and not

different projects or change some of the ideas.

during the events. To improve these practices the paper addresses the C-SanD project (University of Loughborough et al., 2001). C-Sand is driven by the possibility of sustainability of CEn in favour of environment. Creating and sharing knowledge is one of the possibilities for the project to achieve its goals. To purse this objective they report the creation of a website portal called SMAZ. The results after evaluation demonstrate that although the research would be useful for actors in CEn project they need to improve considerably the design of the portal user interface. As also a result they report that groupwork functions are apparently better accepted than information retrieval functions. Kubicki (Kubicki et al., 2006) investigates the cooperative work between heterogeneous actors in a CEn project. The paper includes to the discussion the dynamic of a CEn project especially in the construction phase and the difficulty to create rules when new actors (contractors, security, etc.), objects (material, equipments, tools), and documents are in constant change. Eventually requirements and coordination becomes informal and eventually not registered:
Coordination activities have to determine elementary construction tasks and their time sequence. Planning has to take into account resources (human and material) and technical constraints (Kubicki et al., 2006).

Kubicki quotes Brousseau (Brousseau and Rallet, 1995) on the challenges faced on the codification of actors activities in CEn projects:
The random nature of the activity, the singular requirements of CEn projects and the participation of actors from different sources make difficult a precise codification of rules of interactions. In addition, the effort to represent procedures in a codified way is not

52

justified because of the transient nature of relations. As a consequence, coordination procedures implemented in construction are relatively few formalized. This does not mean that written contracts do not exist, or that the regulations do not define certain coordination procedures. This means that any texts are subject to greater margin of interpretation in the industry (Brousseau and Rallet, 1995).

The paper (Brousseau and Rallet, 1995) finishes reporting the achievements of the designing of a assistance tool for cooperative engineering. A first prototype called “Image.Chantier” is an assistance tool created to coordinate the diffusion of meeting documents between actors. The tool can spread “personalised access to information”, meaning that actors receive only the relevant information for their work. The tool is based on a web browser view with four split windows: Gantt planning view, 3D mock-up, text, and graph. The windows are dedicated to different users like architects (with focus on 3D visualisation) and coordinators (with focus of documentation/organisation). Kathleen (Liston et al., 2001) also argues that actors should concern only to what is really important to share as information. Such sharing should happen only when time requires and tools should be more specific instead of broad designed to all cases. Researchers evaluated the difficulties for actors to share information when using conventional paper-based meeting. Actors spend more time trying to find the information than explaining it to other actors. The situation improves considerably when the meetings are hosted in a Computer Automatic Virtual Environment (CAVE). Information becomes more interactive and easy to find. A large and shared visualisation between AEC helps to keep them with focus in the meeting also improving the interaction between them. The disadvantage is the limitation of essential project information available that can occur. The final evaluation was on meetings with product model support. This technology purses the compatibility of data between different packs of software used by actors making data more consistent and reliable (updated). The major disadvantage is:
Pre-defined semantic models are unlikely to support all the information needs by multidisciplinary project teams because each project needs to address a unique set of requirements, which require the generation of project-specific information views to address project-specific decisions. They (the models) do not yet scale to the multidisciplinary, multi-phase, multi-objective, and multiple level of detail nature of projects.

53

The paper describes a project called iRoom. This interactive workspace is shaped by interactive touchscreens, notebooks, projectors, laser pointers, etc. The meeting room accepts many ways at the hardware level to input and output information. The software controls the information making it easier to share between meeting participants. Information is automatically updated and replicated when users select an item in one of the many screens. The results suggest that users are more focus in relevant information than looking for answers in their on documentation. Complex building designs and the challenges of use 3D tools between actors from different design teams is studied by Ku (Ku et al., 2008). The paper describes 3 cases of studies where companies had to share information between them to implement a CEn project. Usually companies do not have the same set of software making sometimes files incompatibles and improving the chances of misunderstanding. Problems start to increase when the complexity of the structure in construction can not be clearly represented by 2D graphs (some companies still using only 2D graphs for representation) giving margin to ambiguities on the interpretation. The lack of a common channel of communication between the platforms used by constructors and designers can also introduce a lack of coordination on changes to the project. Ku also highlight the liability issues when companies share the same project. Such barrier can force one company to redesign a model increasing the chances of mistakes and misunderstanding due to transcription errors. Scheer (Scheer et al., 2007) describes the results of a found when several professionals involved in a design team worked collaboratively in a system called “System for Project Management” developed at the Federal University of Paraná – Brazil. The system logs in a data base exchangeable information with other actors and users can extract information for particular interest like (personal projects, activities and tasks). The actors of the design team did never have previous experience in tools like this. They were trained to use the system and encouraged to use it. Results of the evaluation report that some of the actors did not fully understand the tool or they were not available to use it showing a moderate optimism in the potentiality of system.

54

2.2

Implementing New Technologies in a CEn Environment

Although companies usually claim that their products are the “best of the market” (sub-chapter 2.1.1) for a range of solutions the papers presented on sub-chapter 2.1.2 showed a different reality. It was shown that researchers still need to face the number of challenges to implement a system that can help to enhance collaboration, communication and mutual understanding between those involved on a CEn project. They also showed the difficulties to introduce/implement information and communication technology in a company when actors are already used to a culture. The situation can get worst when the solution provided does not attend the real users’ desires (Peansupap and Walker, 2005, Mathieson and Keil, 1998, Nitithamyong et al., 2007). In a survey proposed by Nitihamyong (Nitithamyong et al., 2007) with 82 companies evaluating 14 different project management system – application manager providers concluded that benefits are limited to “strategy, time, and communication”. Benefits like “cost savings, quality improvement, and risk management” still need to be improvement – although (Khanzode et al., 2008) proved the opposite regarding to cost and time savings at least. On a collaborative environment system there are also risk to be calculated regarding the coordination of distributed products as explained on Qiu (Qiu et al., 2007a, Qiu et al., 2007b). To know more about the limitation on implementing new technologies in CEn environments the author recommends Peansupap (Peansupap and Walker, 2005) with an extended list of papers reviewing the subject.

2.3

Improving CEn Projects – Communication, Collaboration and Mutual

Understanding The works and products reviewed above confirm that even though the technology in general become more accessible to companies there are still issues to be solved. Problems like how to model the data that will be shared between actors, how to make the software intelligent enough to recognise a problem and provide a right solution from previous cases, how data can be visualised and understood, or even how to input 55

data without loosing long time to do it, are just some of the tasks to be implemented or improved. Some of the projects also mentioned the fact of using 3D visualisation (some of them were using even 4D – time) actors can easily indentify problems and find a solution (Khanzode et al., 2008, Brousseau and Rallet, 1995). This is particular true when investigating complex buildings where 2D representation becomes insufficient (Ku et al., 2008). This thesis concerns about the methodology of how data was collected from AEC and put in the data base but does not get involved on it. The presented research is already a next step that goes beyond the challenges faced by researchers above and explores the possibility of using a CARS to visualise information that most of the time has being explored only in a computer screen. The investigation is directed to user behaviour when exploring such VE in a CEn context. This thesis aims to create and study the initial impact of CARS in a CEn environment, exploring if this technology would be welcome by actors and to instigate communication, collaboration and mutual understanding. Based on the works presented above and the challenges faced by researches this thesis understands that the topics above should drive a research that aims the use of CARS in a CEn context: • Communication – the topic communication in this thesis aims to instigate

verbal and no verbal communication. Users should talk asymmetric with no rules imposing who should speak and who should listen when in a conversation. They are encouraged to ask and to answer and specially to make their on conclusions about what they are exploring in the VE. There is no logging or annotation system to what is being saying. The research is focused first if actors can understand the information. This is only possible with communication available all the time and in special during the final evaluation.

56

•

Collaboration – the second key world for this thesis only happen if firstly

communication is in place. Collaboration is placed when users can help each other inside of the VE. It could happen when the user outdoors cannot find a specific target for instance. A discussion about what is being presented or not is also part of this key world. As mentioned in Chapter 1 collaboration is characterised when two or more people work towards the same objective. • Mutual Understanding – this can only be achieved if communication and collaboration were previous reached. The third key world can be exemplified when outdoor actors cannot understand 3D sections of a building for instances. Through communication and the exchange of information (collaboration) users can talk and explain what each phase of the visualisation represents making the understanding finally happen. • Exploring the VE – users should be able to explore the VE in their computers in stand alone mode. There is no path to be followed or limited area to be explored. There is no need to render view points to create a movie. Everything must be rendered in real time even with the cost the quality of image being produced. The same is true with users exploring the VE outdoors. The system must support one or more users (indoor or outdoor) and allow them to share the same context in real time. • Moving objects in VE – virtual objects can be moved at any time. All modifications are automatically reported to all users that are sharing the VE in that moment. • Animation – this feature help to bring the VE as close as possible of reality. Animation helps also to demonstrate simulations to users like wind corridors, water elevation, increase of traffic, etc. • Accuracy – users and virtual objects should have their positioning as much accurate as possible. The accuracy should happen in real time with no limitations like: pre-determinate surveyed points, huge discrepancies of the real positioning and the calculated one, etc. • User friendly – although the system is a prototype it should be as user friendly as possible. Very few or not instructions should be necessary to use it. Users should be able to rely in their natural instincts to explore the VE. This is particularly true when using the system outdoors.

57

Chapter 3
Technology Review – VR and GPS
As mentioned by James Vandezande (Autodesk Inc., 2008b): “Design and visualisation is an important aspect of a project as people are driven by the visual experience of the architecture”. This state is particularly true in a project that works over the visualisation of VE in complex area like CEn. Lange (Lange and Bishop, 2005) quotes Bruce (Bruce et al., 1996) when talking about the importance of visualisation. They estimate that “80 percent of our impression of our surrounding comes from sight”. Back to the 3 key world of this thesis the work presented by Lange indicates three bullet points very closed related to what this project is focused. Lange and Bishop summarise the idea of visualisation to enhance communication, collaboration and mutual understanding:
o o o We want visualization to give us the opportunity to see, experience and understand environment changes before they occur. Through the ability to share their experience and potential for exploration, visualization will help communities (of whatever size) to build consensus and to make decisions about their future. The relationship of people to their environment is a key contributor to environmental decisions and visualization can help us learn more about that relationship.

58

In VR perspectives, CEn and Architecture are recognised by the wide field of applications that can be created (Woodward et al., 2007a). Tools based on VR concepts are natural medium for ACE professionals to express ideas, manipulate 3D objects in real-time, explore VE from different angles and stages (Bouchlaghem et al., 2004). This chapter makes clear the potential use of VR and in special CARS in CEn projects. The following sub-chapters introduce a review of VR and related work with direct or indirect application in CEn. Although this thesis is not involved in a research aiming to collect information from AEC (like those presented in Chapter 2) it is a complement of it in a near future. The next natural step is the convergence between those researches presented in the Chapter 2 and AR systems (Woodward et al., 2007a). This chapter also introduces an overview of the concepts related to positioning system – Global Positioning System (GPS) – which are relevant to this thesis. It is not intended to include in depth description of the techniques required to generate positioning information from GPS signals. Instead, the following paragraphs provide an overview of the three usual procedures which are commonly applied in AR systems: stand-alone GPS positioning, DGPS and RTK-GPS. Since this chapter is an overview of the applications of GPS, some knowledge of the working of the system is assumed.

3.1

Virtual Reality

The term VR was first coined in the 80’s by Jaron Lainer (Brate, 2002). As mentioned in Koleva (Koleva, 2001), the term MR is proposed by Milgram and Kishino (Milgram and Kishino, 1994) in 1994. They noticed that the term VR is being used not only for full synthetic environments but also for all kinds of computer generated scenarios that combine real and virtual. Milgram and Kishino proposed the Virtuality Continuum to demonstrate where each of the concepts lay (Figure 3.1).

59

Figure 3.5 The “Virtuality Continuum” (Milgram and Kishino, 1994)

The RE, on the left side of the diagram, is defined by Milgram and Kishino based on the difference between real and virtual objects according to three different aspects: • Real objects are any objects that have an actual objective existence.

Virtual objects are objects that exist in essence or effect, but not formally or actually. Real objects can be viewed directly. Virtual objects must be simulated. • In order to be seen, virtual objects must be synthesised and then projected or displayed by an electronic device. Just because an image “looks real” it does not mean that the object being represented is real. Milgram and Kishino developed these aspects on their paper, defining real objects by means of a sort of counterproof. Even though they clarified that virtual and REs are much more comprehensive than visual perception, there is a tendency to work with examples focused on vision. The VE, on the right side of the Virtuality Continuum diagram (Figure 3.1), or VR, are both concepts that can be found in the literature as: “an experience in which a person is surrounded by a three-dimensional computer-generated representation, and is able to move around in the virtual world and see it from different angles, to reach into it, grab it, and reshape it” (Novak, 1999). Brooks (Brooks, 1999) suggests that: “a VR experience is any in which the user is effectively immersed in a responsive virtual world. This implies user dynamic control of viewpoint”. It must be clear that a VR experience is not only related to vision and viewpoint. There are environments where stimulation of the haptic, audio, olfactory and palate senses also exist. All the five senses can work together or separately. The final structure produced for the VE is deeply dependant on the research’s focus.

60

Apart from these concepts, there are three key words to better understand the VR theme: immersion, interactivity and involvement. • Immersion – The idea of immersion comes from the sensation of staying inside the environment, which means being part of it. This key word depends basically on the quality and accuracy of stimulation of the five human senses. It could be achieved by a high graphical resolution on the scene with textures, shapes and simulations to make it very similar to what is known in the physical world, for example. Very accurate devices could be added to it, enabling the user to forget that they are connected on a computer. Such devices must be fast enough to update the changes, light enough in order not to be perceived, and ergonomic enough to allow freedom of movements. • Interactivity: In his report, Pinho (Pinho, 1999) exemplifies a VE where the user can visualise, handle and explore in real-time all the features of the VE without a rigid control of these actions. The user applies his natural and intuitive movements to control the VE. The interactivity is not only related to the object but it is also completed with other users of the system and virtual agents. They share the same senses of space, presence and time. • Involvement: In the same report, Pinho (Pinho, 1999) describes how motivated the user is inside the virtual world. The user can have an active situation when there are interactions or engagements between other users or existing elements. A passive situation is always possible when there is not mutual or reciprocal action (between users or existing elements) in the VE.

3.2

Augmented Reality

If the intention is to overlay a perception of the actual physical and real world with images, sounds, smells, haptic sensations and tastes created by a computer system, the terminology used is named AR. Azuma (Azuma, 1995) gave the following concept about AR:
AR is a variation of VE, or VR as it is more commonly called. VE technologies completely immerse a user inside a synthetic environment. While immersed, the user cannot see the real world around him. In contrast, AR allows the user to see the real

61

world, with virtual objects superimposed upon or composite with the real world. Therefore, AR supplements reality, rather than completely replacing it.

In AR systems, the perceptions can be increased or decreased. Besides adding objects to the RE, AR also has the potential to remove them (Azuma, 1995). It is possible to differ between VE and AR as: • VE is a situation where the user is in a place without external contact. That is, if the VE has the focus in audio, for example, the user will not listen to anything directly from the external world. The sound generated is digital. A cockpit of an aircraft simulator is an example of VE. The user has the sensation of being contained in the VE. • The integration of current information and computer generated (digital) data in a designated area is the concept for AR. This combination supplies the user’s necessity of information that can not be perceived at the moment. The added digital data can represent information from the past, present or future. The inverse could be achieved by taking out information from the RE. It will depend on the applicability of each project. An AR system can, for example, capture sounds from the RE and reproduce it in a directional 3-D VE with a partial inclusion or exclusion of the audio.

3.3

Augmented Virtuality

Milgram and Kishino (Milgram and Kishino, 1994) exemplified the concept of AV on the right side of the Virtuality Continuum diagram (Figure 3.1). The concept proposes integration of information retrieved from the RE on the VE. Schnädelbach et al. (Schnädelbach, 2001) presents an example of AV application in Augurscope project.

62

3.4

Mixed Reality

MR is commonly understood as “seamless integration between the real world and a virtual world”(Novak, 1999). As shown on Figure 3.1, MR is a generic concept involving both AR and AV running from RE to VE. Possible Applications – MR and VE There are different types of applications for MR. Visual applications are the most common work related to this subject. Audio and haptic MR projects are being increasingly explored. The development of new devices that can produce all sorts of smells and tastes opens a wide area to be investigated by researchers. Applications that could be enhanced with works on MR projects are presented on the list bellow. MR combined with other technologies brings information to more accessible way helping researchers with the possibility of simulate substances behaviour, displacement of features, animations, sound simulations, etc. The list is bellow is neither extensive nor complete but illustrates where MR can be used as a tool to enhance understanding over information. • • • • • • • • • Scientific Visualisation: Planets surface visualisation; Virtual tunnel test; Molecular synthesis . Health and Medical Applications: Surgery simulation; Radiotherapy planning; Virtual health; Anatomic visualisation; Training for disabled people. Architecture and CEn: CAD; Decoration of environments; Acoustic estimation; Design of tools. Education: Virtual Laboratories; Planet Exploration; Distance learning; Education for people with mental disabilities. Entertainment: Virtual games; Virtual Cinemas. Training: Flight simulators; Military operations planning; Astronauts training. Arts: Painting; Virtual Sculpture; Music; Virtual museum. Information Control: Financial view; Information view; Virtual information. Telepresence and Telerobotics: Remote control systems; Teleconference; Virtual lecture.

63

3.5

Collaborative Virtual Environments

As mentioned before, one of the main goals of this thesis is related to the aspects of collaboration between the users of the system. A successful co-operation between actors in a CEn environment is achieved easily when well regulated and managed than when developed for a self-contained entity. As cited on (Churchill and Snowdon, 1998): “in ‘real world’ domains, collaborative work involves considerable complex information exchange. Individuals need to negotiate shared understandings of task goals, of task decomposition and sub-task allocation, and task/sub-task progress”. Therefore, collaboration does not mean that there is a complete agreement of the parts involved on the progress of a task. People can have different points of view over a resolution of a task for instance, disagreeing in some topics, but because they are also looking for the same welfare, the dialog and the co-operation must have place for the mutual understanding. No one holds the true on a CE and neither there is imposition of a perspective over a person. There is mutual respect and a wellbalanced division of tasks. Thus, it is important that collaborators know what is currently being done and what has been done in the context of the task goals (Churchill et al., 2001). Collaboration can be observed in several ways in our society: personal contacts, gestures, body expressions, work groups, letters, telephone calls are just few examples which could help to achieve a common goal. More recently, computers have been used on the creation of CVE. CVEs are a digital “space” created over a net of computers where collaboration and interaction create links between users. Each space has a connotation that constitutes a “place”. It gives the users the opportunity to understand the activities performed on the environment as well as on the real world. The places are created to support multiple groups developing multiple activities. As on the real world, people (or the users on a CVE context) understand that even without straight relations the

64

spaces and places are dependent on their actions, behaviour, social skills, manifestations, etc. According to Churchill (Churchill and Snowdon, 1998, Churchill et al., 2001) CVEs are explained as:
CVE’s represent the computer as a malleable space, a space in which to build and utilize shared places for work and leisure. CVE provide a terrain or digital landscape that can be ‘inhabited’ or ‘populated’ by individuals and data, encouraging a sense of shared space or place. Users, in the form of embodiments or avatars, are free to navigate through the space, encountering each other, artefacts and data objects and are free to communicate with each using verbal and non-verbal communication through visual and auditory channels.

The users that explore a CVE are not restricted to a desktop PC. CAVE, wearable computers and PDA, are some examples where CVE can be shared and explored by the users. Greenhalgh (Greenhalgh, 1999a) mentions some common characteristics on CVEs: • Computers spread over the planet are connected over a network sharing the same CVE. The system, as itself, can or cannot be running in a specific machine or set of machines. It can be distributed over the network supporting multiple users. • Collaboration can be done on different ways like: gestures, audio, video, etc. • The idea of space or world is conveyed to the users. The users soon understand the boundaries of the VE and the context of the place. • Users are represented, in a way, as a form inside the VE in order to make them visible. It could be a textual representation as it is done on Multiple User Dimensions/Dungeons (MUD) or as an embodiment (avatar), a humanoid representation on 3D environments. Often the users can choose a representation more appropriate with the VE. • VE is a free space for users to freely go and explore the environment. Certainly, the VE has a virtual boundary delimiting the space. Depending on how detailed the VE is, the users will need to respect basic conventions of the

65

physical world, such as walking over a lake, going straight through a wall; thus, marking the internal boundaries of the system. Natural and Spatial Communication The idea of natural and spatial communication is connected to the knowledge of where the users are positioned in relation to the others and to the objects. On the real world, for example, a simple look around can give to person enough information to identify the approximate number of people on the place, who is physically close to him and who is not, where he is located in relation to the place, the obstacles around, etc. This “glance” over the area that surrounds him is converted to the knowledge of space and time. The CVE should support natural spatial communication. Based on the knowledge of space and time, “…gestures, posture and speed of movement consciously and unconsciously convey information such as availability for conversation, interest and intended actions” (Greenhalgh, 1999a) . The paragraphs above reveal two important points on CVEs. The users on the VEs must have at least an embodiment. This generates the notion of space and time. Comparing two extremes, both 3D and textual CVE follow the same principle. On 3D environments, the perception of scale or obstacles is the most notorious way to know if another user can be reached or not. On textual CVEs inhabited by objects, the user knows if what is being said is reaching the others by a “proximity sensor”. On a richer scenario, space and time are revealed through the information of the object and places that surround a specific user. The second important point is the support of body movements. A CVE can only offer this possibility if, and only if, the embodiment of one user can be observed by the others. On the real world, a good percentage of our communication is done via gestures, movements (it includes body hair movements), expressions, level of voice, colour of skin, transpiration, etc. Currently, much of this “communication” can not be provided by CVEs. However, simple movements are already integrated on the system. On MASSIVE-3 (Greenhalgh, 2000b) the embodiment of the user has its head down if the user is not actively interacting with his avatar for a few minutes. For 66

other users sharing the CVE, it indicates that the person represented by the avatar maybe is not in front of the computer. Sharing Context As cited on (Churchill et al., 2001) a “shared context can mean many things; it can mean shared knowledge of each other’s current activities, shared knowledge of other’s past activities, shared artefacts and shared environment”. Collaboration is based on sharing. When a gesture is done or an object is shared “not only do they become the subject of communication between users, but also the medium of communication” (Churchill et al., 2001). The way that an object is being carried can have different meanings. In a dramatic example, if the sharp blade of a knife is pointing to the person that is holding it in the direction of another, there is no particular reason for a precaution movement from the receptor. The first basic intuitive reaction is that one person is passing a knife to another. In contrast, if the sharp blade is pointing to the receptor, the object transmits the information that something bad is about to happen. The information contained on that specific object being carried on different ways is disseminated to other users. The context of the situation is then shared over the participants. The visual perception of the scene collaborates with the logic of the facts. Peripheral Awareness When a person is driving a car, the environment that surrounds the car provides enough information for the driver to know whether it is time to brake or push the accelerator pedal. What the driver has is a natural and spatial communication that must be shared with all potential hazards that surround him. If the traffic light turns red and the cars in front stop suddenly, the peripheral awareness lets the driver know if it is feasible to avoid the car taking the other lane. In a fraction of time, several calculations must take place, such as: velocity of the car, distance from the car in front, condition of the other lane, break distance of the vehicles behind, etc.

67

Hughes et al., cited on (Churchill et al., 2001), “observed that co-located workers maintain and make use of an ongoing awareness of the activities of others with their environment, even when not explicitly co-operating”.

Negotiation and Communication Churchill (Churchill et al., 2001) explains that:
Conversations are crucial for negotiation and communication about collaborative activities. Collaborative work requires the negotiation not only of task-related content, but also of task structure in terms of roles and activities and task/sub-task allocations.

The exchange of information through limited number of body gestures (vision) and voice (hearing) has been developed with great results on CVE. Such approaches always enhance the comprehension of what is being shared. Wearable tracking devices like (Xsens, 2008, Ubisense, 2008) are examples of how gestures can be added to VE. Freedom Another premise of VE is freedom. The users can explore the VE, maintaining the autonomy of their acts. There are no pre-programmed movements where users should go and scenes are rendered on real time. Eventually, multiple view points can be explored. Instead of being restrained by the embodiment’s line of sight, other shoots can be taken from “virtual eyes”. They could be placed on specific points, either following or not the embodiment movements. Extra powers (flying, diving, etc.) can also be given to the embodiment with the objective of turning the VE flexible, enhancing the perspective of collaboration. Large Number of Participants The size and complexity of the VE added to the number of participants suggests how powerful the computational system is on the background. Usually CVEs are complex, with thousands of polygons to render and textures to add. Video and voice channels

68

are also included on the system’s performance. As cited on Greenhalgh (Greenhalgh, 1999a) “the final advantage of CVEs from the perspective of collaboration is that there is a clear potential to support extremely large number of simultaneous participants”. As a reproduction of the real world, a CVE system must only pay attention to the strict necessities of each user. If an embodiment is placed on a room with some furniture spread around and is talking with a couple of other users, there is no requirement for the system to render the whole building for the three users. The system spends resources on what is relevant for each user – this process is also called Level of Detail (LoD) (Wikipedia, 2008b). As cited on (Greenhalgh, 1999a), “…it allows them to be aware of and to interact with a relatively large, continuous and consistent subset of the total environment”.

3.6

Related Projects

As presented in the previous sub-chapters, MR is an open and wide area of study. The concept can be applied to most of the areas that need to interpret information, which is sometimes not easily understandable by charts, tables, numbers, text, etc. A selection of projects were choose to exemplify how MR, and in special VR, can help CEn actors to perform their works. Although some of the examples introduced at this first moment do not have a direct relation with CEn application it is imaginable that they can be adapted to this area. This selection is based and presented using the following criteria: • From VE to AR – the overview cover the aspects of researches on VE and going towards AR. This approach was used to demonstrate the limits of VE and when AR becomes a necessity. • Visualisation Modes – from computer screens to large concave screens for 3D visualisations. The examples cover the idea of how the VE can be visually explored. The final researches presented the use of HMD devices for AR proposes.

69

•

Collaboration – some of examples are for users exploring the VE in a computer screen without direct interaction with other users. There is no real sense of presence inside of the VE. The information can be shared but not the VE. The examples progress up to where users can share information in a collaborative way. Users can share the same physical or/and virtual space at same time that they collaborate over the information that is being presented.

•

Positioning System – researches have different approaches to calculate the position of a user, especially when outdoors. This survey exemplifies how researchers suggested the resolution of positioning. The examples are focused on GPS solutions once this thesis is working with this methodology for outdoor positioning.

From numerous projects surveyed, the following projects were chosen for review. • • • • • • • • Interactive Visual Simulations (VizSims) (Delaney, 1999) – Applied to Oil Exploration – Petrobras. TerraVision II (Reddy, 1999). VizSims – Applied on Urban Planning (Delaney, 2000, UST, 2004). IMAGINE VirtualGIS – ArcGIS 3-D Analyst (ESRI, 2003, ESRI, 2001). Shiodome CyberDome (Masanori and Sawada, 2004). Cyber Walker 2001 – Wisteria World 2001(Tamura, 2001a, Tamura, 2001b). Towards Outdoor Wearable Navigator (Tamura, 2001a, Satoh et al., 2001) – Air Hockey (Tamura, 2001a) – AquaGauntlet (Tamura, 2001a). Studierstube Augmented Reality Project (Interactive Media Systems Group Institute for Software Technology and Interactive Systems Vienna University of Technology, 2004). • • • • Columbia University AR Projects (Columbia University Computer Graphics and User Interfaces Laboratory, 2004b). Tinmith (Piekarski, 1999, Piekarski, 2001b, Piekarski, 2001d, Piekarski, 2001c, Piekarski, 2001a, Piekarski, 2002). The Augurscope (Koleva et al., 2001). Augmented Reality for Subsurface Data Visualisation (Roberts, 2002, Roberts et al., 2001, Evans, 2001). 70

•

ARMobile, AROnSite, ARWebCam, ARPhone (Woodward et al., 2007a)

VizSims – Applied to Oil Exploration – Petrobras Delaney (Delaney, 1999) describes on his paper a technology that helps find oil faster by using a 3D interactive visualisation. The device collects data from the ground based on acoustic imaging and has its patent dated back to 1923 – US patent no.1451080. Initially, these devices used strip recorders to record the acoustic signal that bounced off from underground rocks. Soon, the number of acoustic receivers increased in order to give a better idea for Petroleum Geologists of what they can expect to find on a possible oil-bearing ground. The big issues were the volume of data captured by the acoustic receivers and the noise caused by the reflection of the signal by the subterranean structures. The data, usually presented as layers of information in 2D charts, was improved in 3D charts with the quality limited by the skill of the geologist, the quality of the data and the time available. In the late 1990s, computers were powerful enough to handle huge amounts of data in real time. 3D charts were substituted by 3D immersive environments where data could be analysed and explored in different angles, and teams collaborated with each other with different expertise. Delaney (Delaney, 1999) cited Loftin – Director of the Virtual Environment Technology Laboratory at the University of Houston – described the rationale:
No one person owns all the knowledge. You have five or six people who work together to manage a reservoir. They have different vocabularies and knowledge. If you put them together in the same data, they communicate better.

The researchers of the Brazilian Oil Company, Petrobras, have worked with the previously described system in order to help geologists and engineers to explore oil on deep oceans.2 With the data from the layers of the ocean ground, 3D images are projected on concave screens. Stereoscope glasses create the sensation of 3D objects “flying” in front of the research team, thus allowing the discussion of ways to perform a new perforation. The system runs on an Onyx with 5 Central Processor
2

This was observed during my visit to this company in 2000.

71

Units (CPU) IP27 250 Mega Hertz (MHz), 8 gigabytes of Random Access Memory (RAM) and a graphic board InfinityReality2E. Even with the high cost of the equipment and the area where the visualisation is performed (estimated at that time in U$2,000,000.00 USA dollars), the cost of positioning an oil platform in the ocean along several days is even higher. With this system they know exactly where the oil platform should be positioned according to the acoustic data received. Figure 3.2 shows a selected number of layers of information collected from the deep ocean to be analysed by Petrobras team. Different colours represent distinct soil layers found by the survey.

Figure 3.6 Potential Oil Field – Petrobras

The system explores an idea created around 1917 when Reginald Fessended received a patent for his “Method and Apparatus for Locating Ore Bodies” which also used acoustic energy to makeup underground formations (Delaney, 1999). Nowadays, users are placed in a collaborative environment that creates a synergy of knowledge, dramatically cutting time and budged needed to analyse a block of seismic data, in some cases from months to hours. TerraVision II Reddy et al. (Reddy, 1999) describe the browser TerraVision that uses the potentialities found in VRML. Their research created the Tile Set Manager Application Program Interface (tsmApi) where they pointed out four fundamental design criteria to develop the browser to explore a virtual terrain: • • Scalability – the browser must support large and small data sets. Composability – the browser must support multiple types of geo-referenced data such as images, cultural features, etc. 72

•

Efficiency – the easy navigation to explore the VE on a standard VRML browser must be maintained or a customisation is required to further increase the navigation efficiency.

•

Data Interchange – the development of generic data representations for georeferenced data in VRML.

TerraVision II (Reddy, 1999) was created as a solution to retrieve and merge massive volumes of remotely located data, including aerial and satellite imagery, topography, weather data, buildings and other cultural features. Such data can be terabytes in size and could be recovered from multiple servers across the Internet. The interface to the data used by TerraVision is GeoWeb, a distributed searchable database of metadata also created by the Stanford Research Institute (SRI) group. TerraVision II constantly queries the indices for any relevant data in the current view. The system starts the most suitable interface depending on the kind of metadata returned from GeoWeb. In order to handle massive data sets TerraVision II does not download the whole high resolution data once, but instead employs a tiled multiresolution data hierarchy. The data is divided in rectangular tiles over a range of resolutions creating the sensation of LoD. This implementation avoids the transmission of dozens of megabytes over the network, leaving the whole system faster even when working with highly accurate data. The browser interprets and predicts the user’s movement over the data and manages to download only the subsequent four tiles in a higher resolution. The process is repeated if the user wants to explore more information from a specific area. Unlike VRML, this approach avoids overspending huge network communication or system processing. The system allows the user to explore a data set through multiple windows from different points of view or different combination of sets. Three-dimensional models can overlay the terrain data adding extra information on the visualisation. Features including buildings and roads, and even simulations of wind vectors and air turbulence can be added.

73

Other features of TerraVision II include (Leclerc et al., 2002): open source, massive scalable dataset, multiple datasets, gazetteer support, OpenGIS Consortium Web Mapping Server support, flight path, viewpoint bookmarks, heads-up display, documentation and cross-platform. Figure 3.3 shows an example of a TerraVision II application. The image represents, with a resolution of one metre, United States Geological Survey (USGS) imagery and elevation for the Menlo Park area. A three dimensional VRML model of the SRI Campus was superimposed by a texture-map extracted from aerial imagery (SRI International, 2002).

Figure 3.7 VRML Model of the SRI Campus (SRI International, 2000a)

Figure 3.4 shows an example of an application created through different sources of information. The view shows an area of Southern California with a high resolution image inserted around the Camp Pendleton area and a VRML model of wind conditions for that region.

74

Figure 3.8 VRML Wind Vector Overlay (SRI International, 2000b)

As described on (Leclerc et al., 2002) “the high performance system enables the user to browse massive, rich, distributed terrain; view embedded multimedia data; and collaborate with other users in the virtual 3D space”. VizSims – Applied to Urban Planning Interactive VizSims were explored in a paper presented by Delaney (Delaney, 2000) with the focus on urban planning. His examples started with Bill Jepson who founded the Urban Simulation Team (UST) at the University of California (UCLA), one of the pioneers of urban simulation. The UST (UST, 2004) produces solutions for real time visual simulation in the field of design, urban planning, emergency response and education. However, the focus of the group is on the development of a model of Los Angeles (LA), an area of several hundred square miles (Delaney, 2000). UST claims that the system is extremely efficient when compared to the time and labour spent to construct a model. The system can be used to visualise any urban environment. The data that feeds the digital environment is acquired through aerial photographs, GPS data, street maps, eye-level photos and three dimensional geometry and creates a realistic 3D visual simulation of the extremely dense Los Angeles urban environment (Delaney, 2000). The VizSim eliminates the necessity of blueprints, charts and other “hard-tounderstand” traditional representational methods. The VE explored allows the users

75

to fly, drive or walk-through in real time giving a better impression of what the architects are doing. On Delaney’s paper (Delaney, 2000), Jepson highlighted the UST work, saying:
We empower the clients to understand what the architects are doing. The architect wants to control the information flowing to the client. We come between the architect and the client. Once the client walks through (the planned building), he feels empowered to take a larger role in the design process…When (their preference) differs from the client’s agenda, problems happen. The model shows the problems.

Even with the focus in modelling LA, the UST team has completed a number of transportation related studies and recreation of historic environments. Transport simulations run from new metro stations to LA freeways development. Some studies are also related with the beautification of Los Angeles International Airport (LAX) (UST, 2003). Figures 3.5 shows an image produced with the results achieved with the modelling of LA. The enhancement proposed to LAX is exemplified by Figures 3.6 and 3.7.

Figure 3.9 Downtown of LA Modelled by UST Team (UST, 2003)

76

Figure 3.10 LAX Original Condition (UST, 1998)

Figure 3.11 LAX Proposed Condition (UST, 1998)

The LAX project is a combination of efforts from different teams of architects, environment graphic designers, landscape, lighting, engineers and art consultants. The first phase of the project was to register the actual condition of the entire environment where LAX is placed. Later, the enhancements on the area were designed and presented to the Los Angeles World Airport’s Board of Commissioners and at public meetings held throughout the process (UST, 2004). Delaney (Delaney, 2000) also reports as experiment with the architect Mike Rosen, in which they created 3D models of houses and office buildings for sales and planning purposes. When working in a virtual baseball stadium, the mayor of Philadelphia, Ed Rendell, was faced with a scoreboard obscuring the City Hall. The solution was to move the scoreboard 40 feet away. It was a simple solution but the system proved to be useful on situations like this. As Delaney (Delaney, 2000) affirmed: 77

…simulation is quickly becoming an essential tool for planners at the town, city and regional levels. As computers become more powerful, we can expect planet-wide simulations to extend our understanding of the impact of development of large areas of the globe.

Interactive VizSim is a term that can be applied in much wider prospects than only urban planning. The application of VizSim for urban design is a facilitator tool that makes the ideas introduced by engineers and architects more accessible. Generally, the public who is affected by the changes produced by CEn schemes have difficulties in assimilating the information presented on 2D layouts, blueprints, charts, and even scale models. Significant changes on the VE can be done almost instantaneously, generating objective and efficient answers. Another example of how far an urban simulator can be easy to understand is the game created by Will Wright titled SimCity. Wright started the project in 1985 (SC3000.COM, 1998) and released the first computer game simulation to accurately model a city in 1989. On a CNN online interview (SC3000.COM, 1998), Will explained:
…in the first two years I worked on "SimCity," I did quite a bit of research and read 20 books at least on urban planning theory. In addition to that, we did a fair amount of research into simulation techniques. And for the next version of "SimCity" (2000), I spent quite a bit of time talking and meeting with various people who had expertise, including city planners, police administrators, public works people, teachers, etc.

Nowadays, SimCity is a VizSim urban simulator played by millions of people around the world. The players can delimit the virtual city areas where industry, shops, rescue teams, etc. will be placed. Roads, power plants, water supply are also allocated and placed by the players. The high graphic resolution and the dynamic way that the virtual city evolves contributes to a sense of reality. Figure 3.8 shows a virtual city named Olympia created by CityScape.

78

Figure 3.12 SimCity – View from Olympia (CityScape, 2004)

IMAGINE VirtualGIS – ArcGIS 3D Analyst The urban visualisation tools that have been developed are getting closer to Geographical Information Systems (GIS) which are, according to (ESRI, 2001): “a system of computer software, hardware and data, and personnel to help manipulate, analyse and present information that is tied to a spatial location”. The same concept is understood by the U.S. Census Bureau (Bureau, 2001) as: “A system of hardware, software and procedures designed to support the capture, management, manipulation, analysis, modelling and display of spatially-referenced data for solving complex planning and management problems”. The following list gives a better idea of what is involved on a GIS system: • • Spatial location – all the data contained on a GIS database must be georeferenced. A reliable result depends on reliable surveys of data collection. Information – a chart of an area containing different colours representing distinct layers of information is better comprehended than a long table with lots of columns and numbers. A visual graph result is one of the goals explored by GIS. Fast analyses of databases, different combinations of layers and a precise result in colour charts, maps, or even 3D VEs contributes to the popularity of this system. • System – the whole system can be exemplified as a bridge between software, hardware and data. The hardware encompasses not only the computers that are

79

part of the management system but also those that capture information that feeds the database. • Personnel – they are the people involved in exploring the potentialities of GIS.

It is not the intention of this thesis to conceptualise GIS. The objective is to describe GIS as an information source to be explored on VEs. The results can be presented to the users in large or reduced scales depending on the application and general specification. Urban simulators are huge databases with the shape, locations and textures of buildings. Telephone and electrical cables, water pipes, street planning, population density, the number of traffic lights; all are examples of data that can be embedded in the visualisation. Figure 3.9 introduces an idea of a GIS visualisation, exploring 3D potentials. The image was created by a combination of layers of information as the Digital Terrain Modelling (DTM), boundaries of the forest, water level, position of the fences, and the position of energy cables. This VE is explored with the help of a VRML browser.

Figure 3.13 Visualising Data from a GIS (Capra, 2000)

The following two examples are commercial products designed for GIS applications. The purpose is to demonstrate how this technology uses VR for visualisation. IMAGINE VirtualGIS by ERDAS is an example (Leica Geosystems LLC., 2001) of a package that allows the user to make an accurate registration between the DTMs and images acquired by aerial photography, satellite imagery and airborne scanner data. Users can add annotations, vector GIS layers and symbols, and can create a VE with significant realism by adding texture on 3D objects. By using mouse control, it is possible to navigate through the VE, exploring different points of view – as if the user

80

had the freedom and eye-sight of a bird – and to record this movement to create a movie. ERDAS improved the LoD algorithms in order to minimise the effect when presenting complex polygons in the DTMs. The software can render datasets with different resolutions that can be merged together on the same visualisation. The screenshot shown in Figure 3.10 is an example of results produced by the software. Some advantages exposed by ERDAS include: • Terrain Database Export – the software can export the objects and its proprieties to VRML, OpenFlight (OSG Community, 2004) and RAW (Wikipedia, 2008d). • • Performance Setting – the users can control the quality and performance of the system to the level that they would like to achieve. Animation Tool – the user can create movies simulating aircraft flight paths, ground vehicle routes, shipping lanes, etc. VirtualGIS is a complementary package of a full system named IMAGINE. The full system is composed of: Advantage, Essential, VirtualGIS and Analyst.

Figure 3.14 Textured 3-D View of Munich (ESRI, 2004b)

ArcGIS 3D Analyst (ESRI, 2004b) is part of a complete system GIS named ArcGIS. This is a browser that integrates the ArcScene application, providing an interface to view multiple layers in a 3D view. With ArcGIS 3D the users can analyse 3D data

81

from several viewpoints to create a realistic perspective, draping raster and vector data over the DTM. The release 9.0 (ESRI, 2003) can manage extremely large set of 3D geographic data. As an extension to the ArcGIS Desktop products, the system allows users to leverage the GIS analysis tool available in ArcView, ArcEditor and ArcInfo to perform geoprocessing tasks in a 3D environment (ESRI, 2003). Some of the characteristics of ArcGIS are: • • • • It can work with both vector and raster data. It can create 3D fly-through animations. It can apply various effects to layers of data, including: transparency, lighting, shading and priority on layers overlap. It allows the visualisation of multiple perspectives simultaneously.

Figure 3.11 shows a screenshot of the ArcGIS 3D Analyst.

Figure 3.15 ArcGIS Screenshot (ESRI, 2004a)

These two tools introduced are a subset of a complete GIS package composed of several other tools. Both IMAGINE Virtual GIS and ArcGIS 3D Analyst are in fact browsers and interfaces between the queries made to the dataset and the 3D visualisation. The major advantage is the full integration between the packages, making it possible to combine 2D and 3D data. The software (Sapeta, 2003) creates an integrated, real-time visual interface for the analysis, mining, linking and

82

visualisation of geographic and related data sources. Also, image registration is a strong component on these tools. It enables the production 3D DTMs with photogrammetric works placed upon the model, providing extra realism to the scene. As cited by Sapeta (Sapeta, 2003), the next generation of GISs with applications for urban analyses are limitless and can include: simulation on emergency and disaster, land development, police operations, urban planning, environmental impact, transportation planning, telecommunications, etc. Shiodome CyberDome Apart from 3D digital environments being explored on computer screens, stereoscopic visualisation presents another possibility for users to be immersed and surrounded by 3D objects. Matsushita Electronics Hemispheric Dome – Shiodome CyberDome – is one of the largest immersive projection based VR systems in the world. The users receive a pair of polarised glasses at the beginning of the presentation. The dome is a hemispherical screen of 8.5m in diameter, displaying to the users a Field of View (FoV) of 180 degrees on the horizontal and 150 degree vertical. On the background, there are 18 liquid crystal projectors, one master Personal Computer (PC) and 18 rendering PCs computing the 3D environment. The high quality stereoscopy projection makes a perfect match between the screens projected by each projector. The software-based distortion-correction technology does not require an expensive special lens, and the projector placement, expressed in parameters, enables projection on a screen dome of any size (Masanori and Sawada, 2004). In a visit to Shiodome, the author could experience the illusion of being immersed in the VE is convincing3. In on of the possible applications, the users can explore part of a city going between buildings, streets and submerging into a river. The sensation is that of a real 3D scale environment, rendered in real-time and commanded by a logical and simple user interface. The interface consists of a ball on a stand where just half of the sphere can be seen. This control allows the forward, backward, left and
3

Based on my visit to this complex on December 2003.

83

right movements, without any body movements from the user, who faces ahead exploring the environment by moving just the sphere. Around the hemisphere there is a disc that controls the right and left turns. Finally, a stick shifts the view up and down. Everything is easy to understand and control, so that users are confident enough to have a close experience and to manipulate it. Figure 3.12 shows a layout of how the CyberDome looks and where the participants are placed. The users are free to explore the 3D VE and to contribute new ideas based on what they see. If the visualisation is related to a new urban plan, for example, the engineers, architects and the other groups involved can maximise effort to find a better solution for ongoing problems.

Figure 3.16 Shiodome CyberDome Structure (Matsushita Electronic Works Ltd, 2003)

Figure 3.13 shows three different users’ interface to control the movements inside the VE. Each interface is strictly related to the approach where the 3D VE is being applied. The frame on the left illustrates the ball interface explained previously. On the middle and on the right, respectively, a virtual walk and a scooter ride are explored as interfaces.

84

Figure 3.17 Shiodome CyberDome User’s Interface (Matsushita Electronic Works Ltd., 2003)

Shiodome CyberDome is a huge state-of-the-art interface created to explore 3D VEs. As mentioned on Virtools (Virtools, 2004), the core application of this CyberDome is the "VR Environmental Project Support Too", which is used for reviewing and consensus-building in CEn and urban developmental projects by construction companies, engineers, planners, designers, residents and users. The range of applicability is unlimited, as are the ways to explore the user’s controls over the VE. Cybercity Walker 2001 and Wisteria World 2001 The Cybercity Walker 2001 by Tamura and Kotake (Tamura, 2001a) uses AV technology to generate a large-scale virtual city. The basic idea was to equip a car with several sensors, including seven video cameras on its roof. The cameras were distributed around a circle on the roof of the car to progressively record images at the same time that the car is driven around the city. The other sensors indicate the car’s position and the angles that the images are being captured. The visualisation is done on a widescreen projection that can change the point of view when the user simulates walking on the street or driving a fast car. The velocity of the projection depends on the amount of force that is being applied to the controls. Virtual buildings can be added to the scene for city planning purposes and virtual tours. Another example introduced by Tamura is the Wisteria World 2001 (Tamura, 2001a), which incorporates MR functions into a telepresence system. Instead of using pre-captured images, the users control a remote video camera pointed at a 1/87 scale model. The scale model is precise enough to resemble a city. The simulation of walking and bird’s eye viewing can be seen on a widescreen projection. As the previous project, virtual buildings can be added to the scene. Both projects work with widescreen projection and can be easily associated with CEn applications. Some enhancements, such as the development of a dedicated hardware to render the images and a robust registration scheme can also described in

85

the paper (Tamura, 2001a). As illustrated on section 3.6.5, the advantage of projections on widescreen is noticed by the number of people in the audience. Towards Outdoor Wearable Navigator - Air Hockey - AquaGauntlet The projects discoursed so far have introduced VEs through computer screens or widescreens with or without stereoscopy. The following sections will explore projects related to AR. Even though some of them are not linked to CEn applications, they are helpful to exemplify the basis of the research proposed of in this thesis, at present, to seamlessly join VE and RE. “Towards Outdoor Wearable Navigator” (Tamura, 2001a, Satoh et al., 2001) is a mobile system where the user is equipped with a HMD and optical fibre gyro, and a portable PC. Instead of the users being tracked by a GPS receiver for example, they announced a system that uses vision-assisted registration to adjust the drift of the optical fibre gyro and presently performs with one of the best spatial registration accuracy in the world. However, the users can only visualise the environment on presurveyed places. There is no flexibility to walk around exploring the VE. Applications including the insertion of new features over specific points in the city were demonstrated in Tamura and Satoh (Tamura, 2001a, Satoh et al., 2001). Tamura and Yamamoto (Tamura, 2000, Yamamoto, 1999) introduced the experience of a case study of collaborative AR. They developed the AR Air Hockey, a game where the players hit a virtual puck with mallets on a table. Thanks to a seethrough HMD, the players can recognise the opponent’s movements (on the real world) at the same time that they see the virtual puck. A sensor and a camera are mounted on the HMD to register the user’s movements. After recognising the player’s movements, the system produces the behaviour of the virtual puck and shares the information on a VE. The authors claim that AR Air Hockey became the seethrough AR system experienced by the largest number of users worldwide. Tamura (Tamura, 2001b, Tamura, 2000) also comments on a multi-player MR entertainment entitled AquaGauntlet. Their objective was to create a system able to support more than three players, multiple virtual objects, occlusion between virtual 86

and real, special sound effects, gestures’ recognition, etc. The players’ movements are logged through magnetic trackers positioned on the HMD and on gloves. The HMD is not a see-through type but a block view. The real environment is recorded by a camera placed on the HMD and then blended on the VE before it is displayed on the glasses. The gloves track the arms’ movements, which are recognised by the system, resulting in different actions to attack the virtual monsters. These three projects exemplify the multipurpose use of AR. The project entitled “Towards Outdoor Wearable Navigator” only works on delimited areas of the city. The users always need to stop over a pre-surveyed base point. Then the users need to tell the system which base point they are stepping over. By choosing this methodology, there is no need of a GPS receiver. Positive points on the research presented by Tamura and Satoh (Tamura, 2001a, Satoh et al., 2001), are as follows: • • • The wearable equipment is considerably light. There is never the problem of GPS signal being blocked or multipath There is no need for radio links to use DGPS or RTK-GPS.

distortion.

However, there are also the following disadvantages: • • The users must stop at one of the base points in order to identify their The system does not allow the exploration of collaborative

location. environments. Air Hockey and AquaGauntlet are two examples of AR systems which allow the exploration of CVE. Users are immersed on the system due to the large amount of interactivity and involvement between players and the objects to be hit. Another positive aspect is the development of interaction in the VE through recognition of a user’s arm movements. In AquaGauntlet (Tamura, 2000), for example, the users have three gestures commands: ready, fire and defence in order to attack virtual jellyfishes, sharks and rockets.

87

Studierstube Augmented Reality Project The Studierstube (Columbia University Computer Graphics and User Interfaces Laboratory, 2004b) is a system that interfaces humans and VE in a CARS context. It was named after the study room where Goethe's famous character, Faust, tried to acquire knowledge and enlightenment. This system has been widely explored over the years, having been involved in around forty projects, according to its website. The most relevant ones are described below. The Mobile Collaborative Augmented Reality is a system created to combine 3D computer generated stereographic images, interaction with virtual objects, collaboration and data sharing between the users (stationary or not). The application integrates a see-through HMD, a web camera, an orientation sensor to be placed on the user’s helmet, a notebook, a pen and a tablet with markers (Reitmayr and Schmalstieg, 2001). Based on ARToolkit (Human Interface Technology Laboratory at the University of Washington, 2004), this system processes the images captured by a camera looking for specific targets. The system overlays the recognised image by a virtual object every time that it recognises a target. The user’s movements are also recognised by the system through the cameras and acted over the virtual objects. In one scenario, two players with different points of view sit at a virtual chess table and manipulate virtual pieces. The Studierstube is the platform used by the players to share the VE. Reitmayr and Schmalstieg describe a CARS for outdoor navigation and information browsing (Reitmayr and Schmalstieg, 2004a). The equipment used to perform the AR system outdoors does not differ much from the one presented on the previous project. Additional integrated hardware consists of a GPS receiver and a wave-LAN network adapter to enable communication with a second rover user. As an example of the application, they developed a tourist guide: the system helps the tourist by indicating the best route to visit interesting locations spread in an area of the City of Vienna, and provides access to useful information related to the area.

88

Information about the city is presented to the users in different ways on the HMD, working as an enhancement or complementary information over a building, images or even text. The users interact with the interface through a 2D touchpad placed on the user’s belt or carried as a handheld. All the devices connected to the notebook are managed by the Studierstube, as the collaborative environment.

Figure 3.18 Additional Information Overlaying a Historic Building (Reitmayr and Schmalstieg, 2004b)

Figure 3.14 depicts an AR view of a tourist attraction overlaid with extra information. The yellow shapes indicate that more information can be acquired about the site through the user’s interface. Figure 3.15 illustrates a box containing text and images supplied through the AR visualisation when the yellow shapes shown on Figure 3.14 are explored. On their website (Interactive Media Systems Group Institute for Software Technology and Interactive Systems Vienna University of Technology, 2004) it is possible to explore other projects. There are researches involving: AR aided surgery, AR locator for publications in a library, VR slide show, occlusion in collaborative augmented environments, etc.

89

Figure 3.19 Additional Information Explored by the Users (Reitmayr and Schmalstieg, 2004c)

In the collaborative AR for outdoor, introduced by Reitmayr and Schmalstieg (Reitmayr and Schmalstieg, 2004a), the authors mentions the low accuracy on GPS positioning. The problem could be reduced, but not completely solved, once the DGPS feature is enabled on the receiver (see more about this subject in §3.7). Columbia University AR Projects Columbia University (Columbia University Computer Graphics and User Interfaces Laboratory, 2004b) has introduced an AR project named Spaceframe (Webster et al., 1996). The project is a test bed of a spaceframe construction aided by an AR system that indicates which piece of a collection should be manipulated in order to assemble a diamond shape. The pieces look very similar, generating confusion about which should be handled by the users. The hardware used consists of a see-through stereoscopic colour HMD with integral headphones and orientation tracker. The position tracking is provided by an Origin Instruments DynaSight (Origin Instruments, 2002) optical radar that tracks small LED targets on the HMD (Columbia University Computer Graphics and User Interfaces Laboratory, 2004a). The system uses verbal or textual instructions to guide a user to a pile containing required parts. It sends confirmation through a barcode reader on whether the user has the correct piece, as well as indicating how to assemble the component. At the same time that the audio and text instructions are conveyed to the user, a virtual object is positioned where the frame should be placed.

90

Feiner (Feiner, 1997) also described the Touring Machine, created to combine 3D graphics of AR with the freedom of mobile computing. The hardware design is based on a DGPS receiver, a see-through HMD, an orientation tracker, a notebook, a handheld PC and a radio modem. This project was developed under the Mobile Augmented Reality System (MARS) concepts (Höllerer, 1999). The application scenario in Feiner’s paper (Feiner, 1997) introduces a user who is wearing the equipment and is standing in the middle of Columbia University. Around the user, buildings are labelled with virtual texts displayed on the see-through HMD. The interface between the user and the system is composed of a menu displaying a set of options. With the help of a touchpad, mounted on the back of the handheld, the user can select the options that give information related to the buildings around, departments and the user’s location. Finally, the chosen option is sent as a Uniform Resource Locator (URL) to a web-browser running on the handheld. The system can communicate via wireless with the web-servers to collect any data that is not cached. If the user stops for a short time in front of a building for example, the label displayed on the middle of the screen changes from grey to yellow and then to green, opening a second menu bar, which offers more links about the place. When the building is selected an indicator remains pointing at the position, then it changes colour, from green to red, if the user’s head turns more than 90 degrees from the target. The system runs on two different platforms: the COTERIE distributed VE infrastructure, based on Modula-3 (Wikipedia, 2008c), for the outdoor backpack system and its indoor simulation version, and Java/Java 3D (Sun Microsystems, 2008) for the indoor desktop and immersive user interface, as well as for the main database interface (Höllerer, 1999). Improvements made in related projects, including the positioning of labels and information on the VE, are described on Bell (Bell, 2001). The distribution of information and annotations inside the VE needs to be displaced in a way to avoid self-occlusion. Excess information can also cause confusion in the VE, making it

91

almost impossible to navigate through the real and virtual data. The paper describes the study of positioning of the information, size, transparency, priority, etc. Höller (Höllerer, 2001b, Höllerer, 2001a) introduced in his paper the early stages of an experimental mobile AR system that employs technologies to position the users in indoor and outdoor areas. These technologies include a ceiling-mounted ultrasonic tracker (used on covered areas) and a RTK-GPS system for open areas. For areas out of range of both systems, he described the results of the development of a deadreckoning approach that combines a pedometer and orientation tracker with environment knowledge expressed in spatial maps and accessibility graphs. According to the example cited in his paper (Höllerer, 2001b), the results achieved with the combination of pedometer and inertial orientation tracker were strongly disturbed by the magnetic field inside of the building. In the second example, an inertial orientation tracker was used and the results were more convincing, apart from the linear degradation of the orientation information due to drift. Finally, good results were achieved when using the combination of pedometer, inertial orientation tracker and environment knowledge. The system is based on a map of the area where the users is walking. It checks whether there is the possibility of a collision or if the directions given by the sensors are just noise caused by orientation drift. Spaceframe (Webster et al., 1996), is an example of an AR system designed for potential industrial applications. Any kind of situation where the users need to integrate their skills with instructions provided by a machine can be shaped on the base of this project. Examples of its application are: automobile industry, aerospace industry, CEn, etc. The system is being modified to support more than one user working on the same frame in real-time, as stated on the website (Columbia University Computer Graphics and User Interfaces Laboratory, 2004b). The Touring Machine (Feiner, 1997) introduced an AR application with a suitable user interface. The evolutions described on Bell (Bell, 2001) and Höller (Höllerer, 2001b) complements the issues of an AR system where the GPS signals are not available. Tinmith

92

The Tinmith system (Piekarski, 1999, Piekarski, 2001b, Piekarski, 2001d, Piekarski, 2001c, Piekarski, 2001a, Piekarski, 2002) from the University of South Australia is a complete architecture project developed for AR purposes. As a wearable system, the equipment attached to the notebook does not differ much from what has been previously introduced in the other AR systems. The main differential is in an interface approach named Tinmith-Hand. Custommade pinch gloves are used to control a finger-mapped menu through sensors placed on the tip of the fingers. The menu system handles eight complex pre-programmed tasks displayed in the bottom of the see-through HMD. Also on the extremity of the thumb in the gloves there are two targets (two small squares with a symbol painted in black and white) that are recognised by the system through to ARToolkit’s (Human Interface Technology Laboratory at the University of Washington, 2004) image recognition. The images captured by a web camera mounted on the user’s head are analysed by the system. Each time that ARToolkit recognises the target it calculates the position according to the tilt sensor, also placed on the user’s head. Figure 3.16 shows the wearable system.

Figure 3.20 Tinmith Wearable System (Piekarski and Thomas, 2002)

With the glove’s targets locked, the user can explore the enhancements achieved by Tinmith-Metro (for outdoors environment) and Tinmith-VR (for indoors environment). Tinmith-Metro (Piekarski, 2002) allows users to capture the outdoor design of a building and its structures using beyond arms reach manipulation techniques. Through Constructive Solid Geometry (CSG) the users can shape

93

complex objects using their hands. The results can be achieved from unions, intersections, or differences between primitive 3D objects (as pyramids or spheres, for example). The user has a choice to position and select the object using 2D or 3D techniques (Piekarski, 2002). If the object to be designed is more complex than can be made with the primitive forms, the users have the option to use the “infinite planes technique” (Piekarski, 2002, Piekarski, 2001d). Two examples are given in order to clarify how the user can design the objects: • Given the location of the thumbs (Piekarski, 2002), the system

overlays registered 3D axes. At the same time, a 2D flat cursor is overlaid on the top. After the user has positioned the cursor with his hands it is possible to activate the select mode using one of the pre-programmed tasks offered in the menu. The users create a virtual mask that superimposes the real object selecting one point for each corner that shapes its boundary. The plane created can be extruded in order to be modelled, with the same dimension of the original, or assembled with other planes created during the process. • A 2D cross-named eye cursor is displayed in the centre of the seethrough HMD. The user needs to match the cross with the corners of the object and select the point. The final result is an object with irregular shape. The process to acquire the planes is repeated until the whole building is modelled. Once set, the planes can be grouped in just one object and processed to be combined with another one. When the model is completed, the VE is fed with the new virtual object. The Tinmith-VR system is used for indoor environments and has the same modelling techniques offered by the Tinmith-Metro system. As cited on Piekarski and Thomas (Piekarski, 2002), the graphical quality of objects created is largely limited by the accuracy of the tracking hardware employed. DGPS has been used for positioning, with a claimed accuracy that ranges from 50 centimetres to 5 metres. Augurscope 94

The Augurscope (Koleva et al., 2001) has been developed at the Mixed Reality Laboratory (MRL) as part of the Equator (Equator, 2005) project. It is a portable MR interface for outdoor use, supporting both AR and AV. The Augurscope is a wheeled stand designed to carry equipment involved in the positioning and display of the VE. On the base of the wheeled stand there is a CPU, which is powered by batteries, which are covered by a shell. Arising from the centre of the base, an arm supports a screen which functions as a window to the VE. This small platform is movable. A compass and a tilt sensor are used to give the orientation of the Augurscope. A DGPS receiver calculates its position. The sensors are integrated on the screen of a CPU that has a wave-LAN card transmitting the information to the computer placed on the base. The data are computed, rendered and then re-transmitted to the display. The system has an attractive and user-friendly interface which facilitates the manipulation and exploration of the VE. The users can manoeuvre the wheeled stand with the advantage of not having to carry any equipment in a backpack. Figure 3.17 shows the Augurscope.

Figure 3.21 Augurscope (Benford et al., 2003b)

Currently the project has its test-bed at Nottingham Castle, where users can see and explore how the outdoor environment of the castle was in the past. For each direction

95

that Augurscope points the user can see the walls, houses, and towers that were once placed there. Two potential applications for the Augurscope are: cultural heritage and environmental planning. In terms of planning, it could be used to explore scenes from the future as a consultation over designs for new buildings, transport systems and other public facilities. Augurscope is marked by the design. It is an easy-to-handle device, which almost invites the users to play with it4. The intuitive handle allows even children to manipulate it. If the user does not like to use the wheeled stand the display can be detached from the stand arm’s support. Because its communication is done via waveLAN with the base, the display always stays within range of the wave-LAN transmission. The display tilt has been exaggerated so that for every sensed degree of tilt two degrees are rendered (Benford et al., 2003a). Additionally between 20˚ to 45˚ the camera pulls upwards. At 45˚ (the limit of sensible movement), the camera tilts to 90˚, raising several metres into the air giving a bird’s eye view to the users. This procedure not only masks the limits of the tilt sensor as well as helps the users to locate areas that were not yet explored. The lack of accuracy of the DGPS was substituted by an interpolation between the previous point and the actual positioning. Thus, with each three or four metre movement the system computes the new positioning. For instance, if the user moves only one or two steps and stops, the system does not compute this change of position. Whereas if the user pushes the trolley two more steps and again stop, the system identifies this movement, not as a drift of the DGPS system, but as a real change of user’s position. Augmented Reality for Subsurface Data Visualisation The Institute of Engineering Survey and Space Geodesy (IESSG) (IESSG, 2008) and the School of Chemical Environmental and Mining Engineering (SChEME) have
4

Based on a demonstration on October 2003

96

developed in cooperation a prototype of an AR system aiming to show the location of underground pipes for engineers and workers involved in a excavation. As cited in Roberts (Roberts, 2002), the risk to a worker of striking a live power cable is very high when there are no plans describing what lays under the ground. This situation has already claimed eighty six lives and caused more than three thousand injures over the last ten years – based on a study conducted by the Automobile Association of Great Britain in 1997. Even if 2D-charts exist, they can be misinterpreted, and the cost per day at digging a hole in the street is high. The developed system integrates RTK-GPS and an Inertial Navigation System (INS), enabling centimetre-level positioning accuracy. Apart from the receiver, which weighs 1.2Kg, the other equipment involved is very light and compact, which makes the whole set suitable to be carried in two small packs. Figure 3.18 shows the RTKGPS receiver placed in the user’s backpack. Another pack then contains a minicomputer connected to a pair of digital goggles, a radio antenna, a digital compass and a tilt sensor.

Figure 3.22 AR Sub-Surface System (Roberts et al., 2004)

Slightly different from the see-through HMD, the digital goggles have a camera on the front, where the RE is recorded. The images received from the camera are placed as textures over a digital rectangle, wide and far enough to be set as background on the VE. Between the background and the user’s view the VE is positioned also as a

97

digital image, respecting the angles of the compass, tilt sensor and RTK-GPS positioning. Figure 3.19 shows the VE projected over the RE. Roberts mentions two other applications for the project, which have been tested in a talc mine in Luzenac, France. The first idea is to supply information through the goggles that is not easily visible for workers in the mine, such as faults, aquifers, old workings, obscured rock surfaces, etc. The second idea is the visualisation of a grid with the geology of the terrain superimposed on the mine face and the visual location of different minerals found but not extracted from the ground.

Figure 3.23 VE Projected over the Real Environment (Roberts et al., 2004)

It is clear that the system, with its focus on sub-surface work, could be used in the same way to explore any sort of AR environment. The author had the opportunity to test the equipment. The system passed the impression of being sufficiently precise for the proposed objective. Dr. Hollands is one of the collaborators in the development of AR for Subsurface Data Visualisation. In an informal interview, the differences from this project to the proposed research introduced in this thesis are:

98

•

The visualisation approach is slightly different. As previously

explained, the images projected onto the pair of goggles are totally digital, making this quality dependent on the quality of the camera and displayers. • • The platform used for the development of the system was DirectX and There is no collaboration between users. The equipment runs in a stand the source code is not in the public domain. alone mode. There are no audio or video channels for the users to exchange ideas and points of view. ARMobile, AROnSite, ARWebCam, ARPhone The Technical Research Centre of Finland (VTT) worked over several researches exploring VR and AR in CEn applications. In the paper (Woodward et al., 2007a) they document the use of applications like ARMobile, AROnSite and ARPhone in the planning and construction phases of VTT’s new head offices named Digitalo. Using basically off the shelf hardware devices the system explored allowed users to explore a VE outdoors. For the positioning perspective they report the use of image pattern recognition through ARToolkit (Human Interface Technology Laboratory at the University of Washington, 2004) placing big printed markers in the streets to be recognised by the system. This solution is combined to another system that also indentifies other features in the RE to help to stabilise the system – when markers are not detected. ARMobile (Figure 3.20) is a system created over a miniature PC enabling rendering of 3D models, adding new features or light adjusting in real time. The paper describes the enhanced version of ARMobile called AROnSite (VTT, 2008, Woodward, 2007) that does not require markers to perform outdoors AR but a GPS. With this system, users are able to place features (i.e. a building) in a specific place using Google Earth (Google, 2006) and explore it using the AR system.

99

Figure 3.24 ARMobile (Woodward et al., 2007b)

ARWebCam (Figure 3.21) is an experiment based in a web camera placed over the roof of one of the buildings that surrounds the construction of Digitalo. The system had some knowing land marks and superimposed virtual images of the new building during the construction phase. Users could compare the actual development of the construction with what is really done or how the place will look after the conclusion of the works. The video streamed from VTT’s website already contained the VE blended, making no requirement of an extra software in the users side.

Figure 3.25 ARWebCam (Woodward et al., 2007c)

ARPhone is a prototype of a system aiming to delivery AR models in pocket devices like mobile phones. They are expecting to delivery 3D view of objects in real time once such devices become more powerful and common between people. Users will need only to point the phone camera to the place where the building is being planned to see how the environment will look in the future.

100

3.7

Global Positioning System

The presented thesis was co-funded by IESSG at The University of Nottingham (The University of Nottingham, 2008). The institute is well known as a centre of excellence in satellite navigation and positioning system, photogrammetry, remote sensing, sensor integration, and GIS. The idea for this project was created inside of IESSG and GPS as positioning system become a natural choice to track outdoor users when testing the CARS created. It is part of the technology review a research to compare most used GPS approaches on AR projects. The results guided this research to the RTK-GPS methodology due to: • Accuracy – RTK-GPS can provide coordinates with approximately 5cm in real time. No other system outdoor can provide such accuracy with such reduced infrastructure. • Infrastructure – to make RTK-GPS coordinates available the user need a reference and a rover equipment – in some cases the reference equipment is substituted by augmented systems. Both equipment should not be more than 20Km far from each other for the radio transmission of the corrections. There is no need of triangulation, cameras for partner recognition, several cell signals, etc. • • • Availability – GPS signal works 24h days, 7 days week in any weather condition. Size of Equipment – compared with other high accuracy GPS systems like DGPS the size of the equipment is the same, as well as the weight. Restrictions – RTK-GPS works over the GPS signal. If there is a limitation to receive the signal like multipath for example, the same limitation will affect other GPS methodologies. The following paragraphs are a review of GPS methodologies. This review is important once it aims to inform the reader (especially those without strong GPS background) the basic principles of GPS that eventually are misunderstood or not comprehended at all. GPS has a great value but also has its limitations. This subchapter was created to “briefly” explain why there are limitations and how different

101

GPS methodologies can provide results with so big differences in accuracy. Some tests were performed to demonstrate how far the coordinates could be when using different methodologies.

3.8

GPS Frequencies and Data – Brief Review

The GPS was primarily designed to provide worldwide instantaneous positioning to US military and allies. It was developed by the United States Department of Defence (USA-DoD), and it reached Full Operational Capability (FOC) on 27 April 1995. GPS is operational 24 hours a day and as a passive method it allows a virtually infinite number of users in any weather condition. At the moment there are twentynine operational satellites in orbit (United States Naval Observatory, 2005). There are different techniques to handle the GPS signals for civilian use. Each technique has been devised to fulfil distinct necessities of the user with different equipment configurations and at varied costs. The DoD of USA created the GPS system for military purposes. However, a second service was also created for civilian users. Following its original purpose, the military system is known as the Precise Positioning System (PPS) and the civilian system is called the Standard Positioning System (SPS). Non-military receivers are enabled to sense GPS signal through L1 and L2 frequencies. L1 frequency contains three different types of data: Coarse/Acquisition (C/A code) and Precise (P) code, the ephemeris and the almanac. L2 frequency contains the same data but not the C/A code. The P code is encrypted (Y code – anti-spoof) which makes it partially useless for civilian users. The encrypted P code is known as P(Y). The C/A code is composed by a sequence of 1023 “0s” and “1s” also called chips. The P code (without the Y code) is 23547x10 14 chips long. Both C/A and P(Y) codes are PRN and they are deliberately added to the signal. Then the signal becomes more reliable against noise and spoofing. The C/A code is repeated every millisecond and it is different for each satellite. Due to its length, the P(Y) code can be transmitted for 266.4 days without 102

repetition (Weimer, 1998). Instead of making the transmission continuously, the code is broken in 37 segments (one for each expected satellite) and reset every seven days to prevent non-military users to understand and to break the code. PRN is not truly random. The codes are known due to their retransmission over a period of time. Both C and P timing-codes are not a secret but the problem lays over the Y code, which is unknown. Nowadays, the algorithms in the receivers that are able to sense both L1 and L2 try to extract the Y code from the P code, making it as useful as possible for timing and positioning. Each satellite sends its status on the ephemeris data. It does not contain only the satellite’s “health” but the current date, orbit parameters and timing. The almanac of each satellite contains in a simplified form, an approximation of its orbit over a period of time and it is logged in the receivers. This information is fundamental when a GPS receiver is switched on due the need to know for example which satellites are crossing the sky at that moment. The observation centres on Earth often update the satellites’ almanac in order to improve the computation of its position by the receivers. The military use of the GPS signal enables the receivers to understand the binary code (Y code – anti-spoof) contained in the P(Y) code. The length of the P code helps the receivers to be more precise due to the algorithms embedded in hardware that can identify the location of the received segments along the entire code. The frequencies L1 and L2 are always the same. The differences are in the approaches used to work with them.

3.9

Stand-Alone GPS Positioning

GPS receivers can be single-frequency (tracking only L1) or double-frequency receivers (tracking both L1 and L2 frequencies). It depends on the hardware, on the embedded algorithm and on the type of antenna that are in the set.

103

The receiver in stand-alone mode uses the signals from GPS satellites to compute position without any sort of correction coming from other receivers or geostationary satellites. As any other receiver, it needs at least the GPS signal from four satellites to compute its three-dimensional position. Stand-alone positioning is based on the C/A-code and on the navigation message. The C/A-code is extracted from the signal received and aligned – cross correlation process – with a copy of the same code produced by the receiver. The difference in the alignment is the time taken by the signal to travel from the satellite to the receiver. This calculated amount of time can be converted to range if multiplied by the velocity of the signal propagation, however it is contaminated by several error sources as mentioned in (Bingley, 2003), such as atmospheric delays biases and clock offsets. This is why such ranges are known as pseudo-ranges. The atomic clock on the satellite is much more accurate than the quartz clock embedded on the GPS receiver, so with the C/A-code aligned, the algorithm must consider the satellite’s time frame and the receiver’s time frame. Figure 3.22 (Kaplan, 1996) shows an overview of what must be considered on the range measurement timing relationships:

Figure 3.26 Range Measurement Timing Relationships (Kaplan, 1996)

Where:
∆ t
Ts Tu

= geometric range time equivalent = System time at which the signal left the satellite = system time at which the signal would have reached the user receiver
t without δ D (theoretical) = System time at which the signal reached the user receiver with δ D t = offset of the satellite clock for system time (advanced is positive;

T δ t

' u

104

retardation (delay) is negative) – this information is transmitted in the ephemeris tu = offset of the receiver clock from system time Ts +δt = satellite clock reading at time which the signal left the satellite Tu' + tu = user receiver clock reading at time when the signal reached the user receiver The pseudo-range is then calculated as shown in the Equations 3.1 to 3.3.
∆t = Tu −Ts
Equation 3.1

pseudo range = Tu' + t u − ( Ts + δt )
Equation 3.2 pseudo range = ∆t + δt D + t u − δt Equation 3.3

The receiver clock offset determination does not solve the problem of error inherent
t on the signal. The description of δ D is given by equation 3.4 (Kaplan, 1996). The

Equation 3.1 shows the other error sources involved on the total time offset:
δt D = δt atm + δt noise & res + δt mp + δt hw
Equation 3.4

Where:
δ atm t δ noise t δ mp t δ hw t

& res

= = = =

delays due to the atmosphere receiver’s noise and resolution offset multipath offset receiver’s hardware offset

The biases caused by the atmosphere effects on the GPS signal can be partially mitigated by mathematical models, like the Klobuchar model, embedded into the hardware of the receiver. The Klobuchar models’ parameters are filled by the information received in the ephemeris, achieving up to 50% of success in the reduction of error.

105

Even with this number of errors sources over the signal, companies that produce GPS receivers claim accuracy between five to twenty-five metres on single frequency receivers when in clear view of sky. The error is even smaller when the receiver can use the signal from augmentation systems as discussed in §3.12. The pseudo-range errors can be reduced but not completely solved, even when using double-frequencies receivers. In this case the receiver uses the carrier phase to calculate errors caused by ionosphere delays. Sub-chapter 3.10 gives a review of the carrier phase approach. For further details on how to reduce the error to a minimum in single frequency receivers the reader is recommended the following literature: (Kaplan, 1996, Bingley, 2001, Aquino, 1992, Aquino, 1988, Aquino, 1999).

3.10 Differential GPS – DGPS
DGPS is by far the most frequently used positioning system in AR system development. There are handheld GPS receivers on the market able to compute and transmit DGPS corrections. Some of them are cheaper and lighter when compared to the receivers that will be introduced in §3.11. According to Roberts (Roberts, 1997) “DGPS was developed to increase the accuracies through the elimination of the clock, atmospheric and orbital errors”. Most of the DGPS receivers work with both timing codes and carrier phase positioning. The carrier phase frequencies correspond to wavelengths of the order of 19cm for L1 (1575.42MHz) and 24cm for L2 (1227.60MHz) (Pendleton, 2002). However, only timing code corrections are broadcasted to rover receivers. With DGPS positioning we will normally refer to receivers called base stations and rovers. The base station is placed on a pre-surveyed coordinate known as a base point. As the name indicates this receiver is left static on the base point, calculating and transmitting the GPS corrections for the rovers. The rover is the receiver that

106

changes its position from time to time. A rover equipment could be a receiver that is being used to guide a person, a car, an airplane or an AR system for example. The pseudo-ranges are contaminated due to the presence of several variables, including: clock offset, atmospheric delays, multipath offset and receiver’s hardware offset. The solution proposed by DGPS is to use the pseudo-range corrections calculated in a reference station for each visible satellite. The results are compared with the true range and the biases are broadcasted via radio-link or other sorts of dissemination to other receivers. The rover equipment also calculate their own pseudo-ranges for each visible satellite, which are then corrected using the adjustments received from the reference station. This type of positioning can achieve the accuracy in the range of two to five metres depending on the distance of the receivers. Pseudo-range corrections can only be applied if the reference station is receiving signal also from all satellites that reach the rover equipment. There is the possibility of the transmitted corrections not being appropriate for the rover equipment if it is too far from the reference station, as the method assumes the errors are the same at both reference and rover. In this case, ephemeris and atmospheric errors cannot be completely solved by this technique. As mentioned in (Aquino, 1999), “an increasing separation makes the ephemeris errors to propagate differently into the user and the reference station positions”. In addition, tropospheric and ionospheric delays can be different for receivers placed a significant distance apart. This problem is known as “spatial decorrelation” and poses the most significant limitation to the technique. A final concern is related to the time that the corrections are applied. A truly real-time correction is not possible due to computational processing delay and the transmission of the corrections. This is generally known as “latency” (Aquino, 1999). (Aquino, 1999, Kaplan, 1996, Bingley, 2003) are recommended for further reading about DGPS.

107

3.11 Real Time Kinematics GPS - RTK-GPS
The pseudo-range methods of GPS positioning have advantages and disadvantages. Single-frequency (stand-alone) receivers work only with timing codes. Hypothetically, if the C/A code was available on the L2, dual frequencies (in standalone mode) receivers could improve the computation by comparing the time taken for the code to travel from the satellite to the receiver in different frequencies (L1 and L2). The atmospheric delays could then be calculated with more accuracy using only one receiver. DGPS accent for this error by receiving corrections from reference stations. However, none of these methods can achieve a centimetre level of precision in real time. High accuracy in real-time can only can be achieved through the RTK-GPS method. RTK-GPS receivers process both L1 and L2 frequencies working exclusively over the carrier phase. The general concept of carrier phase positioning is reviewed bellow. Carrier Phase Positioning As cited in Bingley (Bingley, 2003), “high precision relative positioning requires the use of the carrier phase”. The first step to obtain the carrier phase is to remove the timing codes from the signal. As with the pseudo-range, the phase is also contaminated by errors related to clock synchronisation, ephemeris, ionosphere, etc. For more details in the techniques developed to model or minimise these errors the reader is referred to (Kaplan, 1996, Aquino, 1992). GPS receivers can only calculate the fractional part of the carrier phase leaving the integer number of cycles between the receiver’s antenna and the GPS satellite unknown. This integer number of cycles is called “integer ambiguity”. Figure 2.2 (Aquino, 1992) illustrates the carrier phase and the integer ambiguity on the signal received by the GPS equipment.

108

The measured carrier phase (Figure 3.23) is the fraction limited by the green and red lines on time zero (t0). On the next two epochs, t1 and t2, the satellites’ have moved, altering the fraction value. As demonstrated on Figure 3.23, the integer ambiguity does not change as long as the lock is maintained on the satellite (represented by the number of cycles between the two red lines). As the integer ambiguity is not known the receiver then tries to “guess” the number of integer cycles between the receiver and the satellite (Roberts, 1997, Bingley, 2003) when the satellite was first locked. As cited on Roberts (Roberts, 1997), “the range between the satellite and GPS antenna is a result of the fractional carrier phase plus the integer ambiguity”. This technique is better suited for static relative positioning otherwise the movements produced by the rover could affect the number of “guessed” cycles. This method normally calculates the coordinates of one station in post-processing, using the carrier phase data from adjacent stations.

Integer Ambiguity Integer Ambiguity

Fractional Carrier Phase calculated by the receiver

Figure 3.27 Carrier Phase and Integer Ambiguity (Aquino, 1992)

The approaches raised for integer ambiguity resolution can be found on (Kaplan, 1996, Aquino, 1988, Bingley, 2003). As long as the GPS signal is recorded continuously, the accuracy of the coordinates remains high. But it is difficult to maintain a clear line of sight between

109

the receiver and the satellites when in movement. Trees, bridges, narrow streets with high buildings and changes of height are some examples of what can contribute to block or reflect the GPS signal causing “cycle slips”. The cycle slips can be identified, for example, through the technique called phase smoothing. This technique uses the carrier phase to smooth the pseudo-ranges identifying when the GPS signal was blocked. Phase smoothing prevents the receiver to compute segments of the signal that suffered some disruption helping to keep the number of integer ambiguity. Eventually the technique invalidates itself due to the time that the signal was blocked. The receiver then needs to restart the computation of the integer ambiguity again. The reader is referred to (Roberts, 1997) for further reading about cycle slips recovery techniques. A solution for applications where movement is essential involves the use of a carrier phase base method known as On The Fly (OTF) Integer Ambiguity Resolution as cited on (Roberts, 1997). This method runs in real time and as explained by (Roberts, 1997):
the positioning can be established whilst in the field, and any poor reading and data can be rectified and taken again. The raw data may be recorded, in addition to the positions, and the positions computed by post processing using different variables and options as well as forward and backward processing.

This method requires more expensive equipment if compared with the methods suggested in §3.9 and §3.10. The main advantage is the centimetre level of accuracy in real time. To set up the RTK GPS, the coordinates of a reference point are necessary to transmit the corrections for the rover receiver. A vicinity of up to 20Km is the limit of the range in which the corrections from the static GPS receiver can be effectively used. The limit is based in the differences that could occur in the troposphere over the reference and the rover stations. Such differences could make the corrections transmitted not appropriate, resulting in a poor resolution for the integer ambiguity. Differently from the DGPS positioning system, RTK requires at least five satellites and transmits, from base stations to rovers, not only timing-code corrections

110

but also the corrections for possible solutions for the integer ambiguity. This solution can be based on stochastic estimation process or through the Hatch filter (Yunchun Yang et al., 2002) and there is a wide number of methods and variation aiming to improve accuracy and reduce the time to compute the integer ambiguity. In any case, the solutions go through a mesh of ambiguous combinations rejecting all those with higher residuals and selecting the smallest. If more than one solution suits the final answer the algorithm computes the next epoch’s data. Details of RTK integer ambiguity resolution are beyond the scope of this thesis. Further reading is suggested in (Yunchun Yang et al., 2002, Roberts, 1997).

3.12 Augmentation Systems and Network Services
Perhaps one of the main issues for those aiming to have a better accuracy when using a GPS receiver is the cost of the equipment. As shown on the previous subchapters there are different methods with an extensive list of different approaches to get better resolution in the positioning. The better the accuracy, the more expensive is the receiver. The problem of cost becomes more serious if a base station is required for the solution. If the designers aim to cover a large area with the corrections, more than one base station will be necessary to create a network. If the set up of a network of base stations is not viable, another solution can be provided through services offered by governments or private companies. The following paragraphs exemplify some of the services that help users, equipped with single or double frequency receivers, without a private base station, to obtain better precision on their positioning. The services offered have significant impact on the adjustment of pseudo-ranges especially for navigation (airplanes, boats, cars, personal, etc.), and safety issues. A private base station is only then necessary for remote areas where these systems cannot cover. In the near future, it is expected that the dual frequencies receivers have their prices dropped and their size reduced, providing general users with sub-metre positioning accuracy. Augmentation Systems

111

The Federal Aviation Administration (FAA) is a department of the government of the USA with a mission to provide safety and efficiency on American skies. The FAA has created the WAAS, declared operational in late 2003. As mentioned in (Trimble Navigation, 2002) “the system augments GPS with additional signals that increase the reliability, integrity, precision and availability of GPS signals required for the aviation industry”. The system works with twenty-five Wide Area Reference Stations (WRS) spread over the USA with the mission to receive all possible GPS signals from the GPS constellation. According to (Federal Aviation Administration, 2005a), the information collected is then sent to a WAAS Master Station (WMS). The WMS has the responsibility to create the messages that help GPS receivers to remove errors, increasing the system’s precision. The messages are transmitted using geostationary communication satellites. Perhaps one of the most reliable features in the system is a notification for the users whenever there is any error on the GPS system that could cause serious miscalculation on the user’s position. It is claimed by (Federal Aviation Administration, 2005b) that “messages broadcasted by WAAS improve GPS signal accuracy from one hundred meters to approximately seven meters”. Even though this system was designed for airplane landing (Category 1 – precision approach) nonaviation users can also be benefited. To make use of WAAS corrections, it is just necessary to have a receiver with a particular chipset. The European Space Agency (ESA) also offers, free of charge, the SISNET (European Space Agency, 2005), “a platform to provide worldwide access to the EGNOS System signal through the Internet”. The system works through the WAAS’ principle. The main difference is that the corrections transmitted to the users also contain information about the Global Orbiting Navigation Satellite System (GLONASS). The system comprises thirty base references, four master control centres and six up-link stations. According to (European Space Agency, 2005):
EGNOS signal is broadcast by two Inmarsat-3 satellites, one over the eastern part of the Atlantic, the other over the Indian Ocean, and the ESA Artemis satellite which is in Geostationary Earth orbit above Africa.

European GPS users are supposed to achieve a precision of five metres in their positioning without any extra costs of using this service. All they would need is an EGNOS GPS chipset enabled receiver on the GPS receiver. However, EGNOS’ users 112

can also receive corrections that are being transmitted via Internet. Torán-Marti (Torán-Marti et al., 2003) describes a handheld Personal Digital Assistant (PDA) integrated with a low-cost GPS receiver (without EGNOS chipset enabled) and a General Packet Radio Service (GPRS) wireless link to the Internet. The result is a Graphical Unit Interface (GUI) with pseudo-ranges, ephemeris and ionospheric errors being corrected based on the SISNet information. They claim that the correction applied on the signals received has its accuracy down to one to two metres horizontal and two to three metres vertical. Japan also has its augmented system called Multi-Functional Satellite-based Augmentation System (MSAS), which is based on the same principles described in WAAS system. MSAS is augmented with geostationary satellites called MultiFunctional Transport Satellite (MTSAT). Network Services In Japan, approximately two million GPS navigation-equipped cars are sold annually (Petrovski, 2003), and currently, there are 3.8 million GPS-equipped cellular phones being used. The chipsets implemented for cars and mobile phones are basically the same which are in handheld GPS receivers or even smaller. Petrovski (Petrovski et al., 2000) cites two approaches used to provide DGPS and RTK-GPS corrections in Japan: radio broadcast or the Internet. Asahi TV broadcasts DGPS and RTK-GPS corrections services encoded into a TV audio sub-channel signal. The main disadvantage lies with the covering distances between 40km to 100km. The RTK-GPS system usually works with base points positioned not more than 20km far from the rover equipment. With Internet based services, the disadvantage is the latency to transmit the corrections. Petrovski (Petrovski et al., 2000) proposes a solution using virtual reference stations taking advantage of the low latencies and high capacity of sub-channel signals where the users can select the nearest base point available to receive the corrections choosing which piece of information is necessary for particular purposes. This service is also supported by the augmented system MSAS.

113

By 2008, plans have been made to launch the Quasi-Zenith Satellite System (QZSS) with a constellation of three satellites, where at least one of them will be covering near zenith over Japan at the times. The advantage is a better coverage for GPS augmentation in urban canyons providing high accuracy for rapid moving vehicles. This system will also provide a wide range of services based on communication. Petrovski (Petrovski, 2003) gives a wide overview of the Japanese project. SWEPOS is the Swedish network that provides DGPS and RTK corrections to real time users and carrier phase raw data for post-processing. The network consists of 25 permanent base stations covering the whole country. SWEPOS network offers the following services for the users (Jonsson et al., 2003): • Post processing data via World Wide Web/File Transfer Protocol (WWW/FTP) in Receiver Independent Exchange Format (RINEX)-format. • An Automatic Computation Service on the SWEPOS web. The data collected on dual frequencies in RINEX format is sent to the service. After five to ten minutes the user receives via e-mail the corrected data. • The private company Cartesia broadcasts the DGPS corrections via Radio Data System (RDS) channel on the Frequency Modulation (FM) network. The accuracy achieved is between one to two metres. • The Wide Area DGPS (WADGPS) is also explored by the private company Fugro. It uses geostationary satellites for the distributions of the corrections. The accuracy between one to two and a half metres is achievable. There are several studies of RTK-GPS network corrections described in (Jonsson et al., 2003). The corrections are broadcasted via an FM-sub carrier. Apart from the advantages of this kind of system, such as ‘easy to use’ and no need of base station, the most significant achievement is the millimetre accuracy in real time even with strong indications of a disturbed ionosphere. More about SWEPOS can be found in (SWEPOS Control Centre, 2005). Ordnance Survey (OS) and Leica Geosystems offered the SmartNet to the British market. Currently using 96 reference stations, this system is designed to provide accuracy from centimetre level (RTK-GPS) to sub-metre (DGPS); or raw data for 114

post-processing (Leica Geosystems LLC., 2006). The new models of Leica GPS receivers are able to communicate to the servers via Internet GPRS and Global System for Mobile Communication (GSM). The communication can be done 24 hours per day, 7 days a week, with the promise of a highly dense network and highly redundant. The controller of the network, SpiderNet (Leica Geosystems LLC., 2005), decides which base station should send the correction based on the coordinates of the rover receiver. Usually there is a master station and few more auxiliaries working as backup in case of disconnection. IESSG is involved in this project “undertaking independent monitoring and integrity checking, to validate the user experience”. The institute is using its own technology to compare the results offered by SmartNet. This process also includes several receivers placed strategically, working as users to check the quality. Some of the advantages for users of SmartNet are (Leica Geosystems LLC., 2006): • High speed initialisation for RTK solution. • Redundancy due to the number of reference stations. • Quality assurance once an independent body is also monitoring the service provided. • The cost of RTK-GPS solution is cut almost in half, since a base station is no longer necessary.

3.13 Testing Single Frequency GPS, DGPS and RTK-GPS
Even though the RTK-GPS methodology was adopted as standard for this project due to its high accuracy in real time, some tests were performed to show how much accuracy other GPS approaches can provide. The intention here is just to show visually the differences between the approaches. The test was performed with receivers that were pushed in a trolley following a circular path with 40 metres of diameter. The centre of the circle was surveyed by all methods and the path shaped with the help of a tape measure. The tests performed are:

115

• • •

A single frequency GPS receiver (see §3.9). A dual frequency GPS receiver working in DGPS mode (see §3.10). A dual frequency GPS receiver working in RTK-GPS mode (see §3.11).

The single frequency receiver had its data accumulated for two minutes for each point. Because of this, the graph shown on Figure 3.24 has more coordinates displayed then other graphs. This configuration was chosen because of the popularity of such receivers especially because the low price. The graph shows the instability of the system to accurately retrieve the circular shape of the trekking. A spread of about 10 metres is observed at each of the surveyed points. All data was processed using the software Gringo (IESSG, 2002) which was used to create the images. This solution is clearly not the best if in a given project the expectation is for the positioning data to be consistent and accurate.

Figure 3.28 Single Frequency Receiver

Figure 3.25 shows the example of a single coordinate logged for approximately two minutes using a single frequency receiver. It is important to say that the receiver in this case is not moving. The several samples of this unique coordinate are the result of errors involving this methodology (see §3.9). The figure below clarifies the reason an AR system, as proposed in this thesis, would not work as desired if using this methodology.

116

C:\CAPRA\COORDENADAS\045446.G12.04O (Mobile Stand-Alone Solution) Lat (rms = 2.7 m), Lon (rms = 2.3 m) Distance (rms = 3.5 m, 95% = 5.4 m) 4 Latitude (metres from mean) 2 0 -2 -4 -6 -8 -10 -6 -4 0 2 4 6 -8 -2 8

Longitude (metres from mean)
P4 - Pseudorange and Phase Post-Processor © Copyright The University of Nottingham 1999

Figure 3.29 Single Frequency Receiver – A Single Coordinate

The results of the performance of the DGPS in a dual frequency receiver are shown in Figure 3.26. The circular shape has now been retrieved even though it is not exactly centred. The error is spread to all coordinates as a constant drift only in the Latitude.

Figure 3.30 Dual Frequency Receiver Performing DGPS

A final decision about which receiver would be the best can only be achieved if GPS receivers from different brands could be tested together at the same time and day. The data could then be post-processed and analysed. However, there is always a chance that the solution found would not be definitive, once there are different conditions that the receiver could be exposed to. Narrow urban canyons, different weather conditions, magnetic solar storms or even the chosen antenna of the receiver are just 117

some of the examples that would affect the results. There is no perfect equipment, but one that suits better the needs and budget of the project. The RTK-GPS solution is by far the best amongst all tested methods. The graph shows (Figure 3.27) a circumference drawn with the logged coordinates from the RTK-GPS receiver. All coordinates were collected and processed in real time. No post-processing was required, only Ski-Pro (Leica Geosystems LLC., 2003) to plot the graph.

Figure 3.31 Dual Frequency Receiver Performing RTK-GPS

These tests were particularly important, since they proved that different methodologies can provide totally distinct results. Researchers often try to implement projects requiring good accuracy using the wrong approach in relation to positioning. Eventually they misjudge the limitations of the methodologies and when the results are not as expected, an alternative solution, as reported in (Benford et al., 2004, Tamura, 2001a), is required.

3.14 Other Positioning Systems
For outdoor AR applications GPS is the best available solution. However other positioning techniques like cell phone or Wi-Fi triangulation and self-report positioning are also available to be explored. 118

The main advantage of using a mobile phone for positioning is the availability of the service. Mobile phones are connected to one or more antennas wherever the service is available, inside or outside buildings. However, the accuracy is based on the number of cells that the phone has in its range (Gomez M. et al., 2002) and on how much the signal was affected by the distortion along its travel transmission. Wi-Fi triangulation can use either the tags distributed in a delimited area, or the reference of pre-surveyed wireless points. In both cases, the places where tags or the wireless points are, must be pre-surveyed. The use of the tags gives more accuracy to the system, but it is limited to the area where they were placed. They are normally used as positioning systems inside buildings. Nowadays, wireless points are common in places with high density of offices due to the price and practicality of the system. The same happens in wealthy areas where houses have more than one computer. However, that does not occur when middle and working class areas are surveyed. The result is a positioning system with very poor accuracy because of the lack of wireless points. Self-reported positioning (Benford et al., 2004) is based on the users saying where they are. They explicitly add a new input each time that they are over a pre-surveyed point. This kind of positioning has as its major disadvantage the lack of freedom for users to explore AR environments from different angles. The limitations of GPS when used with AR systems are well known and perhaps one of the main frustrations for AR researchers is when the positioning system cannot achieve the desirable accuracy. It is often observed that researchers who need centimetre precision in real time would wrongly choose to use a modest set up of equipment. As introduced on sub-chapter 3.9, the number of variables that would cause errors on GPS is large, which makes single frequency receivers unable to produce results with good accuracy on positioning in real time. DGPS is the most used methodology, perhaps because most of the medium priced sensors available in the market already come with this feature. There are also an extensive number of libraries on the Internet that can help researches to create a 119

network of base stations and transmit corrections to the rovers. With the complement of additional software, it is possible to explore the hardware of the receiver and extract information that is not often available to ordinary users. However, this takes time for configuration and eventually the final result (the coordinates) is not as good as when using RTK-GPS. This thesis adopted the RTK methodology for positioning. It is a fact that this is the best positioning method for real time. Even though this could be the main reason for using it, this thesis also evaluates the advantages and disadvantages of using this method. Some of the advantages that can be mentioned without further research are: • • • The high accuracy can be achieved using only a couple of receivers. Usually the receiver falls back to DGPS when the signal’s lock is lost. In Top of the range equipment are not cosmetically different from receivers

this case, the positioning error increases but the system keeps running. enabled to compute DGPS only. The size, weight and shapes are the same. The disadvantages when compared with other systems are: • • The price of this technology is far higher than any other. RTK-GPS system requires the GPS signal of five satellites, at least. If the receivers are being used on narrow streets or in regions with poor satellite constellation (low or high latitude) the results may not satisfy the user’s necessities. • • Receivers should not be more than 20Km far from each other. Items of top-of-the-range equipment are heavy – about 1.2Kg and big – the size of a shoe box. The antenna is not embedded to the receiver, forcing the user to carry a pole for the antenna or a backpack with an adapted pole. There are a few cables that need to be connected and the battery consumption is higher than in small devices, which make them require stronger and heavier batteries. The final pack becomes more expensive because of the hightechnology embedded in the peripheral equipment. Antennas, for example, are highly sensitive for dual frequencies. The user must be aware that in order to

120

carry this equipment, he needs to have a bag that can carry the antenna, the receiver and batteries without damaging the equipment. Table 3.1 shows the systems that were reviewed on this thesis and the average accuracy in each of those.

Table 3.1 Different Positioning Systems Positioning System Accuracy Comment Mobile phone 30m to 500m Good precision is only achieved when 16 cells are being tracked at once (Gomez M. et al., 2002). Only few mobile phones can do it and Global System for Mobile Wi-Fi Self-positioning Single receivers code) Single receivers augmented frequency (C/A frequency using system < 5m vertical and < 3m horizontal 95% of time 2 to 5 metres Centimetre – level Same as above but requiring two receivers. Expensive technique. The system requires at least 5 satellites available to calculate it’s coordinate. Around 20m precise 5 to 20 metres Communication (GSM) modems. This system is based on a pre-survey and in areas well populated with Wi-Fi spots. The system is based on pre-surveyed points taking the freedom of the user to explore the digital environment. Minimum of 4 satellites and clear sky for better GPS signal reception. The receiver requires a special chipset that enables it to acquire augmentation signal as EGNOS or WAAS.

(C/A code) DGPS (C/A code) RTK-GPS (carrier phase double frequencies)

3.15 Combining and Integrating Technologies
All the researches presented above have one or more features that overlaps with the CARS introduced in this thesis. They are aiming in common the delivery of information in the most understandable way for users. Some of them aspire for an increasing of communication, collaboration and mutual understanding between users, but not all explore CVE. The way that users explore the VE also changes dramatically from system to system specially between VR and AR. Systems also divert when covering aspects like moving objects, animation, accuracy and user friendly. 121

For the perspective of this thesis such aspects are important and are integrated at the end of the research. • Aspects like the communication, collaboration and mutual understanding

can only be achieved as a result of the implementation/integration of hardware and software as well as the way that the system is exposed to users. • The VE explored by users when indoor requires a system flexible enough to allow other users working from different computers both indoors and outdoors. The position of users when indoors is controlled by a handheld device – mouse. When outdoors users must have a reliable and high accurate positioning system to identify where they are, which direction they are going to and which direction they are looking at. • Very few researches mentioned the possibility of animation and simulation. This is also an important aspect once it brings the CVE closest to the RE. Animation also helps users to understand better information like wind corridors or how a particular object moves after become operational. • The accuracy is not only necessary to know exactly where outdoor users are. It must be as much stable as possible avoiding jerking through the selection of the right methodology of positioning as well as the implementation of filters to reduce false movements. • As easy to work with the system as quick users start to interact and produce results. Users should not face any complex interfaces, especially outdoors users. For complex tasks outdoors users should have the remote assistance from indoor users.

122

Chapter 4
Design, Hardware and Software Overview
Previous chapters exposed the difficulties faced by researchers to make AEC working closer. The review also showed that companies try to push products and standards although the lack of a solid consolidation for what is best solution for CEn market. Researchers with focus on BIM solutions are more concentrated on what kind of information is relevant to store, how to drill correct answers from previous CEn developments or how to exchange information between different solutions – between many other topics. Although there are already tools enhancing 3D visualisation of data, like collision between layers for instance, it does still encapsulated on a VE explored on a PC where only few actors can have access. The market is experiencing a technological revolution and this process can make companies easily behind of high tech solutions. Even today communication in CEn projects is sometimes only possible through traditional methods making 2D sketches very much in use (Woksepp and Olofsson, 2006). Once technology is being introduced gradually in CEn companies (not only because the investment required but also which is the right technology a company should invest on) VR is becoming the first natural step to be explored. As mentioned in (Bouchlaghem et al., 2004)

123

VR models are one way to improve information handling, communication and understanding in construction projects. VR offers a natural medium for the users providing a three-dimensional view that can be manipulated in real-time and used collaboratively to explore design options and simulations of the construction process.

However, in a near future it is not difficult to imagine AR systems displaying BIM information for outdoors users (Woodward et al., 2007a, Woodward, 2007). This thesis shares the view of Woodward, standing perhaps one or two steps ahead of what CEn market is receiving as solution today. The following paragraphs extend the idea of how this thesis aims to achieve the enhancement of communication, collaboration and mutual understanding using CARS. The design overview introduce here creates bridges that integrates the several technologies reviewed in Chapter 3 and some knowledge from Chapter 2 in order to address what is proposed as requirements of the project on Chapter 1. This chapter also reveals the hardware and software used during the implementation of this thesis. The description of what was used complements the idea of why this design was choose and already introduces the way that it was implemented.

4.1

Conceptualisations of User Groups

Figure 4.1 is an illustration of how the project should be presented when completed. In the left side of the illustration there are a couple of actors exploring the CVE and talking about what they can see. They share this CVE with a user outdoors through the computer screen. The actor outdoor (Figure 4.1 right side) explores the CARS though the images projected before his eyes. This exchange of information is explored by interaction between actors through the use of a voice channel and the movements of the avatars (human representation) in the CVE. Such interactivity stimulates: the comprehension of obscure points, the perception of new ones and the creation of new conclusions, helping actors to make decisions of what to do next in the area that is being remodelled.

124

There are two main groups that benefits from the results of this thesis: the office users and the field users. Initially, these groups were not created to distinguish people with different levels of knowledge in CEn. However it is imagined that eventually the combination of who is working in the system as actors in the office and who is in the field reflects different levels of expertise. Example might be the situation where an engineer (office user) explains to a potential customer (field user) where the stairs of the house would be placed.

Figure 4.32 Proposed CARS

Office User The office user (Figure 4.1 – left side) is the actor that explores the CVE displayed on a computer screen with the help of a standard mouse. This actor could be either an AEC or a customer wanting to know more about the new construction. The objects designed to populate this CVE are the representation in the natural scale of everything that could be placed in the RE. The actor has the freedom to explore the CVE without restrictions. The avatar can be directed to go through the objects, take panoramic views and even fly. The first example of configuration can be exemplified when only office users are working on CVE. They can explore the CVE at the same time with the freedom to go

125

to all directions or move all objects. These actors are connected via Internet with the facility of a voice channel. A second configuration is the CVE being explored by both office and field users. Again the voice channel is available and users can interact at the same time that they are exploring the CVE. Field User The actor that explores the RE where a new construction is being built is called field user (Figure 4.1 – right side). As with the office user, the field user is also represented by an avatar inside of the CVE. Field user explores the CARS using the movements of his body instead of a mouse from a computer. The person in the field has with him a notebook and a RTK-GPS receiver in the backpack. The GPS coordinates are associated to the embodiment’s movements. A crash helmet, adapted to fit the inclinometer and the electronic compass, logs the movement of the user’s head updating the angles of where the avatar should look at. A see-through HMD is also adapted in the crash helmet to alleviate the weight over the user’s nose. The images generated are then transmitted to a see-through HMD giving to the field user the view of both the RE and VE projected before his eyes. The see-through HMD also provides an output for an earphone. A microphone is attached to the notebook and they are used to voice communication with other users. A wave-LAN link is used to create the transmission between computers. The office user does not necessarily need to be in the wave-LAN’s range. For instance, users could collaborate over the Internet connected to a host that is in range of the wave-LAN router.

4.2

Design Overview

The diagram on Figure 4.2 is a more detailed view of the design of the project showing what actors (field and office users) have, as hardware and software, when the project is fully operational. The diagram also shows how the connections are

126

between different parts of the system created. Two platforms are presented in this graphic: • • MASSIVE-3 – working as the manager and render of the CVE (see §4.3). Equip – working as integrator between external devices and the CVE,

controls the data base Coordinates, and control the queries and the replies of clients (see §4.3). As general information, each computer involved has a copy of MASSIVE-3 and Equip. They also have a copy of a file containing the entire CVE and the design of the objects that populates it.

Figure 4.33 Design Overview of the Project – Field and Office Users

The diagram above does not represent the design used between office users only. This diagram is presented in §5.2. The following paragraphs go through the design of the project although some of the elements of the diagram (trader, API/comm port and Integrator) receive complementary information on chapter 5 due to their complexity and the level of details.

127

Firstly it is presented the left side of Figure 4.2 (office user) and then the components of the right side (field user).

4.2.1

Office User Design

As mentioned before each computer used by office and field users to interact with the CVE contains the files that populate the VE as well as the files that shape the avatar. Both the files are interpreted and rendered by MASSIVE-3. MASSIVE-3 – Office User Design Actors can use the mouse to navigate inside of the CVE. All the rendering process of the CVE is perform by MASSIVE-3. It contains a manager that keep continuously listening for new event inside of the CVE World data base (Figure 4.2 – right side). Office users can click and hold the right button of the mouse and move it forwards, backwards, and both sides. All these movements are logged and replicated to clients sharing the same CVE. If the avatar is facing to an object the actor can click and hold the middle button. A ray comes from the front head of the avatar towards the object selected. The object can be repositioned in another place and angle. Such movements and clicks are handled by MASSIVE-3. The repository of the CVE is managed and hosted by only one of the computers involved (Figure 4.2 – CVE World). Equip – Office User Design Equip contributes in different ways: • • • The handling of connection between computers. The connection with the data base (Figure 4.2 – Coordinates). The handling of incoming data from external devices.

The use of this platform is not compulsory in the office user side. It could be discarded with actors interacting with the CVE in a simple way, like exploring the CVE or moving objects. However it is requested if office users would like that an external device have its input logged by the Coordinates data base.

128

In the diagram, it is presented the input from a keyboard. To make the keys pressed available in the Coordinate data base (Figure 4.2 – right side) a Java code was created (see §5.3 – keystroke) to keep listening from the keyboard. The following steps document what happens when an office user presses a key in the keyboard: 1. 2. 3. 4. 5. 6. World). 7. MASSIVE-3 clients keep listening for any change in the data base (Figure 4.2 – CVE World). When there is a change the CVE is then modified – in this example CARS window and CVE window. All these steps cover a view of the process involved to make the system running. The complete set of programs running as well as their functionality can be found in §5.3.1. Audio Channel – Office User Design The audio channel (see §5.3.1 – audio_server and audio_test) works completely independent of what is being performed by Equip and MASSIVE-3. The communication is directed between computers through the data communication. It is only necessary to create a server in both computers and to set the acceptance between parts to exchange the audio packages. The key pressed is read and sent through Equip Application Program The key stroked is then logged in the data base (Figure 4.2 – Coordinates This new log triggers an event in Integrator (Figure 4.2 – left side – see Integrator processes this new information (translating the key stroked for Integrator updates the management of MASSIVE-3 (see §5.3.1 MASSIVE-3 management logs the data in CVEWorld (Figure 4.2 – CVE

Interface (API) (see §5.3.1 equip_net_trader, trader). – see §5.3.1 equip_data_server). §5.3.1 Integrator). the real meaning of it). equip_mas3_mgmtserver).

129

4.2.2

Field User Design

The field user side (Figure 4.2 – right side) is slight more complexes than the office user side. On this side there are two data bases running as well as two external devices being systematically checked for new inputs. The configuration of the data bases was selected on this way because historically the project always performed with the faster computer in the field user side. A new device is placed straight to field user whenever a part is exchanged (new hard disk for instance) or a new notebook is purchased. MASSIVE-3 – Field User Design The CVE manager running in the field user side is exactly the same version of the one running in the office user’s computer. Field users could also manipulate the CARS using a handheld keyboard (see §5.3) (it runs in the same way described for office users) and mouse. However this option proved be very difficult for users to get used to (see §6.2.1). All the manipulation of the CVE is performed then by users in the office (see §6.2.2). The CVE World data base is automatically updated when a new image is rendered in CARS. Field users can see the changes before their eyes through a see-thought HMD. The difference relies over an external application that feeds MASSIVE-3 with new information to be rendered. This application is triggered by new data arriving in Coordinates data base logging the data in CVE world (see §5.3.1 – equip_mas3_mgmtserver). Equip – Field User Design As in the office users’ keyboard interactions, field users’ coordinates (GPS, inclinometer and compass) are stored in a data base created and managed by Equip (Figure 4.2 – Coordinates). Although there are two devices connected with Equip API the process is basically the same described when an office user presses a key in the keyboard. The RTK-GPS updates its position every one second as set in the receiver. A Java class keeps reading from the communication port every new data. When a new position arrives it

130

is processed and then published in the data base – Coordinates. The data is then transferred to the CVE world. The same process is performed with the inclinometer and the magnetic compass used to indicate the direction that field user is looking – all these steps are introduced on §5.3.1. Animated objects are also controlled in the right side of the diagram. A file containing the coordinates of where the animated object should follow is interpreted by equip_mas3_entitygenerator (see §5.3.1). The new coordinates are published in the data base Coordinates triggering equip_mas3_mgmserver. The process then is the same as described when a key is stroked or a new GPS coordinate is received. Audio Channel – Field User Design The process is exactly the same as described with the office user. A server is established and the connection between computers set. Actors can then talk to each other with good sound quality.

4.3

Hardware and Software

The following paragraphs describe the hardware, the platforms and interpolator software used on the implementation of this thesis. This text goes through with more details over: • • The GPS receiver and the messages that were extracted from it.

• The compass and inclinometer and also the messages that came from it. The see-through HMD and the angles to achieve parity between what is displayed with what is seen. • The notebook used to process all the integration. • The handheld keyboard. • The wave-LAN set up. • The interpolator software used to create the models after the survey. • Equip. • MASSIVE-3.

131

4.3.1

Hardware

In most of AR projects the price of the receivers may be a relevant factor when the GPS technique is to be used. It is possible to buy for a few hundred dollars good handheld GPS receivers that can compute the position through the C/A code, as mentioned in §3.9. Such equipment is good enough for people that are walking in the moors, for example. The accuracy is around a few tens of meters making this device a useful tool when combined with a map for example. However, such receivers are not convincingly good enough for what this work is proposed to. As described before, the drifts can cause not only discomfort to the users but also a misalignment between the real and VE. GPS Receiver The Leica SR530 is a twenty four channel receiver that tracks both the L1 and the L2 carrier phases with high accuracy and with an on-board RTK capability (Leica Geosystems LLC., 1999). The receiver can offer in RTK mode, two centimetres of accuracy in normal conditions of use, which is appropriate for the application that is being developed. The hardware is modular but reasonably heavy when in conjunction with the other equipment in the backpack. A mini-pack with the receiver, plug-in battery, radio modem, radio antenna and GPS antenna weights around 3.7Kg. The radio for the RTK data transmission can be easily connected with the receiver and the reception is good even for long distances. Leica reports that the maximum RTK range is usually 10Km. The receiver has 16Mb of space in a flash card. A QWERTY keyboard, a numerical keypad, function keys, and arrow keys are available to the system’s interface. There is a screen coupled with the keyboard to activate the submenus, each one with different procedures and very clear, avoiding possible confusions or even mistakes from the user. After the receiver is setup, it is possible to take out the screen-keyboard and leave the receiver working without the risk of accidentally pressing a key. The connection between the receiver and the notebook is done using standard communication interface RS-232C and a 9600 baud rate. It is possible to configure the messages in National Marines Electronics Association (NMEA) or American Standard Code for Information Interchange (ASCII) format. For the present project the second option was chosen. More details describing how

132

the messages are collected can be found on §5.3.1. For the reference receiver, a set of two small Camcoder batteries is enough for approximately one hour and a half. The short life time of the batteries is due to the fact that the reference receiver is sending RTK messages to the rover receiver via radio, which makes it necessary to connect another external battery for long periods of use. The rover receiver can work well for a few hours using only two small batteries. The AT502 is a dual frequency antenna used both in the reference receiver and the rover receiver. With high-accuracy, it is small, light, precision centered and has a build-in groundplane. The antenna orientation is not required. Using the system flexibilities, it was possible to convert the output coordinates from the World Geodetic System 1984 (WGS84) to OS datum with the help of a small program stored in the receiver’s memory. The output format chosen is the Leica Local Positioning and Quality (LLQ). The description of this format is shown in Table 4.1 as was presented in (Leica Geosystems LLC., 1999):
Table 4.2 LLQ – Leica Local Position and Quality

$GPLLQ, hhmmss.ss, ddmmyy, xxxx.xxxx, M, xxxx.xxxx, M, x, x, xx.xx, xxxx.xxxx, M *hh <CR> <LF>

Header, include Talker ID, message sent form Receiver Universal Time Coordinated (UTC) time of position UTC date Grid Easting, metres Metre (fixed text “M”) Grid Northing, metres Metre (fixed text “M”) GPS Quality, “0” = not valid, “1” = GPS Navigation Fix, “2” = DGPS Fix, “3” = RTK Fix Number of satellites used in computation Position Quality, metres Height, metres Metre (fixed text “M”) Checksum Carriage Return Line Feed

Figure 4.3 shows the Leica SR530 connected with the screen-keyboard. In the same figure: GPS antenna (white plate), radio antenna (black stick), flash memory (in the bottom of the receiver) and radio transmitter/receiver (with a red label on centre of the figure) are shown. 133

Figure 4.34 GPS Receiver, Antennas, Flash Card, and Radio

The receivers were lent and supported by IESSG that has the know-how necessary to supervise the use of this equipment for this project. Magnetic Compass and Inclinometer The TCM2-50, by Precision Navigation, is a very light circuit board equipped with an embedded RS-232C interface on its 10 pin connector (PNI Corporation, 2004, Heike, 2002). It was used to capture the movements of the user’s head through the values calculated by the compass and inclinometer. Table 4.2 shows the pins and describes its functions:

Table 4.3 Pin Function of TCM2-50

Pin 1 2 3 4 5 6 7 8 9 10

Function V Supply (5V +/-) V Supply (6 – 18V unregulated) Power Ground RxD (RS232) Input TxD (RS232) Output Mouse Output Data Ground Analog Output 2 Analog Output Data Ground

The RX–input is tolerant to signals from -15V to +15V. The minimum signal difference is ±5V. The TX–output only produces a -5V to +5V signal (Heike, 2002). The electronic compass and tilt sensor utilizes a fixed 3-axis magnetometer in conjunction with a 2-axis inclinometer (tilt sensor). As the system moves and tilts, the TCM2’s strapped-down magnetometer continues to calculate accurate field 134

measurements. The inclinometer provides accurate pitch and roll information to the TCM2’s microprocessor, which then mathematically corrects the tilt (PNI Corporation, 2004). As output the device gives: • • • • Compass heading (360 degrees). Roll output (±50 degrees).

Pitch output (±50 degrees). Temperature output (-25° C to 100° C).

The following limitations have to be considered when operating the TCM2-50 module in any application (PNI Corporation, 2004): • • • Acceleration: earth gravity and artificial acceleration can become a Settling Time: after a sudden change in heading the settling time Vibration: the natural frequency of the standard fluid in the

problem on situations like banking turns and take offs of an airplane. for the inclinometer (limiting factor) is approximately 300ms. inclinometer is approximately 20Hz. Inaccurate readings occur if vibration in and around this frequency is present. Continuous output of the circuit board draw as low as 7mA of power consumption from the battery. A 12V battery was used and it took approximately a month to discharge. The user is able to configure the TCM2-50 with the software offered by the manufacturer based on Disk Operational System (DOS) mode. After the circuit board is configured, it can be switched off as the parameters are stored in an internal memory. The description of how the messages from TCM2-50 are being processed can be found on §5.3.1. The message provided by the circuit board is shown in Table 4.3.
Table 4.4 TCM2-50 Message

$C

compass 135

P R X Y Z T E *

pitch roll x – component of magnetic field y – component of magnetic field z – component of magnetic field temperature error code checksum

Figure 4.4 shows the TCM2-50 circuit board.

Figure 4.35 TCM2-50 Circuit Board

The magnetic compass is the vector result of three solenoids displaced as X, Y, and Z axes. The circuit is sensitive enough to compute any magnetic distortion when the circuit board is rotated generating a new position for this vector. The three solenoids are isolated from the other parts of the circuit board and located on the upper left side of Figure 4.4. The tilt sensor is the small dome placed on the upper middle section of Figure 4.4. There is a liquid that makes three resistors become half immersed in this dome. Each time that the circuit board is turned, the liquid follows the movement allowing the difference of power in each resistor to be calculated. See-Through Head Mounted Display A see-through HMD (more specifically a SONY PCGlasstron HMD) – PLM-S700 (SONY, 1998), was employed to augment the user’s vision. The following bullet points indicate some characteristics of this HMD: • The size of the projection is comparable to watching a 30-inch screen from

approximately 4 feet away. 136

•

Minimum screen resolution for PC video card: Video Graphics Array

(VGA) with 640 x 400 (dot x line); 31.5 horizontal frequency kilo hertz (kHz) and 70 vertical frequency hertz (Hz). • Maximum screen resolution for PC video card: Super Video Graphics Array (SVGA) with 800 x 600 (dot x line); 53.7 horizontal frequency (kHz) and 85 vertical frequency (Hz). • • A pair of stereo earphones is provided with the unit. It is possible to power the see-through display with an electrical socket.

For outdoors the use of a battery pack model NP-F750 is needed. The charging time is approximately 8 hours and the battery runs for up to 2 hours. • The unit weights around 120g and is made of magnesium alloy.

Figure 4.5 shows the PLM-S700.

Figure 4.36 PLM-S700

With the features described above it is possible to calculate the Vertical Field of View (VFoV) of the see-through HMD. This value provides the angle to be followed by the virtual camera inside the VE giving the illusion that the VE has a compatible scale. Figure 4.6 shows the parameters of the VFoV.

Figure 4.37 VFoV Calculation

137

The VE is projected into a screen represented by the green rectangle, as shown in Figure 4.6. The user’s eye is placed behind the lens on the centre of the green rectangle. The VE is explored respecting this window and the whole system must follow this guideline in order to produce a compatible scale sensation. The VE is represented as the black rectangle shown on Figure 4.6. It has been assumed that the user’s eyes would be 122cm far from the projector (4 feet – the blue arrow) and the screen has a diagonal of 76.2cm (30 inches – doted line), with an aspect ratio of 4:3. These values allow the computation of the VFoV as shown below (Equation 4.1):
arctan α = 3 = 0.75 ⇒ α = 36.87 o 4

sin 36 .87 o =

Height ⇒ Height = 45 .72 cm 76 .2
22.86 ⇒ β ' = 10 .61o 122

arctan β ' =

β = VFoV = 2 xβ ' = 21 .22 o
Equation 4.5

This angle is placed on the description of the configuration file that will be used by the VE management (MASSIVE-3) to create and maintain the virtual camera. Notebook A notebook, Toshiba Satellite Pro – 6100 (TOSHIBA, 2002), was used in the project to host the two platforms worked (see below) and on the creation of Java classes and methods to make the communication between this and the external set of equipment. This computer was also used to be the server of the CVE between office and field user as mentioned before. Some of the features are listed bellow: • • 1.6 GHz Mobile Intel Pentium 4 processor with 256 MB of RAM. 16 or 32 MB of RAM is provided for video display on a 3-D accelerator graphic board.

138

•

The battery pack made of lithium-ion can supply approximately 1:30h usage time.

Keyboard One of the requirements of a new project while in the test phases is the constant changes in the parameters and source code. For AR projects it is crucial to have access to the computer without taking out all the equipment from the backpack or having to go back to the laboratory. The evaluation of the system is only possible when it is being tested outdoors, where there are GPS signals. Any corrections need to be made in situ, hence the importance of an easily accessible keyboard. During the early phase of this project a desktop keyboard had been used for a while in order to access the computer. This was not practical because of its large size, heavy weight and the difficulties to “carry and type” at the same time. Another problem appeared when the Personal System (PS)/2 was used with the keyboard, leaving no spare port for the mouse. The solution was found on the handheld keyboard-mouse twiddler2 by Handykey Corporation (Handykey Corporation, 2006). The device is connected in the PS/2 port and can manage both keyboard and mouse at the same time. With only 16 buttons it is possible to get a standard 101 desktop keyboard. Figure 4.7 shows the twiddler2.

Figure 4.38 twiddler2

Wave-LAN The communication between the field and office users is done via wave-LAN. In the early stages of this thesis usually both computers are not far away from each other, 139

making it unnecessary to have a gateway to connect them. The configuration chosen was peer-to-peer wireless workgroup. The model of the wave-LAN card used was an Orinoco Silver PC Card. Orinoco guide (Lucent Technologies - Bell Labs Innovations, 2000) describes all features of the device. For the final evaluation of this thesis (see Chapter 6) a wave-LAN router was adopted. The advantage is the power of the router that can let field user go further without the concern that the link could be break in few metres. A DLink router was chose for any particular reason. The features of this equipment can be found in (D-Link Systems Inc., 2006). Integrating the Crash Helmet to the System Some adaptations were necessary to prevent devices like the TCM2-50 from being damaged by static shock or accidental drop. This circuit board does not have a protection to avoid damages or short circuit caused by static electricity. A plastic box, originally designed for business cards, was used to protect the circuit board. There are 4 plastic pins to fix it inside the box and several small holes in the cover to avoid over-heating. Only plastic material was used to prevent distortions caused by metal pieces common in boxes (like screws, for example). Another precaution was taken with the cable used to connect TCM2-50 with the notebook and battery. This cable should be flexible and at the same time robust. Leica supplies a kind of cable with 5 wires inside and 1.8m long. This cable is the only one in the market that has a connector that fits with the Leica 12V battery. Two of the five pins were used for the power supply. The remaining pins were used to connect the circuit board with the RS232-C connector, splitting the cable in two. Even though the battery is not too light to carry, the result was satisfactory. The advantage to use this configuration was that the power consumption of the circuit board was very low, allowing the battery to last for days without interruption. A few more adaptations were necessary to fit some of the equipment giving more comfort and freedom for the field user.

140

It was noticed that the see-through HMD becomes very uncomfortable on the top of the nose after some time due to its weight. At the same time the TCM2-50 should be placed over the user’s head to track its movements. The solution was to cut the top of a crash helmet to fit the TCM2’s and in helmet’s peak two holders have been made to loop a couple of stripes of Velcro, where the see-through glass could be fixed. The Velcro keeps the PCGlasstron out of contact with the user’s nose, making the experience of visualising the VE more comfortable. Figure 4.8 shows the helmet with some of the devices connected to it.

Figure 4.39 Helmet, PCGlasstron, Leica Battery, See-Through HMD and Cable

4.3.2

Software

During the process of implementation of this project some extra knowledge were required to create the VE and the displacement of the features that configure the area that is being represented. This fidelity of what is being displayed in the VE is particular important when dealing with CEn projects. A misplaced of a pipe representation in the VE for example could expose the CEn project to risks like: financial, time consuming, rebuilding, and the most serious: threat of life (Roberts et al., 2004). The creation of the VE in the scope of this thesis started with a survey of the area where the examples run for the first time. The survey was particular important because:

141

•

To get use to the GPS equipment: o What parts of the equipment should be selected. Between many parts including: batteries, cables, antennas, radios, devices models, tripods, nuts for landmarks, etc. o o o The setting up of the equipment in the field: like how to calibrate The setting up of the equipment configuration. The device should The logistic: charging batteries, transport, packing, booking tripods. be set to output the right information. system, safety issues, etc.

•

The area where the equipment is being placed. There are issues like: thief

hazard, people that would like to touch the equipment, traffic hazard, field dispute (green area can be booked by other users to play soccer or gardeners can arrive to mow the grass). • How the information can be extracted from the GPS receiver and then processed. The first example created between office users (see §5.1.1) was based in a well know area inside of The University of Nottingham Main Campus. It was important for users to identify the area that they are exploring. The use of a DTM technique to collect the points helps to extract extra features from the terrain, giving more realism to the VE created. The quality of the results is dependant on how the control points that represent the terrain’s features were chosen and which technique was used to collect them (Capra, 2000). They can be established on a regular or irregular form. If the control points were surveyed using regular method, several details of the terrain could be missed such as canyons and steps. This technique is usually applied on flat areas. In irregular terrains it is recommended that control points are collected in an irregular form. In this configuration the control points have a stronger meaning for what can be found in that particular area. The coordinate collected are then interpolated in order to create a mesh (regular or irregular) that will represent the area surveyed. Interpolator Software

142

The software Surfer (Golden Software, 2004) was used to interpolate these GPS coordinates. This accessible software leaves the user free to manipulate the data without deep knowledge about methods of interpolation, for example. From a set of control points placed in a table, the data is interpolated following the method chosen by the user. The output is another table with a square mash containing intermediate points. A wireframe view can be produced as an option to see if the result matches with what was expected. Because there was not interested in a specific method but in fact in the file format for a further conversion, the method chosen is not be commented in this thesis. A small program was created in order to convert the DTM generated by Surfer to a file that can be explored on the VE. EQUIP To manage all the devices connected to the computer (i.e. GPS receiver, electronic compass, etc.) and to share the information in a data space, the MRL has developed a platform known as EQUIP. This system can provide the distribution of data in real time for different types of applications running in different machines. It is compatible with C++ and Java, in which it is possible to share and select information from other programs using a simple interface of communication. The distribution of data is organised into “data spaces” which are named using a URL-like scheme. As cited on (Greenhalgh, 2001f):
There can be any number of data space and each data spaces can contain an arbitrary number of data items. Any number of processes can connect to a data space and add to and access the data item in it. Data items are passed between processes using EQUIP’s own serialisation protocol together with dynamic code loading in both C++ and Java.

Once the data is published on the data space one or more clients are notified that there is something new to be collected. This new data is then distributed for clients that are sharing the common resource. Figure 4.9 provides a general view of the system. Through EQUIP’s API the data can be shared on a data space illustrated by the grey rectangle. The green and red blocks represent data coming from different sources. Using basically the same API to

143

publish classes in C++ or JAVA the VE is updated with these new data. The reverse direction can also be achieved when data comes from the VE and is read by external devices.

Figure 4.40 EQUIP Platform (Greenhalgh, 2001f)

Generally, when there is the need to include another device in the system, a new Java or C++ class or method that will enable the communication with EQUIP needs to be implemented, supporting different formats and particular characteristics. In the context of this thesis EQUIP has the function of reading the information from external devices connected to the computer, publishing it in a data bank, as mentioned. The clients read from the data space and update the VE with the oncoming information. The outdoor user’s movements are translated into embodiment actions into the VE. Every time the field user starts to walk, for example, the new coordinates from the RTK-GPS receiver update the data space handled by EQUIP and consequently the position of the embodiment in the VE. The same process is done for the user’s head movements. In summary, EQUIP is the link between external devices and the VE. MASSIVE-3 The creation of the data repository does not mean that the VE is ready. The VE is created by another system, namely MASSIVE-3, which was also developed at the MRL. The VE is composed by objects and avatars that are managed by MASSIVE-3. As cited on (Greenhalgh, 2000d):

144

MASSIVE-3 is a multi-user CVE system. It allows multiple users to view and become embodied in a 3D audio-graphical virtual world. It supports real-time audio communication between users, and dynamic modification of virtual worlds.

MASSIVE-3 allows the user to construct a CVE easily. First, it is necessary to design all the objects that compose it. It can be done using a VRML 1.0 browser designer. The positioning of each element inside the CVE is made by a descriptive file informing the coordinates of the objects, scale, rotation, etc. This file is exemplified in §5.1.1. Because these two platforms (EQUIP and MASSIVE-3) were developed on such a complex level, the reader is encouraged to read the following papers in order to understand not only the contained in them, but the possibilities that these platforms present: (Greenhalgh, 2001f, Greenhalgh, 2000d, Greenhalgh, 1999b, Greenhalgh, 1999a, Greenhalgh, 2000c, Greenhalgh, 2000b, Greenhalgh, 2001d, Greenhalgh et al., 2001, Greenhalgh, 2001c, Greenhalgh, 2001b, Greenhalgh, 2001a, Greenhalgh, 2001e).

4.4

Final Analysis over the Design

The project’s design introduced above is created to improve the gaps faced by AEC regarding to communication, collaboration, and mutual understanding during the CEn project with special attention to design and construction phases. The platforms used to manage and manipulate the CVE (MASSIVE-3) as well as the one use to bridge external devices to the computer, handle one of the data bases and deal with clients requests (Equip) are the best options available due to the knowhow and support inside of MRL. These technologies were being in used for a number of years in different projects and proved that are good options for the implementation of the CARS proposed.

145

Again, as mentioned in the beginning to the chapter, this thesis shares the views presented by (Woodward et al., 2007a, Woodward, 2007, Shin and Dunston, 2008) and strongly believe that technology is increasing its capacity of processing complex data at same time that its reduce its size. AR systems and in special CARS are becoming a common tool for CEn actors during the implementation of CEn projects. As an example of real application of CARS, architect offices propose outstanding design all over the world like David Fisher (Rotating Tower Technology International Limited, 2008) or Skidmore, Owings and Merrill (Wikipedia, 2008e) office responsible for the design of the Freedom Towers, between many others. It is feasible to imagine that AEC cannot stay all the time in the same place for every meeting set during the whole cycle of the CEn project. The CARS proposed on this thesis explores the practicality of a wearable computing to enhance the contact between CEn actors. The field users can receive real time assistance from office users saving time and money, clearing doubts, enhancing once again the 3 key worlds of this thesis. AEC actors can collaborate also using only the CVE between office users. The way that a discussion about a project is set is depending of the requirements of the meetings and what kind of assistance actors need at the time. From the hardware perspective it is possible to extend the following points: • • The hardware involved can produce the positioning of users at centimetre Inclinometer and magnetic compass is eventually affected by magnetic

level of accuracy. This is exactly what an outdoors AR system needs. field distortions. The main problem is the jerking of the CVE visualisation. This can be compensated using a Kalman filter (see §5.3.1). • The see-through HMD was not design for AR solutions although it works very well for a prototype. It is expected that coming technology will address problems of resolution, automatic contrast, weight, battery consumption, etc. • Notebooks are becoming each time more powerful, light, small, and inexpensive.

146

Chapter 5
System Implementation
The practical implementation of the system is performed in three steps (Figure 5.1). The first cycle covered the following points. • • • The familiarisation with the equipment that is being used: RTK-GPS, inclinometer and magnetic compass, see-through HMD, wave LAN cards, etc. The familiarisation with the platform used to manage the CVE – MASSIVE-3. The design of a VE representing the surrounding area of IESSG building inside of The University of Nottingham Main Campus. The first phase finishes with the implementation of office users able to share a CVE, base on a RE. The second phase extends the idea of the CVE to outdoors, creating then the CARS proposed on this thesis. The following steps are taken to achieve this objective:

147

• • • • •

Familiarisation with the platform used to bridge external devices with the Implementation of Java classes to link external devices with Equip API. Tests of performance and reliability. Creation of new CVEs. Logistical and orchestration management of how to deal with a

CVE – Equip.

considerable large number of expensive equipment (particularly RTK-GPS receivers). • Final pre-test.

Figure 5.41 Steps for the System Implementation

The final stage is the evaluation process. Although several small evaluations and demonstration of the project were performed to different users along the whole project cycle, only the last one is documented. This final evaluation is created over a small number of people. The invited users were choose due to they professional competence in their respective areas of knowledge. This chapter does not only report the work perform during the implementation process of this thesis. It also documents how to do it in a simple and practical way. It is noticed that such documentation is not easy understandable, especially for researchers not used to MASSIVE-3 and Equip environments. The following subchapters may do not describe all functionalities of these two platforms (there are 148

plenty of literature for it) but they are extremely useful for users to setup, start up and make run the processes involved in the creation of a CVE as well as the connections of external devices with the data base managed by Equip. Another positive point to document this technology is the change of researchers inside of MRL. Once researchers start to work in different projects or move to other laboratories the knowledge of “how to make it – in a simple way” is in risk to be lost.

5.1

First Phase – Office User and CVE

The following paragraphs describe the first phase of the project implementation.

5.1.1

Survey and Data Conversion

The first practical work of this research was the creation of a DTM in the surroundings area of IESSG building at The University of Nottingham Main Campus. Once the IESSG is a centre of excellence in survey and positioning system and holds all the necessary hardware, software, and knowledge, it is a natural procedure to create this DTM using in-house expertise. The data acquisition stage starts with the setup of a base point where the RTK-GPS receiver would work. This base station was setup in the third pole at the top of the IESSG tower. The coordinates based on WGS84 for that base point are: • Latitude North: 52°.56’.26”.49612. • Longitude West: 01°.11’.32”.32484. • Height: 98.388m. The same coordinates translated to OS National Grid are: • East: 454363.647m. • North: 338474.259m. • Height: 40.679m. The area that surrounds the IESSG building is rich in features. Some of the mapped features were: the slope where the building is, the stairs of the footpath used by the 149

general public, lamp posts, fixed bins, roads around the institute and the car park, and the building itself. SKI-Pro software, by Leica (Leica Geosystems LLC., 2003), was used to extract and convert the data from the RTK-GPS receiver in the manipulation stage. This mass of coordinates was then interpolated using the software Surfer (Golden Software, 2004) to generate a mesh with regular spaces between the coordinates. Control points from previous surveys carried out in (Evans, 2001) were included. Such control points were representing the road kerb that surrounds the IESSG building and shape the streets. Due to the incompatibly between the file format that specifies the VE in MASSIVE-3 and the Surfer’s output, a program in C language was created to do the conversion to an intermediary solution. In this case the output was in VRML file format. This software (Appendix A) was designed to receive the following inputs: • Surfer’s output. This file contains the landscape representation in a regular grid. • A file describing the terrain’s features like: trees and building. The objects’ descriptions contain the coordinates logged in the acquisition stage. The software then generates three outputs: • A file that contains the landscape and the features as they are in the physical world – VRML file. • A file with only the landscape. It proved useful for computers with low processing capacity. This file was used to verify if the landscape created was reflecting the RE. • A file containing the .cve file description. More about the .cve file format is described in the paragraphs to follow. The software is started with the following command line at the DOS:

150

MIX006 <grid>.grd <feature>.txt <outfile>.wrl Where: • • • grid – file where the Surfer’s output is saved feature – file with the features and related coordinates outfile – name of the output file

The file with the surveyed features requires an especial attention if the users of the system want to include a road when creating the VE. The feature “road” is not an object but a set of coordinates that bounds it. These boundaries require a descriptive file that is read by the program changing the colour of the terrain (from green to grey) where the boundaries of the road are. This solution helps to reduce the number of polygons to be processed by the browser used in the visualisation. The structures of these files are described in Appendix A.1. Figure 5.2 shows the terrain (green area) and the roads (grey area) when visualised on a VRML browser. The flat lower area at the base of the landscape, shown in Figure 5.2, is generated by the interpolation process. Once the survey collected the control points in an irregular form, the interpolator software, when processing the date, creates a regular grid where the flat areas show no data.

Figure 5.42 Landscape Surveyed Around IESSG Building

151

A partially finished VE containing a number of features that were surveyed in the surroundings of the IESSG is shown on Figure 5.3. Features including trees and IESSG building were logged.

Figure 5.43 View of a Completed VE on VRML Browser

Both the IESSG building and the trees were modelled for the VE using the AC3D (AC3D, 2004) tool. In order to give a more realistic appearance, digital pictures were taken around the building, which created a recognisable texture (Figure 5.4 later in this section– where the VE is already in MASSIVE-3 environment). The texture does not have the intention to be a photogrammetric work but an enhancement in the visualisation for those who know the area and the IESSG building. The elevation of the real terrain is accurately reproduced in the VE. Differences like the ground level between the road that runs in front of the building and the car park in the back of it can be identified, conveying information with real features for the user and reassuring that the data acquisition and the manipulation stages were successfully completed. The .cve File Format As mentioned previously, the .cve file used on this thesis must run under MASSIVE3 specification. That includes: • The structural description of the CVE specified in a .cve file. • The objects that fill the CVE specified in a .dgl file.

152

Both files must be stored on each computer involved on the CVE exploration. The structural description consists of a textual file that specifies the objects that fill the CVE. The description of each object contains: the path where the .dgl file is stored, position, scale, rotation, etc. Greenhalgh (Greenhalgh, 2000b) offers a full description of how to generate these files and the commands that can be passed to MASSIVE-3. For clarification, Table 5.1 shows a very simple example of a .cve file.
Table 5.5 An Example of .cve File /*An example of .cve file*/ {{ {{ { GEOMETRY_FILE: "cube.dgl"} { GEOMETRY_SCALE: 1.00 } { POSITION: {0 0 0} } }} }}

Where: • • • • • • /**/ comment {{ begin GEOMETRY_FILE: .dgl file with the object’s description that compose the VE GEOMETY_SCALE: scale associate with the object POSITION: object location in the VE }} end

DGL’s Description With a VRML file for each one of the objects created to feed the VE, it is possible to use the program available on MASSIVE-3 library to make the final conversion between VRML to Dave’s Graphic Library (DGL) file format. Dave (Snowdon and Greenhalgh, 1996) gives the following description for the DGL (.dgl extension) library:
DGL was designed to support network applications. It has two components a renderer / platform independent part which provides graphics primitives and basic geometric operations and one or more renderer dependant parts which perform rendering for specific platform / renderer combinations. The renderer supports user input using the

153

mouse and keyboard in addition to notifying clients of other events such as resizing and window close events.

DGL files can be created through the MASSIVE-3 library using a single command line in DOS. The conversion can be done from VRML file specification to .dgl. More information can be found in (Greenhalgh, 1997). The utility process used to do the conversion is initialised in the context of this thesis using the command line: dgl_convert <infile>.wrl <outfile>.dgl Figures 6.3 and 6.4 show the area around the IESSG and the building through screen shots of MASSIVE-3. With the files converted to DGL format and the CVE description finished, the VE can now be explored using the MASISVE-3 environment. Three elements must run: trader, cveworld and bodyexampleclient. All these process are well described in (Greenhalgh, 2000d). The following paragraphs are an overview of what each one of these processes mean and how they were ran within the context of this thesis. The first experiments done using MASSIVE-3 were performed using a single user. Later the CVE was explored for more than one user at the same time.

5.2

First Phase of Practical Work

The sub-chapter 4.2 already introduced how the messages go through the data bases, how they are recovered and how the CVE and CARS windows are refreshed. The complexity of the system required that a more detailed view should be presented. The following paragraphs introduce the complete set of processes involved to make the system run. It fully complements the introduction gave during the system design. The way that the programs below are organised represents the order that they should be started up. Figure 5.4 describes the programs necessary to run the CVE only between office users. This configuration is just a subset of the full design presented on §4.2. At this 154

level Equip is not necessary. Users using this design have limited power over the CVE. They can explore, move objects, and talk to each other.

Figure 5.44 System Requirement between Office Users Only

To start a CVE between two or more office users the following processes need to be started: • • • • • trader cveworld – this process is initialised in only one machine that handles the requests of other clients. bodyexampleclient audio_server audio_test

The processes are described below. trader For any CVE started using MASSIVE-3 resources, the trader process is required to bootstrap it (Greenhalgh, 2000d). As cited on (Purbrick, 2001):

155

…all environments created on MASSIVE-3 are databases which contain part or all of a virtual world as a hierarchical scene graph. The two main types of MASSIVE-3 applications are clients and servers. Servers create an agent and an environment and then publish the existence of the environment with a trader service. Clients create an agent, locate a server by querying the trader service and then ask the server for a replica of its environment. Having received the replica the client can query and update its local replica which will generate events which are first sent to the server and from there to other clients.

This means that the trader creates a “space” for the database leaving a port of communication ready to transmit and receive messages to/from the server. At first, trader processes start in all machines although they cannot communicate with each other, since it has not been established which computer is setup as server (the one holding the VE World data base) and which are the client. When the VE is finally loaded and created – a “place” the VE receives a name and a unique system identifier (Greenhalgh et al., 2000). The trader translates environment names to Identifications (IDs) and IDs to network addresses (Greenhalgh et al., 2000). The trader process is initialised in the context of this thesis using the command line: trader cveworld The VE is started when the cveworld process is activated. This process reads the .cve file in order to place the objects that will compose the VE. This process also reads the .dgl files that describe each one of the objects referenced in the .cve file. For each VE described in the .cve file an ID (as a name for example) is required. The trader is then notified that a new VE was created and all requests of communication are directed by the ID. A complete description of this command can be found in (Greenhalgh, 1997, Greenhalgh, 2000d). The cveworld command is only applied once – for the server. Clients do not start this process. Cveworld process is initialised in the context of this thesis using the command line: cveworld <IP_address>/<ID> <CVE_world>.cve 156

Where: • ID is any significant name that will easily identify the CVE created.

bodyexampleclient The bodyexampleclient is a process started for all clients aiming to have access, with their own embodiment, to a specific CVE through the socket offered by the trader service. Note that the machine that is hosting the CVE World is also a client of itself. The embodiment follows the description given in a file named massive3.config that is fully described on (Greenhalgh, 2000a). The process starts by calling a specific address and ID. The bodyexampleclient process is initialised in the context of this thesis using the command line: bodyexampleclient <IP_address>/<ID> Where: • ID is the same name included in the cveworld command.

From the client’s perspective, this request first goes to the local trader that redirects it to the server’s machine. In the server, the trader identifies the request by the ID. Finally the server returns a copy of the VE requested indicating the displacement, scale, rotation, etc, for each one of the objects. Each client has its own copy of the objects that fill the VE. This process is fully described on (Greenhalgh, 2000d, Greenhalgh, 2000a). After the process has started a window pops up in the computer’s screen displaying a CVE ready to be explored. The VE based on the processes of acquisition and manipulation can now be explored as shown on Figure 5.5 and 5.6. These MASSIVE-3 screenshot also shows

157

the textures created with the pictures that were taken around the IESSG building in the Main Campus of The University of Nottingham.

Figure 5.45 View of the Using MASSIVE-3

In these two images (Figure 5.5 and 5.6) the VE is being explored by a single user who controls the movements using a mouse.

Figure 5.46 Front View of the IESSG Building

The next step was the setup of a CVE where more than one computer could share the same place. Figure 5.7 presents both office users sharing the same environment. The screen shot is a view from one avatar to another. The red avatar in Figure 5.7 represents the second user.

158

Figure 5.47 Red Embodiment Representing a Second Office User

The first stage of this thesis finalised with two actors (office users) sharing the same CVE. Although the audio link was already available it was not explored at this stage.

5.3

Second Phase of the Practical Work

Figure 5.8 shows the design of the system required when field and office users are active. This diagram is the same presented in Figure 4.2 however it is now introduced the complete number of processes required to make the system run.

Figure 5.48 System Requirement between Office and Field Users

From the field users perspective it was necessary the implementation of new Java classes, necessary to connect GPS receives, inclinometer and magnetic compass to the data base Coordinates (Figure 5.8 – right side). From the office user side a class was implemented to continuously read from the keyboard. The following sub-chapter

159

explores the MASSIVE-3, Equip and Java classes created to integrate external devices to the system. The high level algorithm and the class’ constructor are described on Appendix B. Some of these classes are based on/or adapted from previous works developed also in the MRL, like Augurscope (Koleva et al., 2001), and were eventually modified to fulfil the requirements of this project. RTK - GPS Connection Two Java classes were created to handle the information from the RTK-GPS receiver. The first one reads and writes the messages from the receiver and it is named GPSReceiver.java. The second class, GPSPublisher.java, requests the update of Coordinates data base through Equip processes. GPSReceiver.java (Appendix B.1.1) reads the data using a COM1 from the RTKGPS receiver. Before the communication is established, the transferring velocity and parity bits are setup to start to receive the GPS messages. The GPS receiver first converts the coordinates from WGS84 to OS grid in an internal program. The string received is formatted using LLQ message format (see §4.3.1) and decoded to extract east, north and height coordinates. The normalisation between the base point and the coordinates received is the next step. The user’s position is converted to the Cartesian system used in MASSIVE-3. Figure 5.9 shows the axes orientation of Cartesian system used by the platform.

Figure 5.49 MASSIVE-3 Cartesian System

Figure 5.9 shows the North (physical world) represented by a negative Z in the VE. When the RTK-GPS receiver is moved to north, the positive signal of the coordinate should be changed to negative, otherwise the embodiment will be moved to South. 160

The GPSPublisher.java constructs an instance of GPSReceiver.java and acts as the interface between the data coming from the GPS receiver and the Coordinates data space using the services provided by Equip. The GPSPublisher.java requests from the library the creation of a data space where the coordinates coming from the GPSReceiver.java are to be published. This data space is then ready to be accessed by Integrator that interfaces with the CVE World data base. MASSIVE-3 is then finally triggered updating the visualisation (Appendix B.1.2). In resume, the processes involved when a new GPS data is read from the receiver are: • • • • • • GPSReceiver GPSPublisher equip_net_trader equip_data_server – logging and reading from Coordinates data base Integrator equip_mas3_mgmtserver – logging and reading from CVE World data

base. This process also renders the CARS window. All clients sharing the CVE World data base are advised about the new data. The process bodyexampleclient receives and update the view on office user side. Magnetic Compass and Inclinometer Connection Like the RTK-GPS receiver, the magnetic compass and inclinometer circuit board has two main Java classes to control communication and data logging: TCM2.java and TCM2Publisher.java. The device was configured to send information only for the compass direction, pitch and roll. TCM2.Java (Appendix B.2.1) starts the communication between the circuit board and the computer, setting up the baud rate and bit parity. Once communication has been established, the algorithm searches for the message starting with $C (Table 4.2) and finishing with a break line. The tokens are then split and an option for a Kalman filter is available to reduce the jerk. The results are demonstrated on §5.3.1 and the

161

Kalman filter dataflow illustrating how it is implemented is shown on Appendix B.2.3. TCM2Publisher.java (Appendix B.2.2) receives the data and requests an update it in Coordinates data space. Integrator is triggered and requests the input of the new data in the CVE World data base. MASSIVE-3 is then advised about the new data. It renders and updates the visualisation of the CARS window. The contents of the tilt sensor and the magnetic compass messages are described in §4.3.1. The process involved in handling the data coming from TCM2 are the same presented for the data coming from the GPS receiver. The difference relies only in the classes TCM2 and TCM2Publisher. Consideration also had to be taken for the TCM2-50. The default communication port used was COM1. However, this port was already in use by the RKT-GPS receiver. A communication port conversion between USB and the serial port was integrated into the system. Such port emulation makes the circuit board operational on a different address of communication port. The system at this stage is enabled to capture the head movements of the user and his positioning over the ground. The system is then configured to work as a field user in standalone mode where the first experiments merging real and virtual where taken. Keyboard Connection The keystroke capturing in the office user side has the same principles already introduce when new data is coming from RTK-GPS or inclinometer and magnetic compass. Every time that a key is stroked, the data is logged in the data base controlled by Equip. From there to the update of the CVE World and the CARS window the process is the same as explained before. The classes created to read and publish the data are: ReadyKeyStroke (Appendix B.5.1) and KeyStroke (Appendix B.5.2). Integrator The Java class that bridges the Coordinates data base with the CVE was named Integrator. This class was inherited from the Augurscope project (Koleva et al., 2001, 162

Benford et al., 2003a) and several modifications were made. This class was designed to attend to the following tasks: • It creates a set of variables that fits in MASSIVE-3 structure. • It creates a link between the data space Coordinates and MASSIVE-3. • It creates a control for the data coming from keyboard, tilt sensor and electronic compass. The algorithm is aware when a new data is logged into the data space, requesting the update of the virtual camera inside the CVE. • It configures the embodiment in the VE overwriting the configuration file available in MASSIVE-3. The VFoV (see §4.3.1) is set also in this class. • All digital objects requiring dynamic update are also created and controlled by this class. The objects are created and placed in the CVE World. Integrator is in a very broad view the middleware between the Coordinates data base and the CVE World data base. This integration only happens because: • • Every new data from external devices triggers Integrator. The data leaves the Coordinate data base through: o equip_data_server o equip_net_trader o Integrator • • • Integrator processes the data if required – keyboard for example. Integrator updates CVE World data base. equip_mas3_mgmserver receives the new data and renders the image in CARS window. The high level algorithm covering this class is given in Appendix B.2.3.

5.3.1

Integrating Software, Hardware, Field and Office Users

The first step to start the system is to setup a wireless communication between the computers. The procedure of how to setup a wave-LAN is not explained here. But both computer (office and the field user) should be able to identify each other through

163

Packed Internet Gopher (PING) (Wikipedia, 2006) command for example, proving that they can establish the exchange of packages. The following paragraphs present the necessary command lines to run in order the field user. It is not the intention to describe in depth each of the components. Instead, the text aims to be useful as a base reference on “how to start it” in future projects. Commands like: trader and the cveworld that were already introduced in §5.2 and are not reviewed in this section again. Field User Setup The order in which to start the system for the field user is as presented below: • trader • cveworld • equip_net_trader • equip_data_server • equip_mas3_mgmtserver • equip_mas3_entitygenerator • TCM2Publisher • GPSPublisher • Integrator • audio_server • audio_test equip_net_trader In the field user’s context, after the trader and cveworld have been started, the equip_net_trader opens a communication port where the requests from the clients are managed and controlled. The equip_net_trader process is initialised in the context of this thesis using the command line: bamboo ./equip_net_trader 164

Bamboo is the platform used to give EQUIP flexibility to operate with both C++ and Java program languages. Further details about Bamboo can be found in (Bamboo Open Source Community (BOSC), 1999). equip_data_server The data space is opened by equip_data_server who deals with the requests from clients to the data spaces running in the server. All the requests arrive through equip_net_trader. The word space was adopted here to name the data_space_name where the VE was created. This data_space_name was named app. equip_data_server process is initialised in the context of this thesis using the command line: bamboo ./equip_data_server -equip://<IP_address>/<data_space_name> equip_mas3_mgmtserver As indicated in (Greenhalgh, 2001f) the “management server creates and renders a 3D graphical view of a MASSIVE-3 virtual world”. This window is linked with the data_space_name created for the VE and identified by the word space. The equip_mas3_mgmtserver process is initialised in the context of this thesis using the command line: bamboo ./equip_mas3_mgmtserver -<config_file> -equip://<IP_address>/<data_space_name>

Where:

165

•

config_file – this configuration file is used by MASSIVE-3 to setup the embodiment and the virtual camera inside the VE. This file is generally saved in c:\Massive\rev-4c\etc (assuming that Massive is installed in C:\ directory and the version is rev-4c).

equip_mas3_entitygenerator All non static objects are controlled by this component. They are linked with a textual file that addresses the same data space used for the VE. Integrator interacts with this process receiving the new coordinates of the object in a thread. Every time that the thread updates the coordinates of an object Integrator updates the CVE World. All clients receive this update and the CVE and CARS windows are refreshed. equip_mas3_entitygenerator process is initialised in the context of this thesis using the command line: bamboo ./equip_mas3_entitygenerator -<file_name.txt> The file with the address and the name of the space is exemplified below: Name: entgen1 DefaultDataspace: equip://<IP_address>/<data_space_name> The world entgen1 identifies the area where the coordinates (that makes an object move inside the VE, for example) of the object are logged in the data space used for the VE. The world space was substituted by data_space_name. TCM2Publisher TCM2Publisher instantiates TCM2 class to establish the communication with the tilt sensor and magnetic compass. It also creates an identifier inside of the Coordinates data base to log the data in the right place.

166

The coordinates are read by TCM2 and processed by a Kalman Filter (see below). After this process the data is then sent to be logged through the following processes: • • equip_net_trader equip_data_server

For this class, the javax communication libraries (Sun Microsystems, 2004) must be present in the Java lib directory. TCM2Publisher process is initialised in the context of this thesis using the command line: java TCM2Publisher equip://<IP_address>/ <data_space_name> <COM port> The world app was substituted by data_space_name. GPSPublisher The GPS data logging is similar to the TCM2Publisher, but for the fact that it is designed for the GPS receiver. It instantiates GPSReceiver class to establish the connection and process the data coming form the GPS receiver. As in the previous class it also requires the javax communication libraries. The GPSPublisher process is initialised in the context of this thesis using the command line: java GPSPublisher equip://<IP_address>/<data_space_name> <COM port> The world app was substituted by data_space_name. Integrator The Java class Integrator makes the link between Coordinates data base with the data space allocated for the VE – CVE World. The class also controls the dynamic objects

167

updating the VE when necessary. It configures and creates a link with the management server that is used to render and control the actions inside of the VE – equip_mas3_mgmtserver. The Integrator Java class is initialised in the context of this thesis using the command line: java Integrator equip:// <IP_address>/<data_space_name> equip:// <IP_address>/<data_space_name> <config_file> <IP_address>/<ID> Where: • data_space_name – the first data_space_name refers to the data space allocated to the VE. The second one is the space allocated for the data coming from the GPS receiver and the TCM2-50. • config_file – this configuration file is used by MASSIVE-3 to setup the embodiment and the virtual camera inside the VE. This file is generally saved in c:\Massive\rev-4c\etc (assuming that Massive was installed in C:\ directory and the version is rev-4c). audio_server The audio channel is setup after the audio server opens a space where the computers involved in the communication are able to connect. This program should be run in all computers that want to use the audio channel. The audio_server process is initialised in the context of this thesis using the command line: audio_server audio_test

168

This program establishes the link between the two machines through the audio server making the transmission of the audio stream. The command must be run on all machines involved on the communication. The audio_test process is initialised in the context of this thesis using the command line: audio_test -h <IP_address machine_1> accept < IP_address machine_2> and audio_test -h <IP_address machine_1> send <IP_address machine_2> Office User Setup – following Figure 5.8 Commands like: trader and the bodyexampleclient, previously introduced in §5.2, will not be described again. The same applies to equip_net_trader, audio_server and audio_test – see §5.2. The order in which the office user should start the system is as presented below: • trader • bodyexampleclient • equip_net_trader • KeyStroke • audio_server • audio_test KeyStroke This Java program class reads from the keyboard and logs the key pressed in Coordinates database. The process involved in this action is similar to the one used to describe the logging of new GPS coordinates in the beginning of this sub-chapter. The KeyStroke is initialised in the context of this thesis using the command line: java KeyStroke equip://<IP_address>/ <data_space_name>

169

The word app was substituted by data_space_name.

Kalman Filter The class that implements the Kalman filter is called internally by those classes that communicate with the GPS Receiver and with the tilt sensor and the magnetic compass. The implementation of a Kalman filter was necessary after some tests showed that the magnetic compass suffered a constant jerk. Such movement happened even when the TCM2-50 was completely stable. For users, the jerk provoked the uncomfortable sensation that the VE was always “shaking”. Even with constant calibration (recommended by the manufacturer), the tracker’s distortion was aggravated by the soft magnetic field created by the backpack frame plus the usual magnetic distortions present in the RE. Based on (Hide, 2003) this filter is well suited to navigation applications, which contains states that are time-variant, even in real-time. It is a recursive algorithm, which filters measurements using knowledge of the system dynamics and the statistical properties of the system measurement errors. Further details of how the filter was implemented are showed in Appendix B.2.3. The same filter but with different weights was also be applied to the GPS coordinates even though the receivers already come with a Kalman Filter algorithm implemented at hardware level. As reported in (Capra et al., 2003b) the results achieved showed a reduction on the jerking, resulting in a smoother view of the VE for the field users when moving their heads. Figures 5.10 to 5.12 show 60 samples of the data related with magnetic compass, pitch and roll. The blue line shows the raw data from the tilt sensor and the pink line the same data after being processed with the Kalman filter. In all cases, it is noticeable that the lines of the graphs are smoothed. The sharp edges on the blue line were replaced by a bend curve. The result for the field user is an image stabilised with 170

a small delay, from the stop of the head’s movement until the update of the latest frames.

Magnetic Com pass 300 Degrees 200 100 0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 Sam ples Without Kalman Filter With Kalman Filter

Figure 5.50 Magnetic Compass data

Pitch 50 Degrees 0 1 -50 Samples Without Kalman Filter With Kalman Filter 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61

Figure 5.51 Pitch Data

Roll 40 20 0 -20 1 -40 -60

Degrees

5 9 13 17 21 25 29 33 37 41 45 49 53 57 61

Samples Without Kalman Filter With Kalman Filter

Figure 5.52 Roll Data

Calibration Even though calibration is not part of this thesis, at the early stages of the project some tests were performed to identify how much error could be observed visually. To

171

implement the experiment, four stacks with approximately half a metre high and few meters far from each other were surveyed and set as targets in RE. Each target was superimposed by a digital object (a cylinder for each stack). The results showed that visually the digital cylinders were never projected further than a few centimetres. This experiment perhaps was not deeply technical and as accurate as it should have been, especially for an AR system. However, it showed that the projection, as it was, would be good enough for the desired aim of this thesis. Following Azuma (Azuma, 1995) it is possible to list the main sources of static errors such as: distortions in the HMD, mechanical misalignments in the HMD or incorrect viewing parameters and errors in the tracker. This reference is a good source of information if the reader would like to know more about processes of calibration for AR systems.

5.3.2

Notes of How to Use the System

Complementary notice: as soon as the Integrator class starts to run in the field user’s machine, one of processes will ask for a major share of the CPU. The users are advised to access the Windows Task Manager (this thesis was performed using Windows 2000 as operational system) and then Process menu option. Sort the processes by CPU usage. One of the tasks (one of the Bamboo tasks) shows a very high percentage of CPU usage. To reduce it, click with the right button of mouse in the task and change the priority to low. This procedure helps to avoid the freeze of the system by CPU overload.

172

Chapter 6
Evaluation and Results
Once the configurations and settings established in the laboratory have already been explored / taken into consideration, the next stage is to evaluate and analyse the results achieved within this project. Perhaps the most difficult part in the evaluation process of an AR system based outdoors is the fact that it should be performed outside the laboratory. The logistics and the orchestration of all parts involved are extensive, requiring attention to small details. A missing cable or battery that was not charged properly can put at risk the entire operation. At this stage, the integration of the system was concluded giving field and office users the opportunity to share the same CVE. The embodiments helped users with feelings of presence and collaboration and thanks to an audio channel, working similar to a telephone line, field user could always have someone placed remotely to guide and help inside of the CVE. This chapter introduces three examples created to demonstrate the use of this project in the RE. The first two examples are based on real situations. The third example is a fictional construction created to show other potentialities of the tool during the evaluation process. The examples are:

173

• • •

A tram station – based on a real tram station in Nottingham, United Kingdom. A house – based on a real construction in São Francisco de Paula, Brazil. A sketch of a construction – showing some extra features of CARS to users.

The evaluation process was performed with a small group representing potential final users for the project. Those users were choose because their expertise in the areas where they work. They are civil engineers, researchers in computer science, researcher in fluid dynamics, researchers in high accuracy GPS, researchers in photogrammetry, researchers in VR, and a person with background in computer science and state-of-art technology. For the evaluation process it is followed two main techniques to extract information from users when exploring the CARS developed: • Think Aloud (Someren et al., 1994, Mountain and Liarokapis, 2007,

Wikipedia, 2008f) – in this technique users are requested to say loudly whatever they are experiencing during the evaluation. A camera man is hired to stay with the users all the time, recording everything that he is doing and saying. The office user (during the test it was performed by the author of the thesis) also put questions, interacting with the field user (through the voice channel), taking notes of some of the key points. Users in the field kept continuously interacting, mentioning positive and negative aspects about the system. • Cognitive Walkthrough (Angewandte und Sozialorientierte Informatik Universität Hamburg, , Zhang, 2005, Wikipedia, 2008a, Abowd, 1995) – users are instructed to what to do, how to do, what to see, how to communicate with office user, etc. During the presentation of the exercises users are reminded where they could find the VE in the RE and what they would able to see there. Although there is not formal script for the exercises performed, they run in very similar way to each other. All the processes are recorded to be analysed after finished the trials. The results then shows where the project must be

174

better designed, especially where users could not reach the end of the tasks or where the solution was not obvious to be achieved. Historically, MRL uses a formative approach (Abele et al., 2005) to evaluate its projects. Once a laboratory where technology has been developed as research and not as final product, the usual evaluation process is focused on identification of points that require improvement during the development process. Formative approach is mentioned in (Vicente, 1999) as:
Formative approaches focus on identifying requirements – both technological and organizational – that need to be satisfied if a device is going to support work effectively. Although these requirements do not uniquely specify a new design, they are still highly informative and thus very valuable because they can be used to rule out many design alternatives.

The evaluation process followed by this thesis did not concern in how long it took for a user to complete a task or how many tasks were completed. The criteria used are focused in questions like: can the user complete a task; can the user interact with the system at all; is the hardware working as desired; is the VE fulfilling users’ expectations? These are broad questions that were answered when users were interacting with the system during the trial and what they answered along the interview. This chapter contains excerpts from interviews recorded when users when testing the CARS created. The interviews were transcribed in Appendix C.

6.1

Three Exercises for the Evaluation

The way choose to demonstrate a CARS project for CEn purposes was to make it happen with examples that had been or could be implemented in the RE. The first example is the tram system that was constructed throughout the city of Nottingham – UK, with the supervision of the Nottingham Express Transit (NET) and implemented by Carillion.

175

The public tram services started in November 2003, removing the need of two million car journeys a year and helping to make the city cleaner, less congested and even more attractive. It is estimated that around 11 million passengers will be transported every year representing up to 30,000 people in a single day. The first line, named Line 1, has 23 tram stations – including 5 Park & Ride stations and it extends for 14Km. The users have a faster and safer alternative for going through the city, in comparison to another type of transport (NET, 2001).

In this example, a scene involving one of the stations and also an object representing the tram was implemented. The reasons to implement this exercise are: • It is a place where most of the interviewed users saw or visited before. Users already arrived with some previous knowledge of what was being showed before their eyes when exploring CARS. • In a potential real scenario AEC actors could take advantage of CARS to work over the necessities faced when in design and construction phases (see Chapter 2). • • The use of CARS to enhance communication, collaboration and mutual understanding in the place where the construction was set. Field and office users working remotely without the necessity of staying in office. To enrich the scene it was implemented a tram that approaches the virtual tram station, stops for a little while, and moves away again. The benefits to add a dynamic object in the scene are: • • The scene is closed to reality – the virtual tram is also following the scale The potential use of the tool – the dynamic object (the tram) could be

of a real tram, making users feeling more like in a tram station. substituted by any other object in a different context. For instance it could indicate wind corridors in a city or the way that water would flows in a case of torrential rain. The second example is based in a real construction. Although the example looks simple in a first instance, this exercise is created to demonstrate:

176

•

Exploring previous knowledge about a specific scene – users already arrived with a previous knowledge of what is a house. Even though houses can have its rooms arranged differently, they have some common areas like: kitchen, bedroom, bathroom, living room, etc.

•

Checking details of the project – it explores the possibility of using CARS as a tool to keep building projects on schedule, checking details like layers collision, etc. This is an important topic that was already demonstrated in §2.1.2 and a real necessity for ACE actors during the design and construction phases of a CEn project.

•

Control of what was being analysed – this example stresses the collaboration between office and field users. Office users can add and take out layers of what was being demonstrated at same time that communication and mutual understanding happen.

•

Collaboration between AEC – as mentioned in (Khanzode et al., 2008) actors (in special engineers and architects) got surprised by the number of hours spent with constructors inside of the meeting room just to clear obscured points. Once again the exercise aims to demonstrate how practical it is for those doing inspections on site to interact with people in the office.

The final example is another object representing a house. At this time it is not based in any real situation. This example aims to achieve: • Drilling information – instead of only show the shape of a house and its phases of construction users are able to drill different level of information like: o Displacement of windows, doors, and stairs – actors could check for instance if the windows are facing the desired direction. o Water, electricity, gas, etc. – the scene is composed by pipes representing some of the supplies of the house. Users could check if the supplies are well distributed in the house for instance. This idea could be extended if the VE contains house’s furniture. It could help users to make sure that no supply (electrical or water) would get

177

hidden behind a piece of furniture for instance. Users could also check if the furniture will not block a view from a specific angle in RE. For general information, exercises 1 and 2 are based on real blue print specifications. The measures are as correct as possible. However the design of the VE was created using none CAD software by the author – that is not an architect or engineer. The third example was created for illustration proposes only, fowling none blueprints neither CAD software. Tram Station – Exercise 1 – Animation Feature The second phase of this thesis started with the design of the furniture of a tram station. Based on the bi-dimensional plan (Figure 6.1) the objects were designed with the three-dimensional editor AC3D (AC3D, 2004), the same used to draw the IESSG building example. In the plan, one can see the dimensions of the platform and the furniture of this tram stop. The following items were designed: sign, bin, lamp post, tram shelter, and platform – Figures 6.2 to 6.6.

Figure 6.53 Blueprint of Phoenix Tram Station (Carillion, 2000)

178

Figure 6.54 Sign and Bin

Figure 6.55 Sign and Lamp Post

Figure 6.56 Lamp Post

Figure 6.57 Tram Shelter

Figure 6.58 Platform

The scale and colours of each of the furniture are closely related to the specification given by Carillion, apart from the platform that is not blue as shown in Figure 6.6. The displacement of the objects in the VE follows the blue print. The design and

179

creation of the .cve file is facilitated by the know-how of what had previously been done to IESSG VE. Figure 6.7 shows how the model looked when all features were placed together.

Figure 6.59 Tram Station

Figures 6.8 and 6.9 are showing users (field and office users) sharing the same CVE. Figure 6.8 shows the bird’s eye view of office user.

Figure 6.60 Tram Station – Top View

Figure 6.61 Tram Station – CVE

Figure 6.10 shows a model of a tram approaching the station. The tram is a simplified version of a model found on (Gilbert et al., 2002). The model is simplified due to limitations of the number of facets imposed by the program dgl_convert (see §5.1.1). The final model of the VE is a reasonable copy of how the RE is. The animation is a key feature of this example stressing the potentiality of the visual augmentation for applications that could require the simulation results in animated form together with the VE.

180

Figure 6.62 Virtual Train Approaching the Station

Figure 6.11 shows how the platform looks in the RE.

Figure 6.63 Platform in the Physical World (Nottingham Tram Consortium, 2004)

The picture of the actual site where the platform works as a final certification that what had been evaluated by the users during the exercises reasonably corresponds to RE. House1 – Exercise 2 – Phases of an Engineering Project The second example is also based on a real construction. This model was created to show AEC and customer the construction on different phases during the design and construction phases. The blue print chart – Figures 6.12 – shows the internal plant of the house.

181

Figure 6.64 Internal Division (Lucena, 1986)

Figure 6.13 shows the right side of the house following the blue chart print.

Figure 6.65 Right Side Scheme (Lucena, 1986)

This example shows how the house should look and also highlights the expected commitment of those involved in an engineering project. Figure 6.14 is a snap shot of the project with four stages of the construction implemented. Each stage is represented by a different colour. During the process of visualisation users could see the stages being placed on the top of each other or just the required layers. The advantage is to illustrate every single phase and analyse it in a clear way.

182

Figure 6.66 Colours Representing Different Stages of the Engineering Project (Design and Construction Phases)

Figure 6.15 shows the house built with different coloured layers. Each layer can have a specific meaning (that can be stipulate by AEC) during the design and construction phases.

Figure 6.67 The Complete Construction (Design and Construction Phases)

Figure 6.16 shows how the house looked at the end of the project – final construction phase on the RE.

Figure 6.68 The House as it Looks in the Physical World

183

House2 – Exercise 3 – Drilling Information The final example introduced in this thesis is a house where the users could drill through information such as: the structure, the pipes (gas, electricity, etc.) and the distribution of windows. Figure 6.17 shows the house from outside to give an idea of how the shape of it. Figure 6.18 shows where the windows of the house are as well as the positioning of the stairs. In a project of a country house, for example, CARS would show the owners of the property if they would have the best views when examining the displacement of the windows. The final layer of information in this example was the pipes that surround the structure of the house as well as the underground. Figure 6.19 shows different pipes with different colours. Each pipe could represent different meanings and follow specifications.

Figure 6.69 The Structure of the House

Figure 6.70 Windows and Stairs

Figure 6.71 Different Colours of the Pipes Indicating Different Uses

184

6.2

Evaluation

The evaluation process was performed several times during the existence of the project. In the very early stages, most of the evaluation was just to confirm that the system was working reliable when all the hardware and software were running at the same time. Eventually some users were invited to check again the evolution of the project. Throughout the entire process the project had three formal demonstrations and several informal ones. During the first two presentations the evaluation was more concerned with the points on which the tool should be improved. No interviews were done and the users just expressed their comments to the author during conversation. In order to give more flexibility to setup and demonstrate the project in both campuses two GPS references base points were surveyed. The first was taken inside of The University of Nottingham Main Campus close to the Sports Centre. The second was in The University of Nottingham Jubilee Campus, on the island in front of The Atrium building. The coordinates (in WGS84) of the base point used close to the Sports Centre are: • Latitude North: 52° 56’ 26.32994”. • Longitude West: 01° 12’ 22.07388”. • High: 88.7259m. The coordinates (in WGS84) of the base point used on the island are: • Latitude North: 52° 57’ 10.74172”. • Longitude West: 01° 11’ 15.04220”. • High: 78.271m. These sites had advantages and disadvantages to work in. The signal captured in the Sports Centre did not suffer as much multi-path as the experiment that took place in the Jubilee Campus.

185

A multi-path effect can be noticed when the GPS signal reflects on buildings, the terrain or any other object between the satellite and the receiver. The reflected signal arrives with some delay at the receiver causing an inaccurate coordinate (see §3.9). This can be confirmed by the constant automatic changes of the GPS receiver from RTK-GPS to DGPS. Both receivers (reference station and rover) were eventually too close to a number of three-store buildings and trees in Jubilee Campus. Another problem is the water surrounding the island. At least once in early stages, a user was reminded that she was walking too close to the border of the island. However in the Sport Centre the challenge is to keep the demonstration running for long periods of time only using batteries as power supply. There is no electrical point and everything has to be running at the moment that users arrive. In the island on Jubilee Campus this is not a real problem. The place is not far from an electrical socket, making the process free of the stress caused by batteries running flat. Any required modification in the configurations or code that required debugging could be easily done as longer as necessary once the computers are connected to a power line. For the final evaluation the green area behind Willoughby Hall inside The University of Nottingham Main Campus was chosen for the performance and testing. This place has the following advantages: • As the other areas, this place is flat. The evaluation is unlikely to

run the risk of users going up and down small hills or stairs. The area is also not surrounded by water as in Jubilee Campus. • • extension). • equipment. The coordinates (in WGS84) of the base point used in Willoughby Hall are: Willoughby Hall also offered the possibility to store the equipment safely in one of the rooms, reducing drastically the time for setting up the There are no buildings or trees too close to where receivers would Electrical sockets are less then 50 metres (close enough for an be used, reducing the chance of multi-path.

186

• Latitude North: 52° 56' 05.87635". • Longitude West: 1° 12' 20.57715". • High: 77.7971m.

6.2.1

Pre-Test before the Final Evaluation – Final Adjustments

Just before the process of evaluation started, a pre-test was performed to ensure that everything was running as expected. As any other project, there are always some last minute minor adjustments to be done. This final pre-test is valuable because it should be performed not by the author of the project but by an outsider. The advantage of the input of a “fresh mind” that does not know how to use the system is to show to the evaluator some missing points of how the system is going to be used. This final adjustment is probably one of the most important points through the perspective of final evaluation. If the developer and the evaluator are the same person, for sure, he had to undergo a kind of self-adaptation to the system. This adaptation could be the skills developed to type in the handheld keyboard or the best speed to walk with the equipment, for example. Another point is the way that the user would wear the equipment. Not all users know how to wear it and what looks trivial for the developer could not be obvious for users. The process of equipping the user should be as delicate as possible. Not just because of the number of cables and fragile equipment that are involved, but also because the user is receiving an extra layer of “content” over his body. The more the user feel confident the more he will have a good time and his concerns will be focused in the evaluation and not on whether he is looking good or doing well. Positive Points of the Pre-Test Previous tests showed a number of details that could make some difference in a real performance. The required adjustments were implemented to optimise the experience of the field user.

187

A line (a white long tape was used) in the grass marking where the VE should be projected helped the user to identify which direction he should go in the RE. This eliminated the need to tell constantly where users should walk. By following the line, the user would automatically go to the right direction and would not be too far from there. In previous experiences, when users does not have the facility of the white mark they would spend more time wondering around rather than focusing their attention in the CARS. A wave-LAN router was used to create the link between the computers rather than using only simple peer-to-peer configuration with computers linked purely by their network cards. The advantages are: the signal is stronger and the router is placed half way through the computers, enabling users to move further away from each other. This system is more stable when compared with the previous versions. The jerking was drastically reduced by the Kalman filter (see §5.3.1) implementation and the installation of a faster hard disk in the notebook. The GPS signal is not affected by multi-path as much as it is in the Jubilee Campus. This is attested by the significant reduction of the VE drift (visual perception only). The receivers usually locked the signal faster in the chosen green area and kept it for most of the time. Designing more than one exercise can enrich the project significantly. In previous performances the users were introduced only to a virtual tram station. During the final evaluation users had a better overview of the issues covered by the project with the three exercises created. Learning Points from the Pre-Test The wave-LAN router should be placed as close as possible to the area where the VE is being projected as well as far from the ground. It was noticed that the system stopped working – freezes – when the connection between users is lost – around 50 metres from the router. At this stage field users could interact with the VE through a handheld keyboard. This device is not suitable for a user without previous training. The device is helpful if users know how to work with it but it becomes unfriendly for a user that is not used 188

to handle it and is already caring all the equipment in the backpack and wearing the HMD. Instruction “on the fly” is impossible and most certainly the person will not understand how it should work, getting frustrated very quickly. As the user said during the pre-test: “It is phenomenally difficult, unless you know what you are doing”. 6.2.2 The Final Evaluation

The final evaluation of this project is directed by the following steps: • Embedded Test – all users were invited to test the system. All actions were recorded for later analyses. • Post Test with Interview – an interview just after the embedded test to extract more information from users. Users were questioned not only about the system but about themselves when using the system. Most of the questions were repeatedly made to users but some of them were tailored to particular points noticed during the trial. It was invited for the final evaluation a small but well representative number of people to interact with CARS. In total, eight people got involved during the evaluation. These people were chosen based their backgrounds and their previous experience on evaluate projects. • Four civil engineers – with background in space geodesy, fluid simulation, photogrammetry, AR systems, high accuracy GPS, etc. • One architect – with experience in AR systems and VE design, also closely linked with computer sciences projects. • Two computer scientists – with knowledge in simulation, AR systems, VE, GPS positioning, CE and etc. • One ordinary person – a young male, qualified to evaluate AR systems and computer games.

189

The users were invited to come for the evaluation at different times and days. This procedure prevented their opinion being influenced by any discussion between them. The exercises for each individual, three in total, took around fifteen minutes each to be completed and the interviews after the trial around thirty minutes. The exercises are not guided by an exact path where field users should go or by a time frame to complete one or more tasks. However there is a pattern in the exercises covering the following points: Tram Station – exercise 1 • VE Identification – field users should be able to understand and recognise the tram station associating it to their previous background. They should also be able to identify the displacement of the furniture or the colour of it. • • Scale – users are invited to stay on the virtual tram shelter and have the perception of scale. Small Details – users are also motivated to find an object representing a Close-Circuit Television (CCTV) in the top of a virtual lamp post. This task is particular interesting because it required users looking up trying to find a small virtual object. • • Animation – users are also invited to stay over the platform to wait for the tram to approach and depart. Broad View – field users are requested to walk away from the virtual tram station to see the entire VE with the tram approaching and departing. House 1 – exercise 2 • • VE Identification – field users should be able to identify the house and the Layer by Layer – field users are exposed to different segments of the house rooms that shaped it based on previous knowledge of a house. construction. With only few layers with distinctive colours users are invited to explore the rooms, find the doors and windows, stay in the veranda, etc. House 2 – exercise 3

190

•

Change of the VE – field users are exposed to a complete changed of layer of information although all of them have the same reference (a house). Users are again asked to go and explore the VE.

•

Identification of Particular Points of Interest – users are requested to look and walk towards the stairs. When they are closed to stairs the VE is switched again. Users are requested to look up to find a whole in the ceiling where the stairs climbs to.

•

Be surrounded by the VE – this VE is designed larger than the other two examples. Users should have the feeling to be surrounded by the VE where they need to walk around 25 meters to get away from the house.

The evaluation was created to cover the topics shown on Figure 6.20.

Figure 6.72 Topics Covered by the Evaluation

•

Technical – regarding to technical aspect of software and hardware used during the exercises. o Software: the system presented in this thesis does still a prototype. The research is focused on the validation of CARS in CEn projects (design and construction phases). Even though the CARS is not a final product, the evaluation covered how reliable the system is during the exercises. o Technical – Hardware: GPS receiver and quality of output, HMD, wave LAN, and inclinometer and magnetic compass are also

191

evaluated. The output of these devices highly affected the field users’ experience. • Tasks – what users did during the exercises? o Instructions: how instructions are passed to field users. The role of the office user as a collaborator in the VE. o Reaching specific goals: how the tasks are performed by users (although the evaluation run over a formative approach), if they could perform what is requested, the difficulties that they faced, etc. • Interacting with the System – how users bound with the system. o Behaviour: How users behave when exploring the CARS. o Immersion: Is it an AR or a VE experience? • VE Quality of the Scene – could users identify virtual objects/spaces and associate them with real ones? What users expect to see when exploring a VE? o Texture and colours: could users indentify where they are inside of the virtual house (bathroom, living room, etc.)? o Dynamic Objects: Does dynamic object bring any improvement to the scene? o Reference Points: Are users orientated when walking in the RE with the see-through HMD on? o Avatar: The visual aspect of the avatar and how much it represents during the experiment. • Office User’s Control – the aspects of the remote control of the VE by the office user. Communication, collaboration and mutual understanding are not evaluated in this subchapter. The three keywords are commented in the next subchapter (see §6.3)

6.2.3

Outcome from the Exercises

The following paragraphs are structured as the evaluated topics introduced on Figure 6.20. They summarise the main points of the video material recorded by the

192

cameraman and the author – as office user – during the exercises and interviews. The topics are commented along the excerpts. For the sake of anonymity field users are identified as: Engineer1 (E1), Engineer2 (E2), Engineer3 (E3), Engineer4 (E4), Architect1 (A1), Computer Scientist1 (C1), Computer Scientist2 (C2) and Ordinary user (O1). The exercises are identified as: tram station (exercise 1), house 1 (exercise 2) and house 2 (exercise 3). Technical – Software Reliability As mentioned in §5.3.1 the computer carried by field user was running eleven parallel processes with two data bases at same time. The system is constantly reliable until some external factor affects its behaviour. Mostly this is an issue regarding to the wave-LAN. The router is not strong enough to send the signal further then forty to fifty metres. If eventually the field user is out of the range, the audio channel stopped work as well as the other process making the system crash. The lost of connection makes field user completely disorientated without the power to restart the system by himself. E1: I do not know where I am going. I am picking up the dialog screen at the moment that it is picking up some of the environment. The brightness is also wrong and I can see only the physical world and not the VE. If you can get rid of these dialog screens it would be helpful. You cannot probably do it from there. In a second occasion during a different exercise E1 faced the same problem again. E1: Oops, it has failed and I have an error screen now. This is the only moment that a field user went through a problem like this. The solution is to place the wave-LAN router closer to the area where field users are performing the test. The best solution would be to place a powerful antenna to cover all the area plus some hidden points where a standard wave-LAN router could not reach.

193

Technical – Hardware RTK-GPS The RTK-GPS receiver used is not as small as the models used for car navigation for instance. It requires heavy duty battery, cables, radio transmitters and a pole with a specific antenna. The antenna by default should be above the user’s head (around two meters) to reduce multipath. The pole is part of the backpack frame carried by field user. Ergonomic – In overall the system is not ergonomic. It is heavy specially because the weigh and size of RTK-GPS receiver. The number of cable also makes the system complex to setup and to carry. Along the exercises it is noticed that a number of users getting stuck with the GPS antenna in the branches of a tree. E1: I got caught in the tree with the GPS receiver because I could not see the tree! E2: I think that I am catching a tree with the antenna. E3: You have asked how immersed I was and that was how immersed I was in a tree. A1: I need to make sure that I will not run into that tree. For the evaluation point of view there is a mix of conclusions regarding to hardware design and user immersion in the VE. The high of the antenna can make the difference for the ergonomic perspective. Field users like E2 and A1 are no more than 175cm height. It is possible to imagine that they did not think in giving some extra room to pass below the obstacle. However E1 and E3 have something close to 200cm height. In any circumstance they would not go towards the tree direction. This can drive to the HMD setup hypothesis (see below). If the device is too dark the user will not see through the lenses. But the tree was no more then 10 meters far from the users during the briefing of the project. They saw the tree and they knew that it is the only tree around. Users can get too immersed in the VE forgetting completely about the RE and the hazards that this can offer (see below).

194

Accuracy – The default output rate of RTK-GPS receiver is one sample per second. This is the fastest speed that the receiver can work when calculating RTK-GPS coordinate in real time in the receiver. When users were in the field, they needed to bear in mind that they should walk slower than usual. The rendering of the CARS is affected if users speed up the walking, making the coordinates far from each other. The excerpts bellow show users having the sensation that the VE jumping from point to point. E1: The environment is jumping a lot at the moment but if I steady on, it gets better. E2: The image is jumping a little bit. A1: It is difficult to track, it is very jumpy. A1: The frame rate makes it sometimes difficult to orientate yourself. I guess you learn to live with that. It is sometimes difficult to use because sometimes you cannot necessarily fix your view to a digital object. It just jumps over that point. C2: It is jerking a little bit as I move but I still have the impression that it has a consistent position in front of me. The only way to go around this problem was walking slower than usual. It is suggested something between 0.5 to 1 meter per second. However there is no rule for this and users should find the speed where they would feel comfortable with the visualisation. MASSIVE-3 already renders extra frames between two coordinates but this also find a limit by the speed of the person in the RE. The Kalman Filter implemented does not bring too much for the coordinates. In fact, GPS receivers already contain a Kalman Filter to reduce high discrepancies between coordinates. Setting Up – every new exercise requirs users coming back to a place where the system should be setup again with the new VE. This place is very close to a building where the GPS signal is mostly blocked. After the setup of the VE and the restarting of all process, GPS receiver must to stay exposed to clear sky to fix the GPS signal.

195

Finally the process could start again. However eventually the receiver does not fix GPS signal long enough, loosing precision after a short period of time. E1: The main problem is the perspective. I do not feel that I am walking towards or through it. By default CARS is set to be at (0, 0, 0) position when the GPS receiver is not updating the data space with new coordinates. This became very disturbing for the user because the person could walk in the RE but the VE is fixed to one position. Meanwhile inclinometer and magnetic compass are working meaning that the image is following the movement of user head. The solution relies in leave the RTK-GPS receiver fixing GPS signal for longer. Technical – Hardware Inclinometer and Magnetic Compass The inclinometer and the magnetic compass attached in the crash helmet are responsive to the iron in the frame of the backpack. This caused a jerking due to this soft magnetic field. The jerking was considerably reduced by the implementation of a Kalman Filter. The reduction of jerking can be seen in Figures 5.10 to 5.12. Some of the reactions from field users are presented below: A1: When I stand still it is ok but when I move it jumps a lot. I think that it is the frame rate rather than anything else. It is a very low frame rate. C1: Because my head…oh god, this is really creepy… because my head keeps moving… I feel like I’m in the tram. Because my head keeps moving this jumping drives me really really mad. C1: If I stand still it is fine. If I turn my head really slowly it is also fine. I need to restrict my behaviour and movements to suit the VE, what is not natural for me. C2: There is a bit of delay when moving the head. The market already offers some other devices that are not so affected by magnetic fields – (Xsens, 2008) for instance. The reduction of jerk through Kalman Filter brought another problem: the small delay in the movement when users turn their heads too fast. Figure 5.10 to 5.12 already shown this delay when the graph lines are

196

not very close together. This delay can be perceptible by users causing some discomfort. Technical – HMD The HMD used was not designed for AR systems. It is a personal wearable screen used to watch movies for example. The size and the weight of the device are reasonable good as well as the quality of image. However this device was designed to work in controlled environments like inside of a room for example. The light of the room can be controlled making the environment stable. The Sun light and the shadows of clouds makes the setup or contrast and bright barely impossible when using this equipment outdoors. If too dark, users could not see the RE, affecting then the AR experience. Users cannot see the VE if the environment is too bright. There is no auto adjustment for the device for variable conditions. Another problem in the design is the light coming from around the frame of the goggles. The external light is usually much brighter than the one filtered by the lenses. This light became inevitable uncomfortable for users. E1: I can see the tram and the VE but I cannot see the physical world so well behind it. E2: I can see something! It looks like street lights! E2: The lenses are a bit dark but I know where the cameraman is (pointing with his hand) and I can see grass. I have to say that, the part where I see the tram, I do not see-through. I can see the bottom part and sides. I think that there is quite a lot of light coming from the side of the HMD. E2: I trust the cameraman because I do not know where I am going. E2: When I am looking to the direction of the embodiment, there is a lot of light coming from the Sun and I cannot see properly. A1: It is the usual thing; I cannot see anything through the glasses. I have to look under the glasses to not fall over. It is just the VE in the glasses. It is too dark, the HMD set up. A1: I am looking right into the Sun so I cannot see anything at all.

197

O1: I can see the physical environment below but not in front. The HMD shows to be a critical device on the whole experiment. This is in part not a surprise once the device is not designed for this purpose. However this is the only device available and the design architecture created for this project requests a seethrough HMD. Some of the users put their hand in front of the device, blocking the view of the RE to be able to see the VE. There are even users that walked completely “blind” in the RE for long time just relying in what they are seeing in the VE. It is important to mention that none real object was mapped in the RE to populate the VE. The problem of the adjustment of see-through is so critical that during the evaluation E2 is requested to look up to see some details in the VE like CCTVs (exercise 1) or a hole in the first floor of the house (exercise 2). Eventually his vision is blocked by the light from the Sun that came from the opposite direction making him temporarily blind. The opposite happened when he looks at the same direction where the Sun is pointing to. The HMD became too dark to see the RE. When doing the evaluation, one of the concerns is to understand what sort of immersion the user could experience. Users were asked if they could see VE and RE seamless. Eventually, the recorded videos show users totally immersed in the VE without noticing the RE (see more about this topic in the following paragraphs). Technical – Wave-LAN The router used for the wave-LAN connection was designed for domestic use only with a very limited range. As already described there is problem when a user moves over the router’s range causing the crash of the system. For a real application, the solution relies over wireless Internet access or a powerful antenna. Tasks – Instructions One way to make sure that the field user is going in the direction suggested by the office user in the RE is to ask him to look for the office user’s embodiment and to

198

walk towards it. The aim is not only to show the presence of a second user in a collaborative way but also to pass instructions from the right position in the VE. The excerpts and the comments below exemplify the way that the instructions were eventually passed to and received by field users: E1: I can see your embodiment and when you move. Each user has a different approach when receiving a command or orientation. There are people to whom just a request like: “-please turn right and look for the CCTV in the lamp post” is enough as a command. They know what to look for and even if they could not find the target immediately they would glance over the VE on their right side that could look like a lamp post and then a CCTV. For E2 this command for example had to be slightly different. When he is requested to turn right he replied with the question: E2: how many degrees? After that, every time that E2 is requested to turn left or right, the number of degrees has also to be included in the command. During the exercises, E2 could not find the CCTVs and dropped the idea of looking for them. Not letting the user down is one of the points to ensure users engaged in the system. In case they cannot find a particular point, the task is moved to an easier that is prepared to keep the user motivated. In this case, the user is requested to look for a bin very close to the lamp post. The user replied that he could not find it. However, because the office user is following the embodiment of the field user, he knew that from that position it is possible to find the bin. E2: I was looking too high. I have the bin right in front of me. At this point E2 is better understanding the technology and he learns that the movements with his head should be slower and smoother then the usual. By doing this, E2 manages to find the lamp post and the CCTVs.

199

E2: Yeah, I can see two yellow CCTVs! Interacting with the System – Behaviour / Interaction The evaluation shows that users have different behaviour when exploring the CVE. Users that are used to CVE and CARS experiments can just walk, exploring and interacting with the system in a very natural and loose way. There is a feeling of no surprise and they almost can predict what they are due to see and how the system supposes to work. On the opposite way, it is noticed a mixture of surprise and exhilarate for non experienced users since they started to wear the equipment. Everything is a big novelty making them react completely different from those with some background. Some of the excerpts show how field users behaved when exploring the CARS. Some of the users for instance does not like the idea of go through the objects that compose the VE: There is a moment during the exercises that field user is guided through the VE where key points are demonstrated as well as the digital tram (exercise 1) that runs as an animation. E2 walks around the green area following the guidance indications from the office user until the digital tram is in front of him. E2: Do I cross the tram? In the physical world it is not natural to walk through objects. It is noticed, in a number of occasions, that users can became afraid of walking through the elements inside of the VE. They tend to avoid them by walking around or asking for a way in. E2 is them invited to explore the other side of the tram station, walking over the digital platform stopping under the tram shelter. To move from one side to another the user should have walked through some of the digital objects. E2: Can I go through the post? Can I walk around it? I will try to walk around it.

200

E2 walks sideways and very slowly avoiding virtual objects. He eventually manages to find his way around without being run over by the virtual tram. The same behaviour is noticed during the first exercise with E3. He gets surprised to be run over by the virtual tram more than once. The video also shows that when he is exploring the VE, he tries to avoid the objects by walking around them. C1 has difficulties in understanding that it is possible to walk through digital objects. C1: And the fact that I can walk through all these objects is very disturbing. As with other users, C2 “jumps” when he is run over by the digital tram. When comparing the RE with the VE, C2 states that the RE should be marked, indicating the “dangerous areas” like the tram rail. Such boundaries could increase the sense of realism in the scene. Even though field users could not move the arms of the avatar the video shows that gestures representes an important way of communication. Users tried to explain situations or point to places when interacting with the office user. For the prototype presented the only way to point is moving to the place and stop facing to the object. The office user should follow the field user avatar and stop in the same position looking to the same place. This is not a real option especially for complex VE or places where the RE is in constant changes like in a CEn site. Some examples of the body gestures during the exercises are: In the beginning of the second example E3 waves for the office user’s embodiment in the VE saying: “- Hello!”. The video shows that E3 is really talking and waving to the office user in the VE and not for the camera. A1 points the necessity of gestures as way to interact with other users sharing the CARS. Throughout the time that A1 is exploring the first exercise (tram station) he is asked about the effect caused by the office user’s embodiment in the scene. There are advantages and disadvantages and the avatar complements the audio channel

201

communication and collaboration in the VE. The embodiment without the audio would not make too much sense. It would become rather difficult without audio commands like: “Please look to where the embodiment is looking at”. At same time the audio without the embodiment could also make the exercise very difficult. A request to go to a particular place to check a specific detail in the VE could become a hard task, especially if the VE is designed without virtual landmarks. The users could be completely disorientated and frustrated if they could not find the places where they are being requested to go to. In the third exercise, for example, details like showing a hole in the ceiling are better performed when the office user asks the field user to move towards and look to the same direction as him. A1: I like the idea that I can talk with somebody, as an embodiment, and follow it and look to roughly same direction that it is looking at. You can think in the embodiment in the level of expressiveness that it’s got. And currently it does not have that much of it. At the moment, the office user can say “come to where my embodiment is”. And probably you can just about say “look to where my embodiment looks”. But that’s where it ends. So, the principle makes sense but I guess that you cannot do really much with it. You cannot really point things with it. You need the audio. If there is something on my left for example you cannot point and I cannot point either because there is no interface to capture the movement of my arms. C1 also uses his hands to indicate things inside the VE even knowing that his gestures could not be seen. The video shows C1 talking with the office user at the same time that he gesticulates to confirm that he is going towards the right direction. Interacting with the System – Immersion The immersion of the user in the CVE is much higher as desired. Most of the users could not see or have the feeling of VE and RE merged. The problem could rely in several aspects like: the low quality of HMD for the task, the low quality and lack of texture of the VE and the excitement of users in exploring the VE and not the AR concept. This immersion is partially over taken after the first exercise. Users starts to understand how the system works and become more independent of the field user. The paragraphs below show some examples of this point: During the first exercises the video shows that E2 is completely immersed, not taking time to see the VE from some distance. He does not ask if he can take any

202

decision, instead, he just follows the office user. E2 stopped walking when he is sure that the other embodiment is big enough to block his view. E2: I think that I am close to you. You are quite big now. During the first exercise, E4 is so impressed by the office user’s embodiment that he forgets to explore the VE during the initial walk. For a considerable period of time he is fixed with the idea that he should follow the embodiment in order to see what it is doing. The office user has to make sure that E4 understands that he can explore the VE by his own. Inside the VE, the field user is totally independent from where the other embodiment is or what he is looking at. As with E2, E4 understood that if he stays too close to the field user’s embodiment he will not be able to see the VE properly. He decides then to open more space to see the VE better. He is requested to look at the lamp post on his right, once he was over the platform. Only then he notices the lamp post and walked backwards to have a better view and from where he could also see the CCTVs. Some of the users lost the sense of direction when surrounded by a virtual object. This happened when they are going through an object or when the VE is swapped to another one during the exercise 3. Most of them just immediately stopped as they are inside of a virtual object and could not go through it. A1 comments during one of the exercises the lack of sense of direction: A1: I do not have a physical sense that this was the direction that I came from. My physical orientation is completely gone because it is too immersive. Throughout the tour E3 easily identifies all the key points of the VE like the stairs or the gap in the roof of the first floor showing some parts of the second floor of the house (exercise 3). The user is immersed and amused by the system until he lifts up the HMD and says: E3: Oops, I did not realize that I was over here! If you had asked “where are you?”, I would not know. I could see the white line there in the ground, that was quite useful.

203

VE – Quality of the Scene – Texture and Colour One of the points rose by users is the lack of textures in the VE. Exercise one is well accepted because in RE, the tram platform is all painted in solid colours and the VE is created using the same colours tones. However users request more realistic textures like brick walls for example in exercises two and three. Another unexpected problem is regarding to one of the field users (E3) be colour blind. The embodiment in the VE is red with white arms and E3 had some difficulties to find it. The problem increases when the office user embodiment passes in front of an object where the combination of colours made it disappear to E3’s sight. O1 for instance has some trouble in understanding and navigating the CARS with flat colours. O1: It is lot harder to find my way in this one than in the tram station. The colour scheme has some part in it because most of the structure has the same colour. They can look like the same objects. Also the field user’s embodiment does not stand out as it did in the tram station. Even with the lack of texture the system provides a fairly realistic view of virtual objects in the RE. During the third exercise, the office user’s embodiment stopped in front of the stairs indicating where C2 should stop too. The user liked the idea that he could look through the gap in the ceiling to the second floor of the virtual house. From his position he could see a window in the second floor in the VE, which for him it is fairly realistic. VE – Quality of the Scene – Dynamic Objects Users are also questioned about the dynamic object in the scene, in this case the digital tram. A1: I do not think that the tram adds anything in the scene more than the same tram static in that positioning. If you have a static tram there I will understand that there is a tram that passes there. It is actually confusing. Now I am so close and if it moves past I could not say what it is. I cannot recognize anything I can just see polygons. If I have an overview then I can see it. I imagine that, for an

204

example where you could see the flow of air, it will make more sense. With the tram I am not so sure. Because of the field of view, I have the impression that I am standing too close to the train where in the physical world I would never be so close. Perhaps the digital tram made the scene too complex to be seen. The tram is higher that the tram shelter blocking the user’s view temporarily, until it moves away again. There are two other points that could make users also disturbed. The first could be the fact that the tram does not stay away from the digital station for a while, in order to open the view. It keep coming and going constantly not giving proper time for the tram station stays for a little while without the moving digital object (5 minutes away for example). The second problem could be the way that the tram moves. The coordinates should be closer to each other to maintain the smoothness of movement. During the exercises O1 put in a question a good idea that could be possibly become a future implementation of the system. O1: If I go through the tram will I move with it? The user could virtually travel in digital objects each time he approaches one. If he moves again his view would come back to where he was previously in the CVE. VE – Quality of the Scene – Reference Points A1: It is really difficult to see what is what now. I understand that you gave me different phases and I’m just about to understand that they are at the top of each other but basically they just look like a lot of polygons. The field user is lost due to the lack of a clear indication of the rooms that he is visiting in the VE. The physical world contributes to the loss of navigation, once it is flat with no reference points to aim at, such as the tape marking the boundaries in the ground (shaping rooms for instance) or a landmark to indicate where the front and the back of the house are. Users also suggest that the VE could have something like furniture to indicate the rooms of the house at the same time that it would help with the navigation.

205

As other users, A1 thought that the VE should be aligned with the white line in the ground. A1: In my idea if I want to explore the VE I should walk over this line. If I do it now there is nothing to see. It is just a comment it does not stop me from using it. The white tape placed in the ground is just roughly delimiting the position of the virtual tram station and up to where users could explore the VE. This information is passed to the users during the instructions prior to the exercises. However, most of the users argued that the VE (in this case the platform of the exercise 1) is not being projected exactly over the tape. C2 felt that he is being unfairly disconnected with the physical world around him, claiming that there are no landmarks that could serve as reference points, neither in the VE nor in the physical world. Like other users, he questions the relation of the white line in the ground and the VE, believing that the white mark is the limit of the house (exercises 2 and 3). VE – Quality of the Scene – Avatar The field user is then asked about what he thinks of the field user’s embodiment in the VE and whether it makes sense to him. C2: The embodiment as I see is not human enough for me to feel like I am actually “seeing” someone standing in front of me. I have a bit of difficulty with it. It does seem to move around and that does make me feel more like there is actually something there as such, rather then there being nothing there. As I said it is not particularly human enough for me to think that there is anything other than an object in the environment. C2 as well as other users mention there is a lack of empathy in the avatar. As a general conclusion both field and office user’s avatars should look more like a human being as well as be able to move their arms in order to make gestures. More about empathy can be found in the following subchapter. Office User’s Control The office user has not only the important duty to guide and collaborate with the field user when in exploring CARS. The manipulation of the objects is considered a

206

complex task and could not be performed by field user without a previous training. The office user help becomes even more important once none of the invited users could manipulate CARS using the handheld keyboard or understand the logic of commands. Field users could see along some of the exercises the demonstration of how the VE could be rearranged while users are interacting in the CARS. The virtual objects could be dragged and dropped by the office user’s embodiment (one of the features of MASSIVE-3). The field user could then take a better position in the physical world in order to see some of the changes that are being made by the office user. The stages of the construction are also demonstrated with CARS being manipulated remotely the office user. Each of the phases is demonstrated progressively, adding each new layer on the top of the other. After the repetition of this procedure for a few more times, the phases are shown in an alternate order. All these procedures are enriched by the possibility of communication between users.

6.3

Final Points over the Evaluation

The evaluation process helped identify a number of points that could not be predicted during the many phases of the project. This was the moment when all that was developed in the first few years was finally put together. The outdoor evaluation was fairly different from those that were tested in a controlled place. One important decision taken during the evaluation was to invite only one person each day. All the difficulties and unexpected errors were better solved and it was possible to learn from them. Users also did not get influenced by other users during the exercises. The evaluation process was not limited in observe how users interacted and explored the system during the exercises. An interview was performed just after the third exercise in order to complement the users’ feedbacks. Most of users answered the same questions. Only few questions were tailored in order to cover particular points experienced by the users during the exercises. 207

The questions and answers are fully transcript on the documentation provided in Appendix C. The same nomenclature for the field users is used to indicate who is answering when there is an excerpt. Figure 6.21 shows the areas covered in these final points over the evaluation.

Figure 6.73 Grouping Points of Interest for the Complementary Outcome

Using the System – Self adaptation Users had to adapt themselves to the system. The update of the coordinates was not fast enough to make the calculated movement inside of the VE smooth enough. All the users were reminded to walk slower than the usual, but not all followed the instructions. GPS coordinates were being updated each second and the tilt sensor had its output processed by the Kalman Filter to reduce jerk helped to make the refresh rate of the view slower than usual. The system was stable when users stood still and smooth when users made slow movements. Only two users tested the system before. Users that followed the instructions commented that they had enjoyed the view, especially those that had previously tested the system in the early stages remarking that when standing still, the image was perfect. Using the System – Immersion During the exercises it was noticed that users were experiencing a total immersion in the system instead of having an AR experience. There are key factors that could have contributed to this happening: • Regarding to the VE perspective users felt more familiarised with the tram

station than any other example. The colours and the shapes were very close to the ones in the physical world. All users that evaluated the system once saw how the tram station looked. They already had a previous knowledge about the scenario, which made them more comfortable with what was seen. It was 208

noticed that users with previous knowledge of what they were due to see would have a better experience when exploring the VE. Users put their focus in the VE not giving chance for merging real and virtual. • There was clearly a process of selective attention when users explored the VE. As mentioned in (Edgar, 2002) “physiologists often regard attention as acting like a filter”. Edgar quotes (Posner, 2002):
This “attentional spotlight” that illuminates only a small part of the visual field (i.e. only a small proportion of everything that is registered by the eyes) and any items that lie within that spotlight receive priority for processing. Any information lying outside the “beam” of the attentional spotlight receives far less processing.

In this case the RE is “out of the attentional spotlight” making users more focused on what was being projected before their eyes – the VE. • There was a disconnection between VE and RE. Landmarks (like tapes drawing the shape of a house for instance) would help users to fix with the idea of an AR system aiming to project a virtual object representing a house or a tram station in that particular place. • The see-through HMD cannot adjust contrast and brightness itself. If a cloud covers the Sun light the image becomes too dark. If the cloud goes away, the projection of the VE becomes very fade. Another problem was the lack of stereoscopy. The images projected in the HMD are not three dimensional. • Some of the users talked about the lack of texture, which could bring more realism to the scene. Users were looking at objects in the VE, but not connecting them to real objects (i.e. a house did not look as a real house). • One user said that the dynamic object (the tram) had a negative impact during the experiment. Perhaps the object could be better explored in the scene. The tram could run smoother, not so often and with some sound effects helping to pass a better sense of realism. It was noticed that the virtual tram often obscured users’ view of the VE. Virtual Environment - Overview Texture and colours were one of the points that raised more comments from users. Flat colours were not what users were expecting, especially when exploring a VE

209

representing a house. Just one of the users who were interviewed said that plain colours did not affect him. High resolution texture would not be a requirement but something intermediary that could pass the idea of a brick wall in the VE and not flat brick colour only. The outcome from users just confirmed what they expected from an AR system: the VE should be presented as close as possible of reality. Originally the colours of the houses created for the exercise were choose to give as much contrast as possible (to show different phases of a construction) making it easy for users to see it. The tram station in the RE was painted in plain colours and users could easily identify the tram shelters, lamp post, bins, etc. Two of the users shared their view about the tram station: E3: It was clear that it was a tram station and it really helps the tram going in and out. It seems that inside the tram it was very detailed. That was quite realistic. The outside of the tram was also good. I liked the lights as well and when I could see the CCTV cameras up in the pole…and the bins. Yeah, everything was fine. O1: The tram station was really good because it has the same curves and same objects that the real one. So I could say straight away what it was. Users were then questioned about how they imagine that different phases of a project could be presented once texture was a requirement. The suggestion was to use the same texture but with slightly different colours. However, the issue of lack of realism still remains when not using a “realistic” texture. As E3 stated: E3: You’re gone decide what you try to represent. From this point of view you try to represent the reality (pointing to the blueprint) and you see the final thing. So texture helps there. If not, then it becomes close to a graph. It is a threedimensional graph in which case putting colour in, highlight in, putting different colours for different phases makes sense. Another point raised by users was the fact that walls (especially in the second exercises) were represented by thin facets. Users claimed that a wall representation should have at least the same thickness as a brick. A window should also look like

210

one and not like a blue rectangle. Users could not associate the virtual object with the real one. During one of the trials E2 asked: E2: Should I go towards the blue rectangle? If I go back I can see the yellow roof. Some other users commented about the lack of virtual furniture in the houses. The furniture for them could help to better understand a couple of points: • To indicate an approximate size of the room. The previous knowledge of the user about the dimension of a double bed for example, could give an indication of the size of the room. • To make it clear that the visited place was a potential bedroom because there was a virtual bed in that place. To use or not of furniture would depend of what the VE was proposed to represent and in each phase of the CE cycle. During the design phase virtual sofas, beds, kitchen, etc. could be added to enrich the scene with details helping architects to give a final touch in the project. However, if the system is being used along the construction phase it would not make sense to add such representations. More valuable information would be required like technical information over the construction specifications. It was noticeable, both from previous experiences and also in the video of the current evaluation that most of the users tried to avoid the digital objects. Some users got frightened, for instance when something suddenly crossed in front of them, like the tram for example. The answers show that there was a mix of feelings about going through the digital objects. It is not natural for humans to walk through walls but this is one of the features of the technology. The object was not there physically and the field user can go through it because it was a shortcut or it was the best way to avoid a hazard in the physical world (especially when considering a construction site). The design of a realistic VE should not create barriers but extend the experience and create possibilities for users to go beyond their natural limitations. The field user for example could fly over the scene – although this feature was not explored. Such

211

extension of capabilities (to fly for instance) would become very useful if an architect would like to show the design of a house with the gardens and the swimming pool. Landmarks could be better explored during the exercises. A single line was not enough to only indicate where the VE would be projected. The shape of a tram station, house or any other VE would make users better connect what they were exploring together with the RE. This would also demonstrate the level of accuracy when using the RTK-GPS receiver. Regarding the objects that could have dynamic behaviour in the scene, E1 made the following comment that summarises the majority of the user’s views: E1: I really like the ability to have the moving train going through. Not only on the conceptual point of view of being able to do that for example for wind flowing or things that are usefully scientific, but actually I think it helps give reality to the environment. Hardware – Overview For an outdoor AR system, RTK-GPS receivers are usually the most expensive part of the hardware involved. It is especially true when there is the necessity of more then one equipment. The price becomes one of the key points for researchers when choosing which GPS solution they want to add to the design of the project. Apart of the costs, RTK-GPS receivers also have issues regarding to size and weight. The receiver does still big and heavy when comparable to other GPS solutions. The GPS receiver used in this project is not the kind of equipment that is broadly accessible. But as mentioned by A1, costs are decreasing since the number of companies in the market offering high-quality products is increasing. Another limitation is the output frequency of the receiver (1 MHz). However this is temporary obstacle that will be surpassed with the natural enhancement of technology in a near future. The area choose for the demonstration was wide open, with buildings at enough distance of did not block the GPS signal or create too much multi-path error. All the coordinates from the GPS receiver and tilt sensors went through a Kalman filter that reduced the jerk considerably. These precautions resulted in a more stable visualisation when compared to the very first trials. However, if field users walked at

212

their normal speed, the refresh of the frames was not fast enough giving them the sensation of a “jumping” VE. The head movements should be done slower then usual. As a prototype, it was not primary concern how ergonomic the introduced set of equipment was. It was a fact that all parts together made the backpack carried by the user not light. Batteries and GPS receivers, as well as the frame of the backpack, made users tired after some time. However the weight was not the only part that should be reviewed. Throughout the evaluation, at least one user had difficulties to wear the crash helmet. For him the helmet kept loose even with the strap that keeps it stable to the head adjusted as tight as possible. Another point was the number of cables to connect GPS antenna to receiver; receiver to external battery; receiver to computer; etc. During the exercises the cable could get eventual twists making very difficult to handle. There were several fragments of comments about the HMD in the text. One of the questions for the users was: “Was it the correct way to display the VE? What about handheld screens like PDAs and tablet PC?” Some of the users said that it was the right choice for engineering purposes but the wrong one for other applications. At least two of them said that they would prefer to have it as a visor in a helmet like the ones used by jet fight pilots. A1 gave an interesting answer: A1: I do not know if it will work better or not. What is it for? If it is for a client of an architect, I think that a hand held will work better. Because then both will work on it. But if it is a troubleshooting application with someone back in the base and another person in the field this is much better. But to design something for a client I will use something else. I think that one issue with the HMD is the FoV. You just cannot see enough. A1 statement can be complemented by another issue raised by (R.T.Azuma et al., 2001) regarding to the social acceptance and the user’s limitation of interaction. These two points could become a real dilemma when asking people to wear the equipment to explore the VE. The social acceptance dilemma becomes real to this project in the very early stages when people felt reluctant to wear it. The equipment is very different from a normal outfit, making them reluctant to wear/carry something like that in public. The equipment, due to its weight and the cables, also limit the movement. A user with movement disability or not strong enough to carry the 213

backpack would find difficulties to test the experiment. The advantage to use a handheld screen, as mentioned by A1, is the portability and the way that it can be shared with other users. Office and Field User – Communication, Collaboration and Mutual Understanding The evaluation makes it clear that for the AR system proposed on this thesis two people were necessary to orchestrate the work. The introduction of complex tasks made the field user even more dependent on the person in the office. On the other hand, office users placed anywhere around the world would had difficulties to obtain a clear interpretation of their questions if there was no field user to investigate. The collaboration pushed the control of the VE to the office user, but still giving field users autonomy and power. Field users also have some control and responsibility over what both were sharing. The tasks performed by the person in the field could be exemplified/enhanced to: show particular areas where the construction could face temporary delays, point out good and bad aspects in the construction during the investigation, scale and measure objects, measurement of the physical world, identify features, update information, surveying, etc. Communication – The exercises showed how important the office user was to guide the person around the field in RE. The audio channel was essential to complement the experience of collaboration. When field users could not find a particular point in the VE during the exercises, the conversation proved to be fluent and useful. The audio channel enhanced the collaboration and made easier for users to understand each other with the help of the embodiment. From the office user’s perspective to give directions was easy because it was possible to know what digital object was on the left or right of the field user. The indication to where the field user should go were all based in his position related to the VE and if he was turning to the correct direction. E4 contributed with the following phrase that summarises the feeling of the audio and the collaboration: E4: The audio is critical because you are the designer. I did not know until I arrived there what I was going to see. You gave me the instructions before we started explaining the procedures to explore the scenarios. But only when I went 214

to the field I realised in reality what you wanted to achieve. I think that communication is critical for people to understand. Collaboration – The system proved itself that without a second person sharing the VE, field users would easily get lost and unmotivated to keep exploring it. Users were fairly busy trying to understand what was being presented before their eyes. They could perform the control of very simple tasks like to type one of two buttons in a personalised keypad (with only two keys perhaps and not with something like the handheld keyboard introduced). However, more complex tasks would become considerably difficult for field users to perform alone. There are strong reasons for the argument of the need of collaboration, especially after the videos showing some of the field users walking towards a tree in the RE. Field users in this evaluation were just walking. There was no equipment in their hands that could distract their attention. All users that hit the tree were looking towards it. As mentioned before, field user’s “attentional spotlight” was in the VE and not in both VE and RE. To give more control to field user would also mean to put in risk even more the user’s attention. The “attentional spotlight” would change more then ever its focus, switching between: VE, RE, tasks, etc. One of these spotlights can require more attention (resources from the person to focus in what he is doing) from the user, cutting more and more the AR experience. The collaborative process helps field users to be free and keep the focus in what they should do. Mutual Understanding – The commands given by the office user through audio channel were based on the field user’s movements observed in the VE. For example: “Now you should turn to your left until you see a virtual tram shelter”. Commands like this were widely used to show key points in the VE. Such interaction could only be possible because the VE was being shared by both users. The audio channel alone would not be sufficient to transmit all the instructions to the field user. The instructions can only be completed by a combination of audio and the office user’s embodiment in the VE. Even with almost no body language, the embodiment facilitated the indication to field users of where to go and what to see.

215

For some users the embodiment could have been more similar to a human being in order to create empathy, especially if in the future tools like this are introduced to general public. Preece (Preece, 1999) discusses the empathy in symmetric and asymmetric ways of online communication. It is possible to create a parallel between Preece’s study and what could become CARS in the future: Research in psychotherapy supports the assumption that empathy is communicated better face-to-face. Facial expressions, eye gaze, tone of voice, body language and touch all have a significant role in communicating empathy (Eisenberg et al., 1999, Etchegoyen, 1991, Lanzetta J.T. and Englis B.G., 1989). Nonverbal communication can be so subtle yet powerful that the communicators may not even be aware of the messages that they convey via their body language(Ickes, 1997). The experiments showed how common understanding between users was enhanced when they can “see each other” in the VE even with an embodiment shaped by geometric primitives. Field users tend to copy the embodiment’s movement of the office user either walking to the same direction or by looking at the same place. Even with the embodiment’s restricted movements and body expression, the possibility to move through the space and move its head was highly important to enrich the demonstration of some places inside of the VE. The drag and drop feature available on MASSIVE-3 was also helped to point to virtual object even in a far distance. This feature showed a ray coming from the middle of the office user’s embodiment’s forehead towards the aimed object. Users reported that this was a very good example to demonstrate which direction they should look to. Choosing the Right Approach to Demonstrate Design and Construction Engineers and architects where questioned if the AR system presented was the best approach to visualize a construction during the design and construction phases. Mostly they prefer blueprints once they were trained to visualize the third dimension from the printed data. For the non-technical audience, 2D charts are more difficult to understand. In any case, the main issue was the low level of detail of the VE. All participants agreed that a more natural representation would make lots of difference in the way that the information would be interpreted.

216

Engineers mentioned that the tool was useful when applied in the right building project. In small projects like a house extension, for example, it would not make as much difference as if applied to a new residential area. Apart from the aforementioned limitations of the system, they could see the potentiality of this project, especially for underground pipes, health and safety training and Computational Fluid Dynamics (CFD) (CFD Online, 1994). E3 mentioned the idea about collaboration could be arranged between more than one field users. This idea would be particularly interesting because as he affirmed: “site inspectors could be back in a month to see the progress, for example”.

Chapter 7
Conclusions and Future Works
The design of the presented research goes back to 1999 when the author saw for the first time the project of Silva (Silva, 1998) where two military tanks reported their coordinates to a database at the same time that a computer displayed their positioning over a 2D map. The tanks at that time are not more than two little squares in the computer screen with all system working over DGPS corrections. This project is 217

inspired in idea of allowing people to be tracked with high accuracy GPS in real time, feeding and sharing a real collaborative 3D environment. This supposes to make users enhance collaboration, communication and exchange information, placed not only outdoors but whenever Internet connection is possible. The results of the work reached during the elaboration, implementation and evaluation of this thesis are very prominent for coming researches. Even though it does still a prototype and subject to constant improvements regarding to technology and new requirements the achievements are very positive. As a first point in the conclusion it is possible to affirm that the research achieved its objective introduced on Chapter 1:
The objective of this thesis is to investigate, create and evaluate a CARS able to explore communication, collaboration and mutual understanding on CEn environments.

The following paragraphs describe the achievements reached by this thesis with a brief summary over each of the chapters as well as the contributions for the knowledge. During the research period some papers were published as well as a television interview, collaborating with the technology dissemination. Finally a number of future works are introduced aiming to contribute to new ideas or to create a direct evolution from the design of the current research.

7.1

Summary and Reflections

In general the research helped to certify that a second person is necessary when an CARS is used on CEn environments. In order to facilitate the work and the availability of this second person an asymmetric remote allocate user (office user) is suggested to share the CVE. So this research concludes that: If people want to do immersive head mounted augmented reality for civil engineering purposes, users will need a network solution of at least two people, as applied in this project. In the virtual environment, the audio and the embodiment of those users in the office will work as a help resource, interacting with the person in the field. While one person has the immersion and the freedom 218

to explore the virtual environment seamless with the physical world, complex tasks must be performed by a second networked person in the office. This control coming from indoors will ensure that outdoor users are safe as well as looking to the right things while collaborating. To achieve such conclusion the work here presented starts introducing the three key worlds of this project: communication, collaboration and mutual understanding – Chapter 1. The key worlds are presented already towards the potential gaps that can be found in CEn environments. A CEn project is complex and requires several multidisciplinary teams that nevertheless can face difficulties over its ontology. It is essential that team should work collaboratively, cutting edges, helping to reduce time to achieve milestones, and reducing costs. Once the project investigates the use of a CARS in CEn environment such knowledge is introduced together with its several phases over its life cycle. Between many, only two of then are choose as target of the research – design and construction. Although the design of the current project has already some years it can be also certified on an extensive research presented by Shin and Dunston (Shin and Dunston, 2008). Once the initial gaps on CEn are presented Chapter 1 introduces with the help of a simplified diagram (see Figure 1.2) the proposed solution. Office and Field users are introduced for the first time together with some basic concepts of the technology that surrounds and link them. Chapter 1 finishes with the steps followed to achieve the thesis objectives and its structure. Chapter 2 introduces the challenges faced by AEC during a CEn project. The idea of an augmented architect is not new. Brate (Engelbart, 1962) had coined the idea almost fifty years ago. However, just now technology is good enough for such experiments. The text goes over the challenges faced by commercial companies eager to create standards in formats and software to be used by other CEn companies. The solutions provided already prove to worth the high investment – especially for small CEn companies – reducing costs and time during the projects. Researchers are more

219

focused on what to retrieve from data bases, which level of filtering, at what time, etc. They are looking for ways for AEC to collaborate each time more, exchanging information, pursing understanding when working. Chapter 2 makes clear that there are gaps in CEn environment regarding to communication, collaboration and mutual understanding. It also helps to indentify the place for this project not as an overall solution but a part of a solution of how to visualise data on CEn context. Chapter 3 reviews the concepts around the Virtuality Continuum first introduced by Milgram and Kishino (Milgram and Kishino, 1994). However, the core of this subchapter is the long review over CE once this is the focus of the research. This review is created specially for those not so familiarised with the topics covered by this area of Computer Science. An extensive survey of researches is the next topic covered by Chapter 3. The survey covers projects with direct or indirect application over CEn visualisation of information. The projects are selected by: how information is being explored by users; the way that information is being displayed; where information is available to be explored; and the positioning system used to explore the information when outdoors. It is possible to create a link of the necessities on CEn and what is being proposed on this thesis even though some projects have no apparently direct connection with the research. The survey once again makes clear that there is space for the proposed research. Most of the work done so far uses limited resources of AR systems, mostly no CVE, and limited use of positioning system – when in comparison with the freedom required by users when exploring outdoors. Chapter 3 finishes with a well informative sub-chapter covering GPS. This subchapter aims to give an “extra mile” of knowledge for those not so familiarised with this positioning system and makes a balance of the amount of information gave on MR systems. The goals of this sub-chapter are the demonstration of different methodologies applied over GPS and different results achieved when using such methodologies. The results prove that RTK-GPS – between GPS methodologies – is the best solution for outdoors positioning. 220

Once the theories and projects surveyed are reviewed Chapter 4 introduces the design for the solution proposed. A diagram illustrates how the system should work (see Figure 4.1) with Field and Office users sharing the same CARS. During the description of the design overview a more complete diagram introduces all parts involved when the project is fully operational. Such description does still in high level but already makes clear that a single keystroke has a complex and long way to travel from one computer to another passing through device manager, data base, and VE manager. Chapter 4 is complemented by Chapter 5 due to the complexity of the system involved. Hardware, software and platforms used to make the system run are revised also on Chapter 4. Readers are presented to the kind of message coming from GPS receiver, tilt sensor and magnetic compass. Before the initial implementation of the CARS the research puts its focus on surveying and creation of a VE – Chapter 5. The area around the IESSG building in the main campus of the University of Nottingham is mapped together with the features of the area. The project finally enters to a next phase when CVE starts to be implemented putting office users to share the same VE, exploring the area just surveyed in the RE. A diagram (see Figure 5.4) shows how the configuration is when two or more office users are sharing the CVE. This diagram introduces the processes involved to make the system run as a complement of the information already presented on Chapter 4. The processes are then more in deep described becoming an important source of quick information for future researchers aiming to implement the system with some extra functionalities. Chapter 5 also demonstrates the results reached when the Kalman filter is applied over the incoming data from the inclinometer and the magnetic compass (see Figures 5.10 to 5.12). The discomfort of the visualisation due to the jerking are at the time one of the main complains from users during some of the demonstrations. The implementation of the filter reduces considerably the discomfort for users when wearing the see-through HMD. 221

The final stages of implementation and evaluation are demonstrated on Chapter 6. For the evaluation three examples are created aiming to show the potentialities of the tool created, as well as the investigation of how users work over the three key worlds. Users are invited based on background and skills on different subjects. They are researchers, technologists, engineers, etc. The evaluation is taken using a formative approach where users are also motivated to think aloud to a hired cameraman. All users received previous explanation of how to explore and collaborate when using the CARS – cognitive walkthrough. The exercises are not rigid performed but mostly users are invited to perform the same tasks. Along the exercises users are already questioned about what they are experiencing. Answers are recorded for pos-analyses. At the end of the third exercise users are invited to an interview. Most of the questions are the same for users and some are tailored based on their experience during the exercises. One of the major aspects learnt during the evaluation is the real necessity of a remote person to perform complex tasks while there is someone exploring the CARS outdoors. This is especially true for field users that never received a training of how to explore the VE or how to handle a handheld keyboard. Nevertheless to reaffirm that the project is a prototype and it is not only the case of the hardware interface but also the software interface. To change from one VE to another or to move objects, users should know what screens to put the right commands – once there are more than 10 processes/screens running in parallel. All these screens can be displayed at the same time that they walk outdoors trying to understand the information that is being presented before their eyes. The complexity of how to handle the system can grow exponentially and users will get frustrate with the experience. During the evaluation, users are not exposed to such kind of complexity. The result is that no one mentioned about this issue during the interviews. Field users could see that the CVE is being changed, object could be moved but they are far from the complexity handled by the office user that has the facility/flexibility to work on it from his computer’s keyboard. The process of evaluation shows that users are eventually more focused on what they are seeing than with how they are seeing. One of the major points of discussion is the quality of the VE like: lack of texture, colours, shapes, dimensions, etc. The see-through HMD is perhaps the second factor that creates more negative feedback. 222

There is not possibility for auto-adjustment in the device as bright and contrast. Because the system is performed outdoors the luminosity is subject to constant changes due to clouds and sun position in the sky. Another lesson learnt is performance of the evaluation with the right users with the right use case. The exercises create for the evaluation are more focused on construction than design phase. Users liked more the first exercise (tram station – more focused on design) than the other two when comparing the answers. This is because the tram station VE is close to what it is in RE. Users could easily indentify the objects and colours. The same does not happen with the other two examples. The second example showed the different phases of the project of a house – that is not too far from the example shown by Woodward (Woodward et al., 2007a) (see Figure 3.21). Perhaps the right approach to evaluate the project is to take a real use case of a construction and show work over the result of those professionals responsible for CEn project – AEC. Another point is to make sure that the right information is displayed for the right user. High definition textures and virtual furniture is not relevant to all users. More technical specification like pipe dimensions, angles of the structures, etc can be directed to those involved more in planning and construction.

7.2

Thesis Contribution

The development of this project aims to contribute to those developing AR systems for CEn application and have low background knowledge either in CARS or GPS. The thesis shows the several steps regarding to the design, implementation and evaluation of such system. The main contributions can be summarised as follow: • Identification of gaps over communication, collaboration and mutual understanding in multidisciplinary teams on CEn projects. The bibliography review introduces a number of issues regarding to how ontology can become an issue when contributors are not working in the same channel of exchange of information. • A good effort has being done for the creation of a standard in order to bridge knowledge but very little regarding to outdoors activities. AR systems and 223

specially CARS for CEn it does still in its principle specially because: positioning system (lack of accuracy indoors) and HMDs (semi-transparent or not). • CARS should be treated as one of the branches for the output/input of information and not as a total solution. Information should be handled for an external system created to merge, retrieve, delete, add, etc. • An ergonomic system is far for ideal. High accuracy GPS receivers are heavy. The heavy duty batteries are also not compact and light. Computers are getting lighter and powerful but there are still problems over the HMD. There is no genuinely equipment designed for CEn application outdoors. Everything runs over an adaptation and naturally susceptible to downsides. The number of cables is also high making the system very fragile. A system with a label “handle with care” does not fit very well in a construction site. • For a tool like the one proposed it is natural to think that more features can be add (see Future Ideas below). Such features would require a better hardware/software interfaces and an even closer relation between field and office users. For the field user perspective the increase of complexity in the tasks performed does not mean the increase of how it is performed. Interfaces must be as simple as possible, using natural body movements, voice commands, and even eyes commands. In any case, complex tasks should be performed with the support of a second person. User can not handle a complex interface hardware/software while in the construction site. The environment itself has potential hazards like moving objects, holes, construction material, etc. • Chapter 2 demonstrates the number of hours that office workers need to spend with managers in order to fill up the gaps and resolve issues. This can be done remotely and almost instantaneously with a system like this. The system proves to be useful when used in the right place at the right time. • The elaboration of examples should be based on real projects. Better if the CEn project is under construction. AEC professionals will be able to see and say the positive and negative points in a solution like this. • The results achieved already shown the potentiality of the tool although there is no evaluation before the project be proposed of how communication,

224

collaboration and mutual understanding work. All users that invited for the evaluation did not receive any previous training of what to see, how to explore, how to behave, etc. Users arrived and received some information but even blue prints are not displayed to everyone. The communication during the exercises are really high with users putting questions and being instructed of where to go, what the features represent, how the system works. This high interaction between office and field users creates a collaborative environment making those outdoors motivated to keep exploring even when they are eventually loss. The collaboration does not happen in this case on the CEn level but in how everything works together and where it could be improved. Finally mutual understanding is clearly achieved when users can identify the explored places. From the first exercises (tram station) they almost immediately get familiarise with the VE. It comes from a previous background and they just need to apply this knowledge to what is being projected before their eyes. On the second exercises (the house) they can recognise the places when looking for the blue print. The video shows users pointing to the blue print showing and talking about where the living room is, the veranda, and the other rooms of the house. From the third exercise (second house) users can see the potentiality of the tool when exploring the CVE specially when looking through the gaps of virtual objects or exploring the scene between the virtual pipes. The thesis contribution from this level is the confirmation that communication, collaboration and mutual understanding are enhanced when using an application like this.

7.3

Dissemination

Along the years this project produces the following scholarship, poster, articles and television interview: The idea of the project is first presented in to Engineering and Physical Science Research Council. The project receives a grand under the number GR/R42696/01.

225

The poster “Combined use of Virtual Scale Models and Space Geodesy” is presented in Brazil (Capra et al., 2003a) during the VI Symposium of Virtual Reality. The authors are: Capra, M.; Aquino, M. H. O; Dodson, A.; Benford, S. The paper “Civil Engineering Application for Virtual Collaborative Environment” is presented in Japan (Capra et al., 2003b) during the International Conference on Artificial Reality and Telexistence. The authors are: Capra, M.; Aquino, M. H. O; Dodson, A.; Benford, S.; Koleva-Hopkin, B. The paper “Mixed Reality on Civil Engineering” is presented in United Kingdom (Capra et al., 2004a) during the Ubicomp. The authors are: Capra, M.; Aquino, M. H. O; Dodson, A.; Benford, S. The paper “A visualisation Tool for Civil Engineering Using Virtual Reality” is presented in United Kingdom (Capra et al., 2004b) during the 1st FIG International Symposium. The authors are: Capra, M.; Aquino, M. H. O; Dodson, A.; Benford, S. The project is also interviewed during the program Dispatches – “Who’s Digging our Roads?” (Channel 4, 2006)

7.4

Future Work

The following paragraphs propose some topics for future work for CARS on CEn environments.

7.4.1

Future on Hardware

Hardware does still an issue on outdoors AR system. Although there is not too much to do once most of the equipment came from commercial products a list of desirable hardware is presented below:

226

•

See-through HMD: it should be light as a pair of glasses. No wires attached and very light batteries. High resolution and stereoscopic images. Auto adjustable for contrast and bright. Weather proof. Retina tracking.

•

Positioning system: it should be as light as single frequency receivers are nowadays (the size of a match box) with the same low battery consumption. The output should be ten times faster or more – actual frequency is 1MHz. Imbibed antenna and weather proof. High accuracy 100% of time with secondary and tertiary solutions to supplement the accuracy – like cell signal, wireless network signal, etc. It should be reliable both indoors as outdoors. No cables at all.

•

Body movements track: through a kind of fabric (using some kind of nano technology) the movements of users’ body could be tracked with maximum resolution. This information would be useful to feedback the avatars creating more empathy.

•

Data communication: although some cities are already covered by fast wireless network transmission this does still an issue. Fast internet should be available anywhere at any time.

7.4.2

Future on Software

Software is more feasible to expect to change over the coming years. More oriented platforms should extract the maximum from hardware without compromise processing and memory. However this can only be achieved if the right hardware is in place and fully available. One of the common issues over software is the limitation imposed by hardware manufactories to fully access the resources of the devices.

7.4.3

Future Applications

During the development of this research a number of new applications were created. This new ideas could become an extension of the current project or a completely new research. Input Interface for Field Users

227

The work of the Field users could be better explored when in the field. They could collect information in the area where the construction is taking place/will take place, feeding the DB in real time. Examples of valuable data that could be collected in real time: • • • Information about the terrain – especially after torrential showers with surface runoff. Irregularities in the terrain like excavations or erosion. Construction material – what construction material is available, quantity and position of where they are stored in the construction site. Technical specification on construction material – relevant information about what is being installed and the consequences of it. Information about the diameter of pipes, capacity of pressure, different kinds of wires, gas valves, steel specifications, etc. • • Identification and resolution of conflicts – any conflict not detected during the design can be discussed in real time with other AEC team members. Survey with video/photography registered by GPS coordinates – the images can be added to audio documents reporting any issue during the inspection of the site. Potential Hazard and Lack of Navigation The evaluation shows that Field users can put far more attention in the VE then in the RE – “attentional spotlight”. It is a serious issue and can increase the chances for Field users to expose unnecessarily to hazards like: uneven terrain, excavations, moving vehicles and objects, loose construction material, etc. All hazards can be monitored through constant surveys of the area. Every time that Field user is in potential dangerous an alarm could sound and some extra information would be displayed in the CVE. Drivers and operators would also be able to receive information indicating the presence of a person in the area of manoeuvre in case of moving vehicles and cranes. The navigation through the construction area can also be performed by the system. Indications of how to go from one point to other avoiding different hazards can help Field users to keep safe.

228

Annotation Tool on CVE A possible extension of the CVE is a 3D annotation tool. Office users would be able to create 3D marks in different colours that would be visualised by office users. Such indications can be helpful to indicate places and points of disagreements/agreements. The annotation tool could also place icons and any other form to indicate a point in the CVE. The 3D annotation tool can also be linked with audio, videos and pictures annotation. Non Immersive Hardware Interface A study comparing the acceptance of non immersive hardware interface (PDAs, cell phones, see-through screens, etc.) against immersive interfaces (HMD). The evaluation should be aware of the discrepancy in technology between Liquid Cristal Displays (LCD) and HMD creating an even way to compare both approaches when exploring the same CARS. The evaluation should also compare the level of freedom required by different users when working with the equipment. Question like: Do users need both hands free all the time? Which equipment is more appropriate to perform complex tasks? are some of the examples to be explored.

7.4.4

Further Evaluation

The thesis uses the formative approach to perform the evaluation. This approach is based on the scale of the project and the culture of MRL and becomes appropriated for an initial outcome from the kind of technology proposed. However a more quantitative and comparative work is required in the future. Exercises can be better designed following a more realistic example like a real construction taking place. Users can also be choose and arranged in subjects of knowledge. Professionals in AEC that are current working in a project would be invited to compare their working methods with the CARS. This evaluation should be done in different construction sites with approximately same scale. The time frame of the evaluation should cover fully the design and the construction phases.

229

7.4.5

Concluding Remarks

The work of the presented thesis consolidates the design, development and evaluation of a CARS used on CEn – design and construction phases. The works starts with the identification of potential gaps over communication, collaboration and mutual understanding on groups shaped by AEC. The extensive survey done in both Engineering and Computer Science projects reaffirms that there are spaces for a research like the one proposed. There are a considerable number of work over the theme but very few using subjects like high accuracy GPS and CVE. The advantage to use RTK-GPS is the assurance that any surveyed feature of the terrain has no more than few centimetres of error. Users have their real position distribute to other users in real time when walking in the field with the equipment. The error on positioning is minimum for the trials that users are invited to perform. The tool enhances collaboration, communication and mutual understanding as demonstrated during the evaluation. Users demonstrate fluency after a short period of time even though when receiving very few amount of information of how the system works or what to do inside of the CVE. This can only be possible with the asynchronous collaboration of a second person inside of the VE. Through the avatar and the voice channel users are able to find objects, recognise places, and express their point of view. The second person inside of the CVE works almost as a facilitator between field users and information. It is easy for the field user to follow an avatar. It is also easy for the office user to explain something once he is exploring the same contend in the CVE. The evaluation shows the necessity of a better understanding of what users expect to see in a VE. This topic is strongly linked with who the users are: architects, engineers, or constructors. They have different ways to see the same information. The way that visual information is passed to them should be also different – coming from textures, virtual furniture, etc. going to more technical and time schedule information. This project aims to give to future researchers a solid view of from where to start on CARS although it does not exhausts the technology neither has an absolute

230

statement. Hopefully this work can contribute further in the conceptual understanding of the requirements on CARS in CEn projects.

231

Appendix A
Parameter & Implementation Reference
As introduced in §5.1, Appendix A demonstrates the structures of the files that are used in the creation of the VE. Such file is passed as second parameter in the call: MIX006 <grid>.grd <feature>.txt <outfile>.wrl This file contains the objects (features) found in the terrain that feed the VE, as well as the coordinates surveyed for each one of them. When a road is present, the algorithm paints the altimetric mesh with a different colour to indicate where it is. In §A.1 there is a description of how to create the files that shape the VE. The algorithm created to process the files and create the VE is explained on §A.1. • • • • The generated output consists of: A VRML file with only the landscape of the area that was surveyed. A second VRML file with the CVE as it will be explored in MASSIVE-3. A .cve file that contains the CVE description to be used by MASSIVE-3.

232

The code is created using the C language and the §A.2 is structured as: • • • A description of the global variables. The documentation for each function created. A high level algorithm.

A.1

Features File

The second parameter when MIX006 is called is <features>.txt. This text file has the positions and objects (features) surveyed on the area that VE was based. Each object that is indicated there corresponds to a VRML file and the coordinates follows the OS grid. The facility is to use the positions saved on the receiver without the need of conversions for Cartesian coordinates. The software does it based on the minimums and maximums of coordinates read on the square mash created by the Surfer software. Structure of <feature>.txt In this example the following features were surveyed in the data acquisition step: • • • Roads. Trees. Buildings.

Apart from the road, each feature had a VRML file with its own design. If a road is included in the VE, it must to be done as the first line of the file. The road is the unique feature that does not require any coordinate below its name. The algorithm follows the file indicated and adds the road to the VE (A.2.1). Table A.1 exemplifies the structure of the file that contains the features.

Table A.6 Features’ File Structure

road.txt tree.wrl 454279 338391 31.9924 454279 338391 31.9924 building.wrl 233

454299 338399 31.9924 454239 338351 31.9924 For all other features, their coordinates must be specified by a couple of control points. There are some cases, such as lamp posts and trees, in which both control points are the same. Buildings and any other shape that requires a positioning in the VE have their coordinates specified by two control points, which could be placed depending on the user criteria. Each line contains the position of the features specified as three numbers representing: East, North, and High. With these values it is possible to calculate the necessary rotation to place the object in the correct position. The road file (first line of Table A.1) contains only the East and North values, ignoring height. Usually, a road surveys in this project takes both sides of the road instead of only the middle. When interpolation process (see §5.1) takes place the coordinates surveyed becomes the vertices of the polygons that will shape the road. A final modification is required in the VRML files that shape the features. Usually the design of a new object on a three dimensional editor is done using the coordinates (0, 0, 0) for X, Y, and Z. Even though, it is a difficult task to design the object exactly on the position that it should be placed on the VE. Firstly, the same corners chosen to be the control points of the features are identified on the object. These coordinates are then explicitly at the third line of VRML file. The solution is then given by the calculation of the distance between the minimum coordinates of the square mesh and the place where the object must be placed. An object’s translation is applied and finally a rotation. Table A.2 exemplifies a header of a VRML file. The first line is standard for VRML 1.0. The following line consists of a blank line and the control points on Cartesian terms. Table A.2 shows an illustrative example of control points.
Table A.7 VRML Header’s Modified

#VRML V1.0 ascii # #20 10 1, 38.5 6 0.8

234

A.2

Source Code Reference

The source code was written in C language. It is expected that the reader has some knowledge in structured programming and C syntax. The following paragraphs describe: • • • The global variables used The functions programmed A high level algorithm

Global Variables The global variables on the code are related with the input and output files that are handled. All of them are pointers to a file. Table A3 introduce these variables:
Table A.8 Global Variable

entrada It is a pointer to a file that contains the Surfer output altimetric file. entrada2 It is a pointer to each one of the files that contains an object that will feed the VE. saida It is a pointer to a file that contains the VRML file with all objects inserted. saida2 It is a pointer to a file that contains the VRML file with only the terrain surveyed. mundo It is a pointer to a file that contains the CVE description. Road It is a pointer to a textual file that contains the coordinates of the road that was surveyed.

Functions Description The Tables A.4 to A.12 describe the functions created by the following steps: • • • • Name. Description. Prototype. Parameters passed when the function is started.

Returns given by the function when the process is finished.

235

Table A.9 Function main

Name: Description:

main This function is called when the program starts. From the main function all other functions are called. Prototype: void main (int argc, char *argv[]) Parameter Passed: int argc – number of arguments passed when the program starts. char *argv[] – the arguments passed when the program starts. Return: There is no return.
Table A.10 Function open_arquivo

Name: Description:

open_arquivo This function opens the files passed as parameters when the program is started. The pointers for a file: entrada, entrada2, saida and saida2 are started on this function. Prototype: void open_arquivo(int argc, char *argv[]) Parameter Passed: int argc – number of arguments passed when the program starts. char *argv[] – the arguments passed when the program starts. Return: There is no return.
Table A.11 Function take_information

Name: Description:

take_information This function reads the square mesh file created by Surfer software extracting the minimum and maximum values of the area. It also creates the header of VRML files generated as output of the system. The function reads from the pointer named entrada and write on saida and saida2. Prototype: void take_information() Parameter Passed: There is no parameter passed on this function. Return: There is no return.
Table A.12 Function vrml_generator

Name: Description:

vrml_generator This function reads the altimetric data from the file pointed by entrada and places the coordinates using the VRML syntax on saida and saida2. For each feature read on entrada2 an action is taken. If a road that is being added, the function input_road is called. Otherwise, it is if a feature, the algorithm calls input_feature function. Prototype: void vrml_generator(int ln, int cl, float min_h, long int x_mn, long int y_mn, long int x_mx, long int y_mx, float h_mn) Parameter Passed: int ln – number of lines contained on the square mesh. int cl – number of columns contained on the square mesh. float min_h – minimum value for high. long int x_mn – minimum value for East. long int y_mn – minimum value for North.

236

Return: Name: Description:

long int x_mx – maximum value for East. long int y_mx – maximum value for North. float h_mn – maximum value for high. There is no return.

Table A.13 Function input_feature

input_feature It extracts the features from the file pointed by entrada2 and places the objects on the output file respecting the translation and rotation. Prototype: void input_feature(long int x_min, long int y_min, float h_min) Parameter Passed: long int x_min – minimum value for East. long int y_min – minimum value for North. float h_min – minimum value for high. Return: There is no return.
Table A.14 Function build_cve

Name: Description:

build_cve This function creates the CVE file to be used further on MASSIVE-3. Prototype: void build_cve(float tra_x, float tra_y, float tra_z, double rot, int flag, ch ar feature[13]) Parameter Passed: float tra_x – Cartesian coordinate X of the object. float tra_y – Cartesian coordinate Y of the object. float tra_z – Cartesian coordinate Z of the object. double rot – Rotation angle in radians. int flag – Flag used to indicate what action the algorithm need to follow. char feature[13] – Feature’s name. Return: There is no return.
Table A.15 Function tabela_poligono

Name: Description: Prototype: Parameter Passed: Return:

tabela_poligono It fills a table with the coordinates of the road surveyed. long int tabela_poligono(float mat[256][2]) float mat[256][2] – matrix with the road coordinates. It returns the number of polygon vertices created by the surveyed road.

Table A.16 Function fbguPnpoly

Name: Description:

fbguPnpoly This function verifies if a coordinate is in, out or in the polygon border. Prototype: long fbguPnpoly (long Npoly, float *Poly, float *Pto) Parameter Passed: long Npoly – number of polygon vertices. float *Poly – array of polygon vertices. float *Pto – coordinate that will be checked.

237

Return: OBS:

-1 if the coordinate is out of polygon boundaries. 0 if the coordinate is out the boundaries of polygon. 1 if the coordinate is inside of polygon boundaries. This function was consented by Luiz Eduardo Seabra Varella.

Table A.17 Function input_road

Name: Description:

input_road This function sets the colour for each node of the square mesh. If the node is part of the road the colour is grey otherwise green. Prototype: void input_road(long int x_min, long int y_min, long int x_max, long int y_max) Parameter Passed: long int x_min – minimum value for East. long int y_min – minimum value for North. long int x_mix – maximum value for East. long int y_mix – maximum value for North. Return: There is no return. Algorithm The high level algorithm presented below offers for those who would like to use the code, an easy understanding of how it works. The following steps documents only what was judged as important for the understanding of the algorithm. Table A.13 shows the high level algorithm. The highlighted worlds identify the functions and file pointers.
Table A.18 High Level Algorithm

void main (int argc, char *argv[]) { open_arquivo(argc,argv); take_information(); fclose(entrada); fclose(saida); fclose(entrada2); } void open_arquivo(int argc, char *argv[]) { if number of parametres wrong exit; open entrada; open entrada2; open saida; open saida2; open mundo; } void take_information() 238

{ it reads minimums and maximums from the square mesh; It adds the headers of saida and saida2; vrml_generator(lin, col, h_min, (long int)x_min, (long int)y_min,(long int) x_max, (long int) y_max, h_min); } void vrml_generator(int ln, int cl, float min_h, long int x_mn, long int y_mn, long int x_mx, long int y_mx, float h_mn) { It completes the headers of saida and saida2; The coordinates from the .grd file is transferred to .wrl file format; if (file name road.txt is found in entrada2) input_road(x_mn,y_mn,x_mx,y_mx); input_feature(x_mn, y_mn, h_mn); } void input_road(long int x_min, long int y_min, long int x_max, long int y_max) { tabela_poligono(matriz); for (each row and line) fbguPnpoly(vertices, Poly, Pto); The altimetric model is painted with the right colour; } void input_feature(long int x_min, long int y_min, float h_min) { build_cve(0, 0, 0, 0, 10, "land"); while(It is not end of file (entrada2)) { The control points of the object are read; The translation and rotation are calculated and written on the output file; build_cve(ponto[0].x, ponto[0].y, ponto[0].z, alpha, flag, nome); } } long int tabela_poligono(float mat[256][2]) { It reads from the road.txt file and logs the coordinates in a table; The return is the number of the vertices read; } long fbguPnpoly (long Npoly, float *Poly, float *Pto) { It checks if the coordinate is inside, outside or in the polygon boundary;

239

} void build_cve(float tra_x, float tra_y, float tra_z, double rot, int flag, char feature[13]) { An header describing the name of the object, positioning and rotation is created; }

240

Appendix B
Implementation Reference
Appendix B brings relevant details about the source code implemented for this project. Some knowledge of Object Oriented (OO) structure in Java language is expected from the reader. Each sub-item on this Appendix refers to a class that explores the most important methods in order to facilitate the comprehension of those that could use this work as a base for future research. The classes developed for this research complement the facilities provided by EQUIP’s to work with external devices connected to the computer. As soon as the data is received and filtered, there are EQUIP methods to publish the new information on the data space, as it will be demonstrated below. The sub-items in this appendix as follows: • • • Overview – summarises what the class does. High Level Algorithm – as a complement to the overview. Class Constructor – how the class is inherited.

241

In the high level algorithm the Begin and End indicate where the class starts and finishes. However, one or more classes can be inside of the main class. Even though Java is an OO language, the algorithm was written to be the most linear possible for the sake of clarity.

B.1

RTK-GPS Class

As mentioned in §5.2 the communication with the RTK-GPS receiver and the publication of the data on the data space are shared in two classes named: GPSReceiver.java and GPSPublisher.java.

B.1.1 GPSReveiver.java In the class GPSReceiver.java, the receiver transmits all data using as bytes via COM1 port. The data is analysed and if it starts with the characters “GPLLQ” it is split into tokens where only the significant part is retained, as east, north, and height. The data is then normalised for the Cartesian coordinates followed by MASSIVE-3 and finally published on a data space. There is the option to process the data using a Kalman Filter. The processing of the data to a Kalman Filter class is done before the data is published. Table B.1 shows the high level algorithm. In the source code the variables are set as: • • • x = east. y = high. z = north.

Table B.19 GPSReceiver.java Algorithm

Begin initCommPort() initiation of communication port setup of communication port initiation of variables that do the in/out interface with the receiver Begin run() While (there is a byte coming) read byte convert to a character type and add it on a string If (the character received is equal to a brake line) decode(String with character received)

242

clean String EndIf EndWhile End decode(String with data received from RTK-GPS receiver) initiation of variable that receives the String in tokens For (each token) add it on a array of tokens EndFor associate to a variable token number 3 and subtracting it from the OS value of the base point associate to a variable token number 5 and subtracting it from the OS value of the base point associate to a variable token number 10 and subtracting it from the OS value of the base point Kalman(x, 2.0, (-)z, 1) updateCoordinates(z, x, 2.0) End Class Constructor The GPSReceiver class constructor is: • Where: GPSPublisher – is an instance of the class GPSPublisher String – is communication port identifier GPSReceiver(GPSPublisher, String)

B.1.2 GPSPublisher.java For each new set of data received and processed from the RTK-GPS receiver, this class makes it available on the data space. This data space is created at the beginning of this class and identified for further utilisation. Table B.2 shows the high level algorithm.
Table B.20 GPSPublisher.java Algorithm

Begin checks if the URL received is valid checks if the data service can be activated starts the data space with an identification creates an instance of the GPSReceiver Java class

243

updateCoordinates add a new 3D position on the data space End Class Constructor The GPSPublisher class constructor is: • Where: String – is the IP address followed by the data space identification String – is communication port identifier GPSReceiver(String, String)

B.2

Tilt-Sensor and Magnetic Compass Class

As introduced in §5.2 the communication with the tilt sensor and magnetic compass circuit board and the publication of the data on the data space were shared in two classes named: TCM2.java and TCM2Publisher.java.

B.2.1 TCM2.java This class is similar to the previous class showed in B.1.1. The connection is opened in the early stages of the algorithm. If successfully opened, the class starts to listen to the port collecting the data until it finds a line break. The beginning of the line is then analysed. If the line started with the characters “$C” it is known that the message is not broken. The message is then split into tokens and only the values of pitch, row and yaw are retained. An option for a Kalman filter is available that can be used or not. The data is then sent to TCMPublisher class to be stored on the data space. Table B.3 shows the high level algorithm. The values are converted to two digits of precision in order to reduce the small drift.
Table B.21 TCM2.java Algorithm

Begin initCommPort() initiation of communication port setup of communication port initiation of variables that do the in/out interface with the receiver 244

Begin run() While (there is a byte coming) read byte convert to a character type and add it on a string If (the character received is equal to a brake line) decode(String with character received) clean String EndIf EndWhile End decode(String with data received from tilt sensor and magnetic compass) initiation of variable that receives the String in tokens pass the values of the correct tokens into the correct variable Kalman(x, 2.0, (-)z, 0) updateCoordinates(xtcm, ytcm, ztcm) End Class Constructor The TCM2 class constructor is: • Where: TCM2Publisher – is an instance of the class TCM2Publisher String – is communication port identification TCM2(TCM2Publisher, String)

B.2.2 TCM2Publisher.java This class has the same function introduced in B.1.2. It creates another reference in the data space with an identity and logs the data received from TCM2.java. Table B.4 shows the high level algorithm.
Table B.22 TCM2Publisher.java Algorithm

Begin checks if the URL received is valid checks if the data service can be activated starts the data space with an identification creates an instance of the TCM2 Java class updateCoordinates add a new 3D position on the data space End

245

Class Constructor The TCM2Publihser class constructor is: • Where: String – is the IP address followed by the data space identification String – is communication port identification TCM2Publisher(String, String)

B.3

Reading from the Database

The following class is designed to check any new data that was published in the data space such as: GPS coordinates, tilt sensor and compass data or any key command pressed by the office user. Each new data is a trigger for an action performed by the Integrator class.

B.3.1 Integrator.java The Integrator class is used by the field user when all devices are connected to the computer carried on the backpack. GPS, tilt sensor and magnetic compass data, when available, are read from the data space and pass to the variables into MASSIVE-3. The update corrects the users’ visualisation of the VE in relation to the real environment. Because the office user can control the VE the database can de updated each time that a command key is pressed in his keyboard. This means that field and office user receive a new object or even a new a VE depending on the exercise being performed when each new income data from this particular area of the database is read. Table B.5 shows the high level algorithm. As mentioned previously the Integrator class inherited its structure from AugurscopeControl class created for the Augurscope project (Koleva et al., 2001, Benford et al., 2003a).

246

Table B.23 Integrator.java Algorithm

Begin constructor initialisation GUID factory dataspaces activate management server key stroke input session compass input session gps input session management server initialisation dynamic objects initialisation a thread controls the multiple process running in parallel as dynamic objects and positions update update dynamic objects update positions Begin management server control class End Begin Key stroke event handler class End Begin GPS event handler class End Begin GPS event handler class End End Class Constructor The Integrator class constructor is: • Where: String – is the IP address followed by the VE identification String – is the IP address followed by the data space identification private Integrator(String, String, String, String)

247

String – configuration file used by MASSIVE-3 String – is the same name used to start the cveworld command

B.4

Kalman Filter

The filter is implemented to reduce the jerk of: GPS coordinates, tilt sensor and compass.

B.4.1 Matriz.java This class implements a Kalman filter to be used on the data received from the magnetic compass, the tilt sensor and the RTK-GPS receiver. The implementation of the filter is necessary in order to reduce jerk in the data received. Two matrixes supply the models for the error and noise processed by the Kalman filter. The values of such matrixes are set up in accordance with the equipment in use. Instead of an algorithm, a diagram is chosen to introduce the concept due to the simplified way it demonstrates the process. Figure B.1 shows the diagram. There is a set of important variables that contribute to the final result. These variables are: • • • • • • • x – matrix with the predicted coordinates q – matrix of noise - covariance p – matrix with estimated error r – matrix with the noise from TCM2 k – Kalman gain number weight z – matrix with the data from TCM2 – innovation vector phi – matrix identity

The filter is well suited to navigation applications, which contain states that are timevariant, even in real-time. This is a recursive algorithm, which filters measurements using knowledge of the system measurement error (Hide, 2003). Even being recursive, the algorithm is based on the last estimate and current measurements. There 248

is no necessity to log all data to process again. The result is a computationally efficient algorithm that allows real-time processing, preventing the swing of the HMD image and smoothing the movements registered by the tilt sensor and the RTK-GPS receiver. It is not within the scope of this thesis to describe the Kalman filter equations. The reader is invited to read (Welch and Bishop, 2001, Hide, 2003) for a better description of the equations. The following dataflow (Figure B.1) describes in a few steps the Kalman filter that is implemented. The first step of the filter is the calculation of the estimated error (p). This matrix is the result of the sum of (p) and the matrix of noise - covariance (q) that is propagated to the next epoch (Figure B.1 – step 1). It is convenient to mention that (q) is static and contains an approximate model of the signal that is being filtered.

Figure B.74Kalman Filter Dataflow

The second step (Figure B.1 – step 2), (k) (Kalman gain number weight) receives the matrix with the estimated error (p) multiplied by the inverse of the result of (p) added to the matrix with the noise from TCM2 (r). In this step, (k) is receiving the adjustments based in the last sample of data that will be applied in the next epoch. On the third step the algorithm combines the data from the sensor (z) (innovation vector) with (x) (the predicted coordinates) through a subtraction (Figure B.1 – step 3). The result is then multiplied by (k) (with the adjustments based on the previous samples) and added to (x).

249

Finally (Figure B.1 – step 4), (p) receives the new Kalman values (k) after a subtraction with a (phi) matrix identity.

Class Constructor The Matriz class constructor is: • Matriz()

B.5

Office User Key Stroke Command

The Office user can control the VE from his keyboard. Each time that a new command is typed a new object can be added to the VE or the entire scene can be changed. This flexibility gives freedom for the Field user.

B.5.1 ReadKeyStroke.java This class is used by the Office user to type any command in the keyboard that will be logged in the database by the KeyStroke class. The commands are at the moment the letters from “a” to “f”. Each letter when read by the Integrator (see §B.3.1) is interpreted as a command. At the moment the VE can be changed completely or objects can be put in or out of the visualisation. Table B.6 shows the high level algorithm.
Table B.24 ReadKeyStroke.java Algorithm

Begin run() While (true) Thread (it sleeps each 1000 milliseconds) read byte If (character typed) update database with new character EndIf EndWhile End Class Constructor The ReadKeyStroke class constructor is:

250

• Where:

ReadKeyStroke (KeyStroke)

KeyStroke – is an instance of the class KeyStroke

B.5.2 KeyStroke.java This class has the same function introduced on B.1.2. It creates another reference in the data space with an identity and logs the data received from ReadKeyStroke.java. Table B.7 shows the high level algorithm.
Table B.25 KeyStroke.java Algorithm

Begin checks if the URL received is valid checks if the data service can be activated starts the data space with an identification creates an instance of the ReadKeyStroke Java class updateCoordinates the key stroke is translated to a number to be logged in the database End Class Constructor The KeyStroke class constructor is: • Where: String – is the IP address followed by the data space identification KeyStroke(String)

251

Appendix C
User Interviews – Questions and Answers
The questions and answers made throughout the interview during the evaluation process (Chapter 7) are shown here. All the comments can be found in §6.2.3. The codes used to identify the users were maintained: Engineer1 (E1), Engineer2 (E2), Engineer3 (E3), Engineer4 (E4), Architect1 (A1), Computer Scientist1 (C1), Computer Scientist2 (C2) and ordinary user (O1). The area covered by the questions are as follows: • • • • • • Merging real and digital environments. Hardware. Digital environment. Usability. Different/complementary approaches. Office and Field Users.

252

C.1

Merging Real and Digital Environment

The question involves the main topic of this thesis: AR. The users were questioned if they needed to adapt themselves to the VE and if the scale and distances walked were the same that they were used to see in the physical world. Self adaptation Did you have to adapt the way that you move in order to have a better perception of the VE? E2: Actually the turning of my head is much better. I think that the difficulty is when you come closer to an obstacle and the program decides, I guess, whether you are half way through it, through it or just in front of it. This is the most, I would say, annoying. But moving my head or walking towards when I was quite far was not a problem. Quite far is something around two metres. E3: Yes, but I was ok with that. It was a bit juggling but you explained to me that it was the GPS. I was not expecting it to be so “shaking”. Then I learned that I have to move my body much slower to look around. O1: Yes, I had to. I noticed that I moved my head a lot less. Immersion Did you feel immersed in the VE or did you have an AR experience? E1: I did it in the tram station. I did not so much in the buildings. I think that is to do with the texture and the reality. E2: In the beginning when the see-through of the HMD was not set as clear, maybe because we have a sunny day today, I did not realise that I went so far. For the tram station I was immersed completely in the VE. When you adjusted the see-through then I was a bit more conscious of where I was. For my eyesight I have two layers so if I want to see the grass I change my eyes focus and it takes a second for me to seethrough. If I focus on my goggles alone then I am immersed in the VE. E3: Yes, you saw me walking through the tree (a tree in the physical world). I was quite happy to concentrate on that. Because of the brightness and contrast I could not see much of the background. I could mostly see the VE and I was quite comfortable. A1: I think so. You cannot get any sense of distance because you cannot see the physical world. I was often amazed when I looked around to see where I was in the physical world. The augmented process did not work really. For me there was not a 253

relation between the physical and the digital environments. I was immersed. It was not mixed reality but only the VE there to be seen. And I think that part of the problem is that there are not physical features to relate to it. C1: After a while I was totally looking into the VE. I was trying to find things in there, trying to find what was going on. I think that the physical world went to the back of my mind. But then when you said: “-Turn around to my physical location” the two worlds just swapped their positions. I was again able to see-through the goggles. But while in the VE I forgot sometimes that I could see through the HMD. C2: The tram driven through me took away the sensation of immersion because it was difficult to work out how to avoid the tram. However, I think that there is some relation about this and how dark the glasses are. As you make the glasses darker you feel more involved in the digital environment aspect of the mixed reality stuff and obviously less involved in the physical world. O1: Partially. From the local view yes but at the same time you know that it will not be there. Why? The floor. When you walk you can feel the grass. Scale What can you say about the scale? Was it coherent or not? Did you have any difficulty to know how high a wall or a tram shelter was? A2: I can imagine but it is difficult to know without a proper scale. Is the Office User’s embodiment to scale? It has about two metres high. It is possible because I could still see it after the first phase. I have no idea how tall the house is in reality but I would say three and a half metres tall or something like this. The first phase I would have said was…I could still see the top of the embodiment’s head, one metre sixty or something like that, then one third plus one third. A3: One of the difficulties is to get scale. It would be nice if you put some image that we could recognize. Like you, it was useful to have you wondering around that gave me something to compare with scale. From the perspective point of view it will be nice if there is something I could make some comparison. E4: If the pipes were not in the right scale (but) the idea of different colours was good. You just need to make sure that each colour is representing the right thing in the proper diameter. C1: The scale was not very precise. Some more information is necessary to know how far you are from the house or how high is a wall. When I was walking towards the house I did not have the feeling that I was getting close to the house. I just kept walking until I eventually got there. When I walked around the house the scale looked ok. Another point is how high you put the “eyes of the Field User’s Embodiment”. I could misunderstand a view because I think that an object is too high or too low based in the perception of my own body.

254

C.2

Usability

Did you feel annoyed to walk through digital objects? E1: No. E2: No, but each time I asked you if I could go through them. It also means that you need to re-collect what you have in front of you. If I cannot find the Office User’s embodiment I need to imagine where was it at the last time or any other digital object that could help me in the navigation. I keep going through until I find it again. But it means in a complex environment if you need to use another application where people may be moving or things may be moving, it is a bit annoying, but other than this, there is no problem. E3: It was a little perplexing to sort it out. I was quite happy to walk through a wall but for a long time you could just see blue for example (when going through the windows in the third exercise). I was not sure there if I was getting any closer or any further away. C1: Extremely! It throws the user off, because the user is not expecting to go through it. I think that for me it reduces the experience. I started to disbelieve in a lot of other things within that environment and that represented what it should be. Scale, how big things are, movements or if I go through this wall maybe it is not even positioned there. O1: I did it first, but once you got used to it, it becomes fairly easily. Why did you feel annoyed? Partially because the frame rate that was hard to judge how is your movement. Because when you move in the physical world you have your view changing straight away. The system has a delay that you have to anticipate the movement to make it stop in the right position. C.3 Hardware

What could you say about the hardware used? Did the GPS receiver and the tilt sensor work adequately? E1: I think that when you stand still or walk really slowly is much better. It is better than in the previous prototypes. The positioning and orientation are quite good. The orientation changes quite quickly, as you turn round it follows you quite quickly. I think the problem is more in the position of the GPS when you move. For GPS equipment is coming down in price. I think in overall the problem is the weight and the number of cables. E2: The only jumping action was when you become close to something. Let’s say that I want to touch these pipes then I want to be close to them. Maybe I want to touch

255

them physically and in the VE. I noticed when I was coming close to the Office User in the tram station the issue of the positioning that made me go through the wall due to some drifting and also the frame rate. E3: I think so. It helped me when you told “do not look around so fast”. When I felt more in control of that I did look around much more slowly and waited for the thing to catch up. That was much better and gave me a much better smoother view. A1: I know that the GPS was being updated each second but it still is very jumpy. It does not matter when you are outside of the digital house for example, but when you are inside the house where it is pretty narrow it becomes a problem. C2: It seems to not have too much drifting at all. The whole scene does not seem to drift from the physical world. It was jerking a little bit but for me it was not a real problem. As soon as I stopped moving everything seemed to settle down.

C.4

Digital Environment

Texture and Colours What can you say about the VE? Where the colours used ok? How could the design be more appealing, attracting more users to it? Should texture make more sense than only colour? E1: Yes. So it is to make clear that you are looking at bricks, wood, glass or something else. It helps you to visualise what you’re looking at. I think that it is quite difficult to visualise what it is, with only solid colours. And it is not only shadows but it does not have any point of realism to relate to. If you look at a wall and you have bricks you know that it is a wall. If you look through glass you know that you have a window. The perspective of what are you looking at, how far away it is, is quite difficult again when you have not got the texture. But when you have not got texture it is really difficult to visualise the size of what you are looking at. In a blue print, there are the dimensions or something indicating what that room is. When you look at the elevation on a blue print you have a brick texture there. It is not always coloured but there is some attempt to show what is a brick, what is a window and what is a roof. It just helps your conceptualisation of what you are looking at. E2: It does not bother me if the wall in one of the phases was blue. But for somebody maybe less able to think in space maybe they will want to see something that could help in the visualisation, like bricks or cement textures. But it is not a big deal if you can imagine that it is a new geometry. If you imagine that it is a house and you cannot imagine the geometry, maybe you want to see more. E3: The colours sometimes confuse, I had to turn my head to get a view of what was going on and what the colours represented. Because there is no stereo you need to

256

move your head to actually see it. For the external wall I suggest a brick texture making it more photorealistic and perhaps the internal walls as well. E4: You could use different or higher resolution to present a house. So now, because you assembled everything together the viewer has a rough idea about how the house will be and how it will be integrated with the environment. That is a good idea particularly for the environmentalists and planners. Its assisting software is good. A1: It is not the problem with the concept but the VE for example could be more realistic. I would like to see more textures that could help you to distinguish between the walls from the floor. Of course that there are different colours but it is more difficult to see. Is that right that parts of the walls have different colours? Yes. Of course you do not get there in a real building. Usually the walls have the same colour from bottom to top. Sometimes you do not know if there is a real window or something else. It is because each colour represents a different phase in the project. Ok, but for orientation it does not help. The windows in the third example could also be improved if a more realistic representation is put there. I did not have any sense of wall thickness, what was odd. Are there any? Or the walls are just polygons? They are all polygons. This is also weird because if you go around the corner it is just flat. It becomes more difficult to know if you come through something or not. It is difficult to say if you are in one side or in the other side of the polygon. And you should not be able to go through walls. How do you imagine it working? You could make the animation stop for example. I do not know if something like it will work. Probably not. But it makes it weird because we all know that that we can go through them, but because of the rendering you cannot really see what side you are. You never know when you are in it, or walking through it or just stuff inside of it. C1: Most of our video processing uses colours. It is natural to relate things to colours like when you think in a brick you think red. If you think a tyre you think black. There are lots of ways that you can make it help to visually stimulate the user adding a bit more of details like in the wall, the floor and the ceiling. This sort of texture helps to emphasise things like: that is a wall and that is a wall further away from you because it is slightly darker for example. Lightening is another thing. If I have a blue wall a few centimetres from my face the colour is different from if I had the same wall a few metres away. At the moment the system does not support this kind of lightening process. The user automatically loses the sense of depth perception. C2: The fine details should be right to become a real good tool. The objects do not really look like in the physical world. Like texture for example. A window is not just a square or a rectangle shape with glass that allows him to see through it with some particular properties. For me to see a blue block is a bit confusing. O1: Yes you can tell which is which part of the project. Did you get confused with the colours? I did with the pipes (third example) because the floor and the walls are very similar colours. Perhaps because the colours are flat shaded. When the surface merges with each other, each time that he moves because there is no difference in contrast between the two polygons, that represents different walls.

257

C.5

Office User and Field User

Collaboration How did the collaboration between Field and Office User work? Did you feel the necessity to have two people to guide you (Office User’s embodiment) and control the VE on your behalf? Even with very limited gestures and movements the Office User’s embodiment could point or turn its head. What do you think about it? E1: What it did help me to do, is if you said to me to look to a certain direction it is easy to say: “-Look where my embodiment is”. It gives me direction to look at. I think that having not necessarily being you but having something as a pointer and then you say: “-Look where this pointer is” then it gives you something to relate to. I think this is useful and also the collaborative discussion is useful as well. Not only because of the complexity of controlling the environment as a user, but also once the user is in there they may want to ask questions about what is he looking at which in a collaborative environment someone can talk to him and discuss about it and ask for changes. Even if it was a man user in the field, if he’s got a question, how does he ask? Because the Office User knows where I am in the VE and what I am looking at and then you can answer questions about what I am looking at. E2: Well, the Office User pointed out things with the embodiment’s head to me. So I used that. There were instances that the embodiment did a movement up and down with its head to show me something I was able to associate this as a human interaction. Walking towards me also worked well to indicate direction. When I was on my own I was also able to explore things like in the second example. The only doubt that I had was when I was exploring the bathroom because in my country that place does not have big windows. Does it make a difference to have the embodiment? Yes it does if the Office User is pointing things or like when you are moving thinks about I can see the string coming from your head when it moves its head I could associate. But if there is no embodiment there and the explanation is verbally clear enough. I think that I managed to find my way around it. E3: For me it was useful to have you to follow as a guide. I felt less secure when I could not see you. When you passed close to an object where I could not see you I did not know where I was going. When I could see you I was happy to follow you around and look to what you were looking. When I did lose myself I found myself wanting to see you. In relation to gestures, it was really good when I looked to the stairs in the third exercise. When you stood in the stairs and looked up that helped me to look in the right direction because you have something specific to look out there. E4: In the beginning I did not understand your “role in the game”. And then you told me follow me and I did looking at the same places that the Office User’s embodiment was looking at. You were there to guide the other users to see for example the future of the city. The embodiment is really necessary. I was never lost I just followed you

258

and your instructions. I think that you could implement a kind of laser ray coming from the embodiment’s eyes to point at things in the VE. It should be easier than to make the embodiment gesticulate to the right position or digital object. It could not make a big difference if the VE was controlled by you or by me. The result will be the same but the viewer is who will make the decision if it is good or not. To have someone working in front of the computer to relieve the hassle of the Field User, to press buttons is really good. If you have asked me if I would like to control the VE by my own I would say no. I would certainly refuse. Along the exercise we were continually exchanging information and you gave me the guidance of what I should do and what action I should take. That was good. A1: I think that it was useful. You could say something like: “-come over here”. That was useful. Or “look where I’m looking”, that also worked. And to be able to talk with each other is useful as well. I think that the Field Users could have a kind of keyboard with only two keys and flip through them. Like go up and go down. C1: Definitely! I would be lost on that VE where I was. The most that I could do was to walk blindly just going through different rooms. The overall structure that the Office User explained while I walked, I could not understand if alone. After all I was able to explain where things were event though there was no feature like living room sofas or tables to indicate that I was in the living room. But because the Office User could explain the layout of the house I was able to create a mental map of how the house was spatially distributed. It worked almost as a tour guide. The embodiment made me feel like having a tour guide. And to see you move away from me and move towards me is quite reassuring because besides being lost in that environment I would not know where I started from. When you stood in a particular point and you said to me: “-Look at where I am looking” and I can see where you are facing and I know where to go. It also helps me to explore the VE in a more secure perspective. C2: There is a lot of cognitive effort to working your way in the environment. Because you are missing other senses, information, or because you are just getting visual information, you have to think harder how to navigate around. And because it is jerking, you need to think slightly more and memorise how things are before you start to move and go back to this. For your brain is working quite hard to navigate around the scene without having someone else showing you around and changing the VE for you. The control in the Office User’s hands takes one task away from your brain allowing you to focus more on the building. O1: It helped me to see what you tried to show me. If I just had the sound I would find it really hard to find the things. Wherever the Office User was in the VE, I could spot him and go towards his direction. What about the head movements? Did it make any difference when I gesticulated with my head looking up or down? Yes, it helped me to know where I should look.

259

C.6

Different/Complementary Approaches

Merging blue prints, scale models and AR When you look at the blue print could you associate what you saw, with the VE that you walked through? What about the picture of the house in the second exercise? Could you make any association? E2: I could imagine the house made of bricks as you showed in the picture but I thought that the pillars were made of wood. E3: Ah, yeah, yeah, yeah! Most of the time I stood over here (pointing to where he stood in the blue print). I definitely saw that in there. I can recognise it because mainly I stood here (pointing to the place in the blue print). This is the elevation of the house. I did not really see the external, I was standing there when we lost contact. I think if I stood there in the balcony (in the physical world) rather than this distant view here (the picture) it would be closer to what I saw (in the VE). This pillars where not there (pointing to the picture). They were there. I saw these two in one side but not on the other side. A1: Yes and no. I mean most half of it. I understood that there was a veranda. I understood that part. I have no concept of this. I remember that you told me what is what, but I think that these walls were not there. Partially I think…looking at the elevation in the blue print I can bring all the view together but I cannot remember to see this view at all in the VE. And I also did not have an idea of how the shape of the roof is. Can you imagine the house in the physical world (questioning A1)? I can imagine very bright colours because this is what sticks in my mind. I know that there was some windows but I could not say where is where. I can relate this picture to the plan but it is because I used to read plans. I can relate the veranda that I saw in the VE. I cannot relate this window…oh I remember now that I saw this window when I was inside the house. Although that on, in the VE, looks completely different, much wider. If you asked me how the window looks like on that side I would draw a window that goes around the corner very thin and long. Maybe you are confusing with the kitchen’s window because that one is like this. There is also a door not far from the window at the corner of the house. Ah…but that is weird because I am pretty sure that I turned round. Do you remember when we talked about a very small window here (pointing in the blue print to a small window between the living room and the veranda). But I thought that it was inside the house between the kitchen and the bedroom. I did not realise that it was in the outside wall of the house. So you cannot imagine the house based in the VE (questioning A1)? I would not go that far. I could imagine part of it as I also found partly confusing as well. It is not necessary the principle of it. But of course the model that you have is

260

extremely simple and the colours which ever direction you look you have the same colours. You could not say from the colours that I was looking this way or that way. You have to take all the other features. And that is probably where the confusion comes in. I also did not have any physical orientation. Obviously when you are in the physical world you can easily say that that tree is there and that other tree is there without even looking at. And if I turn around I still know where the trees are. But in the VE I could not tell you if when I turned left, I would see something that I want to see. I will have to turn around to find it again. I do not think that necessary you have a mental map or it is very abstract. I could not just say that the kitchen was in my left. C1: It definitely did, I could visualise the house although I may have a different colour in my head. But it was very similar. So if you show me a picture of the house I can tell you that it looks something that I could visualise. On paper, on two dimensional, it is hard to tell heights or the third dimension. This is basically what you can get from VE. You can have a better perspective and distances and how things overall look. With the blue print it is very difficult. O1: Yes, but it is difficult to say the size of the bathroom. You can tell for example that in a direction there is a bedroom but without furniture you cannot tell how big it will be. Best approach to explain a new development What do you prefer to see if someone comes to your house with a new project that will be implemented in few days time: a blue print, a scale mode, an artistic drawing or the tool in this project? E1: For someone that is an engineer and used to look to blue prints and is not used to look at VE, it is a bit difficult comparison to give. I think that the VE, the rendering, is quite simplistic, so realising what it is you’re looking at is more difficult. As soon as you put the roof on, because of the shape of the roof it is obviously that it is a roof. But because all the other squares and blocks are solid coloured it is more difficult to understand. So I think the important development will be to get the rendering more appropriate to what are you looking at and then it will make it easy to understand. C1: I think because an artist can paint in tree dimensions like in a picture it will give more information than a blue print. In the VE you can experience, you can walk through, you can feel the surroundings. You are immersed. For me the VE as shown is more helpful than the other options. O1: I will prefer the blue print. Between the scale model and the tool presented I will prefer the second option because it is possible to move around. It is more useful. What about the level of realism of the model? It did not make that much difference. You do not need fantastic graphic to know the shapes, you still can find your way around.

261

Real potentiality of the tool In a real situation do you think that a system like this would be worth? E2: If there is a need for it or a use for it is quite easily to visualise. For a house I do not think that you need it but for a big project I think that could be useful. E3: I think that the level of detail in the tram station was quite impressive. The buildings with the colours you got to have to sell, people understand what it is. When people want to see how high the construction could go, you need to relate it with some sort of scale and for me this is what was missing in the buildings. E4: The quality of the VE in the first exercise and the LoD like the CCTV were amazing. There is potential use for security issues. A1: I think that it is going to be useful. I have issues with the HMD. This area is flat there is nothing to fall from, but it is just a dangerous thing to have. Because you cannot really see where you are going to. It can be useful for the construction application as you suggested. Do you have something else to add to your comments? E2: We were quite unlucky to have too much light coming from the Sun and shades from the clouds intercalated. You asked me to look at things in the roof, because it is in the VE, I do not have the benefit of the shade of the physical roof, I was looking direct to the Sun or to the sky. It is impossible to see even when the see-through is in its darkest. I found it very easy to use. I am not good in visualising things, but I found it particularly easy to use. Usefulness yes, the pipes and things that you told me. All the underground networks, all sorts of pipes, cables, telephone, gas. I worked on this when I was in the army looking at underground systems and very often the only map that you got is somebody taking the measurement with the reference to a wall or to a curb. Of course that if somebody moved the curb or the wall you lost the reference. Very often what you do is to take one metre and a half between the curb and the access. You marked it. And this is not very accurate. You may be ending digging some area that it is not suitable. Maybe with GPS system and the see-through you could visualise things better. For building houses I would say that it is suitable for bigger projects. Maybe also for people who is buying on plants that could be useful. Either for a promoter or a buyer to see how it will look like. Maybe with a better rendering you could make it more appealing. I cold see this idea working indoors in the future for heath and safety where you could even couple with other things. You could create a digital training environment for the fire brigade without having to burn something. You could put them in a situation, or to police forces too. We could bring some physical modelling or CFD to the research.

262

E3: I will be interested to find how you work with a group of people as Field Users. Let’s say that if you want to sell the idea of design of a tram station or a building to a group rather than an individual. I can imagine that they will want to interact and they could individually go to explore the VE. I am not quite sure how could they make it as a group in the VE. I could imagine site inspectors coming and saying that they will be back in a moth to see the progress for example. The same could happen if you are an architect and want to sell something to a panel and not to an individual.

263

Bibliography
Abele, E., Anderl, R. and Birkhofer, H. (2005) Springer, pp. 282. Abowd, G. (1995), Vol. 2008. AC3D (2004), Vol. 2004. AceCad (2004), Vol. 2008 AceCad. AEC Design Group (2008), Vol. 2008 AEC Design Group,. Ammar-Khodja, S., Perry, N. and Bernard, A. (2008) Concurrent Engineering, 16, 89-101. Angewandte und Sozialorientierte Informatik - Universität Hamburg. Aquino, M. H. O. d. (1988) In Institute of Engineering Surveying and Space GeodesyUnivesity of Nottingham, Nottingham, pp. 155. Aquino, M. H. O. d. (1992) Petrobras, Rio de Janeiro - Brasil. Aquino, M. H. O. d. (1999) In Institute of Engineering Surveying and Space GeodesyUniversity of Nottingham, Nottingham - UK. Askew, B. (2007) The University of Nottingham - School of Civil Engineering, Nottingham, pp. 39. Augenbroe, G., Verheij, H. and Schwarzmüller, G. (2002) In Engineering, Construction and Architectural Management, Vol. 3 Blackwell Science Ltd, pp. 259-271. Autodesk Inc. (2008a), Vol. 2008 San Rafael, CA, USA. Autodesk Inc. (2008b), Vol. 2008 San Rafael, CA, USA. Azuma, R., T. (1995) In Department of Computer ScienceUniversity of North Carolina, Chapel Hill - USA, pp. 262 pages. Bamboo Open Source Community (BOSC) (1999), Vol. 2004. Bell, B., Feiner, S., Höllerer, T. (2001) In ACM Symposium on User Interface Software and TechnologyOrlando, Florida, USA. Benford, S., Schnädelbach, H., Koleva, B., Gaver, B., Schmidt, A., Boucher, A., Steed, A., Anastasi, R., Greenhalgh, C., Rodden, T. and Gellersen, H. (2003a) University of Nottingham - Mixed Reality Laboratory, Nottingham - UK, pp. 19.

264

Benford, S., Schnädelbach, H., Koleva, B., Gaver, B., Schmidt, A., Boucher, A., Steed, A., Anastasi, R., Greenhalgh, C., Rodden, T. and Gellersen, H. (2003b), Vol. 373x503 (Ed, augurscope2.jpg) Nottingham - UK. Benford, S., Seagar, W., Flintham, M., Anastasi, R., Rowland, D., Humble, J., Stanton, D., Bowers, J., Tandavanitj, N., Adams, M., Farr, J. R., Oldroyd, A. and Sutton, J. (2004) In UbiComp 2004(Eds, Davies, N., Mynatt, E. and Siio, I.) Springer, Nottingham - UK, pp. 70-87. Bentley Systems Incorporated (2008) In Buildings, Vol. 2008 Bentley Systems Incorporated,. Bingley, R., M., Roberts, G., W. (2001) Institute of Engineering Surveying and Space Geodesy - University of Nottingham, Nottingham - UK. Bingley, R. M. (2003), Vol. 2 Institute of Engineering Surveying and Space Geodesy. Bouchlaghem, D., Shang, H., Whyte, J. and Ganah, A. (2004) Automation in Construction, 287-295. Brate, A. (2002) Technomanifestos: Visions of the Information Revolutionaries, Texere; 1 edition (July 4, 2002). Brooks, F. P. (1999) IEEE Computer Graphics and Applications, 19, 16-27. Brousseau, É. and Rallet, A. (1995) Revue d’Économie Industrielle,, 74, 1-20. Bruce, V., Green, P. R. and Georgeson, M. A. (1996) In Visualization in Landscape and Environmental Planning: Technology and Application(Eds, Lange, E. and Bishop, I. D.) Taylor & Francis, pp. 296. Building Integration Software Company Ltd. (2008), Vol. 2008 (Ed, Building Integration Software Company Ltd.). BulidingExplorer.com (2007), Vol. 2008 BulidingExplorer.com,. Bureau, U. S. C. (2001) In GIS Frequently Asked Questions and General Info List Index, Vol. 2004. Capra, M. (2000) In Departamento de Engenharia CartográficaInstituto Militar de Engenharia, Rio de Janeiro - Brasil, pp. 144. Capra, M., Aquino, M., Benford, S. and Dodson, A. (2004a) In Ubicomp, Vol. 1 Nottingham, UK. Capra, M., Aquino, M., Dodson, A. and Benford, S. (2004b) In 1st FIG International Symposium - NottinghamNottingham, pp. 3. Capra, M., Aquino, M. H. O., Dodson, A. and Benford, S. (2003a) In VI Symposium on Virtual RealityBrazil. Capra, M., Aquino, M. H. O., Dodson, A., Benford, S. and Koleva-Hopkin, B. (2003b) In International Conference on Artificial Reality and Telexistence, Vol. 1 Tokyo - Japan, pp. 314. Carillion (2000) Nottingham. CFD Online (1994), Vol. 2006 http://www.cfd-online.com/Links. Channel 4 (2006) In DispatchesChannel 4, UK, pp. 60 min. Churchill, E. F. and Snowdon, D. N. (1998) Virtual Reality: Research, Development and Applications, 3, 3-15. Churchill, E. F., Snowdon, D. N. and Munro, A. J. (2001) Collaborative virtual environments: digital places and spaces for interaction, Springer, London UK. CityScape (2004), Vol. 800x600 (Ed, Olympia311.jpg) CityScape. Columbia University Computer Graphics and User Interfaces Laboratory (2004a), Vol. 2004 New York - USA. Columbia University Computer Graphics and User Interfaces Laboratory (2004b), Vol. 2004 New York - USA.

265

Delaney, B. (1999) IEEE Computer Graphics and Applications, Mar/Apr Edition. Delaney, B. (2000) IEEE Computer Graphics and Applications, May/Jun Edition. D-Link Systems Inc. (2006) D-Link Systems. Edgar, G. (2002) In Mapping Psycology 2(Eds, Miell, D., Phoenix, A. and Thomas, K.), pp. 3-56. (Eds.) Simpson J. A. and Weiner E. S. C. (1989) In Collaboration(Ed, Public Domain / Wikipedia). Eisenberg, N., Fabes, R. A., Schaller, M. and Miller, P. A. (1999) In Empathy online, Vol. 4 (Ed, Preece, J.) Springer London, London, pp. 74-84. Encyclopedia Brittanica Online (2007) In Collaboration, Vol. 2008 (Ed, Public Domain / Wikipedia) Wikipedia. Engelbart, D. (1962) In Technomanifestos(Ed, Brate, A.) Texere LCC, New York and London. Equator (2005) In www.equator.ac.uk, Vol. January 2005. ESRI (2001) In Your Internet Guide to Geographic Information Systems, Vol. 2004 California. ESRI (2003), Vol. 2004 ESRI. ESRI (2004a), Vol. 1280x995 (Ed, 3dextensionsshotgreen3_lg.jpg) ESRI, Redlands California - USA. ESRI (2004b), Vol. 2004 ESRI, Redlands - California - USA. Etchegoyen, R. H. (1991) In Empathy online, Vol. 4 (Ed, Preece, J.) Springer London, London, pp. 74-84. European Space Agency (2005) In ESA - Navigation, Vol. 2005 Paris - France. Evans, A., J., Roberts, G., W., Dodson, A., H., Denby, B., Hollands, R., Cooper, S. (2001), Vol. 2001. Federal Aviation Administration (2005a) In Programs - Wide Area Augmentation System, Vol. 2005 U.S. Department of Transportation, Washington. Federal Aviation Administration (2005b) In Programs - Wide Area Augmentation System, Vol. 2005 U.S. Department of Transportation, Washington - USA. Feiner, S., MacIntyre, B., Höllerer, T., Webster, A. (1997) In International Symposium on Wearable Computing'97Cambridge, Manchester, UK, pp. 7481. Fiksel, J. and Hayes-Roth, F. (1993) In Establishing and processing client requirements-a key aspect of concurrent engineering in construction, Vol. 1 (Eds, Kamara J. M., Anumba C. J. and O., E. N. F.) Blackwell Science Ltd. Gilbert, S., Mitchell-Jones, I. and Vardy, M. (2002) In Virtual Reality for Engineers J1C VRE, Vol. 2004 (Ed, Schofield, D.) Nottingham. Godfried Augenbroe, Hans Verheij and Schwarzmüller, G. (2002) In Engineering, Construction and Architectural Management, Vol. 3 Blackwell Science Ltd, pp. 259-271. Golden Software (2004), Vol. 2004 Golden, Colorado, USA. Gomez M., Sun D., Waelchli C., Balsiger P. and F., P. (2002), Vol. 2006 (Ed, Neuchâtel, I. o. M. U. o.) Neuchâtel. Google (2006). Graphisoft R&D Zrt. (2007), Vol. 2008 (Ed, Graphisoft R&D Zrt.) Budapest, Hungary. Gray, B. (1989) In Engineering, Construction and Architectural Management, Vol. 3 (Eds, Godfried Augenbroe, Hans Verheij and Schwarzmüller, G.) Blackwell Science Ltd, pp. 259-271.

266

Greene, J. H. (1984) In Civil Engineering Practice in the Twenty-first Century: Knowledge and Skills(Ed, Grigg, N. S.) Reston Pub. Co. Greenhalgh, C. (1997) (Ed, 7) University of Nottingham, Nottingham, pp. 19. Greenhalgh, C. (1999a) Large scale collaborative virtual environments, Springer, London - UK. Greenhalgh, C. (1999b), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2000a) (Ed, 3) University of Nottingham, Nottingham, pp. 8. Greenhalgh, C. (2000b), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2000c), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2000d), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2001a), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2001b), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2001c), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C. (2001d), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C., Benford, S. (2001e), Vol. 2001 Mixed Reality Laboratory University of Nottingham - UK. Greenhalgh, C., Izadi, S., Rodden, T. and Benford, S. (2001) Mixed Reality Laboratory - University of Nottingham - UK. Greenhalgh, C., Izadi, S., Rodden, T., Benford, S. (2001f) Mixed Reality Laboratory University of Nottingham - UK. Greenhalgh, C., Purbrick, J. and Snowdon, D. (2000) In CVE2000San Francisco USA, pp. 8. Handykey Corporation (2006), Vol. 2007 Denver, CO, USA. Hartshorn, J., Maher, M., Crooks, J., Stahl, R. and Bond, Z. (2005) Canadian Journal of Civil Engineering, 32, 170-180. Haymaker, J., Fischer, M., Kunz, J. and Suter, B. (2004) ITcon, 9, 419-441. Heike, F. (2002) In Lehrstuhl für Realzeit - ComputersystemeTechische Universität München, München - Germany, pp. 163. Heintz, J. L. (2002) In Engineering, Construction and Architectural Management 2002, Vol. 3 Blackwell Science Ltd, pp. 181-191. Hide, C. D. (2003) In IESSGUniversity of Nottingham, Nottingham. Höllerer, T., Feiner, S., Bell, B., Hallaway, D. (2001a) IEEE Computer Graphics and Applications. Höllerer, T., Feiner, S., Terauchi, T., Hallaway, D., Rashid, G (1999) IEEE Computer Graphics and Applications, August Edition. Höllerer, T., Feiner, S., Tinna, N., Hallaway, D. (2001b) In Second International Workshop on Artificial Intelligence in mobile SystemsSeattle, Washington, USA. Human Interface Technology Laboratory at the University of Washington (2004) Washington - USA. Huss, E. (1997), Vol. 2008.

267

Ickes, W. E. (1997) In Empathy Online, Vol. 4 (Ed, Preece, J.) Springer London, London, pp. 74-84. IESSG (2002), Vol. 2004 IESSG, Nottingham - UK. IESSG (2008) In www.nottingham.ac.uk/iessg, Vol. 2008 IESSG, Nottingham - UK. Interactive Media Systems Group Institute for Software Technology and Interactive Systems Vienna University of Technology (2004) Vienna - Austria. Jonsson, B., Hedling, G. and Wiklund, P. (2003) In ION GPS/GNSSPortland, Oregon, U.S., pp. 10. Kamara J. M., Anumba C. J. and Evbuomwan N. F. O. (2000a) In Engineering, Construction and Architectural Management 2000, Vol. 1 Blackwell Science Ltd, pp. 15-28. Kamara J. M., Anumba C. J. and Evbuomwan N. F. O. (2000b) In Concurrent Engineering: Research and Applications, Vol. 1 (Eds, Fiksel, J. and HayesRoth, F.) Blackwell Science Ltd. Kamara J. M., Anumba C. J. and Evbuomwan N. F. O. (2000c) In Establishing and processing client requirements—a key aspect of concurrent engineering in construction, Vol. 1 (Eds, J. M. Kamara, C. J. Anumba and Evbuomwan, N. F. O.) Blackwell Science Ltd, Engineering, Construction and Architectural Management 2000, pp. 15-28. Kaplan, E. D. (1996) In Understanding GPS principles and applicationsArtech House Publishers, Norwood, pp. 237-383. Khanzode, A., Fischer, M. and Reed, D. (2008) ITcon, 13, 324-342. Koleva, B. (2001) In Computer ScienceThe University of Nottingham, Nottingham, pp. 239. Koleva, B., Adams, M., Taylor, I., Benford, S., Fraser, M., Greenhalgh, C., Schnädelbach, H., Lehn, D. v., Heath, C. and Row-Farr, J. (2001) In SIGCHI conference on Human factors in computing systemsACM Press, Seattle, Washington, United States. Ku, K., Pollalis, S. N., Fischer, M. A. and Shelden, D. R. (2008) ITcon, 13, 458-485. Kubicki, S., Bignon, J. C., Halin, G. and Humbert, P. (2006) ITcon, 11, 566-586. Lange, E. and Bishop, I. D. (2005) Visualization in Landscape and Environmental Planning: Technology and Application, Taylor & Francis. Lanzetta J.T. and Englis B.G. (1989) In Empathy online, Vol. 4 (Ed, Preece, J.) Springer London, London, pp. 74-84. Leclerc, Y., Reddy, M., Eriksen, M., Brecht, J. and Colleen, D. (2002) In SRI InternationalSRI, Menlo Park, pp. 45. Leica Geosystems LLC. (Ed.) (1999) Technical Reference Manual Version 2.0, Printed in Switzerland, Heerbrugg, Switzerland. Leica Geosystems LLC. (2001), Vol. 2004 Leica, Atlanta - Georgia - USA. Leica Geosystems LLC. (2003), Vol. 2004 (Ed, 3.0) Heerbrugg Switzerland. Leica Geosystems LLC. (2005) In Product News, Vol. 2006. Leica Geosystems LLC. (2006) In TS 3 – Forum for Providers and Users of Real Time Correction ServicesMunich Germany. Liston, K., Fischer, M. and Winograd, T. (2001) ITcon, 6, 69-82. Lucena, V. (1986). Lucent Technologies - Bell Labs Innovations (2000). Maldovan, K. (2007) CONVR 2007. Masanori, N. S. and Sawada, K. K. (2004) In Technical Report: Matsushita Electric Works, Vol. 2004 (Ed, Matsushita Electrocnic Works Ltd) 81, Tokyo. Mathieson, K. and Keil, M. (1998) Information & Management, 34, 221-230.

268

Matsushita Electronic Works Ltd (2003), Vol. 1054x464 (Ed, shiodome_cyberdome_2.jpg) MEW, Tokyo. Matsushita Electronic Works Ltd. (2003), Vol. 675x269 (Ed, shiodome_cyberdome_1.jpg) MEW, Tokyo. Merrian-Webster (2007) In Collaboration, Vol. 2008 (Ed, Public Domain / Wikipedia) Wikipedia. Merrian-Webster (2008) In Merrian-Webster Online Dictionary, Vol. 2008 (Ed, Merrian-Webster) Merrian-Webster, Springfield, MA, USA. Milgram, P. and Kishino, F. (1994) In IEICE Transactions on Information Systems, Vol. E77-D (Ed, IEICE) IEICE. Mokhtar, A. H. M. (2002) In Engineering, Construction and Architectural Management 2002, Vol. 3 Blackwell Science Ltd, pp. 111-231. Mountain, D. and Liarokapis, F. (2007) Aslib Proceedings: New Information Perspectives, 59, 422-436. Neil S. Grigg, Marvin E. Criswell, Darrell G. Fontane and Thomas J. Siller (2001) In Civil Engineering Practice in the Twenty-first CenturyASCE Publications, pp. 83-108. Nemetschek North America (2008) In Vectorworks, Vol. 2008 Nemetschek North America,. NET (2001), Vol. 2001 Nottingham Express Transit. Nitithamyong, P., Skibniewski, M. J. and Endowed, A. J. C. (2007) ITcon, 12, 39-59. Nottingham Tram Consortium (2004), Vol. 2006 Nottingham Tram Consortium, Nottingham. Novak, J. (1999) In Electrical Engineering and ComputingFaculty of Electrical Engineering and Computing, Zagreb - Croatia, pp. 110. Office of Superintendent of Public Instruction In Communication, Vol. 2008. Origin Instruments (2002) Origin Instruments, Grand Prairie - USA, pp. 6. OSG Community (2004) In OpenFlight, Vol. 2008. Peansupap, V. and Walker, D. H. T. (2005) ITcon, 10, 193-218. Pendleton, G. (2002) In Directions Magazine. Petrovski, I. (2003), Vol. 2004 (Ed, Communications, A.) GPS World. Petrovski, I., Kawaguchi, S., ISHII, M., Torimoto, H., Fujii, K., Sasano, K., Kondo, M., Shoji, K., Hada, H., Uehara, K., Kawakita, Y., Murai, J., Townsend, B., Cannon, M. E. and Lachapelle, G. (2000) In GPS2000Salt Lake City. Piekarski, W. (2002) In Proceedings of IEEE Virtual Reality'2002IEEE, Orlando, Florida, USA. Piekarski, W., Bernard, G., Thomas, B. (2001a) In 2nd International Symposium on Augmented RealitySan Francisco, USA. Piekarski, W. and Thomas, B. (2002), Vol. 664x577 (Ed, tinmith.jpg) Adelaide Australia. Piekarski, W., Thomas, B. (2001b) In Linux Conference, Vol. 2001. Piekarski, W., Thomas, B. (2001c) In 2nd International Symposium on Augmented RealityNew York, USA. Piekarski, W., Thomas, B. (2001d) In International Symposium on Wearable ComputersZurich, Switzerland. Piekarski, W., Thomas, B., Gunther, B. (1999) In DCNET'99Sidney, Australia. Pinho, M., S. (1999), Vol. 2002 Pontifícia Universidade Católica - RS - Brasil. PNI Corporation (2004) Santa Rosa, CA, USA. Posner, M. (2002) In Perception and Attention(Ed, Edgar, G.) Open University, pp. 15.

269

Preece, J. J. (1999) Virtual Reality, 4, 74-84. Public Domain / Wikipedia, Vol. 2008 Wikipedia. Public Domain / Wikipedia (2008) In Wikipedia - The Free Encyclopedia, Vol. 2008 Wikipedia. Purbrick, J. C. R. (2001) In Computer ScienceUniversity of Nottingham, Nottingham, pp. 176. Qiu, Y., Ge, P. and Yim, S. C. (2007a) Concurrent Engineering, 15, 357-368. Qiu, Y., Ge, P. and Yim, S. C. (2007b) Concurrent Engineering, 15, 369-380. R.T.Azuma, Baillot, Y., Behringer, R., Feiner, S., Julier, S. and Mac-Intyre, B. (2001) In Augmented reality navigation systems, Vol. 4 (Eds, Narzt, W., Pomberger, G., Ferscha, A., Kolb, D., Muller, R., Wieghardt, J., Hortner, H. and Lindinger, C.) Springer Berlin / Heidelberg, pp. 177-187. Reddy, M., Leclerc, Y., Iverson, L., Bletter, N. (1999) IEEE Computer Graphics and Applications, Mar/Apr Edition. Reitmayr, G. and Schmalstieg, D. (2001) In ISCW 2001 Zurich, Switzerland. Reitmayr, G. and Schmalstieg, D. (2004a) In Symposium Location Based Services and TeleCartography 2004 Vienna - Austria. Reitmayr, G. and Schmalstieg, D. (2004b), Vol. 681x494 (Ed, reitmayrlbs2004.pdf) Vienna - Austria. Reitmayr, G. and Schmalstieg, D. (2004c), Vol. 686x513 (Ed, reitmayrlbs2004.pdf) Vienna - Austria. Roberts, G., Evans, A., Dodson, A., Denby, B., Cooper, S. and Hollands, R. (2001) In ION DPS 2001Salt Lake City, USA. Roberts, G., Evans, A., Dodson, A., Denby, B., Cooper, S. and Hollands, R. (2004) Nottingham - UK. Roberts, G., W. (1997) In Institute of Engineering Surveying and Space GeodesyUniversity of Nottingham, Nottingham - UK. Roberts, G., W., Evans, A., Dodson, A., Denby, B., Cooper, S., Hollands, R. (2002) In ION DPS 2001, GPS WorldUK. Rotating Tower Technology International Limited (2008), Vol. 2008 Florence, Italy. Salazar, G., Mokbel, H. and Aboulezz, M. (2006) In ASEE New England Section 2006American Society for Engineering Education, New England, pp. 10. Sapeta, K. (2003) In Photogrammetry & Remote Sensing, Vol. 69 ASPRS, Bethesda Maryland - USA, pp. 114-115. Sarshar, M., Tanyer, A. M., Aouad, G. and Underwood, J. (2002) In Engineering, Construction and Architectural Management 2002, Vol. 2 Blackwell Science Ltd, pp. 152-160. Satoh, K., Hara, K., Anabuki, M., Yamamoto, H. and Tamura, H. (2001) In ISMR 2001(Ed, ISMR) Yokohama - Japan, pp. 2. SC3000.COM (1998) In SimCity 3000 Resource Center, Vol. 2004 Wren M. Weburg. Scheer, S., Junior, R. M., Quevedo, J. R. S., Jr, J. M. and Fontoura, P. S. (2007) ITcon, 12, 221-230. Schildt, H. (2001) Java™ 2: The Complete Reference, The McGraw-Hill Companies, Berkeley - USA. Schnädelbach, H., Koleva, K., Flintham, M., Fraser, M., Chandler, P., Foster, M., Benford, S., Greenhalgh, C., Izadi, S., Rodden, T. (2001), Vol. 2001 Mixed Reality Laboratory - University of Nottingham - UK. Shelbourn, M. A., Bouchlaghem, D. M., Anumba, C. J., Carillo, P. M., Khalfan, M. M. K. and Glass, J. (2006) ITcon, 11, 57-71. Shin, D. H. and Dunston, P. S. (2008) Automation in Construction, 17, 882-894.

270

Silva, L. F. C. F. d. (1998) In Engenharia de TransportesUniversidade de São Paulo, São Paulo, Brasil. Snowdon, D. and Greenhalgh, C. (1996) University of Nottingham, Nottingham, pp. 10. Someren, M. W. v., Barnard, Y. F. and Sandberg, J. A. C. (1994) The Think Aloud Method - A practical guide to modelling cognitive processes, Academic Press, London, UK. SONY (1998). SRI International (2000a), Vol. 654x543 (Ed, campus.gif) Menlo Park - CA - USA. SRI International (2000b), Vol. 579x519 (Ed, win.gif) Menlo Park - CA - USA. SRI International (2002), Vol. 2004 Menlo Park - CA - USA. Sun Microsystems (2004), Vol. 2001. Sun Microsystems (2008) In Java 3D API, Vol. 2008. SWEPOS Control Centre (2005), Vol. 2005 Gävle - Sweden. Tamura, H. (2000) In Imagina 2000. Tamura, H. (2001a) Japan, pp. 97-104. Tamura, H., Yamamoto, H., Katayama, A. (2001b) IEEE Computer Graphics and Applications, Nov/Dec Edition. Tekla (2008), Vol. 2008 Tekla. The University of Nottingham (2008) In www.nottingham.ac.uk, Vol. 2008. Thesaurus (2008) In The Free Dictionary, Vol. 2008 Farlex. Tilley, P. and Barton, R. (1997) In Engineering, Construction and Architectural Management 2002, Vol. 3 (Ed, Mokhtar, A. H. M.) Blackwell Science Ltd, pp. 111-231. Torán-Marti, F., Ventura-Traveset, J. and Chen, R. (2003) In GNSS2003, pp. 15. TOSHIBA (Ed.) (2002) TOSHIBA Satellite Pro 6100 Portable Personal Computer User's Manual. Trimble Navigation (2002), Vol. 2005 Sunnyvale - USA. Tuzmen, A. (2002) In Engineering, Construction and Architectural Management, Vol. 3, pp. 209–221. Ubisense (2008), Vol. 2008. Ugwu O. O., Anumba C. J. and Thorpe A. (2001) In Engineering, Construction and Architectural Management, Vol. 3 Blackwell Science Ltd, pp. 211-214. United States Naval Observatory (2005), Vol. 2005. University of Loughborough, University of Salford and London School of Economics (2001), Vol. 2008. Uschold, M. and Gruninger, M. (1996) In Knowledge Engineering Review, Vol. 2 Cambridge Journals. UST (1998), Vol. 1024x610 (Ed, lax_05_big.jpeg) UST website, Los Angeles. UST (2003), Vol. 1024x610 (Ed, la_08_big.jpeg) UST web-page, Los Angeles. UST (2004), Vol. 2004 Los Angeles. Vicente, K. J. (1999) Lawrence Erlbaum Associates. Virtools (2004) In Applications > Industry, Vol. 2004 Montreal - Canada. VTT (2008), Vol. 2008 VTT. W3C (1995), Vol. 2008. Webster, A., Feiner, S., MacIntyre, B., Massie, W. and Krueger, T. (1996) In Computing in Civil Engineering California - USA, pp. 1090. Weimer, N. (1998), Vol. 2005 (Ed, Canada, U. o. A.-.). Welch, G. and Bishop, G. (2001) In SIGGRAPH(Ed, ACM, I.) Los Angeles, California, USA, pp. 46.

271

White, C. (2006) In BIM Panel Discussion Denver, CO, USA. Wikipedia (2006), Vol. 2006. Wikipedia (2008a), Vol. 2008 (Ed, Public Domain). Wikipedia (2008b), Vol. 2008 (Ed, public Domain). Wikipedia (2008c), Vol. 2008 (Ed, Public Domain). Wikipedia (2008d), Vol. 2008 (Ed, Public Domain). Wikipedia (2008e), Vol. 2008 (Ed, Public Domain). Wikipedia (2008f), Vol. 2008 (Ed, Public Domain). Wilkinson, S. (2001) In Engineering, Construction and Architectural Management 2001, Vol. 3 Blackwell Science Ltd, pp. 160-170. Winner, R. I., Pennell, J. P., Bertrend, H. E. and Slusarczuk, M. M. G. (1998) In Establishing and processing client requirements-a key aspect of concurrent engineering in construction, Vol. 1 (Eds, Kamara J. M., Anumba C. J. and Evbuomwan N. F. O.) Blackwell Science Ltd, pp. 15-28. Woksepp, S. and Olofsson, T. (2006) ITcon, 11, 267-240. Woodward, C. (2007) In IBM iForum 2007 Zurich, Switzerland. Woodward, C., Lahti, J., Rönkkö, J., Honkamaa, P., Hakkarainen, M., Jäppinen, J., Rainio, K., Siltanen, S. and Hyväkkä, J. (2007a) In W78 Conference Maribor. Woodward, C., Lahti, J., Rönkkö, J., Honkamaa, P., Hakkarainen, M., Jäppinen, J., Rainio, K., Siltanen, S. and Hyväkkä, J. (2007b) In W78 Conference Maribor, Vol. 309x233 (Ed, ARMobile.jpg) Finland. Woodward, C., Lahti, J., Rönkkö, J., Honkamaa, P., Hakkarainen, M., Jäppinen, J., Rainio, K., Siltanen, S. and Hyväkkä, J. (2007c) In W78 Conference Maribor, Vol. 432x343 (Ed, ARWebCam.jpg) Finland. Xsens (2008), Vol. 2008. Yamamoto, H. (1999) In 1999 IEEE Systems, Man and Cybernetics SocietyTokyo, Japan. Yunchun Yang, Richard T. Sharpe and Hatch, R. R. (2002) In ION 2002Portland, Oregon, USA, pp. 953-961. Zhang, Z. (2005), Vol. 2008.

272