This unit was edited by C. Peter Keller, Department of Geography, University of Victoria, Canada.

This unit is part of the NCGIA Core Curriculum in Geographic Information Science. These materials may be used for study, research, and education, but please credit the authors Robert Haining and Stephen Wise, and the project, NCGIA Core Curriculum in GIScience. All commercial rights reserved. Copyright 1997 by Haining and Wise.

Your comments on these materials are welcome. A link to an evaluation form is provided at the end of this document.

---

Advanced Organizer

Unit Topics and  Learning Outcomes

Instructors' Notes

Full Table of Contents

Metadata and Revision History

1. Introduction to Multimedia and Virtual Reality

Multimedia (MM) - Computer systems allowing for integrated access to a range of data through the means of stiimulating human senses using digital technologies

Virtual Reality (VR) - Computer systems able to combine a mixture of real world experiences and computer generated material to allow for simulated real world representation

• Closely related technologies sharing some similar hardware and data usage

• From a Geographical Information System (GIS) perspective MM and VR are the means to an end - handling (integrating, storing, accessing and viewing) a multitude of spatial data using a variety of tools. Can be considered under the general heading of visualisation: methods therefore vary depending on whether usage is for private investigation or for public demonstration; whether data is accessed interactively or in a pre-determined manner; and whether there is data investigation and interrogation or whether mere presentation suffices.

• Multimedia data
• multimedia covers the integration of:
• images, video and graphics (both still and animated); including raster and vector data, maps, photographs
• text; in a variety of forms including alphanumeric databases
• sound
• (potentially) smell and taste

• Multimedia tools
• under computer control, allowing interaction with real world digital data in the form mentioned above (including spatial digital data) with 'hyper-card' tools, visualisation software, audio and video players

• Virtual Reality data
• VR addresses the construction of artificial worlds, with clear spatial dimensions databases for VR can structure and store data using methods beyond the conventional abstractions of GIS

• Virtual Reality tools
• under computer control allowing access to the artificial worlds with internet viewers, VR navigators and dedicated stand-alone hardware stations

• Among the important concepts in MM and VR are database construction and integration, and user navigation and interaction
• the former can be achieved
• using existing datasets or 'on-the-fly'
• the latter can be done
• real-time or can be pre-determined
• can be in a fixed sequence or can be interactively led

• geographical data has a role in the enabling of such implementations, although there are implications for geographical data when included in MM and VR systems
• data structuring; difficulty in altering or enhancing data; lessening the importance of a reference system for the data; user interaction with the data and the importance of representation, particularly in three and four dimensions

2. Technological Issues

• The hardware components of a multimedia and/or a Virtual Reality PC or workstation

• Multimedia requires perception and interaction with use of visual and auditory participation, i.e. the production of vision and sound, Virtual Reality additionally requires tactile and vestibular participation

• A typical multimedia PC must comprise of the following:-
• A powerful PC with one of the most up-to-date processors
• large amounts of on board memory
• large capacity hard disk
• e.g. in 1996, an Intel Pentium or Pentium Pro, 32MB RAM, high speed bus and SCSI 2.0GB hard drive
• A high performance high resolution graphics processing capability
• dedicated memory
• specialist graphics microchips
• high speed bus
• e.g. in 1996 1024 x 768 resolution with 64k colour, PCI local bus, 2MB graphics memory
• Very fast, multi-session, Kodak Photo-CD compatiable CD ROM drive
• full screen, 25 frames per second Video-CD and MPEG playback
• e.g. in 1996 Multimedia PC3 (MPC3) standard
• High quality stereo sound and high quality wavetable sampling
• Sound Blaster compatable card with 16-bit stereo sound
• headphones
• mains powered speakers
• High resolution colour printer/plotter device
• laser, ink jet, bubble jet
• A mouse and keyboard

Figure 1.

• A Virtual Reality system may be considered to be an expansion of a multimedia system into a multi-sensory system

• The additional components of a Virtual Reality PC or workstation may include any of the following
• Tactile interaction
• Head Mounted Display (HMD) - wide field of view, anamoriphically projected stereo
• Tactile feedback devices, vibrotactile displays - teletactile feedback glove, virtual joystick
• Force feedback
• Teleoperation systems - force feedback joystick, remote manipulator arm, joystring
• Vestibular
• Motion platforms - flight simulators, motion simulators
• Other interactive devices
• 2 degrees of freedom (DOF) - mouse, joystick, 2-d tablet with gesture recognition, touch screen
• 6 degrees of freedom - wand, 6 DOF mouse, dataglove, force ball
• Wired clothing - datasuit
• Biological input (biosensor) - voice recogniser, skin temperature probe, myoelectric (muscle) sensor, cerebroelectric (brain) sensor

• For the processing of geographical data, e.g. a multimedia GIS, graphical output in both digital and hard copy form are of the utmost importance

• Graphics
• Visual Display Unit (VDU)
• cathode ray tube (CRT), traditional TV/monitor device
• liquid crystal display (LCD), flat screen
• Three dimensional (3D) display, this requires perspective and stereoscopic vision.
• A single display unit presenting two images alternatively, arranged so that one eye sees each. This requires a rate of change greater than 50 times a second and the use of special spectacles with liquid crystal shutters, synchronised with the display, or polarised glasses and an alternating polarising of the display
• Immersive graphical display devices such as head mounted display consisting of two miniature VDUs
• Direct Volume Display Device (DVDD) generate images directly in a volume, users of DVDD see a solid volume that they can walk around
• High Definition Television (HDTV)

• Interactive interface tools for access and manipulation
• Keyboard
• Mouse, trackball and joystick
• A digitising tablet
• Light pen - contains a photodetector at it's tip that enables the screen position to be calculated from refresh rate information
• 3D pen position sensor
• Data glove, transmits positional data from the wearer's hand movements

• Data storage
• Hard Disk, usually internal, with capacities of up to tens of Gigabytes (GB)
• Floppy magnetic disk, with capacities of up to tens of Megabytes (MB)
• Magnetic tape, used for large capacity backup/archive storage, with capacities of up to units of GB
• Optical storage, compact disk (CD), with a typical capacity of 650MB would hold over 1000 large scale (1:1250) 500km x 500km digital map sheets
• Digital Video Disks, with a capacity of 4.5GB

• Multimedia on networks
• Hypermedia systems are multimedia systems with link-based navigation
• Distributed databases on Local Area Networks (LANs) and Wide Area Networks (WANs) and the international network, Internet
• A World Wide Web (WWW) page or series of pages form a route through a hierarchical or network model using Uniform Resource Locations (URLs) to both graphical (spatial) and textual (aspatial) data. Additionally to other media such as sound and video data
• Access to the Internet using a Web browser such as Nescape Navigator or Microsoft Internet Explorer

---

3. Computer Science Aspects

• Multimedia object modelling in GIS
• Geographical access, coordinate based, place name, map region, arbitrary point
• Hypermap concept - similar to hypertext document where texts are organised bysemantic units called nodes and associated with links
• Icons, windows etc. represent multimedia entities in a database, linked by graphics means on the screen, e.g. a line, area, point or several of these
• Many current 'relational' database management systems (RDMS) will store pointers to graphical, sound and video files using appropriate executable programs to process them
• Simple relational model for multimedia data

PICTURE1 (Image_ID, Format, Resolution, Capture_date, Filename)
VIDEO1 (Film_ID, Time, Format, Filename)
SOUND1 (Song_ID, Time, Format, Filename)

Diagram of a more complex object model for multimedia data

• The data would be processed for retrieval by table name and format attribute value

Diagram of a hypermap with multimedia data linked to an area

• Queries for retrieving Hypermap nodes would use conventional GIS spatial query (select graphical node, define area, buffer zone etc.) and SQL, textual, semantic matching

• Advanced Hypermap systems will match images, video and sound sequences, with data of the appropriate media, held in the database

• Standards exist for multimedia data
• Image formats - TIFF, BMP, GIF, JPG, PCX, WPG, etc.
• Video Data - NTSC (National Television Standards Committee), PAL (PhaseAlternate Line), SECAM (Sequential Colour and Memory)
• CD-ROM - CD-RX (CD ROM Read-Only Data Exchange), DXS (Data Exchange Standard) for CD-ROM file systems. Colour Book Standards (Philips and SonyCorps)

• Compression of graphical data
• Uncompressed Image
• true colour 800 x 600 pixel image requires 1.44MB of disk space
• a 10 second video clip played at 30 frames per a second, with a resolution of 320 x   200 pixels, and in true colour requires 57.6MB of storage
• Compressed bitmap formats
• Still image
• JPEG (Joint Photographic Experts Group) can compress images from 2:1to 160:1 using a symmetrical compression algorithm. This is a 'lossy' scheme as reconstructing an exact replica of the original is not possible
• LZW (developed by Lempel, Ziv and Welch) is a 'lossless' scheme which substitutes more efficient codes for the data
• Fractal image compression (used by Microsoft in the Encarta multimedia encyclopedia) uses Fractal segments and three-dimensionalaffine transformations
• Compression and decompression for JPEG and LZW take about thesame time. However Fractal compression takes much longer; eight minutes compared to 41 seconds (JPEG) while decompression is faster, seven seconds compared to 41 seconds (JPEG)
• Compressed images are resolution dependent
• Video, moving images
• MPEG (Motion Picture Experts Group) is the standard, it uses intraframe coding, which removes redundancies within individual frames,50:1 compression rates are possible
• Px64 is the video conferencing standard compression algorithm of CCITT Consultative Committee International Telegraph and Telephone)

4. User Interface

• Multimedia relies on ‘hyper-links’ which ensure integration of ‘documents’ or ‘pages’

• Hyper-links can be initiated through ‘hot-spots’ defined by:
• text
• coordinated position
• area of image
• embedded objects

• VR can incorporate similar linkages and also requires sophisticated graphic displays, possibly including stereo viewers, moving chambers, audio, etc (See Section 2), along with'cursor' positioning, possibly in four dimensions
• Types of interaction
• passive or active
• natural (language, spoken word) or artificial (‘fly-bys’)

• ‘Free-form’ active navigation of data is user-controlled and needs
• navigation tools
• joysticks, mouse, pointer
• browsing and searching tools
• querying and reporting capability
• visual ‘trails’ to monitor navigation
• preview of any 'pre-set' routes
• overview showing complete 'map' or virtual world being used
• ‘you are here’ indicating relative position
• recording the thread
• producing a route plan of locations accessed or journey undertaken

• Cognitive aspects of interfacing with MM and VR have been researched
• 'free-form' navigation possibilities may lead to a lack of focus and inefficient interaction
• real world problem solving may be difficult in these environments
• multi-sensory access to spatial data may lead to information overload
• improvements in realism and more naturalistic interaction with data may improve decision making

5. Interaction with Geographic Information

• New search languages are being developed such as OO-SQL and MM-SQL both part of SQL3 to aid in searching MM data

• 3D and MM data can either be attached to the GIS as attributes of 'standard' datasets or be used as data sources directly to which further information is then added

• MM data allows for
• opening of GIS to more people
• change in the application of spatial information
• aid in enticing a younger audience to be involved
• creation of 3D models from still and video imagery

• VR can be used in a GIS in two ways
• a tool for purely viewing three dimensional models of data
• this can be purely in an office situation or in the field overlaying three dimensional data on top of real world data
• applications of the latter in underground pipe work, user can 'see' network under their feet
• the whole user interface to the GIS dataset, allowing for the display of VR, MM and standard data in three dimensions
• this would envolve the creation of a virtual interface
• possibility of viewing any data easily from any angle

6. Applications

• Anywhere the "limited sensory bandwidth of current GIS representations of the world" (Shepherd, 1991) needs to be extended
• to overcome the stylised and conventionalised picture of the real world which GISoften gives, by constructing VR interfaces to spatial databases and by using MM integration of disparate data sources

• Education
• self-led interaction with the real world, especially for children
• introducing geographical concepts, displaying distant 'realities'
• possibilities, using MM and VR, of the 'virtual fieldtrip'
• use of MM for local studies and global geography knowledge building, whilst integrating with other National Curriculum subjects such as history, economics, biology, geologyand information technology

• Scientific research
• creating three and four dimensional views of spatial data
• preliminary views of integrated data sets prior to verification of data linkages and casualtiess
• MM integration and overlay of datasets, for example; vector data with attribute information on raster satellite imagery
• exploratory data analysis in ‘virtual worlds’
• physical geography data, e.g. meteorological, geological, oceanographic data ideallysuited to its four dimensional nature
• VR applications include environmental monitoring, hazard and risk assessment, atmospheric modelling, planning and forecasting, pollution analysis, terrain visualisation, multi-variate analysis

• Military
• for training purposes and scenario building, particularly VR representations of terrain

• Entertainment
• improving realism of interaction with spatial data

• Built environment
• VR applications in architectural simulation, urban planning, resource modelling

• Archival of geographic information
• MM storage of the disparate range of data which can convey geographical information

7. Potential

• The future for both MM and VR is developing rapidly, input is coming from both the entertainment industry and military as well as major software houses

• Increased usage of all five senses
• in 1996, only have sight, sound, touch
• experimental ideas in smell
• little approach to the idea of taste

• Advancement to the masses requires more computer power at lower price range giving
• photorealistic shading of VR models in realtime
• immediate response and display update
• high resolution images

• VR and MM across the web
• Already have basic MM and VR across the internet using Web browsers
• Developments need to be made in the searching and indexing of multimedia data sources
• HTML, VRML, Java are 'on-going' developments to aid in distributed data
• The Superscape product is currently, in 1996, the closest so far to true VR across the WWW
• Development of the WWW into a better, more organised, system than in 1996

• Open system technology
• creation of distributed computing for GIS in general
• seemless remote data object access
• use of remote application objects
• driving force is the Open GIS Consortium

• Data availability
• still only really use and collect static 2D
• move to full 4D data sets
• data collection with MM and VR applications in mind
• removal or lowering of cost and copyright restrictions

• More public applications
• Currently just visualisation in main
• More analysis of MM and VR data

8. Reference Materials

Botto, F., (1994), Multimedia on your PC, Sigma Press, Wilmslow.

Krummenacher, B. and Hersch, R., (1993), Parallel Image Storage and Retrieval, in Thalmann, N.M. and Thalmann, D. (eds), Virtual Worlds and Multimedia, John Wiley, Chichester, pp. 13-21.

Brodie, K.W., Carpenter, L.A., Earnshaw, R.A., Gallop, J.R., Hubbold, R.J., Mumford, A.M.,
Osland C.D., Quarendon P. (eds.), (1992), Scientific Visualisation: Techniques and Applications, Springer-Verlag, Berlin.

Buttenfield, B.P., (1991), Visualization, in Maguire, D.J., Goodchild, M.F. and Rhind, D.W. (eds.), Geographical Information Systems, Vol 1, Longman, Harlow, pp 427-443.

Cartwright, W., (1994), Interactive Multimedia for Mapping, in MacEachren, A.M. and Taylor, D.R.F. (eds.), Visualisation in modern cartography, Elsevier Science, Oxford, pp. 63-89.

Cheiney, J. and Kerherv, B., (1990), Image Data Storage and Manipulation for Multimedia Database Systems, in Brassel, K. and Kishimoto, H., Proceedings of SDH90, pp. 611-620.

Cook, B.M. and White, N.H., (1995), Computer Peripherals, Edward Arnold, London.

Foley, J.D., van Dam, A., Ferrier, S.K. and Hughs, J.F., (1990), Computer Graphics
Principled and Practice, Addison-Wesley.

Fuma, F. and Bradley, J., (1991), A real-time display for tactile images, in Klinger, A.(ed.), Human-machine interactive systems, Plenum Press, New York. pp. 269-276.

Hearnshaw, H.M. and Unwin D.J., (1994), Visualisation in Geographical Information
Systems, Wiley.

Hersner, W. and Kappe, F. (eds.), (1994), Multimedia/Hypermedia in Open Distributed Environments, Springer-Verlag, Berlin.

Huber, M., (1994), Mutlimedia enhances GIS applications, in GIS World, Vol. 7, No. 8, pp.
51-52.

Jacobson, R., (1994), Virtual worlds capture spatial reality, in GIS World, Vol. 7, No. 12, pp. 36-39.

Kubo, S., Takamura, S. and Yoshino, S., (1990), Mutimedia GIS on PC, in Brassel, K. and Kishimoto, H., Proceedings of SDH90, pp. 363-370.

Laurini, R. and Thompson, D. (1992), Fundamentals of Spatial Information Systems,
Academic Press, London, pp. 594-619.

Shepherd, I.D.H., (1991), Information integration and GIS, in Maguire, D.J., Goodchild, M.F. and Rhind, D.W. (eds.), Geographical Information Systems, Vol 1, Longman, Harlow, pp     337-360.

Shepherd, I.D.H., (1994), Multi-sensory GIS: Mapping out the research frontier, in Waugh,
T.C. and Healey, R.G. (eds.), Proceedings of SDH94, pp. 356-391.

van Oosterom, Peter, (1993), Reactive Data Structures for Geographic Information Systems, Oxford University Press, Oxford.

Worboys, M.F., (1995), GIS A Computing Perspective, Taylor & Francis, London, pp. 23-43,
287-299.

9. Web References

Fairbairn, D., Parsley, S., (1996), The Use of VRML for Cartographic Presentation http://www.ncl.ac.uk/~nsurv/campus/vr_carto.htm

Parsley, S., (1996), A Three Dimensional, Multimedia, Geographical InformationSystem Across the Internet
http://www.ncl.ac.uk/~n230208/research/mmgis.html

Evaluation

Citation

Taylor, G.E., Fairbairn, D. J., Parsley, S., (1996) Multimedia and Virtual Reality,, NCGIA Core Curriculum in GIScience, http://www.ncgia.ucsb.edu/giscc/units/u131/u131.html, posted December 17, 1997.