I have written previously on the conceptual and likely practical relevance of immersive documents to the library and information science community. I have defined immersive documents as those which deliver an unreal reality to the ‘reader’. Reader, in this context is a loosely defined term, as the concept of the document is expanded to embrace the type of experience afforded by technologies such as virtual reality (VR), pervasive computing and the multisensory Internet. In this case, the reader may also be described as a viewer, a player, a user or a participant, but in writing from the perspective of LIS, the term reader seems to be an apt, all-encompassing descriptive.
This idea was sparked by Shuman’s scenario of the library as the ‘experience parlour’, described in “The Library of the Future”, published by Libraries Unlimited in 1989.
Immersive documents, wherein reader engagement delivers the perception of an unreal, computer generated world as indistinguishable from reality, do not yet exist. This type of intangible document would, like other digital entities, only exist when its overarching computer program executes, and the associated file of content is read and processed. An immersive document, would in addition, allow for real-time reader behavior and interaction data to be processed along with the base content, and to influence the narrative outcome. The term ‘narrative’ can imply that the immersive document delivers the perception of being within a fictional novel. Whilst this is certainly one example of an immersive document, the ‘narrative’ could be a rendering of an historical event, a travel or news documentary, or of a training scenario.
Immersive documents comprise technology, software, and novel narratives, and developments in all three component areas will be needed for such documents to be realized, although an additional, important driving factor will be the strong desire for participatory experiences from readers.
At the time of writing, Spring 2016, we are anticipating the first wave of commercial VR head mounted displays, (HMDs), from Oculus, Samsung, HTC and Sony, which work with compatible computers and software to render stereoscopic, computer generated, virtual worlds, accompanied by sophisticated sound. These environments are compelling, and the ones I have tried deliver a realistic experience of being in a virtual world as an observer.
At this stage however, the reader is not able to fully interact with the environment or content; in programs which support telepresence, this only extends to feeling parts of the body. Doubtless, as technology advances, a fuller sense of body presence in the unreal reality will emerge, but this needs to be matched by the authoring of scripted worlds, to allow for more reader determined behavior and interaction with elements of the unreal world portrayed.
In support of an enhanced feeling of immersion, the HMD interface needs improvement, as at the moment it is rather clumsy and restricts the sense of full immersion, or the suspension of disbelief. Pervasive, wearable technology will undoubtedly improve to the point at which we become less/(un) aware of the interface, and we can look forward to contributions from the fields of neuroscience and psychology in reducing the friction between reality and unreality. Sensory implants already allow merging of corporeal functions with the machine, so it will doubtless be an incremental step for this type of data recording to feed into immersive documents to simulate all five senses.
Advances in multisensory transmission over the Internet, i.e. smell, taste and touch, will further enhance our ability to make the unreal, real, and at a distance.
Although fully immersive documents do not yet exist, it would be prudent for the LIS community to consider at this stage, whether the sector should play any part in the handling of these entities, and if so, in which ways. It will be easier to record and collect the documents as they emerge, if frameworks for understanding and description are already in place – thus avoiding the enormous retro-conversion efforts needed to redesign and extend current bibliographic data to enable semantic web functionality and promote discovery.
There is, already extant, a growing body of partially immersive documents, which has prompted enquiry into how these could be recorded, stored, described, discovered, shared and preserved. Whilst LIS related work on these partially immersive entities is scattered amongst other disciplines, and in no way comprehensive, it is a worthwhile source of material relevant to the handling of future immersive documents, and is by its nature surely of interest to the LIS community.
Some of these partially immersive documents are born digital entities, such as: animated texts, animations, visualizations, simulations, interactive narratives, videogames, virtual worlds, 360 digital video recordings and digital artworks. The documents all furnish the reader with varying degrees of unreal reality, but the reader remains aware of the physical world, and its relationship with the immersive experience.
As with fully immersive documents, these partially immersive document entities also exist as computer files containing content together with display or processing instructions, which require specific technological platforms on which to run. They exist only when the content data is acted upon by the software instructions. In many cases, such as with interactive narratives, there is scope for real-time data input from the reader, which generates novel content. Formats such as visualisations, simulations and digital art can all rely on other program data for input as well as a human reader.
This complex, dynamic nature demands a more detailed approach to document handling than that used for digital files representing more conventional types of static document, (born digital or scanned) such as books, journals, manuscripts, datasets, sound, images or films, even though standard metadata for these more familiar documents may still need to be agreed, and issues of preservation in perpetuity remain.
Other types of partially-immersive document are ephemeral, temporal, real world activities, which need first to be recorded in order to be preserved for future access and understanding. These include participatory theatre performances, dance performances and a myriad of installation art.
Augmented reality, and mixed reality events offer yet more time dependent document forms, blending physical world immersive events with the digital.
Immersive and partially-immersive documents may be thought of as ‘complex documents’, for which an interdisciplinary approach to their information communication chain journey may be beneficial, in contrast to the solely LIS focused efforts to record more usual document forms.
One area in which a significant amount or research has been done is that of preservation, and there are several disciplines which have started to consider how to describe and record complex documents within their domain.
The JISC (Joint Information Systems Committee) organized the POCOS project (Preservation of Complex Objects Symposia) in 2010, which considered complex documents as complex digital objects. To simplify the many types of objects under scrutiny, they were divided into three catagories and investigatied from the perspectives of simulations and visualisations, software-based art, and gaming environments and virtual worlds. The original publications from the project are published in “Preserving Complex Digital Objects” edited by Janet Delve and David Anderson, Facet, 2014.
Perhaps the main challenge to handling complex documents within the information communication chain comes from reader participation.
Participatory behavior can take many forms, from allowing software to read facial expressions, or to measure pulse or heart rate, to allowing full interaction with objects, characters or avatars within the simulated environment. This data is complex to record and process, but is also complex to add to document description. In some cases, there is more than one participant, adding to the complexity. For the purposes of recording and preservation, there is the question of what is being recorded, and how authentic is the replay?
In some games for example, there are hundreds of participants. Likewise for immersive, participatory theatre. Once we consider participation, we are faced with an additional layer to the recording, description and sharing of the document – do we record the viewpoint/experience of the participant? Can we? There are thus dual (possibly multiple) reader modes; that of a first time reader, or that of previous readers.
It is perhaps surprising that the question ‘what is a document?’ is still unclear, five and a half thousand years after records began. Humanity nonetheless, still endeavors to record and preserve more and more layers of the human condition.