Properties of metaverse applications

Three key properties define metaverse applications in this report: a persistent virtual world, presence, and social connectivity. This section examines each of these properties and the various technologies that enable them.

Property 1: A persistent virtual world

Metaverse applications are typically 3D persistent virtual worlds experienced in the first-person, similar to 'first-person' computer games. There are locations users can travel to and objects users can interact with. They are typically persistent, in that they do not reset to an initial configuration but change over time (possibly through interactions with users). Users are present in the virtual world through their 'avatar': a 3D representation of themselves that may or may not resemble how they look in reality.

Enabling technologies

3D modelling

Techniques and tools to construct, texture, and animate 3D objects and environments are well-established in computer-aided design (CAD) and computer game development, as well as TV and cinema. These tools support output that can be highly detailed, realistic or stylized. Through 3D scanning and photogrammetry, real objects or locations can be recreated digitally with high fidelity.

Game engines

Game engines are software tools used to create virtual worlds. Unlike the rendering tools used to generate TV or cinema effects, game engines render 3D sound and vision in real time: meaning it is possible for the world to be interactive, reacting to the inputs of users as they occur. Game engines are responsible for a number of tasks in a virtual world including

rendering, which turns mathematical representations of digital objects into visual output for display in a headset or screen that accounts for lighting, shadows, colours and textures
simulating physics and other processes, such as ensuring that a lamp falls to the ground and smashes if a user pushes it off a table
simulating 'non-player-characters' (NPCs); avatars controlled by software rather than a human.

Examples of game engines commonly used for metaverse-style worlds include Epic Games' Unreal engine and Unity Technology's Unity engine. ^[17], ^[18]

Typical elements

A well-defined geography: The virtual world has locations users can travel to, perhaps corresponding to the real world's geography or a completely invented one. The world might have buildings and open spaces like a modern city, or be under water, in space or in some other fantastic setting.
3D representation: The virtual world has length, depth and height just like the real world, and can be examined from different positions, angles and locations. This is contrasted with many 2D computer games that show a flat, drawing-like world or a pseudo-3D world rendered from a fixed perspective.
Interactivity: The virtual world has interactive elements: objects can be picked up, manipulated and interacted with. Potentially, buildings or the landscape itself may be able to be altered by users.
Persistence: Changes made to the world persist over time. This is in contrast to a world that resets every time a user logs in. Changes may be caused by users' interactions, developers' interventions, or simulated dynamics of the world itself. Time in the world is sometimes frozen when no user is present, or it may continue to pass. This means, for example, that a user may notice changes that have occurred while they have been disconnected from the metaverse application.

Current limitations

Mark Zuckerberg takes a selfie in Meta's VR Horizon Worlds. Low geometric and lighting complexity are a consequence of the limited computing power of the Meta Quest 2 headset for which Horizon Worlds is designed. [19]

Rendering fidelity: There are currently significant limitations on the complexity and visual fidelity of virtual worlds when using virtual or augmented reality technologies. The computing power available for rendering on headsets requires compromises between:
- high frame rates required to combat cyber-sickness
- high resolutions needed to render to screens very near the eye
- high levels of geometric and lighting complexity to reproduce many natural scenes.

These issues may be ameliorated by offloading some or all rendering to a dedicated desktop or server, however the network bandwidth and latency to that server then imposes its own limitations.

Simulation complexity: Many natural and human processes are too computationally expensive or too poorly understood to simulate in a virtual world. Phenomena such as object deformation, turbulent fluid flow, chemical and biological processes and many others are infeasible to simulate today in many cases, for example. Virtual worlds use simplifications and/or visual tricks to give the appearance of complex phenomena like these.

Property 2: Presence

Metaverse applications use virtual and/or augmented reality to immerse users in their virtual worlds, giving them a strong sense of presence. Rather than viewing the world from the outside, akin to watching a movie, users get the sense that they are 'really there'.

Enabling technologies

Virtual reality (VR)

Researchers in Germany using a VR headset and motion controllers to demonstrate how astronauts might use VR to train how to extinguish a fire inside a lunar habitat. [20]

Virtual reality replaces the real world around the user with an alternative. A user may be sitting in their living room, but in virtual reality they are taking a tour of (a computer recreation of) the Eiffel Tower. To an extent the user is perceiving the virtual world as if they are there: they can turn their head to view the scene from different angles, and stand up from their living room chair to help them gain a better view.

Today, virtual reality experiences are most commonly produced by headsets, devices somewhat akin to large ski goggles that users wear over their eyes. Each eye sees its own small video display to allow for stereoscopic 3D imagery to be presented, and hear spatial audio from headphones either built into the headset or worn separately.

Augmented reality (AR)

Another approach to completely replacing the real world with a virtual alternative, is to 'augment' the real world with additional (virtual) features. This is the approach taken by augmented reality (AR), which for example, can place virtual posters on real buildings, or display information about a person that appears to hover over their head. ^[21]

A smartphone augmented reality application by artist Vishal Dar, that adds abstract light sculptures in response to the environment. [22]

Like virtual reality, augmented reality can be delivered via a headset-like or glasses-like device, however at its current stage of development it is most commonly accessed via mobile or tablet camera overlays. Users wearing AR headsets can navigate without bumping into things, but their visual and audible experience is augmented with sights and sounds the headset is overlaying on the real world in such a way as to make them seem real. AR headsets may use semi-transparent displays that let light from the real world in, or may be more similar to VR headsets and use outward-facing cameras to recreate the augmented scene in its entirety on opaque displays.

One feature of augmented reality is that additional information can be provided to users in their current sensory context: for example, a user conducting a field repair of a generator can have the service manual in their field of vision, perhaps even with relevant parts visually highlighted as needed.

Mixed reality (XR)

When speaking generally about virtual reality, augmented reality, or a mixed experience in which some participants are using either technology, it is common to use the term mixed reality (XR).

Typical elements

Experienced in the first person: The virtual world is experienced in the first-person, similar to 'first-person' computer games. This means that the view into the world is rendered from the perspective of the user's eyes. First-person contrasts to a third-person view, common in 'role-playing' computer games in which the view into the world is rendered from the perspective of a floating camera tracking the user from above or behind.
Stereoscopic visuals: A headset presents each eye with a different image, calculated so that the brain perceives depth correctly as it would in a real scene.
Head tracking: In order to adjust the images displayed to each eye as the user moves their head, headsets must track the head's position in space. Head-tracking can be accomplished with the help of external sensors or markers placed about the user's room, or through sensors (cameras, gyroscopes and accelerometers) built into the headset.
Spatial audio: Sounds appear to come from the direction they should, get softer as the source gets further away in the virtual world, and can even seem to be muffled when occluded by virtual objects.
Avatar: Users typically inhabit a virtual body in the metaverse world called an avatar. This may resemble them physically or not, and may not even be human. Users look out the avatar's eyes' and control its motions through keyboard, hand controls or motion tracking.
Body tracking: Tracking more aspects of the users body allows for a higher fidelity recreation of them in the virtual world. For example, modern headsets typically track the hands to allow users to see them to aid picking up virtual objects. ^[23] New headsets arriving on the market will track gaze direction, facial expressions and positions of more of the body (like legs) to improve the fidelity of avatars from the perspective of other people in the virtual world. ^[24], ^[25]

Current limitations

Cyber sickness: Human balance systems can be sensitive to inconsistencies between their visual inputs and those from their inner ear. Such inconsistencies cause not only sea-sickness, but 'headset-sickness', where users of XR headsets feel dizzy, nauseous and get headaches after prolonged use. ^[26] Improving headsets' head tracking capabilities and decreasing the latency in their displays will likely reduce the frequency and severity of this effect. However, many users today find the current generation of headsets impossible to wear for extended periods. ^[26]
Placing objects in a complex scene: AR, despite making fewer changes to the user's perception, is more difficult to implement than fully replacing a user's perception as with VR. Not only does the headset have to account for the virtual elements in a scene, it also must be able to understand the geometry and lighting of the user's real-world environment to be able to integrate those new elements convincingly. Many of the current 'augmented reality' products are really 'heads-up displays (HUDS)' which are standard screens overlaying information onto the users' field of vision without attempting to integrate that information into the scene.
Senses beyond hearing and vision: Other sensory phenomena are increasingly being added to the virtual reality experience but the work is at an early stage and is low fidelity. Haptic gloves or suits can mimic the physical sensation of interacting with objects and are expected to reach the mainstream market shortly ^[27], and some companies have experimented with smell-dispensers to further increase the sensation of immersion. ^[28]

Property 3: Social connectivity

Metaverse applications allow multiple users to join virtual worlds together and interact with each other. Current metaverse applications include the ability for people to have virtual meetings, play games, create art, and design objects or buildings. ^[29]

Enabling technologies

Social networks

Social media such as Facebook, Instagram, Twitter and TikTok rely on an underlying social network of connections and interactions between individuals. This network informs inferences about what content may be of interest to a particular individual, who they might interact with in the future, and what content may 'go viral'.

Given the interest in the metaverse from social media companies, it is reasonable to expect that these same approaches will find use in the metaverse. Explicit friendship links, similar to social media, exist in many metaverse applications. These may soon be augmented with 'implicit' links, inferred by the metaverse application from user behaviours. Implicit social relationships could be inferred, for example, by noticing people stopping to talk to each other, sharing a physical space, or even making eye contact.

Typical elements

Shared presence: Users can simultaneously navigate and interact with one another in a virtual space as if they were physically in the same room.
Long-distance communication: Users in the same world but not located nearby may be able to teleport to each other's locations or send text or audio messages.

Current limitations

Content moderation: Content moderation is an unsolved problem in existing social networks and internet forums. The addition of more flexible means of expression (for example, through body language) opens up new difficulties for platforms trying to remove abusive or unacceptable behaviour and content.
Scalability of simultaneous users: The technical challenges involved in creating metaverse worlds increase with the number of users present in an environment simultaneously. VR applications with higher fidelity avatars are currently limited to small numbers of participants (for example, 16 in the case of Meta's Horizons Workrooms application), ^[30] whilst those with larger numbers of users typically have simple, mostly static geometry and highly stylised avatars that do not attempt to reproduce accurate body pose. ^[31] Currently, network and computing power limitations make it difficult or impossible to support more than a few thousand concurrent users in a virtual space. ^[32]
Requirements on network infrastructure: Metaverse applications are heavily dependent on high-bandwidth, low-latency network connections to achieve real-time interaction between users. For people in remote areas with lower-quality internet connections, this may represent a significant barrier to their access to the metaverse.

Other technologies associated with the metaverse

This section outlines some technologies that are commonly associated with metaverse applications. These are not enabling technologies in the sense that they contribute to the properties of metaverse applications. Rather, they are associated with the metaverse because some application developers choose to integrate them. From the perspective of the NSW Government, these technologies are mainly relevant to this report because of the risks they create for users.

Cryptocurrencies

Commerce is an important part of the metaverse. Many metaverse application developers create virtual property including clothing, items and 'land' in their worlds which is for sale to users. ^[14] Gambling, especially virtual poker and casino games, is also offered in several metaverse applications. ^[33] Brands also perceive the benefit that a virtual world can be used to showcase real items: users could, for example, examine a new car or accessory in VR before purchasing it for themselves in real life. ^[34]

The mechanisms on which transactions within the metaverse are based are being explored by application and platform developers, but some are building their efforts on cryptocurrencies and non-fungible tokens (NFTs). ^[35] Cryptocurrencies such as bitcoin or ethereum take the place of traditional national currencies in these worlds. For example, a user may purchase a piece of virtual land in a metaverse world by transferring a certain amount of bitcoin to the metaverse application developers. ^[36]

The utility of cryptocurrencies for digital transactions is contentious, and is also largely independent of the metaverse as a particular application: advocates claim that such currencies are inherently 'private, safe and untraceable' ^[37], however, the bulk of both technical and financial experts without direct stakes in the success of cryptocurrency believe it has limited utility as a payment method, and is largely a way to circumvent banking regulation for the purposes of money laundering and fraud. ^[38]

Technical details aside, the volume of documented instances of large-scale fraud and ransomware attacks enabled by cryptocurrency suggests that if these currencies do become an integral part of the metaverse, users will be exposed to significant novel risks. ^[39], ^[40]

Non-fungible tokens (NFTs)

Some metaverse applications record the purchase of an item or land in a metaverse world with non-fungible tokens (NFTs). These are records on a decentralised public database (a 'blockchain') that record a particular user has purchased a particular item in the virtual world. In principle, this item can then be on-sold by the user to another party, independent of the metaverse developer. Advocates of using NFTs to record metaverse ownership also contend that the public nature of the blockchain facilitates owners moving their items between metaverse worlds. ^[41]

Technical experts argue about the degree to which NFT technology is required or even useful in enabling this notion of ownership, or the ability to transfer items between virtual worlds. One uncontested fact is that the relationship between the record of ownership of an item and the effective rights that grants a user in a particular virtual world is entirely at the whim of the developers of that world. ^[42] A user purchasing a virtual pair of sneakers for their avatar in one world, for example, cannot have their avatar in another world wear those sneakers unless that second world creates the in-game asset and chooses to acknowledge the user's ownership of it. Similar to cryptocurrencies, the world of NFTs is replete with fraud, and its use in metaverse applications is a source of significant risks for users.

[14] Marr, B. How To Buy Land & Real Estate In The Metaverse. Forbes (2022).

[17] Epic Games. Unreal Engine | The most powerful real-time 3D creation tool. Epic Games.

[18] Unity. Unity Real-Time Development Platform | 3D, 2D VR & AR Engine. Unity.

[19] Zuckerberg, M. We’re launching Horizon Worlds in France and Spain today! Looking forward to seeing people explore and build immersive worlds, and to bringing this to more countries soon. Facebook (2022).

[20] Advanced Ground Software Applications Laboratory. Reality check. The European Space Agency (2017).

[21] IEEE Computer Society. Personal AR for Data Visualization. (2021).

[22] Florin, F. Edge Of See: Twilight Engines. (2019).

[23] Donald, A. How To Use Hand Tracking With the Oculus. Alphr (2022).

[24] Vive. VIVE Tracker (3.0) | VIVE Australia. (2022).

[25] Pultarova, T. Emotion tracking sensors add feelings to virtual reality. (2016).

[26] Biener, V. et al. Quantifying the Effects of Working in VR for One Week. Preprint at (2022).

[27] Goode, L. Facebook Reaches for More Realistic VR With Haptic Gloves. Wired (2021).

[28] Stone, Z. Inside the smell-o-verse: Meet the companies trying to bring smell to the metaverse. (2022).

[29] Oculus VR. Introducing Studio Share: Sculpt with Friends in Oculus Medium. Meta Quest (2017).

[30] Meta. Horizon Workrooms for VR Remote Collaboration. Meta (2021).

[31] VRChat. VRChat. VRChat (2017).

[32] Takahashi, D. Improbable is attacking the metaverse’s networking problem. The Machine: Making sense of AI (2022).

[33] Lenihan, R. The Metaverse Has a Winner: Casinos, Gamblers. TheStreet (2022).

[34] Wong, Q. Shopping in the Metaverse Could Be More Fun Than You Think. CNET (2022).

[35] Fonarov, O. What Is The Role Of NFTs In The Metaverse? Forbes (2022).

[36] Binance Academy. How to Buy Land in the Metaverse? Binance Academy (2022).

[37] Edwards, D. How Cryptocurrency will be used in the Metaverse. Robotics & Automation News (2022).

[38] Zeitchik, S. The crypto-skeptics’ voices are getting louder. Washington Post (2022).

[39] Hanoch, Y. & Wood, S. Scams and cryptocurrency can go hand in hand – here’s how they work and what to watch out for. The Conversation (2022).

[40] Bainbridge, A. & Kent, L. Cryptocurrency scams targeting Australians as scammers bank more than $100 million. ABC News (2021).

[41] Ratan, R. & Meshi, D. The metaverse is money and crypto is king – why you’ll be on a blockchain when you’re virtual-world hopping. The Conversation (2022).

[42] Marinotti, J. Can you truly own anything in the metaverse? A law professor explains how blockchains and NFTs don’t protect virtual property. The Conversation (2021).