Application class: Assisting people with contextual information about their activities
This class is primarily relevant to AR applications, which are able to inject additional information, objects or features into a user's existing perception of their environment.
Users equip an AR-enabled headset that can scan a real-life environment and overlay the image with data-based contextual information in real-time.
Existing alternatives
The obvious alternative to AR applications is experiencing reality without information overlay, in other words, continuing to operate business-as-usual.
Potential benefits
In AR, users can see hidden information about their surroundings as they are experiencing them in real-time. This can be useful for
- tourism: for example, walking past points of interest and seeing information about the height of a building and its history
- multi-tasking: undertaking difficult or mentally taxing tasks without needing to switch contexts or use other devices
- obtaining remote assistance: drawing on expertise remotely or whilst in the field, which will improve work efficiency and upskill the workforce. The ability to channel expertise remotely reduces the costs and time pressure of bringing an expert onto the physical scene.
Potential harms
- We can identify few risks of using AR technology in this context, outside of the risks already occurring with business-as-usual. For example, errors in maintenance from using an outdated AR schematic map is akin to having used an outdated paper map, but AR schematic maps can be more easily updated and distributed, provided other workers use the same AR systems.
- Unexplored technical limitations: As AR tech is still in its infancy, the technical limitations are unclear. As a result, setup could be expensive, only purchase-able from certain vendors, and may require certain environments to work (e.g. good lighting conditions, good bandwidth).
Example applications
Application: Rail corridor maintenance activities
Description: AR headsets with automatic digital schematic overlay to assist workers in rail corridors.
Role of government: Application user
Rationale: By generating a digital schematic (e.g. of a railway network or an electrical network) with corresponding geographical coordinates, this information can be superimposed on an AR device, such as headsets or smart glasses. A rail-corridor maintenance worker could use this digital overlay to obtain a 360° view of the underlying corridor infrastructure (e.g. wires, pipes) while standing in the corridors themselves. This would be comparable to the existing Sydney Trains smartphone-based AR maintenance approach, but the automated overlay on an AR headset would remove the degree of separation between the phone and the scene.
Assumptions: Maintenance areas — especially ones underground — may lack the lighting conditions and sufficient internet download speeds for live AR schematics overlay, unless the schematics have been stored in the AR device itself. In that case, a sufficient storage space on the device would be essential.
Specific considerations: The success of AR overlay maintenance procedures depends on the accuracy and completeness of the schematics used. If necessary information that impedes maintenance (e.g. electrical network routes used by private companies) is not included in the schematics, it could disrupt networks and the NSW Government could be liable for damages.
Existing implementations: None for AR headsets, however Sydney Trains has implemented AR rail corridor safety applications for mobile devices. [132]
Application: NSW food safety certificate AR overlay
Description: A metaverse company or third party developer could provide a Restaurant Finder app. A user would walk through a restaurant area and see properties of restaurants displayed on the facade of the restaurant. This could include properties such as:
- restaurant review ratings (from a platform provider, third party user rating service or a critical specialist restaurant review site)
- restaurant hygiene and food safety scores ('Scores on Doors' provided by the NSW Food Authority) [133]
- restaurant offences (from the 'Name and Shame' register provided by the NSW Food Authority) [134]
- Business licences such as food handling licences from NSW Food Authority. [135]
Role of government: Service provider
Rationale: This AR certificate overlay could replace restaurant guide phone applications with no significant risks to the community. Having a wearable AR device that automatically overlays information about a restaurant would forgo needing to manually search and cross-reference restaurants using a phone, and would have few or no additional requirements on the data APIs. It could encourage the public to try different businesses, raise public awareness of the hygiene statuses of businesses, and thus generate further incentive for businesses to maintain good hygiene standards. However, as this AR overlay would be bespoke, it could be more expensive and complex to implement compared to using a phone, and have more significant hardware requirements.
Assumptions: The wide adoption of AR interfaces commonly worn or used in public
Specific considerations: To achieve this, The NSW Government would need to
- Provide business data through an API
- Provide hygiene and food safety scores for each business through an API
- Provide business offence data through an API
- Provide building data from digital twin to assist visualisation of virtual facade
Existing implementations: There are no AR headset examples with similar use cases to Scores on Doors, however source code for automatic QR tracking has been created for AR headsets such as HoloLens 2. [136] Combining the two may be feasible to create the desired functionality.
[132] Crozier, R. Sydney Trains progresses AR protection for rail workers. iTnews (2022).
[133] NSW Food Authority. Scores on Doors. NSW Food Authority.
[134] NSW Food Authority. Name and shame. NSW Food Authority.
[135] NSW Food Authority. Licensing and notifying. NSW Food Authority.
[136] qianw211. QR code tracking overview - Mixed Reality. Microsoft (2022).