Technical Retrospective: Elixar Remote Expert

Dial in help without dropping tools! Elixar Remote Expert is a smart-glasses enabled video conferencing platform for remote instruction, inspection and certification.

Bondi Labs has a long history of delivering technologies for government and industry, specialising in education and training through simulations, including Virtual and Augmented Reality.

Contents

• High Risk Workers Need Their Hands
  • Device Safety
• MVP to Production
  • An Unexpected Catalyst and Industry Validation
• Smart Glasses Demand Smart Features
  • Features that drove adoption
• From Solo Developer to Product Team Lead
• Smartglasses User Experience
  • Challenge: Head mounted camera
  • Challenge: No keyboard
  • Challenge: Small HUD screen
• Final Thoughts
  • The future of human-computer interaction is integration with the real world
• External Publications

High Risk Workers Need Their Hands

Picture a biosecurity inspector examining shipping containers for hazardous materials, or engineers handling high-voltage power lines.

These high risk occupations have two very important things in common; the field worker's lives, are literally in their hands.

They need handsfree operation, high quality evidence capture and reliable communication channels.

Device Safety

Safety requirements often mean smartphones can't be present on-site, devices like the Vuzix M400 and Realware HMT-1 have adapters that attach the device directly to safety equipement.

Realware also offers 'intrinsically safe' variants, certified for use in hazardous, potentially explosive electrical environments.

MVP to Production

In 2019 CTO Sam Clifton (Bondi Labs CTO) handed me a Vuzix M300 and pointed me at the open-source Web Real-Time Communication library (WebRTC).

A few weeks later the MVP was ready. Android/Java/Kotlin, Socket.io, React and the Android WebRTC library enabled realtime peer-to-peer (p2p) connectivity and the Vuzix voice control SDK for complete handsfree operation.

As demand increased, static AWS Cloud deployments became scalable cloud-agnostic Kubernetes Serverless (infrastructure-as-code with Pulumi was a huge time-saver 👏) and paired with Coturn for NAT traversal and media relay.

Alibaba Cloud was leveraged to service China on it's new Serverless Cluster service.

An Unexpected Catalyst and Industry Validation

When COVID-19 hit, travel restrictions affected industry and government and Elixar Remote Assist was thrust into the spotlight.

The Department of Environment, Land, Water & Planning (DELWP - now DEECA) trialled remote biosecurity inspection.

Manufacturers began testing remote expert assistance and compliance auditing.

Industry partnerships with Bondi Labs enabled corporations to begin in-depth assessments of worker augmentation, particularly in rural food-processing centers where a single sick expert can lead to a complete shutdown of operations.

5G had a timely launch in Australia (2019) and Bondi Labs was awarded a government-funded trial for improve site connectivity and AI experimentation. As all Australians know, internet connectivity can be a serious challenge.

Smart Glasses Demand Smart Features

Connecting more individuals without degrading stream quality is not trivial, particularly when the most important peer connected often has the weakest network connection.

We COULD show 10 (or 100) video streams at once...

But the smartglasses user only needs to see the speaking individual, so why transmit all videos (or even audio...)

We ran our daily stand-up on Elixar to stress test media relay capabilities and stream prioritisation controls every day.

Native voice control provided 100% command coverage for Elixar field users wearing RealWear (HMT-1, HMT-1Z1) and Vuzix Devices (M Series, Blade).

Our biggest competitive advantage was high-resolution still captures that could be taken on-device, during calls and shared in the background without interrupting the video feed, even during low connectivity sessions, essential for inspection in rural areas.

I compiled modified WebRTC libraries for Android to achieve this, I'll spare you the detail, just know Google's monorepo was PAINFUL to deal with 🤯

Features that drove adoption

We prioritised features in product cycles that solved customer needs and encouraged adoption over our competitors:

• Live document and map sharing
• Zoom, flash-light and field-of-view controls
• QR code device actions (e.g., secure device authentication)
• Remote device actions (permission based remote control)
• Live feed annotation (also great fun in stand-up)
• Client and server-side recording
• Priority streams (selectively prioritising video streams)
• Adaptive stream quality controls (network performance -> stream priority management)
• Session timeline review (session and evidence playback)

From Solo Developer to Product Team Lead

By 2022 Elixar has evolved into a fully featured SaaS and my role had evolved with it. From solo greenfield developer to coordinating product delivery with a team of four other exceptional engineers:

• Mun Fei (Ben) Tan
• Jason Lee
• Samuel Clifton
• Phong Lee

All in the vision of Jonathan Marshall (our fearless serial entrepreneur CEO) and the extremely talented Bondi Labs product team.

• UX Designer and Researcher Dr Joshua Hall
• Design Director Jeremy Hamilton
• Security Advisors Omar Jarkas and Joshua Scarsbrook from the University of Queensland (UQ)
• AI expert Parham Khojasteh
• QA specialist Tristan Dubois
• Industry researcher Nanditha Sirigiri
• Sales managers Stuart Smith and Graeme Baxter (who were never afraid to get their boots dirty all across Australia to demo the platform).

My proudest achievement, the three graduates we brought into the Elixar team for mentorship (via Monash University) became strong ICs across this stack.

Balancing hands-on development, feature prioritisation and mentorship; "trial by fire" often came to mind, but Jira and effective collaboration kept us focussed.

I am still yet to find a team that rivals the development velocity we achieved for Elixar.

Smartglasses User Experience

For now, smartglasses are an uncommon but extremely effective tool for augmenting a workforce, but only when done right.

They pose UX challenges that we don't see anywhere else. Small screens, voice control and navigation, head-mounted cameras, the list goes on...

But as we start to solve for these constraints, the software becomes more intuitive, the user is less distracted and can be better engaged with the real world.

---

Challenge: Head mounted camera

Solution: Hardware + software video stabilisation

Less pointing cameras, more operational awareness

Challenge: No keyboard

Solution: Voice controls, passive QR scanning and AI object detection

Less typing, more data captured

Challenge: Small HUD screen

Solution: Fewer UI elements and optimized navigation trees

Less distraction, more focus on task

constraint + creativity = opportunity

Jeremy Hamilton (design director at Bondi Labs) has a detailed portfolio piece on the Elixar UX and design process

Final Thoughts

The future of human-computer interaction is integration with the real world

Smartglasses and augmenting technologies are still in their infancy. If this project is any example to go by, I'm excited to see where this takes us.

Harry Martin

External Publications

• Jeremy Hamilton, Design Director Post-Mortem
• BAI Communications, Innovative 5G technology trial, 2022
• AMCA (Australian Communications and Media Authority), 5G in Australia - Market Study, 2022
• BeefCentral, Remote veterinary inspections under study, using ‘smart’ glasses, 2023
• AMPC, Remote inspection and auditing for the Australian red meat processing industry, 2023
• AMPC, Remote Vet Inspection
• Vuzix, Vuzix Delivers Smart Glasses to Assist with Plant Audits During COVID-19 Pandemic, 2020
• UQ, ARM Hub Launch Partnership for Applied Cybersecurity Research with Manufacturers, 2022
• ARM Hub, Leading Australian university partners with ARM Hub to improve the cyber security for manufacturers, 2021