A Tale of Two Timelines

(Note: In June of 2022 I was asked to give a brief talk on my upcoming book and also about WebXR to the New York Augmented Reality (NYAR) meetup group. Here are my remarks as originally prepared, lightly edited. This post will be updated when a recording of the event is made available)

07-Jul-2022 Update: Video of the event has been posted! View the full talk here (I start at the 37:00 mark or so) or via the below embed.

Tech is a funny business to be in - you never really know when something will come around that upends everything. Most progress in tech is far more evolutionary than revolutionary but despite that I suppose it's part of human nature that we tend to focus on flashy large changes than we do for a gradual but steady ones.

This is a tale of two timelines. Like any good story involving a multiverse, our shared universe diverges from the other at a specific point in time. No, Steve Jobs doesn't survive in either timeline, but our tale does involve his legacy in a non-trivial way.

The year is 2017, and Apple is one of the most successful corporations in history. Their market share in the US smartphone market exceeds 60%, giving Apple a huge amount of influence when it comes to setting standards in the like. The AppStore is the only real way to distribute applications on iOS that make use of any native hardware functionality. At this time as well, There was growing demand in the market for augmented reality applications at a consumer level. Previous effort at marketing dedicated AR hardware devices largely had failed over the preceding half decade or so (anyone remember glass?) with the somewhat notable exception of Microsoft HoloLens that has made a name for itself in the commercial and industrial sectors. Rumors of Apple working on their own set of AR hardware have been abounding for only slightly less longer than the iPhone has been in existence. Indeed, a recent Ars Technica article puts their availability at sometime in 2023, allegedly. Hold on to that thought for a minute while we continue to explore that long lost pre COVID past era known as 2017.

In the Year 2017

It was obvious to everyone in the tech industry that augmented reality was going to be big, And it wasn't unexpected. It's not like the sci-fi authors and video games and films of the previous decades hadn't been preparing the public for some form of augmented reality for a long time , it's more that the combination of hardware software and bandwidth hadn't yet been available at a consumer level. There are many different ways to define augmented realit and many different approaches. When I talk about augmented reality here I'm referring to the more colloquial generic catch-all term for it that encompasses everything from Garage Band's face-controlled Wah pedals all the way to turning a living room table into The Cones of Dunshire.

Now, love it or hate it , Apple's App Store is a walled off garden that is extremely effective at giving Apple control over what developers can publish on their platform. Smartphones were and still remain an obvious platform for targeting AR applications and it was also obvious that the best vendor agnostic way of putting the smartphone into that role would be via the web browser.

That’s on the entire opposite end of the spectrum from WebXR. WebXR is the resulting web standard that establishes browser API’s relevant to both VR and AR applications - open by nature, and despite this seeming contradictory interest (or perhaps, the cynic says, because of that contradictory interest), Apple and its Webkit engine joined the WebXR consortium as one of its founding members. They, along with Google, Microsoft, Mozilla, Samsung, (then) Facebook and others, have the goal of lowering the cost of entry and maintenance for developers as an end to the means of the primary goal of making it possible for web applications to access device hardware.

At the same time WebXR was launching, Apple launched its’ own proprietary closed-source ARKit that provides developers powerful API’s that can drive cutting-edge AR apps from the latest and greatest hardware – Apple’s of course.

A Reality Disjunction

  • What if ARKit released with WebXR support?
  • How much richer would the AR ecosystem be if it were?

The launch of WebXR in 2018 is it – this is our disjunction of realities. In our timeline, WebXR has evolved and grown as a standard along with all of its’ members save Apple, who has yet to support WebXR in its’ iOS WebKit engine. It should be a refresher for most that WebKit is the underlying rendering engine that Apple’s walled ecosystem requires browser vendors to use on iOS, with all the implications that entails.

If Apple had been truly dedicated to implementing the WebXR standard, then WebKit today would have more than just do-nothing feature toggles. In our sibling reality though, Apple worked to release an ARKit-to-WebXR interface. In this other reality, AR applications have flourished in the intervening years, to the point where AR applications have become a standard “table stakes” feature for that any retailer selling physical goods offers as part of their online shopping experiences.

Customers are happy to browse, shop, and ultimately purchase from multiple outlets without having to download, install, and setup multiple different smartphone apps from their App Store – all they launch is a URL. Museums, Artistic installations, and memorials are able to provide high-quality AR-based experiences over the web that reach almost everyone with a smartphone capable of it, and which gracefully degrade when viewed by those which aren’t.

Theatrical performances and Operas provide live translation and captioning services for attendees who need assistance – or who just want a different view of the action – without needing to allocate large portions of already-tight budgets to a new technology with uneven adoption. Whatever else has gone on differently bad or good in our worlds, in that timeline there has been much greater progress with Augmented Reality than in ours.

Obviously, things didn’t quite turn out that way in our timeline. Apple’s ARKit, instead of being used to forge a bridge to the walled-off iOS hardware, has instead been a lever to keep applications and the users who use them on Apple hardware.

It’s completely understandable why Apple would see it as being in their best interests to slow-walk WebXR. Their rival, really their only real competitor in the smartphone and tablet market, is Google and their Android OS, and if their view is zero-sum, then a gain for the web (which de facto is controlled in this context by Chromium and thereby Google, to a large extent) is a loss for Apple.

The Darkest Timeline?

I should take a moment to caveat my words a bit here, both for the sake of legal liability but also for full honesty. A lot of what you’re hearing has been logical speculation, inferred conclusions gleaned from existing facts. The alternate timeline projection of course, is an entirely made-up projection on my part, part aspirational and partly logical anticipation.

I cannot of course speak to Apple’s actual motives, only attempt to outline what can be observed from their actions. Here the evidence of Apple’s action – or rather, lack of it – comes from the WebKit project’s publicly searchable bug tracker. The meta-issue tracking WebXR work shows that only one actual work ticket remains, a ticket to enable cross-platform tests for WebXR on iOS. It’s unclear why or when this will change, but some further digging illuminates some potential explanations that aren’t directly documented.

A search for WebXR in the issue tracker turns up a few issues which weren’t linked to or included in the meta-issue tracking WebXR features. What caught my eye was this feature: Bug 229752 - [WebXR] Support Raw Camera Access in WebXR

This ticket is still in the NEW status despite being open for the better part of a year, indicating that work hasn’t yet commenced or begun planning on it. Pretty much any meaningful AR-related is impacted by this, which is why I predict that WebKit will not get WebXR support in 2022, and probably not even in 2023.

This is all a real bummer, and I’ll bet the organizers (Hi Andy!) are questioning their decision to invite me here to speak.

Hear me out though, please.

There's Light at the End of the Tunnel

Every journey has its’ dark hours and it has its’ light ones. This story, this tale of two timelines doesn’t end here, tonight, with this event. This isn’t even the middle of the story!

This is just the beginning of the story, because the darkness has to have somewhere to go when chased out by the light of dawn – or something suitably inspiring that you’d read on a poster somewhere. There are some amazing tools and frameworks being worked on by a lot of incredibly talented people all working towards the goal of making 3D a first-class citizen of the web, no matter how it’s used or experienced. Babylon.js and the Babylon Native (inc. Babylon React Native) toolset collection provide a rock-solid foundation amidst the roiling sea of constantly evolving standards that allow you to create a cross-platform, web first experience that can even target iOS, although of course publishing to the App Store is still a requirement.

My book, Going the Distance with Babylon.js, isn’t centered around how to build an AR application, it’s focus is on giving readers the knowledge and learning tools needed to build any type of 3D web application experience – and that can include AR with WebXR. Space-Truckers is the game you’ll learn about building in the book, but as an Open Source Project, it stands by itself as a reference repository of in vivo tools and techniques for building, maintaining, and extending quality 3D content.

Thank you to the NYAR organizers (Andy, and co) for inviting me here to speak, it’s an honor to have the chance to share my work with you all and I look forward to answering your questions!

Josh Elster

Josh Elster

A long-time veteran of the software industry ranging between enterprise architecture to creative site design and many points in-between, each challenge only shows how much there is to learn out there!

Read More