How might we prototype a bridge between the reading experiences of paper books and their digital version?
Background
Have you ever found the dilemma of whether to read a book on a print copy or a digital device?
Better digital reading devices aim to provide similar experiences to reading print books while delivering many other traits: quick look-up, easy annotation, and synced progress between devices. However, it probably could never replicate the tangible touch of flipping through pages on a print copy, which most readers still prefer.
Ideation
What if we can have the best of both worlds by integrating digital annotation to physical touch?
Existing "Bridges"
We are not the first to ask this question. Several existing products are designed to digitize print books and physical annotation with different approaches.
A progressive OCR text scanner for print books, featuring vocabulary look-up and excerpt extraction.
I personally own a Hanvon E-Pen, mainly to look up words in my non-native languages. Using it has been quite pleasing, but there are two significant shortcomings:
A real-time text scanner for handwriting featuring note synchronization to other digital devices.
Although Livescribe integrates better with other devices, it deviates much from the use case of reading but more on the notetaking side.
These defects of existing "bridges" urged us think out of the box:
The limited options for interfaces and lack of connectivity are hindering the user experience in these attempts.
Imagining AR
Our team happened to have our hands on a Microsoft Hololens. What if we utilize augmented reality to smooth out any bumps on the "bridge" in existing solutions.
We settled on two fundamental interactions to build a seamless hybrid reading experience:
AR Quick Look-Up
By overlaying information on top of a paper book, we could mitigate the hassle of going back and forth between devices – just wear Hololens while reading. To look up an obscure word, just tap your finger on it!
Cross-Platform Annotation Syncing
Handwritten annotations will be parsed and digitized to be accessible across digital reading devices, such as Kindle.
Concept Video
Introducing Holo Reader, an AR application that bridges your digital and physical reading experience.
Implementation
These are technically challenging ideas, and we managed to implement a part of them during a short hackathon at NYU.
The client side of our prototype is built with Unity with Mixed Reality Toolkit and Vuforia. On top of that, we developed a back-end server with Flask using Python to handle data transfer and image processing.
Using Vuforia, the app constantly looks for user's fingertips and the word on top of it. After sending back to our server, the app receives the word definition to overlay on top of book.
For further details, check out our
→ Github Repo.
