Reversing and making it compatible with Apple iOS devices.
ABOUT
Hi all, I am Matteo Pisani a creative, curious and inspired software developer with hacking attitude and strong disposition toward reverse-engineering.I matured several years of experience in IoT, embedded systems development and in bridging the gap between the physical reality and the digital world.
E-mail: matteo.pisani.91@gmail.com | LinkedIn: https://it.linkedin.com/in/matteopisani
ABSTRACT
Google Daydream controller and headset
Mobile virtual reality is growing rapidly. The Google Daydream platform was launched just last month and it suggested that compelling VR experiences might become widely accessible to consumers sooner than expected. Today, solutions like smartphone + headset + bluetooth controller are very appreciated by developers, media and entertainment companies, but… There’s one problem: compatibility. As announced, the Daydream controller binds only with a bunch of Daydream-ready smartphones running Android 7.0 Nougat. Moreover, as reported by Clay Bavor (VP, Virtual Reality at Google), the Google Daydream ”It’s not currently compatible with iOS and won’t be for several years probably.”.
Original post: https://www.quora.com/Is-iPhone-compatible-with-Googles-Daydream-VR
Since I like challenges I decided to hack the Google Daydream controller using code, reverse-engineering skills and some math, to extend the compatibility also on Apple iOS devices: it was a success.
ANALYSIS
Google Daydream controller works via Bluetooth LE (Low Energy) but I wasn’t able to discover it in Bluetooth settings of my iPhone 5, so I used the BlueCap (github.com/troystribling/BlueCapapp) which allows to easily implement Central and Peripheral applications, serialize and deserialize messages exchanged with bluetooth devices and define reusable GATT profile definitions.
I had a look at the data available for each Service: there were known services like Device Information and Battery but I also found something interesting inside an unkown one, the FE55:
As soon I explored inside the first Characteristic with the UUID 00000001–1000–1000–8000–00805f9b34fb and turning On the Notifications, BlueCap started showing BLE packets. Waving the Daydream controller in the air, I could see the incoming data changing in real-time. Same thing happened by touching the pad on top or randomly by pressing the buttons.
According to Bluetooth LE standard each packet should weigh 20 bytes: 7be85b3ff13b48003bf1ffa00000000000000070
The packets anatomy revealed that they were encoded and represented into Hexadecimal notation. Behind the masked data laid the whole status of the controller, including accelerometer, gyroscope, magnetometer, touchpad, buttons and more.
ENVIRONMENT
The first step was to setup a testing environment to facilitate all the debug processes. I decided to start from scratch: I developed a sandbox with Apple XCode (working on a MacBook Pro) and an iOS app (with some Objective-C) that included the CoreBluetooth\CoreBluetooth.h framework (developer.apple.com/reference/corebluetooth). Thanks to this, I could establish and manage communications and data flows over Bluetooth GATT protocol.
After choosing the Service FE55 and requesting notifications for the Characteristic 00000001–1000–1000–8000–00805f9b34fb I was able to get the data output flowing through the console:
Once the data was collected and opportunely decoded I decided to represent it into a 3D view. So, I migrated all the iOS native code to a Hybrid environment wrapping it all into a Cordova plugin: thanks to this process, I was able to save time and perform several optimizations.
The use of JavaScript reduced the overall complexity, speeded up the experiments and allowed me to improve the data visualization embedding also thanks to the amazing A-Frame WebGL framework (aframe.io) inside a HTML5+CSS3 view.
(the entire environment stack)
With the use of Blender, the open-source 3D creation suite, I was able to edit a bulky Google Daydream controller model found on the internet, making it suitable for my purpose. After the editing, I exported it to an A-Frame compliant format (.obj).
In few lines of code, I was able to finish the whole setup and this was the result:
REVERSING
Now for the hardest part: understanding the raw data.Starting from an average knowledge about Hexadecimal to Decimal conversion, I split up the 40 chars in 20 chunks of 2 chars then converted to Binary:
7b e8 5b 3f f1 3b 48 00 3b f1 ff a0 00 00 00 00 00 00 00 70
I just wanted to give it a try, so I tested an online Hexadecimal to Decimal converter and this was the output
Later, I also tried the Decimal to Binary converter.
Bringing everything to JavaScript
The output expected was 160 bits length chain (8 bits * 20 chunks) for each packet:
I got only 94 instead of 160 bits expected so I realized that something was wrong. After going deep into the issue, I found that the hexadecimal values converted in bits sometimes produced results shorter than 8, in other words, were not stuffed in groups of 8: the zeropad to 8 solved all the problems.
Once I added the zeropad method and changed the code in:
this time the expected result was correct.
After a couple of sleepless nights I started to give a shape to this mesmerizing bitchain: a comprehensive knowledge about IMU (Inertial Measurement Unit) and MEMS (Micro Electro-Mechanical Systems) sensors paired with a great patience and good observation skills helped me to figure out the sense of what was happening.
The crucial points were:
* observate all the oscillating bits;* play a little more with offests.
This allowed me to recognize, extract and categorize the values. I reported all of them below:
SENSORS (12 bits for the value + 1 bit for the sign)
TOUCHPAD (8 bits for the value)
BUTTONS (1 bit for the value)
Once I achieved this goal, I tried to manipulate all these data to give a coherent orientation to the 3D Google Daydream controller model, through the A-Frame canvas: unfortunately the output on the screen resulted in a tilting controller with meaningless movements.
Reversing some of the .apk of the Google VR Services (found inside the Google Pixel OS and that allows native communication with Google Daydream controller via BLE), I was able to get my hands on useful information.
TOOLS
Through reverse-engineering of Android Java app using apktool, dex2jar, jd-gui to convert .apk file to .java, it was possible to:
* understand how a particular UI in an App is constructed* reading AndroidManifest.xml, permissions, activities, intents etc in the App * discover native libraries and images used in that App* find obsfucated code (Android SDK, by default, uses ProGuard tool which shrinks, optimizes, and obfuscates the code by removing unused code and renaming classes, fields, and methods with semantically obscure names.
The tools I used:
ApkTool (from http://code.google.com/p/android-apktool/)
to extract AndroidManifest.xml and everything in res folder(layout xml files, images, htmls used on webview etc..), run the following command:
It also extracts the .smali file of all .class files, but which is difficult to read.
Dex2jar (from http://code.google.com/p/dex2jar/)To generate .jar file from .apk file, we need JD-GUI to view the source code from this .jar.Run the following command:
JD-GUI (from http://java.decompiler.free.fr/?q=jdgui) It decompiles the .class files (obsfucated, in case of Android app, but readable original code is obtained in case of other .jar file). i.e., we get .java back from the application.Just Run the jd-gui executables on your OS and after, File->Open to view Java code from .jar or .class file.
In particular, I found interesting information inside:
* com.google.android.vr.home.apk* com.google.vr.vrcore.apk
(magnetometer event)
(accelerometer event)
(gyroscope event)
(compensation methods)
Collecting all my developer-thoughts and making them fit together, I realized that the best solution was to use the AHRS (Attitude Heading Reference Systems) calculation for JavaScript (npmjs.com/package/ahrs).
This calculates the attitude and heading for a device with all of the following sensors: magnetometer, gyroscope and accelerometer. The Madgwick or Mahony algorithms can be used to filter data in real time from these sensors, obtaining a great accuracy.
The getEulerAnglesDegrees method returns an object with the Euler angles (heading/yaw, pitch, roll), in degrees.
The return Object contains:
* heading is from north, going west (about z-axis).* pitch is from vertical, going forward (about y-axis).* roll is from vertical, going right (about x-axis).
Finally, it was possible for me to set the model orientation to the right coordinates
The result was brilliant: as you can see in the YouTube video below that I recorded to show the potential of the entire hack
Google Daydream controller running on iOS (on an iPhone 5)
The responsiveness is extremely fluid, according to the PPS (packets per second) parameter, ~ 60 are enough to cover a VR game or a 3D experience as well.
(Google Daydream controller running on an iPhone)
CONCLUSION
The scenarios that this hack opens are various. Now that the secret sauce has been exposed and the compatibility extended to iOS devices, it is possible to replicate the job to include all the desktop platforms. This would help the developers debugging their own software in a desktop environment, without passing through deploying an app on the smartphone every time. On the Android side, this hack will unleash the whole potential of the Daydream controller as it would be no longer restricted to the OS Nougat 7.0.
In this perspective, it is possible to see the Daydream controller working with older versions of Android OS. On the other hand, binding this controller with open source platforms like Raspberry PI or Arduino, will extend the horizons of makers and creatives. Do you imagine using the Daydream controller to pilot your drone or your RC-car, playing a virtual drumset or maybe, making some sounds with a virtual synth?
Do you wanna see Google Daydream controller running on Linux? Check this out: How I hacked Google Daydream controller (Part II).