Learning Three.js

or WebGL for Dummies

Hatsune Miku Dancing in Augmented Reality

This demo shows Hatsune Miku dancing in augmented reality within your browser! This is great and surprisingly easy to do. All that running on your browser, based on web standards. I did it to show it was possible to do AR within the browser. I wanted to share the code to see what you guys can do with it.

Best of all, it even runs on mobile phones that support WebGL and WebRTC. The screenshot on the right is made on a nexus. So we can do augmented reality within the browser on mobile

We can do augmented reality within the browser on currently deployed mobile phones today! There is nothing to wait for.

How is it coded ?

It is mainly a link of 2 parts. First webar extensions for three.js. They bundle what is needed to handle augmented reality with three.js, from the webcam setup to the marker localisation. Second part is the mmd loader which loads the model. It loads the Hatsune Miku model and its animations.

Once you got those 2 parts, things are simple :) you just have to display your model where your marker is. To know the tech details, you will have to dig in the code. I will likely talk more about AR tho.

Now Let’s Do a Bit of History

I did this to show it was possible to do AR within the browser and to share the code to see what you guys can do with it. It was for a AR oriented hackathon in Dublin. It recently landed a new job at Daqri. Part of my job is developer relations, so I get to help people doing AR. I love it :)

All the code to handle Hatsune Miku is from Takahiro Aoyagi. He did a fork on three.js to load mmd formats, the format used to store Hatsune Miku. It may be included in three.js soon. Here is his example for three.js It all started with his mmd viewer in pure webgl. The original contains a LOT of neat features like inverse kinetic, physics for the hair, a toon shader to make the color more cartoonish, a edge shader to enhance the outline of the model like a drawing. He is doing excellent work. Check him out on twitter as @superhoge.

Let’s have a word about Hatsune Miku herself. It is such a nice story. She is a Japanese star but she is purely virtual. She is a humanoid persona which appears as a drawing or as a hologram. There is no human behind, even for the voice! When she sings, what you hear is a voice synthesizer from crypton All that is crazy but true :)

She even does concerts where thousand of fans are go to see her. Actual human fans I mean, not virtual ones. As you can see, Hatsune Miku is no stranger to augmented reality. I love her!

How To Run The Demo ?

You may want to run this demo obviously. Here is how to do it. First you put the demo url in your browser. It will read your webcam using getUserMedia. When it asks for permission, allow it :) Then you need to put this AR marker in front of the camera. You can print it and point the camera toward the paper or you can load the marker web page and put the phone in front of the camera. And now you can see Hatsune Mike Dancing in Augmented Reality.

Now let’s look at it in action in this little screencast.


If you want to experiment with augmented reality and three.js, checkout threex.webar. It is so cool to make augmented reality on the web! I hope you have as much fun as I had doing it :) I can’t wait to see what augmented reality will do on mobile’s browsers.

That’s all folks. Have fun.