Facial Mocap (Motion Capture) with Apple’s iphone

Facial Expression Animation with Blendshapes

Facial motion capture with an iphone is very popular these days. With the advancement of Apple’s TrueDepth cameras, it makes facial mocap very affordable. Capturing all micro-expression on the face and being able to remap that onto a 3D character for a very lively render.

The character on the right was done with blendshapes that came with the character when I generated it from Character Creator (more about this software down below).

Microsoft Kinect v2

How to go about Facial Mocap without breaking the bank?

Most facial mocap software available for home-studio level grade quality may involve one of Apple’s iphone device and Apple’s TrueDepth sensors. The TrueDepth sensor is one of Apple’s proud flagship novelties that has proven to be extremely powerful for motion capture for the face. It’s a camera attached to the iphone that has a lot of fancy sensors shooting out infrared rays that scans the face in realtime. The main application Apple intended for the TrueDepth sensor was to scan a person’s face and use that as a way to unlock your phone. So you can imagine how accurate this device is! I’m not sure how many people use it as a security device, but I know that many motion capture software applications use this device as a way to provide a budget friendly facial motion capture studio.

Blender

Facial Rig generated by FaceIt Blender addon

There’s a simple Blender addon called FaceIt on the blendermarket.com that relies on an iphone app titled, Face Cap App and the iphone’s TrueDepth Camera to capture the expressions on your face. What would take a few minutes of recording time using this addon would save you hours in rigging! The best thing is that this is extremely budget friendly.

Cost Breakdown

The addon costs $78 USD (indie license), but you own it for life. The iphone device, which many already have, but I’ll list it as a cost because a lot of people own one. The iphone device will put you back around $700 USD. The Face Cap app which is required if you want to do realtime facial motion capture with this addon will cost you around $60 USD in addon prices inside the app. This totals an estimated $838 USD (before tax), which I admit is still quite a bit of money. But if you think about how far motion capture has gone, this would have cost a lot more in the past. And if you already own an iphone device with the TrueDepth camera then that’s really just $138 USD, but that’s just positive thinking.

This is probably the most affordable option to get a very decent motion capture experience.

Rokoko

Rokoko is famous for making full body motion capture suits that is professional quality for a home-studio price. And now Rokoko is adding facial motion capture to its whole suite of body motion capture suits. Rokoko is also relying on the Apple’s iphone TrueDepth cameras as its main source of input to capture your facial expressions and offers its own proprietary software to translate that onto 3D characters for an easy user experience.

Cost Breakdown

The whole setup is very impressive, but comes with a price tag of $695 USD and that doesn’t include the iphone. So you’ll have to get that on your own and will set you back an estimated total cost of $1395 USD including the iphone camera you’ll need to buy.

Reallusion and Faceware

Faceware is the one I am most interested in, but sadly it’s also the most expensive option to facial motion capture on the consumer level. Faceware uses AI and neural networks and a webcam to capture the facial expressions. The results from their demonstration videos looks VERY impressive. Just look at the demo video from 9 years ago from the time of writing this blog post. The unfortunate thing is that this is also the most expensive solution out of the ones I mentioned in this blog post.

Cost Breakdown

The whole iClone plus Faceware addon, Motion Live addon, and 3dXExchange addon will put you back $2500 USD. Details on each addon scroll below.

Faceware Addon (for iClone)

The part that interests me the most is the fact they use a regular webcam and nothing fancy, which makes this possible for a director to make a film with remote actors and actresses. Imagine directing a film remotely where all your cast are sitting at home and you yell out “ACTION”. This can bring the filming industry to a whole new level!

Reallusion has a special Faceware face profile that you can buy as an addon for Reallusion’s iClone software. The special Faceware addon allows you to stream the data collected and analyzed by Faceware and feed in all that data into iClone for ease of use.

Just to clearify, Faceware by itself is a web service, so that means you’ll have to pay a monthly fee to use its independent services, which is the Faceware software by itself. The Faceware addon for Reallusion’s iClone is different and is a one time payment that offers part of the Faceware software to be used within the iClone application.

You take a video recording of a real live person with the head clearly visible and input it into the iClone with Faceware addon and it runs through Faceware’s AI and machine learning software as part of the addon. It outputs a bunch of analyzed data and the data can then be fed into Reallusion’s iClone. But it’s not limited to just iClone, Faceware offers other clients like Unity and Unreal Engine. I’m not sure if the recording needs to be done live in person or can it be a prerecorded video clip that you can feed into the iClone software anytime.

Motion Live Addon (for iClone)

This addon is mandatory requirement for the Faceware addon to work with iClone. iClone itself is an animation software developed by Reallusion and is capable of doing manual animation, rigging, and the basics. The Faceware Addon gives you a way to animate the character easily with a webcam so you don’t have to sit there and keyframe each expression. You’ll need the Motion Live addon to allow iClone to be able to receive all the data from the Faceware addon, otherwise the Faceware addon isn’t much use for iClone.

Cost Breakdown

The Motion Live addon for iClone costs $200 USD. So if you’re planning on getting the Faceware addon, you MUST get the Motion Live addon as well for everything to function properly.

3dX Exchange Addon (for iClone)

This is an addon that allows iClone to export its animated assets to a third party program. Surprisingly this isn’t free. If you’re like me and prefer to do your rendering in Blender or any other rendering software like Redshift and RenderMan, you’ll need to get 3dX Exchange in order to export the animations and import it into your desired software. Otherwise you’re stuck with iClone and you’ll have to use Iray rendering engine that is built-in iClone.

Cost Breakdown

The 3dXExchange will cost you $500 USD.