SentiMask SDK: creating Blender project


Hello, in this video I will show you how to
create a Blender project for SentiMask SDK. We will be using shape keys in Blender also
known as blend shapes or morph targets in some other modeling software. And we will use AddOSC plugin for Blender
as well. This is a third party plugin for Blender so
you have to install it separately. If you have not done this yet please, follow
the link to one of our videos that has a step by step instructions on how to install and
enable AddOSC plugin for Blender. It only takes a couple of minutes to set everything
up. Although, I am using Blender 3D, the same
principles should apply to other software as well. First of all, I want to say that, this video
is not about a character modelling or Blender interface. You can find a lots of tutorials online on
these topics, I will also put a couple of links in the description of this video. So let’s get started. I will be using a simple face models with
expressions. You can download these models if you want. If you would like to use your own character
you must have a separate model for every expression, that is neutral expression, right eye closed,
left eye closed, right eyebrow up, left eyebrow up, jaw to the right, jaw to the left, mouth
right corner up, mouth left corner up, mouth open and lips forward, it’s like if you
want to give someone a kiss. Now let’s start the Blender. You will see Blender‘s default scene with
a cube, camera and a light source. Object selection in Blender is done with the
right mouse button. To select all visible objects just press ’A‘
on the keyboard. To unselect all press ’A‘ again. So select all objects by pressing ’A‘
and delete them pressing ’DELETE‘ or ’X‘ followed by ’ENTER ‘ to complete the operation. Now let’s import our models with an expressions. Go to File, Import, Wavefront (.obj). On the left bottom side of the Blender you
will find Import OBJ options. Make sure that Keep Vertex Order is selected. It is important so all our models must have
the same number and order of the vertices, otherwise the shape keys will not work correctly. And keep that in mind then preparing your
own models. Now navigate to our models and import 000_expression_right_eye_closed.obj. I on purpose added indices from 000 to 010
at the beginning of the file names. They will be useful later then we will map
OSC commands to our model. Now import the rest of our models. Then finished you will notice that our neutral
expression has index of 010. Let’s hide it for now, by pressing the eye
icon next to it. If your keyboard has a numpad, press ’1‘
and then ’5‘, which will bring the viewport to front orthographic mode. Otherwise use View menu at the bottom left
side of the Blender. Now with the right mouse button select any
of the face models and press ’A‘ once or twice so everything gets selected. Now with left mouse button press and drag
the tip of the red arrow to move all the selected models along the x axis. Now unhide our neutral expression model and
while holding ’SHIFT‘ key select it with a right mouse button. The neutral expression will be added to the
selection. It is important that our neutral expression
is selected last. Now let’s go to Object data tab, the icon
with the three points connected in to a triangle. If it is not visible just move the panels
a little bit. Under Shape Keys press little dark arrow to
bring the context menu and select Join as Shapes. The reason why we selected our neutral expression
last is that the last selection becomes as Basis when we join shapes together and all
the rest of the shapes are the part of that basis. Notice how shapes are arranged by the indices
from 0 to 9. Now again let’s hide neutral expression
and select everything else by pressing ’A‘ once or twice. Press ’DELETE‘ or ’X‘ and ’ENTER‘
to confirm the operation. Now unhide our model. In the bottom of the Blender let’s change
display mode from solid to texture. Now let’s check our expressions. Select an expression, let’s say mouth open. You can change the value of the expression
by dragging slider at the bottom of Shape Keys or click with a left mouse button and
entering a value from the keyboard. Also you can enter a value next to the shape
itself. The same applies to the rest of the shapes. The default values of Shape Keys in Blender
goes from 0 to 1, we will change that from -1 to 1 for all of our shapes. Simply select the shape and change the value
of the minimum in the Range settings from 0 to -1. Once we have done that, we need to add values
of the Shape Keys to Blender’s keying set. To do that, click with the right mouse button
on the value next to the shape name of our shape with index 0 and select Add to Keying
Set. Let’s repeat that for all of our shapes
taking the order of indices in to account. It will make things easier later. Now let’s go to Object tab and under Rotation
in one of the fields click the right mouse button and select Add All to Keying Set. This option will add all the values of rotation
to the keying set at once instead of adding every value one by one. Now go to AddOSC plugin tab on the left side
of the Blender. Press Import Keying Set button. This will import all our created keying sets
in the order that we have created them. Now I will open Visual Studio solution of
our sample, so you can understand what is going on. The solution itself can be found in Samples,
SentiMask folder of our SDK. We need to open C sample not CPP. At the top of main file you will see HOST_ADDRESS
and HOST_PORT defined. This means that our sample will send OSC commands
to the localhost through port 9001. Now if you scroll down all the way to the
line around 480. You will see the line that formats a string
according the OSC protocol. This particular line means that face pose
angles pitch, yaw and roll will be sent as three floating point values and these values
can be accessed using a path or id named “/pose”. A few lines below you will see another message,
dealing with the expressions. The id or path for expressions is “/expressions”
and this message has 10 floating point values that controls every expression. Notice the order of the expressions, they
match the order of every file name for relevant expression model and the order of our created
shape keys in blender. Finally these two messages are packed into
a single bundle message and sent to the Blender. Now let’s get back to the Blender. In AddOSC Settings you will see that Blender
is listening for the commands using localhost and input port is set to 9001. These are the same settings as in a source
code of our sample. As for Imported Keys, we need to change the
address from default value to “/expressions” for every key controlling certain expression. And we do the same with the pose address by
giving the address “/pose”. The AddOSC plugin decodes received values
into array of numbers so we need to set indices for every expression. That is why I tried to keep the names and
the order of keying set as close as possible to the same order that is sent from the sample
program. So now I can simply change indices just increasing
by one for every expression from 0 to 9. As for neutral shape its index is 10, but
we do not control this shape in any way it is just a basis for all other expression. In general case you can give any name to your
models, but then you will have to be careful and try not to mix the order of shape and
the order of value that will control corresponding shape. Finally let’s change the indices for our
pose values as well. This is a rotation about x, y and z axis. Now you can press start button at the top
of AddOSC plugin, the Blender will start listening for the OSC commands. Now let’s start our SentiMask OSC Sample. You will have to switch to the top view in
Blender. You can do it by pressing ’7‘ on numpad
or go to view and select top from the menu. I hope this video was heplpfull and thank
you for watching it.

1 thought on “SentiMask SDK: creating Blender project

Leave a Reply

Your email address will not be published. Required fields are marked *