1.什么是人脸姿态估计问题

人脸姿态估计主要是获得脸部朝向的角度信息。一般可以用旋转矩阵、旋转向量、四元数或欧拉角表示(这四个量也可以互相转换)。一般而言,欧拉角可读性更好一些,使用更为广泛。本文获得的人脸姿态信息用三个欧拉角(pitch,yaw,roll)表示。

欧拉角动图注解

pitch:俯仰角,表示物体绕x轴旋转

yaw:偏航角,表示物体绕y轴旋转

roll:翻滚角,表示物体绕z轴旋转

2.计算步骤

1)首先定义一个具有n个关键点的3D脸部模型,n可以根据自己对准确度的容忍程度进行定义,以下示例定义6个关键点的3D脸部模型(左眼角,右眼角,鼻尖,左嘴角,右嘴角,下颌);

2)采用人脸检测以及面部关键点检测得到上述3D脸部对应的2D人脸关键点;

3)采用Opencv的solvePnP函数解出旋转向量;

4)将旋转向量转换为欧拉角;

3.定义6关键点的3D Model

C++

// 3D model points.
    std::vector<cv::Point3d> model_points;
    model_points.push_back(cv::Point3d(0.0f, 0.0f, 0.0f));               // Nose tip
    model_points.push_back(cv::Point3d(0.0f, -330.0f, -65.0f));          // Chin
    model_points.push_back(cv::Point3d(-225.0f, 170.0f, -135.0f));       // Left eye left corner
    model_points.push_back(cv::Point3d(225.0f, 170.0f, -135.0f));        // Right eye right corner
    model_points.push_back(cv::Point3d(-150.0f, -150.0f, -125.0f));      // Left Mouth corner
    model_points.push_back(cv::Point3d(150.0f, -150.0f, -125.0f));       // Right mouth corner


python

# 3D model points.
model_points = np.array([
                            (0.0, 0.0, 0.0),             # Nose tip
                            (0.0, -330.0, -65.0),        # Chin
                            (-225.0, 170.0, -135.0),     # Left eye left corner
                            (225.0, 170.0, -135.0),      # Right eye right corne
                            (-150.0, -150.0, -125.0),    # Left Mouth corner
                            (150.0, -150.0, -125.0)      # Right mouth corner
                         
                        ])

68点的3D model下载链接:链接:https://pan.baidu.com/s/162szx0PWl6dvnEHbDTDPAw 密码:ip7v

4.关键点检测

利用相关算法进行人脸关键点检测,一般常见68个关键点检测模型,其具体顺序如下所示,而6个关键点对应的索引id分别为:

下巴:8

鼻尖:30

左眼角:36

右眼角:45

左嘴角:48

右嘴角:54

C++

// 2D image points. If you change the image, you need to change vector
    std::vector<cv::Point2d> image_points;
    image_points.push_back( cv::Point2d(359, 391) );    // Nose tip
    image_points.push_back( cv::Point2d(399, 561) );    // Chin
    image_points.push_back( cv::Point2d(337, 297) );     // Left eye left corner
    image_points.push_back( cv::Point2d(513, 301) );    // Right eye right corner
    image_points.push_back( cv::Point2d(345, 465) );    // Left Mouth corner
    image_points.push_back( cv::Point2d(453, 469) );    // Right mouth corner


python

#2D image points. If you change the image, you need to change vector
image_points = np.array([
                            (359, 391),     # Nose tip
                            (399, 561),     # Chin
                            (337, 297),     # Left eye left corner
                            (513, 301),     # Right eye right corne
                            (345, 465),     # Left Mouth corner
                            (453, 469)      # Right mouth corner
                        ], dtype="double")

5.用Opencv的solvePnP函数解出旋转向量

OpenCV中solvePnP 和 solvePnPRansac都可以用来估计Pose。第二个solvePnPRansac利用随机抽样一致算法(Random sample consensus,RANSAC)的思想,虽然计算出的姿态更加精确,但速度慢、没法实现实时系统,所以我们这里只关注第一个solvePnP函数,具体的参数可以参见Opencv文档。

solvePnP implements several algorithms for pose estimation which can be selected using the parameter 
flag. By default it uses the  flag SOLVEPNP_ITERATIVE which is essentially the DLT solution followed by 
Levenberg-Marquardt optimization. SOLVEPNP_P3P uses  only 3 points for calculating the pose and it 
should be used only when using solvePnPRansac.

C++: bool solvePnP(InputArray objectPoints, InputArray imagePoints, InputArray cameraMatrix, InputArray 
distCoeffs, OutputArray rvec, OutputArray tvec, bool useExtrinsicGuess=false, int 
flags=SOLVEPNP_ITERATIVE )

确定pose也就是确定从3D model到图片中人脸的仿射变换矩阵,它包含旋转和平移的信息。solvePnP函数输出结果包括旋转向量(roatation vector)和平移向量(translation vector)。这里我们只关心旋转信息,所以主要将对 roatation vector进行操作。 
在调用solvePnP函数前需要初始化cameraMatrix,也就是相机内参,并调用solvePnP函数:
 

c++
 
// Camera internals
    double focal_length = im.cols; // Approximate focal length.
    cv::Point2d center = cv::Point2d(im.cols / 2, im.rows / 2);
    cv::Mat camera_matrix = (cv::Mat_<double>(3, 3) << focal_length, 0, center.x, 0, focal_length, center.y, 0, 0, 1);
    cv::Mat dist_coeffs = cv::Mat::zeros(4, 1, cv::DataType<double>::type); // Assuming no lens distortion

    cv::Mat rotation_vector; // Rotation in axis-angle form
    cv::Mat translation_vector;

    // Solve for pose
    cv::solvePnP(model_points, landmarks, camera_matrix, dist_coeffs, rotation_vector, translation_vector);


python

# Camera internals
 
focal_length = size[1]
center = (size[1]/2, size[0]/2)
camera_matrix = np.array(
                         [[focal_length, 0, center[0]],
                         [0, focal_length, center[1]],
                         [0, 0, 1]], dtype = "double"
                         )
 
print "Camera Matrix :\n {0}".format(camera_matrix)
 
dist_coeffs = np.zeros((4,1)) # Assuming no lens distortion
(success, rotation_vector, translation_vector) = cv2.solvePnP(model_points, image_points, camera_matrix, dist_coeffs, flags=cv2.CV_ITERATIVE)
 
print "Rotation Vector:\n {0}".format(rotation_vector)
print "Translation Vector:\n {0}".format(translation_vector)
 

6.将旋转向量转换为欧拉角

1)旋转向量—>旋转矩阵—>欧拉角

旋转向量转旋转矩阵
theta = np.linalg.norm(rvec)
r = rvec / theta
R_ = np.array([[0, -r[2][0], r[1][0]],
               [r[2][0], 0, -r[0][0]],
               [-r[1][0], r[0][0], 0]])
R = np.cos(theta) * np.eye(3) + (1 - np.cos(theta)) * r * r.T + np.sin(theta) * R_
print('旋转矩阵')
print(R)
旋转矩阵转欧拉角
def isRotationMatrix(R):
    Rt = np.transpose(R)   #旋转矩阵R的转置
    shouldBeIdentity = np.dot(Rt, R)   #R的转置矩阵乘以R
    I = np.identity(3, dtype=R.dtype)           # 3阶单位矩阵
    n = np.linalg.norm(I - shouldBeIdentity)   #np.linalg.norm默认求二范数
    return n < 1e-6                            # 目的是判断矩阵R是否正交矩阵(旋转矩阵按道理须为正交矩阵,如此其返回值理论为0)
 
 
def rotationMatrixToAngles(R):
    assert (isRotationMatrix(R))   #判断是否是旋转矩阵(用到正交矩阵特性)
 
    sy = math.sqrt(R[0, 0] * R[0, 0] + R[1, 0] * R[1, 0])  #矩阵元素下标都从0开始(对应公式中是sqrt(r11*r11+r21*r21)),sy=sqrt(cosβ*cosβ)
 
    singular = sy < 1e-6   # 判断β是否为正负90°
 
    if not singular:   #β不是正负90°
        x = math.atan2(R[2, 1], R[2, 2])
        y = math.atan2(-R[2, 0], sy)
        z = math.atan2(R[1, 0], R[0, 0])
    else:              #β是正负90°
        x = math.atan2(-R[1, 2], R[1, 1])
        y = math.atan2(-R[2, 0], sy)   #当z=0时,此公式也OK,上面图片中的公式也是OK的
        z = 0
    
    x = x*180.0/3.141592653589793
    y = y*180.0/3.141592653589793
    z = z*180.0/3.141592653589793

    return np.array([x, y, z])

2)旋转向量—>四元数—>欧拉角

# 从旋转向量转换为欧拉角
def get_euler_angle(rotation_vector):
    # calculate rotation angles
    theta = cv2.norm(rotation_vector, cv2.NORM_L2)
    
    # transformed to quaterniond
    w = math.cos(theta / 2)
    x = math.sin(theta / 2)*rotation_vector[0][0] / theta
    y = math.sin(theta / 2)*rotation_vector[1][0] / theta
    z = math.sin(theta / 2)*rotation_vector[2][0] / theta
    
    ysqr = y * y
    # pitch (x-axis rotation)
    t0 = 2.0 * (w * x + y * z)
    t1 = 1.0 - 2.0 * (x * x + ysqr)
    print('t0:{}, t1:{}'.format(t0, t1))
    pitch = math.atan2(t0, t1)
    
    # yaw (y-axis rotation)
    t2 = 2.0 * (w * y - z * x)
    if t2 > 1.0:
        t2 = 1.0
    if t2 < -1.0:
        t2 = -1.0
    yaw = math.asin(t2)
    
    # roll (z-axis rotation)
    t3 = 2.0 * (w * z + x * y)
    t4 = 1.0 - 2.0 * (ysqr + z * z)
    roll = math.atan2(t3, t4)
    
    print('pitch:{}, yaw:{}, roll:{}'.format(pitch, yaw, roll))
    
	# 单位转换:将弧度转换为度
    Y = int((pitch/math.pi)*180)
    X = int((yaw/math.pi)*180)
    Z = int((roll/math.pi)*180)
    
    return 0, Y, X, Z

PS:

人脸旋转的极限角度:

Pitch -60.4~69.6
Yaw   -79.8~75.3
Roll  -40.9~63.3

误差范围:

Pitch: 5.10
Yaw:   4.20
Roll:  2.40

参考博客链接:https://blog.csdn.net/c20081052/article/details/89479970

                         https://github.com/yuenshome/yuenshome.github.io/issues/9

                         https://www.learnopencv.com/head-pose-estimation-using-opencv-and-dlib/

                         https://blog.csdn.net/u013512448/article/details/77804161

                         https://www.jianshu.com/p/5e130c04a602

参考github链接:https://github.com/guozhongluo/head-pose-estimation-and-face-landmark

                            https://github.com/yinguobing/head-pose-estimation

                            https://github.com/lsy17096535/face-landmark

比较有用的github链接:https://github.com/TadasBaltrusaitis/OpenFace

                                       https://github.com/genshengye/IntraFace

Logo

开放原子开发者工作坊旨在鼓励更多人参与开源活动,与志同道合的开发者们相互交流开发经验、分享开发心得、获取前沿技术趋势。工作坊有多种形式的开发者活动,如meetup、训练营等,主打技术交流,干货满满,真诚地邀请各位开发者共同参与!

更多推荐