01-26 14:51 常州大学 C++ 发布于四川

关注

神经网络基础——矩阵求导运算

本篇文章不涉及过多理论推导。关于本篇文章，在实践中会用即可。矩阵求导有以下9种可能情况：

自变量/因变量	标量 $y$	向量 $\mathbf y$	矩阵 $Y$
标量 $x$	$\frac{dy}{dx}$	$\frac{d\mathbf y}{dx}$	$\frac{dY}{dx}$
向量 $\mathbf x$	$\frac{dy}{d\mathbf x}$	$\frac{d\mathbf y}{d\mathbf x}$	$\frac{dY}{d\mathbf x}$
矩阵 $X$	$\frac{dy}{dX}$	$\frac{d\mathbf y}{dX}$	$\frac{dY}{dX}$

以下分别讨论。

一阶导数

标量对标量求导

这里是最简单的情况，本篇默认读者已掌握，不再赘述。

向量对标量求导

行向量对标量求导

设行向量 $\mathbf y=(y_1,y_2,\cdots,y_n)$ ，则有

\frac{\partial\mathbf y}{\partial x}=\begin{bmatrix} \frac{\partial y_1}{\partial x} & \frac{\partial y_2}{\partial x} & \cdots & \frac{\partial y_n}{\partial x} \end{bmatrix}

列向量对标量求导

设列向量 $\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix}$ ，则有

\frac{\partial\mathbf y}{\partial x}=\begin{bmatrix} \frac{\partial y_1}{\partial x}\\ \frac{\partial y_2}{\partial x}\\ \vdots\\ \frac{\partial y_n}{\partial x} \end{bmatrix}

矩阵对标量求导

设 $Y=\begin{bmatrix} y_{11} & y_{12} & \cdots & y_{1n}\\ y_{21} & y_{22} & \cdots & y_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ y_{m1} & y_{m2} & \cdots & y_{mn} \end{bmatrix}$ ，则有

\frac{\partial Y}{\partial x}=\begin{bmatrix} \frac{\partial y_{11}}{\partial x} & \frac{\partial y_{12}}{\partial x} & \cdots & \frac{\partial y_{1n}}{\partial x}\\ \frac{\partial y_{21}}{\partial x} & \frac{\partial y_{22}}{\partial x} & \cdots & \frac{\partial y_{2n}}{\partial x}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_{m1}}{\partial x} & \frac{\partial y_{m2}}{\partial x} & \cdots & \frac{\partial y_{mn}}{\partial x} \end{bmatrix}

标量对向量求导

标量对行向量求导

设 $\mathbf x=(x_1,x_2,\cdots,x_m)$ ，则有

\frac{\partial y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y}{\partial x_1} & \frac{\partial y}{\partial x_2} & \cdots & \frac{\partial y}{\partial x_m} \end{bmatrix}

标量对列向量求导

设 $\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_m \end{bmatrix}$ ，则有

\frac{\partial y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2}\\ \vdots\\ \frac{\partial y}{\partial x_m} \end{bmatrix}

标量对矩阵求导

设 $X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1n}\\ x_{21} & x_{22} & \cdots & x_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ x_{m1} & x_{m2} & \cdots & x_{mn} \end{bmatrix}$ ，则有

\frac{\partial y}{\partial X}=\begin{bmatrix} \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{12}} & \cdots & \frac{\partial y}{\partial x_{1n}}\\ \frac{\partial y}{\partial x_{21}} & \frac{\partial y}{\partial x_{22}} & \cdots & \frac{\partial y}{\partial x_{2n}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y}{\partial x_{m1}} & \frac{\partial y}{\partial x_{m2}} & \cdots & \frac{\partial y}{\partial x_{mn}} \end{bmatrix}

向量对向量求导

列向量对行向量求导(雅可比矩阵)

设 $\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix},\mathbf x=(x_1,x_2,\cdots,x_m)$ ，则有

\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_m}\\ \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_2}{\partial x_m}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_n}{\partial x_1} & \frac{\partial y_n}{\partial x_2} & \cdots & \frac{\partial y_n}{\partial x_m} \end{bmatrix}

上式也称为雅可比矩阵

行向量对列向量求导

设 $\mathbf y=(y_1,y_2,\cdots,y_n),\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_m \end{bmatrix}$ ，则有

\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_n}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_n}{\partial x_2}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_1}{\partial x_m} & \frac{\partial y_2}{\partial x_m} & \cdots & \frac{\partial y_n}{\partial x_m} \end{bmatrix}

行向量对行向量求导

设 $\mathbf y=(y_1,y_2,\cdots,y_n),\mathbf x=(x_1,x_2,\cdots,x_m)$ ，则有

\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial\mathbf{y}}{\partial x_1} & \frac{\partial\mathbf{y}}{\partial x_2} & \cdots & \frac{\partial\mathbf{y}}{\partial x_m} \end{bmatrix}

其中 $\frac{\partial\mathbf{y}}{x_i}$ 的定义见行向量对标量求导

列向量对列向量求导

设 $\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix},\mathbf x=\begin{bmatrix} x_1\\ x_2\\ \vdots\\ x_m \end{bmatrix}$ ，则有

\frac{\partial\mathbf y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial\mathbf{y}}{\partial x_1}\\ \frac{\partial\mathbf{y}}{\partial x_2}\\ \vdots\\ \frac{\partial\mathbf{y}}{\partial x_m} \end{bmatrix}

其中 $\frac{\partial\mathbf{y}}{x_i}$ 的定义见列向量对标量求导

矩阵对向量求导

矩阵对行向量求导

设 $Y=\begin{bmatrix} y_{11} & y_{12} & \cdots & y_{1n}\\ y_{21} & y_{22} & \cdots & y_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ y_{m1} & y_{m2} & \cdots & y_{mn} \end{bmatrix},\mathbf x=(x_1,x_2,\cdots,x_c)$ ，则有

\frac{\partial Y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial Y}{\partial x_1} & \frac{\partial Y}{\partial x_2} & \cdots & \frac{\partial Y}{\partial x_c} \end{bmatrix}

其中 $\frac{\partial Y}{x_i}$ 的定义见矩阵对标量求导

矩阵对列向量求导

\frac{\partial Y}{\partial\mathbf x}=\begin{bmatrix} \frac{\partial Y}{\partial x_1}\\ \frac{\partial Y}{\partial x_2}\\ \vdots\\ \frac{\partial Y}{\partial x_r} \end{bmatrix}

其中 $\frac{\partial Y}{x_i}$ 的定义见矩阵对标量求导

向量对矩阵求导

行向量对矩阵求导

设 $\mathbf y=(y_1,y_2,\cdots,y_n),X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1c}\\ x_{21} & x_{22} & \cdots & x_{2c}\\ \vdots & \vdots & \ddots & \vdots\\ x_{r1} & x_{r2} & \cdots & x_{rc} \end{bmatrix}$ ，则有

\frac{\partial\mathbf y}{\partial X}=\begin{bmatrix} \frac{\partial\mathbf y}{\partial x_{11}} & \frac{\partial\mathbf y}{\partial x_{12}} & \cdots & \frac{\partial\mathbf y}{\partial x_{1c}}\\ \frac{\partial\mathbf y}{\partial x_{21}} & \frac{\partial\mathbf y}{\partial x_{22}} & \cdots & \frac{\partial\mathbf y}{\partial x_{2c}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial\mathbf y}{\partial x_{r1}} & \frac{\partial\mathbf y}{\partial x_{r2}} & \cdots & \frac{\partial\mathbf y}{\partial x_{rc}} \end{bmatrix}

其中 $\frac{\partial\mathbf y}{x_{ij}}$ 的定义见行向量对标量求导

列向量对矩阵求导

设 $\mathbf y=\begin{bmatrix} y_1\\ y_2\\ \vdots\\ y_n \end{bmatrix},X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1c}\\ x_{21} & x_{22} & \cdots & x_{2c}\\ \vdots & \vdots & \ddots & \vdots\\ x_{r1} & x_{r2} & \cdots & x_{rc} \end{bmatrix}$ ，则有

\frac{\partial\mathbf y}{\partial X}=\begin{bmatrix} \frac{\partial\mathbf y}{\partial x_{11}} & \frac{\partial\mathbf y}{\partial x_{12}} & \cdots & \frac{\partial\mathbf y}{\partial x_{1c}}\\ \frac{\partial\mathbf y}{\partial x_{21}} & \frac{\partial\mathbf y}{\partial x_{22}} & \cdots & \frac{\partial\mathbf y}{\partial x_{2c}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial\mathbf y}{\partial x_{r1}} & \frac{\partial\mathbf y}{\partial x_{r2}} & \cdots & \frac{\partial\mathbf y}{\partial x_{rc}} \end{bmatrix}

其中 $\frac{\partial\mathbf y}{x_{ij}}$ 的定义见列向量对标量求导

矩阵对矩阵求导

设 $Y=\begin{bmatrix} y_{11} y_{12} \cdots y_{1n}\\ y_{21} y_{22} \cdots y_{2n}\\ \vdots \vdots \ddots \vdots\\ y_{m1} y_{m2} \cdots y_{mn} \end{bmatrix},X=\begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1c}\\ x_{21} & x_{22} & \cdots & x_{2c}\\ \vdots & \vdots & \ddots & \vdots\\ x_{r1} & x_{r2} & \cdots & x_{rc} \end{bmatrix}$ ，则有