There's something that's always been deeply confusing to me about comparing the Jacobian and the Hessian because their nature is very different.
The Hessian shouldn't have been called a matrix.
The Jacobian describes all the first order derivatives of a vector valued function (of multiple inputs), while the Hessian is all the second order derivatives of a scalar valued output function (of multiple inputs). Why doesn't the number of dimensions of the array increase by one as the derivation order increases? It does! The object that fully describes second order derivation of a vector valued function of multiple inputs is actually a 3 dimensionnal tensor. One dimension for the original vector valued output, and one for each derivation order. Mathematicians are afraid of tensors of more than 2 dimensions for some reason and want everything to be a matrix.
In other words, given a function R^n -> R^m:
Order 0: Output value: 1d array of shape (m) (a vector)
Order 1: First order derivative: 2d array of shape (m, n) (Jacobian matrix)
Order 2: Second order derivative: 3d array of shape (m, n, n) (array of Hessian matrices)
It all makes sense!
Talking about "Jacobian and Hessian" matrices as if they are both naturally matrices is highly misleading.
Mathematicians are afraid of higher order tensors because they are unruly monsters.
There's a whole workshop of useful matrix tools. Decompositions, spectral theory, etc. These tools really break down when you generalize them to k-tensors. Even basic concepts like rank become sticky. (Iirc, the set of 3-tensors of tensor rank ≤k is not even topologically closed in general. Terrifying.) If you hand me some random 5-tensor, it's quite difficult to begin to understand it without somehow turning it into a matrix first by flattening or slicing or whatever.
Don't get me wrong. People work with these things. They do their best. But in general, mathematicians are afraid of higher order tensors. You should be too.
I'm always surprised how many other mathematicians don't know what I'm talking about when I reference this paper. It should be in the canon of math essays.
Readit News
The Hessian shouldn't have been called a matrix.
The Jacobian describes all the first order derivatives of a vector valued function (of multiple inputs), while the Hessian is all the second order derivatives of a scalar valued output function (of multiple inputs). Why doesn't the number of dimensions of the array increase by one as the derivation order increases? It does! The object that fully describes second order derivation of a vector valued function of multiple inputs is actually a 3 dimensionnal tensor. One dimension for the original vector valued output, and one for each derivation order. Mathematicians are afraid of tensors of more than 2 dimensions for some reason and want everything to be a matrix.
In other words, given a function R^n -> R^m:
Order 0: Output value: 1d array of shape (m) (a vector)
Order 1: First order derivative: 2d array of shape (m, n) (Jacobian matrix)
Order 2: Second order derivative: 3d array of shape (m, n, n) (array of Hessian matrices)
It all makes sense!
Talking about "Jacobian and Hessian" matrices as if they are both naturally matrices is highly misleading.
There's a whole workshop of useful matrix tools. Decompositions, spectral theory, etc. These tools really break down when you generalize them to k-tensors. Even basic concepts like rank become sticky. (Iirc, the set of 3-tensors of tensor rank ≤k is not even topologically closed in general. Terrifying.) If you hand me some random 5-tensor, it's quite difficult to begin to understand it without somehow turning it into a matrix first by flattening or slicing or whatever.
Don't get me wrong. People work with these things. They do their best. But in general, mathematicians are afraid of higher order tensors. You should be too.