Yeah I think almost anyone will agree that these matrix formulations are more confusing than helpful. The subject itself isn't really that complicated, but the jargon and notation makes it a lot less approachable than it is. Unwrapping the operations with explicit summation makes it a heck of a lot clearer what is happening.
There are places where matrix expressions are informative and helpful, but at least in the context of teaching machine learning, this isn't one of them.
Hard disagree, to the point where I'm not sure there's not some confusion at play here.
I find using indices and summations generally confusing and hard to read/follow/understand. To me, it's so much simpler to draw out the computational graph, derive the gradient with a single element (so I'm in agreement with you guys here?), see how that applies the same to every element the same, then scale up to the vectorized/matrix version.
> and you need to memorize a bunch of new rules to apply it
There's nothing to memorize though..?
Is there something called "matrix calculus" that's different than just drawing the computational graph and deriving the gradient in the way I described..?
Now that einsum syntax is supported in languages like Python and Julia this is more true.
Before einsum syntax, it helped if the result could be written in terms of matrix operation because then one could utilise the primitives offered by matrix libraries. The alternative was to write the raw low level index loops, tedious and errorprone.
For some cases though the matrix result is just more compact. For example the derivative of log determinant of matrix wrt the matrix.
Same point made by bollu here
https://news.ycombinator.com/item?id=23359467
Yeah I think almost anyone will agree that these matrix formulations are more confusing than helpful. The subject itself isn't really that complicated, but the jargon and notation makes it a lot less approachable than it is. Unwrapping the operations with explicit summation makes it a heck of a lot clearer what is happening.
There are places where matrix expressions are informative and helpful, but at least in the context of teaching machine learning, this isn't one of them.
Hard disagree, to the point where I'm not sure there's not some confusion at play here.
I find using indices and summations generally confusing and hard to read/follow/understand. To me, it's so much simpler to draw out the computational graph, derive the gradient with a single element (so I'm in agreement with you guys here?), see how that applies the same to every element the same, then scale up to the vectorized/matrix version.
> and you need to memorize a bunch of new rules to apply it
There's nothing to memorize though..?
Is there something called "matrix calculus" that's different than just drawing the computational graph and deriving the gradient in the way I described..?
Like, is this "matrix calculus"?
- https://cs231n.github.io/optimization-2/
If so, what is there to memorize here?
Not memorize but one may need to derive the result. Consider differentiating the matrix logarithm of a complex matrix.
Now that einsum syntax is supported in languages like Python and Julia this is more true.
Before einsum syntax, it helped if the result could be written in terms of matrix operation because then one could utilise the primitives offered by matrix libraries. The alternative was to write the raw low level index loops, tedious and errorprone.
For some cases though the matrix result is just more compact. For example the derivative of log determinant of matrix wrt the matrix.