It is possible to constrain objects or bones to vertex groups using Child Of, copy rotation, etc. However, when you do this, you tend to get odd behavior, especially with bones. They tend to roll over or otherwise flip strangely.
However, if you parent an empty to 3 vertices, it works reliably. You can then constrain the bone to the empty. But this is a lot more setup.
Can anyone explain why these behave differently? Why is constraining directly to vertex groups not viable, and is there a way to solve it? I remember reading about it somewhere in detail a year or two ago, but cannot find it now.
When using a "Child Of" constraint with the vertex group option, because there can be any number of vertices, the rotation must be derived from a mix of all the vertex normals.
Even if you just use 3 vertices in the vertex group, the normals of those three vertices will be affected by other vertices that are not part of the vertex group. If those three vertices are connected by edges to other vertices, the connected vertices will affect their normals.
When doing a vertex parent, there is a guarantee of only three vertices so the algorithm is a simple barycentric algorithm.
If you make a duplicate of the three vertices in your group and detach them from other vertices you should get the same behavior with a "Child Of" constraint that you would get with parenting to three vertices.
One possible workaround would be to duplicate the vertices you want to use with the constraint so they are not connected to any neighboring verts that could affect their normals. You could delete their faces so they don't show up in render. I tried the mask modifier but it kills the constraint.