In mathematics, invariant theory refers to the study of invariant algebraic forms (equivalently, symmetric tensors) for the action of linear transformations. This was a major field of study in the latter part of the nineteenth century, when it appeared that progress in this particular field (out of any number of possible mathematical formulations of invariance with respect to symmetry) was the key algorithmic discipline. Despite some heroic efforts that promise was not fulfilled, one can say; but many spin-off advances can be shown to have been connected. Current theories relating to the symmetric group and symmetric functions, commutative algebra, moduli spaces and the representations of Lie groups are rooted in this area.

In greater detail, given a finite-dimensional vector space V we can consider the symmetric algebra S(V), and the action on it of GL(V). It is actually more accurate to consider the projective representation of GL(V), if we are going to speak of invariants: that's because a scalar multiple of the identity will act on a tensor of rank r in S(V) through the r-th power 'weight' of the scalar. The point is then to define the subalgebra of invariants I(V) for the (projective) action. We are, in classical language, looking at n-ary r-ics, where n is the dimension of V.

These days it might be more natural to look to decompose the degree r part of S(V) into irreducible representations of GL(V): the formulation just given is the same as saying we are concerned only with the occurrence of one-dimensional representations. The representation theory required came later, though, with Schur.

It is customary to say that the work of David Hilbert, proving abstractly that I(V) was finitely presented, put an end to classical invariant theory. That is far from being true: the classical epoch in the subject may have continued to the final publications of Alfred Young, more than 50 years later. Explicit calculations for particular purposes have been known in modern times (for example Shioda, with the binary octavics).

The modern formulation of geometric invariant theory is due to David Mumford, and emphasizes the construction of a quotient by the group action that should capture invariant information through its coordinate ring. It is a subtle theory, in that success is obtained by excluding some 'bad' orbits and identifying others with 'good' orbits. In a separate development the symbolic method of invariant theory, an apparently heuristic combinatorial notation, has been rehabilitated.

For the invariant theory of finite groups, see Molien series.