Non-abelian (Yang-Mills) gauge theories generalize electromagnetism from the abelian group U(1) to non-abelian groups like SU(2) and SU(3). The crucial difference is that the gauge bosons themselves carry charge and interact with each other. This self-interaction is responsible for asymptotic freedom in QCD and for the rich structure of the Standard Model.
Quantum electrodynamics is a gauge theory based on the abelian group U(1): the gauge transformation is multiplication by a phase e^{i alpha(x)}, and the single gauge boson (the photon) is electrically neutral. Non-abelian gauge theories, introduced by Yang and Mills in 1954, generalize this to non-commutative groups like SU(2) and SU(3). The gauge field A^a_mu now carries an index a labeling the generators of the group, and the gauge bosons themselves carry charge under the gauge group.
The key mathematical difference is in the field strength tensor. In QED, F_{mu nu} = partial_mu A_nu - partial_nu A_mu is linear in the gauge field. In a non-abelian theory, F^a_{mu nu} = partial_mu A^a_nu - partial_nu A^a_mu + g f^{abc} A^b_mu A^c_nu, where f^{abc} are the structure constants of the group (encoding the commutation relations of the generators). The extra term, quadratic in A, means that the kinetic energy -(1/4)F^a_{mu nu}F^{a mu nu} contains cubic and quartic terms in the gauge field. These are the self-interaction vertices of the gauge bosons -- a three-gluon vertex and a four-gluon vertex -- which have no counterpart in QED.
Quantization of non-abelian gauge theories introduces additional complications. The gauge freedom must be fixed to avoid integrating over physically equivalent field configurations. The standard method (Faddeev-Popov procedure) introduces ghost fields -- anticommuting scalar fields that are not physical particles but are needed to maintain unitarity in covariant gauges. In Feynman diagrams, ghosts appear as internal lines (drawn as dashed lines) in loop calculations, canceling the contributions of unphysical gauge boson polarizations. In abelian gauge theories, ghosts decouple and can be ignored.
The physical consequences of gluon self-interaction are profound. The most important is asymptotic freedom: unlike QED where the coupling grows at high energies, the QCD coupling decreases at high energies. This is because gluon loop contributions to the vacuum polarization (which come from the self-interaction) overwhelm the fermion loop contributions and have the opposite sign. Asymptotic freedom means QCD is perturbative at short distances (explaining the success of perturbative QCD in describing hard scattering) but strongly coupled at long distances (explaining confinement). The entire structure of the strong interaction -- from the proton mass to jet production -- follows from the non-abelian nature of SU(3).