We propose an electric-magnetic symmetry group in non-abelian gauge theory, which we call the skeleton group. We work in the context of non-abelian unbroken gauge symmetry, and provide evidence for our proposal by relating the representation theory of the skeleton group to the labelling and fusion rules of charge sectors. We show that the labels of electric, magnetic and dyonic sectors in non-abelian Yang-Mills theory can be interpreted in terms of irreducible representations of the skeleton group. Decomposing tensor products of these representations thus gives a set of fusion rules which contain information about the full fusion rules of these charge sectors. We demonstrate consistency of the skeleton's fusion rules with the known fusion rules of the purely electric and purely magnetic magnetic sectors, and extract new predictions for the fusion rules of dyonic sectors in particular cases. We also implement S-duality and show that the fusion rules obtained from the skeleton group commute with S-duality.