We examine how best to design qubits for use in topological quantum computation. These qubits are topological Hilbert spaces associated with small groups of anyons. Operations are performed on these by exchanging the anyons. One might argue that in order to have as many simple single-qubit operations as possible, the number of anyons per group should be maximized. However, we show that there is a maximal number of particles per qubit, namely 4, and more generally a maximal number of particles for qudits of dimension d . We also look at the possibility of having topological qubits for which one can perform two-qubit gates without leakage into non-computational states. It turns out that the requirement that all two-qubit gates are leakage free is very restrictive and this property can only be realized for two-qubit systems related to Ising-like anyon models, which do not allow for universal quantum computation by braiding. Our results follow directly from the representation theory of braid groups, which implies that they are valid for all anyon models. We also make some remarks about generalizations to other exchange groups.