We present a decentralized algorithm for estimating mutual poses (relative positions and orientations) in a group of mobile robots. The algorithm uses relative-bearing measurements, which, for example, can be obtained from onboard cameras, and information about the motion of the robots, such as inertial measurements. It is assumed that all relative-bearing measurements are anonymous; i.e., each specifies a direction along which another robot is located but not its identity. This situation, which is often ignored in the literature, frequently arises in practice and remarkably increases the complexity of the problem. The proposed solution is based on a two-step approach: in the first step, the most likely unscaled relative configurations with identities are computed from anonymous measurements by using geometric arguments, while in the second step, the scale is determined by numeric Bayesian filtering based on the motion model. The solution is first developed for ground robots in SE (2) and then for aerial robots in SE (3). Experiments using Khepera III ground mobile robots and quadrotor aerial robots confirm that the proposed method is effective and robust w.r.t. false positives and negatives of the relative-bearing measuring process.