The formation of stellar mass black holes from the remnants of Population (Pop) III stars provides a source of initial black hole seeds with the potential to grow into intermediate or, in rare cases, possibly supermassive black holes. We use the Renaissance simulation suite to follow the growth of over 15 000 black holes born into mini-haloes in the early Universe. We compute the evolution of the black holes by post-processing individual remnant Pop III star particles in the Renaissance simulation snapshots. The black holes populate haloes from 106 up to 109 M. We find that all of the black holes display very inefficient growth. On average, the black holes increase their initial mass by a of factor 10−5, with the most active black holes increasing their mass by approximately 10 per cent. Only a single black hole experiences any period of super-Eddington accretion, but the duration is very short and not repeated. Furthermore, we find no correlation of black hole accretion with halo mass in the mass range sampled. Within most haloes, we identify clumps of cool, dense gas for which accretion rates would be high, but instances of black holes encountering these clumps are rare and short-lived. Star formation competes with black hole growth by consuming available gas and driving down accretion rates through feedback. We conclude that the black holes born from Popu III remnants do not form a significant population of intermediate mass black holes in the early Universe and will need to wait until later times to undergo significant accretion, if at all.