In this paper, we revisit the fundamental problem of computing MIMO capacity under per-antenna power constraint (PAPC). Unlike the sum power constraint counterpart which likely admits water-filling-like solutions, MIMO capacity with PAPC has been largely studied under the framework of generic convex optimization. The two main shortcomings of these approaches are (i) their complexity scales quickly with the problem size, which is not appealing for large-scale antenna systems, and/or (ii) their convergence properties are sensitive to the problem data. As a starting point, we first consider a single user MIMO scenario and propose two provably-convergent iterative algorithms to find its capacity, the first method based on fixedpoint iteration and the other based on alternating optimization and minimax duality. In particular, the two proposed methods can leverage the water-filling algorithm in each iteration and converge faster, compared to current methods. We then extend the proposed solutions to multiuser MIMO systems with dirty paper coding (DPC) based transmission strategies. In this regard, capacity regions of Gaussian broadcast channels with PAPC are also computed using closed-form expressions. Numerical results are provided to demonstrate the outperformance of the proposed solutions over existing approaches.