Bayesian additive regression trees (BART) is a Bayesian tree-based model which can provide high predictive accuracy in both classification and regression problems. Within the Bayesian paradigm, regularisation is achieved by defining priors which ensure that each tree contributes modestly to the overall ensemble, thereby enhancing generalisation. Consequently, BART has proven to be very useful in a wide array of applications. However, the standard BART model is limited in certain respects. This thesis introduces some novel extensions to the BART framework to address certain key shortcomings. The inherent lack of smoothness, which is intrinsic to the piecewise-constant nature of the decision trees, is the motivation behind two of our proposals. The first involves the incorporation of Gaussian processes while the second uses penalised splines in the terminal nodes. Both of these novel approaches yield demonstrable improvements from the points of view of predictive accuracy and uncertainty calibration in extensive simulations and real-world applications. Another drawback of the standard BART model is that it is designed for predicting univariate outcomes. We introduce a third extension to embed BART in the seemingly unrelated regression framework to deal with multiple outcomes and model the covariance structure arising from their joint distribution. The method is applied in a causal setting in order to determine the cost-effectiveness of a novel medical intervention. The incorporation of penalised splines is designed to introduce smoothness to BART’s predictions. Concurrently, the extension to model multivariate outcomes within a seemingly unrelated regression framework enhances BART by structuring the covariance among responses. The synthesis of Gaussian processes with BART exemplifies this dual enhancement, simultaneously facilitating smooth predictive surfaces and capturing structured dependency, although the latter is within the feature space.