Traditional Machine Learning (ML) methodologies typically involve aggregating datasets on a central server for analysis and model training. While effective in certain contexts, this centralized approach presents significant limitations, particularly concerning data sensitivity, privacy, and security. These constraints hinder the full realization of ML’s potential, thereby slowing progress across a range of applications. Distributed Machine Learning (DML) offers a promising alternative by decentralizing the learning process. In this paradigm, data remains at its source, while learning models are transmitted to the data, thereby eliminating the need for data extraction. Federated Learning (FL), a prominent subset of DML, is specifically designed to address challenges related to data privacy. FL commonly employs distributed stochastic gradient descent (DSGD) for optimization, yet it faces several practical challenges, including slow convergence and high communication overhead. This thesis investigates enhanced alternatives to conventional FL through the application of Hessian-based optimization methods. By leveraging second-order information, the learning process can be accelerated, requiring fewer iterations and reduced communication to accomplish learning tasks efficiently. The work addresses key challenges in FL and Fully Distributed Learning (FDL) over wireless networks, with a particular emphasis on minimizing communication costs while exploiting the advantages of second-order optimization. In the FL setting, we propose Distributed Approximate Newton with Determinantal Averaging (DANDA), a Newton-type method that significantly reduces the number of communication rounds required for convergence. To accommodate the limited computational capabilities of client devices, we incorporate approximation techniques for Hessian computation. DANDA operates over over-the-air Multiple Access Channels (MAC), and its performance is analyzed under realistic wireless conditions with channel fading and noise. Building on this, we develop Lazy-DANDA and Lazy-DANTA, approximate Newtonbased algorithms tailored for fading MAC environments. These methods integrate subsampled Hessians, weighted Hessian averaging, and an adaptive Hessian update strategy that transmits updates only when necessary, thereby further reducing communication overhead. To counteract distortions from channel fading, we incorporate channel inversion and power control mechanisms, preserving signal quality while regulating power consumption. In the FDL scenario, we extend the server-dependent GIANT algorithm into Network- GIANT, enabling decentralized node-to-node learning without a central server. This is achieved by combining gradient tracking, Newton-type iterations, and consensus-based averaging of Newton updates. Network-GIANT achieves semi-global exponential convergence under strong convexity and smoothness assumptions, addressing the slow convergence issue inherent to fully distributed setups. We further refine this approach into Network Exact Convergence-GIANT, which employs finite-time distributed consensus to match the exact convergence properties of GIANT in the FL setting. Collectively, these contributions advance the state of the art in second-order optimization for FL and FDL, delivering faster convergence, reduced communication overhead, and improved robustness under realistic wireless network conditions.