Enhancing Sustainability in LLM Training: Leveraging Federated Learning and Parameter-Efficient Fine-Tuning

Large language models deliver strong performance but at the cost of high energy use and carbon emissions, limiting their accessibility. Federated Learning (FL) addresses energy efficiency by utilizing distributed data sources, coordinating learning from diverse, energy-constrained clients, and allowing the training process to occur without the need for large centralized datasets. This paper evaluates LLM training using various FL client selection strategies and a centralized approach, integrating Parameter-Efficient Fine-Tuning (PEFT). The assessment focuses on carbon emissions, energy consumption, and model performance, providing insights into the environmental impact of different training methods and identifying sustainable practices. Results show that centralized training achieves the highest accuracy but lacks privacy and scalability, whereas FL methods like FedFull+LoRa provide comparable accuracy but incur higher energy consumption and emissions. FedProx+LoRa and FedNorm+LoRa balance energy efficiency and accuracy. We also extended our FedSustain client selection strategy by incorporating real-time renewable-energy signals from the Electricity Maps API alongside on-device energy buffers. On an IMDB/DistilBERT benchmark, FedSustain matches leading FL baselines in both accuracy and convergence speed while reducing total energy use by up to 38% and CO $_{2}$ emissions by up to 46%. Furthermore, to address the limitations of the current client selection strategy, we present an enhanced strategy for future evaluation on more complex tasks and larger models. These results highlight the potential of energy-aware client selection to make large-model training more sustainable.