The Carbon Footprint of Machine Learning – The New Stack

Machine learning models are becoming increasingly powerful in their capabilities, whether it’s processing natural language, tackling the intricacies of computer vision, or any number of exciting applications that are emerging. .

But we find that as these models get bigger and bigger, their corresponding carbon footprints also increase, especially when it comes to creating and training these complex models.

This trend has led some experts to develop tools capable of monitoring the carbon impact of machine learning models, as well as laying the groundwork for mitigation approaches such as carbon-aware computing.

Yet most of these tools measuring the carbon footprint of AI are still in relatively early stages of development, and as some recent studies show, their mileage can vary depending on the context.

Now, an interdisciplinary team from the Allen Institute for AI, Microsoft, Hebrew University, Carnegie Mellon University, and AI startup Hugging Face presents a more sophisticated approach that uses location-based data and to more accurately measure operational carbon emissions. popular AI models.

The new work, which was recently presented at the ACM Fairness, Accountability, and Transparency (FAccT) conference, differs from previous methods because it documents the power consumption of server GPUs as a series of measures, rather than oversimplifying it as a counted sum. 0over the entire training period of the model.

Further, the team’s approach also tracks these fluctuating measurements along with a series of data points that indicate local emissions per kilowatt-hour (kWh) of energy used.

These calculations can ultimately help users understand the hidden carbon costs of their cloud-based workloads, or the carbon intensity of the amount of carbon emissions produced per kilowatt-hour of electricity consumed, wrote the team: “Perhaps the most important contribution of this document is the simplest: a presentation of software carbon intensity (SCI) as an indicator of carbon emissions for a given cloud instance while it is running. »

Comparison of different carbon intensities

The team’s work involved training 11 machine learning models of varying sizes and applications, with training periods ranging from one hour on one GPU to eight days on 256 GPUs.

Energy consumption was documented at regular intervals, in addition to tracking different carbon intensities at different times of day, across 16 geographic locations at 5-minute intervals. This approach allowed the researchers to compare the energy consumption of the models according to geography, time and type of energy production.

“The carbon intensity of a network is sensitive to small changes in carbon-intensive output and can vary with both location and time,” the study co-author explained. , Jesse Dodge, in a blog post.

“Each region incorporates a different mix of energy sources, which can therefore vary significantly. Carbon intensity varies by time of day, day and season due to variations in electricity demand, low-power generation and carbon emissions (wind, solar, hydroelectric, nuclear, biomass) and conventional hydrocarbon production.As a result, there are many opportunities to move computing resources to take advantage of these variations: this is what we calls carbon-aware computing.

During their experiments, the team made some eye-opening discoveries. For example, they found that the carbon emissions generated when training some of the lightest AI models were equivalent to those of charging a mobile phone, while one of the largest models tested – as a 6.1 billion parameter “transformer” model – was trained to only 13% complete, and yet produced an “astonishing” amount of carbon equal to powering a home for a year in the States -United. One can only imagine the emissions involved in training even larger models like OpenAI’s groundbreaking GPT-3 NLP model, which consists of 100 billion parameters.

The team found that the biggest factor in reducing emissions was selecting the best geographic location. For example, when training the commonly used BERT model of NLP, the team discovered that emissions could be cut in half by training it using renewable energy sources located in countries like Norway or Turkey. France.

The researchers found that even the time of day the training took place had an impact; forming a pattern in Washington State overnight has resulted in reduced emissions, because that’s when electricity is produced solely by hydroelectric power, rather than being mixed with power from the gas-fired power plants during the day.

In addition to these results, the team also evaluated two optimization algorithms to reduce emissions from AI workloads on the Microsoft Azure cloud computing platform.

The first of these was “Flexible Start”, which delayed training for 24 hours and benefited the smaller models the most. In contrast, the team’s “Pause and Resume” approach would pause training during peak emission hours, but favor larger models as long as it didn’t double the training time.

While the team’s work focuses only on the operational carbon emissions of the training AI models and does not consider the carbon emissions associated with building hardware, cooling data centers, etc. , the team nevertheless stressed that more comprehensive approaches to carbon will become vital to ensure the future sustainability of machine learning models.

“Carbon accounting and reporting is becoming more mainstream in machine learning, with conferences such as NeurIPS requiring submissions to report their emissions and recent work reporting emissions incurred,” the team noted. “However, this has not yet become the norm in our field, and we still lack systematic information regarding the environmental footprint of training machine learning models and how we can reduce it. In this article, we argue that if members of the machine learning community had access to information about the carbon emissions of their actions, they could tailor their decisions to reduce those emissions while meeting the computational needs for their final tasks.

Read more in the team diary.

Images: Luis Ruiz via Pexels; Allen Institute for AI.

Sherry J. Basler