AI impact on Emissions and Implications for SCI

[Henry-WattTime]

Starting a discussion around the emissions implications of training large AI models.

[will-iamalpine]

How about inference? That's the biggest culprit at this point

[Henry-WattTime]

This should certainly be included. Do you know if there are any studies or references that show the total or relative energy consumption of training and inference. I think Marcos is going to post some resources, but we want to collect as many as we can.

[will-iamalpine]

There are lots of great sources listed here:
https://news.climate.columbia.edu/2023/06/09/ais-growing-carbon-footprint/#:~:text=Google%20estimated%20that%20of%20the,of%20CO2%20in%20a%20year.

Google estimated that of the energy used in AI for training and inference, 60 percent goes towards inference, and 40 percent for training.

And this paper is great:
https://arxiv.org/pdf/2211.02001.pdf
While a 2019 article estimated that 80–90% of Nvidia’s ML workload is inference processing [20] – a figure that was cited in a recent article by Patterson et al [28], a recent publication by Meta reported that inference accounted for approximately one-third of their end-to-end ML carbon footprint while the remainder owes to data management, storage, and training [36].

In total, the instance used for the BLOOM model API consumed 914 kWh of electricity – of this amount, 22.7% was consumed by the RAM (207.2 kWh), 2% by the CPU (18.5 kWh) and 75.3% by the GPU (688.38 kWh). It is hard to disaggregate this number into idle versus dynamic consumption because we do not have access to the GCP platform as we did for Jean Zay, but we can nonetheless compare the energy consumed by the instance versus the number of requests that it received. We do so in Figure 3, where we plot the number of incoming requests to the BLOOM inference API and the energy consumption of the GCP instance where it is running. It can be seen in the Figure that even when there are almost no incoming requests during a 10 minute interval, there is still ∼0.28kWh of energy that is consumed during this interval, which represents the energy consumption of the model when it is not responding to any user requests. While more experimentation is needed in order to further disaggregate these numbers, we believe it is worth noting the high proportion of energy dedicated to maintaining a LLM like BLOOM in memory (approximately 75% of the total energy consumed by the instance), without it being used. Going further, given that the GCP instance used for deploying the BLOOM model is running in the us-central1 region, which has a carbon intensity of 394 gCO2eq/kWh [10], this results in approximately 19 kgs of CO2eq emitted per day of API deployment, or 340 kg over the total period during which we were tracking emissions.

[bgamazay]

At the moment, there are 2 inference energy efficiency benchmarks that I've seen: Hugging Face's LLM-Perf leaderboard and the ML.ENERGY leaderboard, both with slightly different approaches. I compare and contrast the two here

[Henry-WattTime]

@mvaltas, might this be helpful for the SCER project?

[navveenb]

Adding link to previous GSF work - Evaluating SCI for foundation models at Inference Stage - https://github.com/Green-Software-Foundation/eval_sci_of_foundation_models/blob/main/Report/Final_Report.pdf

[marcoshidalgonunes]

CodeCarbon is doing a great job about measure carbon emissions using AI.

Besides that, there are good articles in Frontiers in Environmental Science related to Artificial Intelligence Applications in Reduction of Carbon Emissions: Step Towards Sustainable Environment.

And the Reducing the carbon emissions of AI article is a good summary about how to reduce carbon emissions in AI.

[KasperGroesLudvigsen]

I've written about the electricity consumption and carbon footprint of training vs inference in the links below. For large scale deployment of LLMs, the deployment / inference stage of the life cycle is for sure the bigger culprit.

https://www.linkedin.com/posts/kaspergroesludvigsen_sustainabledatascience-greenai-sustainability-activity-7089122064632033280-OHgf?utm_source=share&utm_medium=member_ios

https://towardsdatascience.com/chatgpts-electricity-consumption-7873483feac4

https://towardsdatascience.com/environmental-impact-of-ubiquitous-generative-ai-9e061bac6800

[KasperGroesLudvigsen]

This diagram shows the estimated carbon footprint of the hardware manufacturing, model training and model deployment stages in a large scale generative AI adoption scenario (large scale = 3.5B users, 30 daily queries each). Even at smaller user or query numbers, the picture is the same: deployment uses much more energy, thus has a larger footprint.

Source: https://towardsdatascience.com/environmental-impact-of-ubiquitous-generative-ai-9e061bac6800

[adrianco]

The rate of change is very high in this space, and there is a lot of research and progress into drastic reductions in the compute, cost and hence carbon footprint of training, inference, fine tuning and prompting. The growth of fine tuning and prompting as workloads means that new applications can be developed without training from scratch, which drastically reduces the emissions. However Jevons paradox applies, and we are seeing a big increase in the number of possible applications enabled by the reductions in cost.

[Henry-WattTime]

@atg-abhishek wrote this interesting article: https://thegradient.pub/sustainable-ai/

[seanmcilroy29]

AI article overview provided by @atg-abhishek - https://thegradient.pub/sustainable-ai/

WG agree to close Issue