Watts Driving the Cost of AI Deployment? (2024)

1. Introduction

Understanding the environmental impacts of different industries is an important first step towards developing effective strategies to mitigate those impacts. For newer industries such as information and communication technologies (ICT) of which Artificial Intelligence (AI) and Machine Learning (ML) are considered to be a part of, more work is needed to understand the extent of their environmental impacts and the factors that influence it. Between 2017 and 2021, the electricity used by Meta, Amazon, Microsoft, and Google, the main providers of commercially-available cloud compute, more than doubled (International Energy Authority, 2023). According to the most recent figures available, global data centre electricity consumption has grown by 20-40% annually in recent years, reaching 1-1.3% of global electricity demand and contributing 1% of energy-related greenhouse gas emissions in 2022(Hintemann and Hinterholzer, 2022). However the contribution of the AI sector specifically towards these figures is unclear.

Recent work documenting the environmental impacts of ML has focused largely on quantifying the operational energy and carbon required to perform the training phase of the ML model life cycle(Strubell etal., 2019; Patterson etal., 2021; Dodge etal., 2022; Luccioni and Hernandez-Garcia, 2023) due to the relative ease of measuring per-model energy use for that phase and the impressive quantity of energy required to perform a single training run(Strubell etal., 2019; Patterson etal., 2021). Yet, other phases of the ML model life cycle, such as inference, stand to impact the environment just as much, or more, than training due to the computational resources required to deploy modern models at scale. While inference on a single example requires much less computation than that required to train the same model, inference happens far more frequently than model training — as many as billions of times a day for a model powering a popular user-facing product such as Google Translate.¹¹1Google reported translating more than 100 billion words per day in 2016, assuming an average query length of 100 words yields an estimate of 1 billion queries to the model per day. Source: https://blog.google/products/translate/ten-years-of-google-translate/ Yet, in-depth work quantifying the costs of model inference and deployment is limited and their environmental impacts, in terms of energy and carbon as well as water and mining of rare earth minerals, have yet to be estimated. According to AWS, the largest global cloud provider, inference is estimated to make up 80 to 90% of total ML cloud computing demand(Barr, 2019; Leopold, 2019), whereas a 2021 publication by Meta attributed approximately one-third of their internal end-to-end ML carbon footprint to model inference, with the remainder produced by data management, storage, and training(Wu etal., 2021); similarly, a 2022 study from Google attributed 60% of its ML energy use to inference, compared to 40% for training(Patterson etal., 2022). Given the increasing ubiquity of AI model deployment, it is crucial to go beyond these high-level statistics to get a better idea of the energy requirements and carbon emissions of model inference for different models and tasks.In particular, looking at inference rather than training leads to drastically different conclusions when considering the multi-purpose (or “general-purpose”) aspect specifically. Training a single model for multiple tasks can indeed be more energy-efficient when considering training costs only, but these gains can easily be lost and even reversed over the course of the model’s lifetime, given how much inference is carried out when these models are deployed in user-facing applications like chat and web search.

To help shed light on this issue, we perform an extensive study measuring the amount of energy required to deploy various ML models and architectures, including large language models (LLMs)- as such, our study is, to our knowledge, the first to focus solely on the inference phase of the ML model life cycle. We study 88 models across 10 tasks and 30 datasets, spanning applications in natural language and computer vision, analyzing the impact of end task, modality, model size, architecture, and learning paradigm (i.e. task-specific or multi-task/multi-purpose) on energy efficiency. We identify orders-of-magnitude differences in the amount of energy required per inference across models, modalities and tasks and shine light on an important trade-off between the benefit of multi-purpose systems, their energy cost, and ensuing carbon emissions.By painting a more detailed picture of widely varying energy requirements for ML model inference, we hope this study can be useful for practitioners to better understand accuracy-efficiency trade-offs across tasks and models, as well as enabling better estimates, and projections and policy decisions at the sector level.

2. Previous Work

Estimating the energy and emissions of ML models has remains a relatively under-explored topic, albeit one that has been gathering traction since Strubell et al’s seminal article quantifying the energy and carbon emissions of a variety of then-large NLP models(2019). Since then, most studies have focused on estimating the energy consumed and carbon emitted during the training phase of neural networks – this includes studies by Patterson et al.(2021; 2022), who compared different models and analyzed factors influencing their emissions. There have also been studies of specific model architectures, e.g. BLOOM(Luccioni etal., 2022) and Nour(Lakim etal., 2022), which carried out in-depth analyses of the different steps in the models’ life cycle and their relative contribution towards the final quantity of carbon emissions.Given the increasing deployment of ML models in the cloud, several studies have therefore looked at cloud-specific ways to reduce the emissions of ML models such as delayed scheduling, workload elasticity and choosing the least carbon-intensive electricity availableDodge etal. (2022); Hanafy etal. (2023); Chien etal. (2023).

Despite these empirical studies, there is currently a lack of standardized methodology for quantifying and comparing the energy consumption and carbon emissions of ML models. There are several tools that exist, such as Code Carbon(Schmidt etal., 2021), MLCO2(Lacoste etal., 2019) and LLMCarbon(Faiz etal., 2023), all of which adopt different approaches and output different results (see(Bannour etal., 2021) for a detailed comparison). It is therefore difficult to systematically compare the carbon footprints of different models. Existing tools and studies have also largely focused on the dynamic power consumption (i.e. the electricity necessary for powering hardware) and its resulting emissions. However, there have been several proposals to also take into account the embodied emissions of ML models (i.e. the emissions that can be attributed to the manufacturing of computing equipment) into carbon emissions estimates. This has been impeded by a lack of transparency from the designers of common computing hardware such as GPUs, although recent estimates have revealed that the embodied carbon footprint of an LLM trained and deployed on Meta’s compute cluster constitutes up to 50% of its carbon footprint(Wu etal., 2021).While the majority of existing work has been focused on ML model training given that it is a more tractable part of the model life cycle (i.e. it is most often carried out over a set period of time on a specific compute instance), model inference has started to also become the subject of scholarship(Desislavov etal., 2021; Chien etal., 2023). Luccioni et al.’s study of BLOOM was the first of its kind to look at the specific energy costs related to deploying an LLM(Luccioni etal., 2022) and found that, over time, this can represent a significant portion of a model’s overall carbon footprint.

The current study further pursues this line of work, delving deeper into the inference stage of ML models, the energy it consumes and the carbon it emits. By testing a variety of architectures on different tasks and datasets, we aim to gain a better understanding of the degree of variance that can be observed and how seemingly small user choices can result in large differences in model’s environmental impacts.

3. Methodology

As stated above, our study focuses on the inference (i.e. deployment) stage in the model life cycle, aiming to address the knowledge gaps that currently exist with regards to its energy consumption and ensuing emissions. We describe how we chose the tasks, datasets and models in the sections below, and present the results of our analysis in Section4.

4. Results

We present our results in the subsections below: in Section4.1, we analyze the range of energy used and carbon emitted for each task for task-specific models. In Section4.2, we shift our focus to multi-purpose (i.e. ‘zero-shot‘ models), looking at the variation between different sizes and architectures of multi-purpose models and the difference in the energy consumption and emissions between task-specific and multi-purpose models. In Section4.3, we carry out a comparison between model training and inference costs for models of different sizes, calculating when parity is reached.

4.1. Task-specific model analysis

We start by analyzing the degree of variability in terms of the energy cost of ML models specifically trained for a variety of tasks. Table2 shows each of the ten tasks that we analyzed as well as the mean energy used across all models for 1,000 inferences and its standard deviation. We can see that classification tasks for both images and text are on the lower end of the spectrum in terms of emissions (ranging between 0.002 and 0.007 kWh for 1,000 inferences), whereas generative tasks such as text generation and summarization use, on average, over 10 times more energy for the same number of inferences (around 0.05 kWh for 1,000 inferences), and multimodal tasks such as image captioning and image generation are on the highest end of the spectrum (0.06-2.9 kWh for 1,000 inferences). Text-based tasks are, all things considered, more energy-efficient than image-based tasks, with image classification requiring less energy (median of 0.0068 kWh for 1,000 inferences) than image generation (1.35 kWh) and, conversely, text generation (0.042 KwH) requiring more than text classification (0.0023 kWh). For comparison, charging the average smartphone requires 0.022 kWh of energy(US Environmental Protection Agencyy, 2024), which means that the most efficient text generation model uses as much energy as 9% of a full smartphone charge for 1,000 inferences, whereas the least efficient image generation model uses as much energy as 522 smartphone charges (11.49 kWh), or around half a charge per image generation⁴⁴4Before January 2024, the EPA website estimated a smartphone charge to consume 0.012 kWh of energy, which was the number used for comparisons in an earlier version of this study., although there is also a large variation between image generation models, depending on the size of image that they generate.

	inference energy (kWh)
task	mean	std
text classification	0.002	0.001
extractive QA	0.003	0.001
masked language modeling	0.003	0.001
token classification	0.004	0.002
image classification	0.007	0.001
object detection	0.038	0.02
text generation	0.047	0.03
summarization	0.049	0.01
image captioning	0.063	0.02
image generation	2.907	3.31

We can also observe that there is a large variation in the amount of energy used, from the least energy-intensive task, text classification, with mean consumption of 0.002 KwH per 1,000 inferences, to the most energy-intensive one, image generation, whose mean consumption is 2.9kWh. This means that the different models examined in our study can vary by a factor of over 1450 in terms of the energy required to perform the same number of inferences.Intuitively, this is coherent given the decision space that different types of models have - from a binary classification task such as sentiment analysis (which can only output, for instance, a 0 for negative sentiment and a 1 for positive) to an entire vocabulary for text generation and summarization models. The length of text generated also impacts energy usage: on average, text generation uses 15 times more energy than masked language modeling, which makes sense given that the masked language modeling task only generates a single token, whereas in our setup the text generation task generates 10 new tokens for each input text, with the length of the input text rising as new tokens are generated, since each sequence of tokens gets fed back into the model to generate subsequent tokens. Finally, for image-based tasks, the level of abstraction is lower and the decision space is larger given that they generate raw pixels as opposed to tokens for text, making image-based tasks more energy intensive than text based ones, e.g. image classification uses over 3 times more energy than text classification (0.007 vs. 0.002 kWh) and image generation uses, on average, over 60 times more energy than text generation (0.047 vs. 2.9 kWh).

Next, we examine the respective influences of model size and task structure on model emissions. Figure2 shows the relationship between model emissions (in grams of $CO_{2}eq$ per 1,000 inferences) and sizes (in terms of the number of parameters) across the task categories listed in Section3.1.We do observe a relationship between model size and quantity of emissions produced during inference, with differing progressions for each modality – however, the task structure accounts for more of the variation than the model size does. We can observe once again that text-to-image is by far the most carbon- and energy-intensive task, with smaller image generation models such as segmind/tiny-sd that have around 500M parameters producing magnitudes more carbon than text-to-category models (100g vs. 0.6g of $CO_{2}eq$ per 1,000 inferences). Within the text-to-text tasks, we see two separate sets of models: the masked language modeling task following a lower trend, producing emissions akin to text-to-category models, compared to text generation and summarization tasks, which produce similar amounts of carbon to the image captioning models with a similar number of parameters. For context, the most carbon-intensive image generation model (stable-diffusion-xl-base-1.0) generates 1,594 grams of $CO_{2}eq$ for 1,000 inferences, which is roughly the equivalent to 4.1 miles driven by an average gasoline-powered passenger vehicle(US Environmental Protection Agencyy, 2024), whereas the least carbon-intensive text generation model (distilbert-base-uncased) generates as much carbon as 0.0006 miles driven by a similar vehicle, i.e. 6,833 times less. This can add up quickly when image generation models such as Dall·E and MidJourney are deployed in user-facing applications and used by millions of users globally (we discuss this point further in Section5).

The (high-level) takeaway of this analysis is that even for models specifically trained to carry out a single task, there is a large level of variation both within each task and an even larger one between tasks from different modalities. In essence, tasks that map both image and text inputs to categorical outputs are less energy- and carbon-intensive than those that generate text or images. Making these distinctions can help inform policies seeking to mitigate the environmental impacts of AI, given that it is important to be aware of this variation, which can sometimes reach several orders of magnitude. In the next section, we delve deeper into multi-purpose systems, which are meant to carry out several tasks concurrently, to better understand their environmental impacts and how they compare to task-specific models.

4.2. The environmental cost of multi-purpose systems

The second part of our analysis examines multi-task models of two types: decoder only, from the BLOOMz family, and sequence-to-sequence models from the FLAN-T5 family, with the goal of comparing energy intensity and carbon emissions of models with differing numbers of parameters when applied to different tasks. To address this question, we selected a subset of 3 tasks – text classification, extractive question answering, and summarization – given their diversity and broad applicability in a variety of settings, and compare the 8 zero-shot models of different sizes, based on the same 3 datasets per task as described in Table1.

4.2.1. Emissions of task-specific and multi-task architectures

To start our analysis, we examined how the choice of model and architecture type impacts emissions given a specific task and dataset. For this analysis, we took the same 8 task-specific models described in Section3.2 and compared their emissions to the 8 multi-purpose models described above. In Figure3, we plot the mean query emissions for each model on a dataset-by-dataset basis. We can see that for the two discriminative tasks, sentiment analysis (which includes SST 2, Rotten Tomatoes and IMDB datasets) and question answering (which encompasses SciQ, SQuAD and SQuAD v2) there is a clear distinction between task-specific discriminative models (in blue), which have less emissions than both multi-purpose sequence-to-sequence (in yellow) and decoder-only generative models (in green). Given that the y axis in Figure3 is in logarithmic scale, this indicates that the difference is several orders of magnitude, with the most efficient task-specific models emiting 0.3g of $CO_{2}eq$ per 1,000 inferences for extractive question answering on a dataset like SciQ, multi-purpose models emit 10g for the same task. This result follows intuitions derived from the model structures: while a task-specific model trained on binary text classification will carry out a softmax on a two-category vector to predict a class, a multi-purpose model will generate ‘positive’ or ‘negative’, which logically requires more energy because the prediction is based on the model’s entire vocabulary.

For the generative task, summarization (represented by the SAMsum, XSum and CNN-Daily Mail datasets), the task-specific and multi-purpose models are closer in terms of emissions: task-specific sequence-to-sequence models generate 4-10g of $CO_{2}eq$ for 1,000 inferences, while multi-purpose models emit 20-30g for the same task. The difference appears to mostly come from model size – all of the task-specific summarization models we looked at were 600 million parameters at most, compared to the larger multi-purpose architectures, which attained the 11 billion parameters.

seq2seq models				decoder-only models
model name	number of parameters	emissions (g $CO_{2}eq$)	energy (kWh)	model name	number of parameters	emissions (g $CO_{2}eq$)	energy (kWh)
Flan-T5-base	222M	3.67	0.026	BLOOMz-560M	559M	7.5	0.054
Flan-T5-large	750M	7.68	0.055	BLOOMz-1B	1.7B	8.66	0.062
Flan-T5-xl	2.8B	8.08	0.058	BLOOMz-3B	3B	10.17	0.073
Flan-T5-xxl	11B	11.48	0.083	BLOOMz-7B	7B	14.46	0.104

We also carry out an evaluation of both the task-specific and multi-purpose models examined in our study to ensure that they have comparable performance. For task-specific models, we used the evaluate library(VonWerra etal., 2022) and the LM Evaluation Harness(Gao etal., 2021) for zero-shot models. Fundamentally speaking, it is hard to compare task-specific and multi-purpose models using the same metrics, given that task-specific models have a much more constrained decision space (e.g. two classes in the case of binary text classification), whereas multi-purpose models have a large output vocabulary to choose from, and are dependent upon the prompt schema and prompting strategy used. However, by utilizing two standardized packages (evaluate and lm-evaluation-harness) and keeping the prompting approach stable across zero-shot models, we endeavor to standardize our evaluation approach as much as possible.

		BLOOMz 560M	BLOOMz 1B	BLOOMz 3B	BLOOMz 7B	Flan-T5 base	Flan-T5 large	Flan-T5 xl	Flan-T5 xxl
dataset	input	output	output	output	output	output	output	output	output
IMDB	58.73	1.64	2.61	1.72	1.53	1.00	1.00	1.00	1.00
Rotten Tomatoes	30.08	1.00	0.99	1.03	1.00	1.00	1.00	1.00	1.00
SST 2	28.35	0.98	0.99	1.01	1.02	1.00	1.00	1.00	1.00
SciQ	113.12	1.28	1.25	1.10	1.10	2.03	5.41	3.12	2.42
SQuAD	134.00	1.93	1.96	2.02	1.95	2.01	2.15	2.16	2.13
SQuAD 2	115.85	2.33	2.54	2.58	2.41	2.28	2.74	2.71	2.58
CNN	54.00	12.05	11.91	11.73	10.34	8.52	11.34	11.34	10.68
SamSUM	47.82	9.54	9.41	9.75	9.85	10.56	11.05	10.18	10.57
XSum	53.85	11.53	12.22	11.94	11.92	12.95	13.62	13.49	13.09

We hone in on one specific task, text classification, in Figure4, which illustrates the relationship between model size (x axis, in logarithmic scale), accuracy (y axis) and emissions (dot size, in logarithmic scale). Among task-specific encoder models, we observe that accuracy varies more widely, i.e. there are several smaller models of similar size and comparably small amounts of carbon emissions, with widely varying levels of accuracy. The multi-purpose models vary less in terms of accuracy, having higher average accuracy overall. Both sequence-to-sequence and decoder-only models produce comparable amounts of emissions (several orders of magnitude more than task-specific models).We can see that mid-size multi-purpose models (in the 3B parameter range) may have slightly better accuracy compared to both larger and smaller models. However, given the many caveats and specificities involved in multi-purpose LLM evaluation, this difference may not be significant. We present the full results of our evaluation, which include the other 2 tasks, in SectionB in the Supplementary Materials.

4.2.2. Differences within multi-purpose architectures

Beyond the differences between task-specific and multi-purpose models generally, we also observed variation within the multi-purpose models that we examined. We present our results in Table3; in it, we can observe that on a per-architecture basis (i.e. within the family of decoder-only models and the family of sequence-to-sequence models), size and emissions are correlated, with smaller models emitting less carbon and using less energy. However, sequence-to-sequence models are more efficient than their decoder-only counterparts when models of the same size are compared: for instance, Flan-T5-XL and BLOOMz-3B are both of a similar size (around 3B parameters), but the former generates, on average, 2 grams of emissions less for 1,000 inferences than the latter. This difference holds when comparing Flan-T5-XXL, which is the biggest model in terms of parameter count in the multi-purpose models that we tested (11 billion), yet it has lower emissions (11.48g on average) compared to the smaller BLOOMz-7B. Comparing the models on a per-task basis in Figure5, we can see the same pattern for zero-shot models as for task-specific ones, with text classification a less carbon-intensive task compared to question answering, and summarization the most intensive one of the three. The spread between the tasks is smaller for sequence-to-sequence models (indicated with dots in Figure5), whereas for decoder-only models (indicated with crosses), the difference between the different tasks is more significant.

We can analyse the relationship between sequence-to-sequence and decoder-only models noted in Table3: whereas for tasks such as summarization, decoder models do generate more emissions than sequence-to-sequence models of a similar size, for question answering and text classification, the two architectures have similar emissions.This can again be explained by the differences in the model structures, specifically the attention mechanism: while sequence-to-sequence models only attend to the last layer of the input when producing their answers, decoder-only architectures attend to all layers for the full sequence – leading to a stronger dependency on the output length for the number of operations, resulting in more emissions for tasks with longer outputs.

We further verify this intuition in Table4 and Figure6: while there is some variation between models and datasets in Table4, the distribution of output lengths is consistent with our expectations for the different task categories: tasks with longer outputs result in more emissions, especially for decoder-only models. Figure6 delves further into the relationship between average output length, carbon emissions, and model structures for the different summarization datasets. It shows a clear correlation between output length and measured emissions, with a higher slope for the decoder-only architectures (the BLOOMz family of models) than for the sequence-to-sequence architectures (the Flan-T5 family).

As we have observed in the current section, there is no ‘one-size-fits-all’ pattern for multi-purpose models either – they too exhibit variation in terms of their emissions and energy usage, which can be attributed to different factors, including model size and output length. This would indicate that more careful consideration is needed when making choices to deploy these models for different tasks and applying them in different scenarios. We further discuss our results and further avenues of research in the next and final section.

	BLOOMz-7B	BLOOMz-3B	BLOOMz-1B	BLOOMz-560M
Training energy (kWh)	51,686	25,634	17,052	10,505
Finetuning energy (kWh)	7,571	3,242	1,081	543
Inference energy (kWh)	1.0 × $10^{-4}$	7.3 × $10^{-5}$	6.2 × $10^{-5}$	5.4 × $10^{-5}$
Cost parity (# inferences)	592,570,000	395,602,740	292,467,741	204,592,592

5. Discussion

There have been limited studies regarding the energy consumption and carbon emissions of LLM inference, largely due to its distributed nature — compared to the relatively time- and location-constrained nature of training — making it difficult to make meaningful comparisons between different models and tasks. In this work, we have endeavored to keep as many parameters stable as possible, including the code, hardware, datasets, batch size and Python library. We provide all of the code that we used for our analysis as well as an interactive tool to allow users to more deeply explore the results we present here. We also highlight the main high-level takeaways of our study below:

Ethical Considerations Statement

The main ethical concerns that we faced in our experimentation is the sheer amount of energy needed and carbon emissions generated by our study, given that we ran each of the 88 models on 3 datasets 10 times to ensure statistical significance of our measurements. In total, for all of model experimentation and evaluation, we used a total of 754.66 kWh of energy and emitted 178.97 kg of $CO_{2}eq$. In order to reduce our impacts as much as possible, we did all up-front experimentations on smaller portions of the dataset (to reduce wasted resources).

Researcher Positionality Statement

The authors of this paper have backgrounds in theoretical and applied machine learning and work in institutions based in North America. We therefore recognize that our way of planning and running experiments is not necessarily reflective of other institutions from other regions, or the constraints faced by researchers from institutions with more limited access to compute.

Adverse Impacts Statement

We recognize that our work can be perceived as a critique of ML deployment in general, given the analysis that we provide of its environmental impacts. This could be used as an argument to stop pursuing ML research and development, or as a way of targeting specific companies or organizations. Our intention, however, is to shed additional light on the environmental impacts of ML, in order to help model developers and researchers make more informed choices as a function of their environmental footprint or energy usage.

References

• (1)
• Bannour etal. (2021)Nesrine Bannour, Sahar Ghannay, Aurélie Névéol, and Anne-Laure Ligozat. 2021.Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools. In EMNLP, Workshop SustaiNLP.
• Barr (2019)Jeff Barr. 2019.Amazon ec2 update–inf1 instances with AWS inferentia chips for high performance cost-effective inferencing.https://aws.amazon.com/blogs/aws/amazon-ec2-update-inf1-instances-with-aws-inferentia-chips-for-high-performance-cost-effective-inferencing/
• Bing (2019)Bing. 2019.Bing delivers its largest improvement in search experience using Azure GPUs.https://azure.microsoft.com/en-us/blog/bing-delivers-its-largest-improvement-in-search-experience-using-azure-gpus/
• Bing (2023)Bing. 2023.Confirmed: the new Bing runs on OpenAI’s GPT-4.https://blogs.bing.com/search/march_2023/Confirmed-the-new-Bing-runs-on-OpenAI%E2%80%99s-GPT-4
• Brown etal. (2020)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, JaredD Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, etal. 2020.Language models are few-shot learners.Advances in neural information processing systems 33 (2020), 1877–1901.
• Chien etal. (2023)AndrewA Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. 2023.Reducing the Carbon Impact of Generative AI Inference (today and in 2035). In Proceedings of the 2nd Workshop on Sustainable Computer Systems. 1–7.
• Chowdhery etal. (2022)Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, HyungWon Chung, Charles Sutton, Sebastian Gehrmann, etal. 2022.Palm: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311 (2022).
• Chung etal. (2022)HyungWon Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, ShixiangShane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, EdH. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, QuocV. Le, and Jason Wei.2022.Scaling Instruction-Finetuned Language Models.https://doi.org/10.48550/ARXIV.2210.11416
• Dagli and Shaikh (2021)Rish*t Dagli and AliMustufa Shaikh. 2021.CPPE-5: Medical Personal Protective Equipment Dataset.arXiv:2112.09569[cs.CV]
• Desai etal. (2021)Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. 2021.RedCaps: web-curated image-text data created by the people, for the people.arXiv:2111.11431[cs.CV]
• Desislavov etal. (2021)Radosvet Desislavov, Fernando Martínez-Plumed, and José Hernández-Orallo. 2021.Compute and energy consumption trends in deep learning inference.arXiv preprint arXiv:2109.05472 (2021).
• Dodge etal. (2022)Jesse Dodge, Taylor Prewitt, Remi Tachetdes Combes, Erika Odmark, Roy Schwartz, Emma Strubell, AlexandraSasha Luccioni, NoahA Smith, Nicole DeCario, and Will Buchanan. 2022.Measuring the carbon intensity of AI in cloud instances. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1877–1894.
• Faiz etal. (2023)Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang. 2023.LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models.arXiv preprint arXiv:2309.14393 (2023).
• Gao etal. (2021)Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021.A framework for few-shot language model evaluation.https://doi.org/10.5281/zenodo.5371628
• Gliwa etal. (2019)Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics, Hong Kong, China, 70–79.https://doi.org/10.18653/v1/D19-5409
• Google (2019)Google. 2019.Understanding searches better than ever before.https://blog.google/products/search/search-language-understanding-bert/
• Google (2023a)Google. 2023a.Bard can now connect to your Google apps and services.https://blog.google/products/bard/google-bard-new-features-update-sept-2023/
• Google (2023b)Google. 2023b.An important next step on our AI journey.https://blog.google/technology/ai/bard-google-ai-search-updates/
• Hanafy etal. (2023)WalidA Hanafy, Qianlin Liang, Noman Bashir, David Irwin, and Prashant Shenoy. 2023.CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency.arXiv preprint arXiv:2302.08681 (2023).
• Hermann etal. (2015)KarlMoritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.Teaching Machines to Read and Comprehend. In NeurIPS. 1693–1701.http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend
• Hintemann and Hinterholzer (2022)Ralph Hintemann and Simon Hinterholzer. 2022.Cloud computing drives the growth of the data center industry and its energy consumption.Data centers 2022. ResearchGate (2022).
• International Energy Authority (2023)International Energy Authority. 2023.Data Centres and Data Transmission Networks.https://www.iea.org/energy-system/buildings/data-centres-and-data-transmission-networks
• JohannesWelbl (2017)MattGardner JohannesWelbl, Nelson F.Liu. 2017.Crowdsourcing Multiple Choice Science Questions.arXiv:1707.06209v1.
• Krishna etal. (2017)Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, DavidA. Shamma, MichaelS. Bernstein, and Li Fei-Fei. 2017.Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations.International Journal of Computer Vision 123 (2017), 32–73.https://doi.org/10.1007/s11263-016-0981-7
• Krizhevsky (2009)Alex Krizhevsky. 2009.Learning multiple layers of features from tiny images.Technical Report.
• Lacoste etal. (2019)Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019.Quantifying the carbon emissions of machine learning.arXiv preprint arXiv:1910.09700 (2019).
• Lakim etal. (2022)Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah, and Julien Launay. 2022.A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models. Association for Computational Linguistics, virtual+Dublin, 84–94.https://doi.org/10.18653/v1/2022.bigscience-1.8
• Leopold (2019)George Leopold. 2019.AWS to Offer NVIDIA’s T4 GPUs for AI Inferencing.www.hpcwire.com/2019/03/19/aws-upgrades-its-gpu-backed-ai-inference-platform/
• Lin etal. (2014)Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’ar, and CLawrence Zitnick. 2014.Microsoft COCO: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 740–755.
• Luccioni and Hernandez-Garcia (2023)AlexandraSasha Luccioni and Alex Hernandez-Garcia. 2023.Counting carbon: A survey of factors influencing the emissions of machine learning.arXiv preprint arXiv:2302.08476 (2023).
• Luccioni etal. (2022)AlexandraSasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. 2022.Estimating the carbon footprint of BLOOM, a 176B parameter language model.arXiv preprint arXiv:2211.02001 (2022).
• Maas etal. (2011)AndrewL. Maas, RaymondE. Daly, PeterT. Pham, Dan Huang, AndrewY. Ng, and Christopher Potts. 2011.Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, 142–150.http://www.aclweb.org/anthology/P11-1015
• Merity etal. (2016)Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.Pointer Sentinel Mixture Models.arXiv:1609.07843[cs.CL]
• Muennighoff etal. (2022)Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, TevenLe Scao, MSaiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, etal. 2022.Crosslingual generalization through multitask finetuning.arXiv preprint arXiv:2211.01786 (2022).
• Narayan etal. (2018)Shashi Narayan, ShayB. Cohen, and Mirella Lapata. 2018.Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization.ArXiv abs/1808.08745 (2018).
• Oremus (2023)Will Oremus. 2023.AI chatbots lose money every time you use them. That is a problem.Washington Post (2023).https://www.washingtonpost.com/technology/2023/06/05/chatgpt-hidden-cost-gpu-compute/
• Ortiz Su’arez etal. (2019)PedroJavier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019.Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures (Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019), Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald L”ungen, and Caroline Iliadi (Eds.). Leibniz-Institut f”ur Deutsche Sprache, Mannheim, 9 – 16.https://doi.org/10.14618/ids-pub-9021
• Pan etal. (2017)Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.Cross-lingual Name Tagging and Linking for 282 Languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1946–1958.https://doi.org/10.18653/v1/P17-1178
• Pang and Lee (2005)Bo Pang and Lillian Lee. 2005.Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL.
• Patterson etal. (2022)David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2022.The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink.https://doi.org/10.48550/ARXIV.2204.05149
• Patterson etal. (2021)David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021.Carbon emissions and large neural network training.arXiv preprint arXiv:2104.10350 (2021).
• Raffel etal. (2019)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu. 2019.Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.arXiv e-prints (2019).arXiv:1910.10683
• Rajpurkar etal. (2018)Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.Know What You Don’t Know: Unanswerable Questions for SQuAD.arXiv:1806.03822[cs.CL]
• Rajpurkar etal. (2016)Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.SQuAD: 100,000+ Questions for Machine Comprehension of Text.arXiv:1606.05250 (2016).arXiv:1606.05250
• Russakovsky etal. (2015)Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, AlexanderC. Berg, and Li Fei-Fei. 2015.ImageNet Large Scale Visual Recognition Challenge.International Journal of Computer Vision (IJCV) 115, 3 (2015), 211–252.https://doi.org/10.1007/s11263-015-0816-y
• Santana (2023)Gustavo Santana. 2023.Stable Diffusion Prompts.https://huggingface.co/datasets/Gustavosta/Stable-Diffusion-Prompts
• Schmidt etal. (2021)Victor Schmidt, Kamal Goyal, Aditya Joshi, Boris Feld, Liam Conell, Nikolas Laskaris, Doug Blank, Jonathan Wilson, Sorelle Friedler, and Sasha Luccioni. 2021.CodeCarbon: Estimate and Track Carbon Emissions from Machine Learning Computing.
• Socher etal. (2013)Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, ChristopherD. Manning, Andrew Ng, and Christopher Potts. 2013.Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, 1631–1642.https://www.aclweb.org/anthology/D13-1170
• Strubell etal. (2019)Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.Energy and policy considerations for deep learning in NLP.arXiv preprint arXiv:1906.02243 (2019).
• Tjong KimSang and DeMeulder (2003)ErikF. Tjong KimSang and Fien DeMeulder. 2003.Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003. 142–147.https://www.aclweb.org/anthology/W03-0419
• US Environmental Protection Agencyy (2024)US Environmental Protection Agencyy. 2024.Greenhouse Gases Equivalencies Calculator - Calculations and References.https://www.epa.gov/energy/greenhouse-gases-equivalencies-calculator-calculations-and-references
• VonWerra etal. (2022)Leandro VonWerra, Lewis Tunstall, Abhishek Thakur, AlexandraSasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, Helen Ngo, etal. 2022.Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurement.arXiv preprint arXiv:2210.01970 (2022).
• Wang etal. (2019)Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and SamuelR Bowman. 2019.SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.arXiv preprint arXiv:1905.00537 (2019).
• Wang etal. (2022)ZijieJ. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and DuenHorng Chau. 2022.DiffusionDB: A Large-Scale Prompt Gallery Dataset for Text-to-Image Generative Models.arXiv:2210.14896 [cs] (2022).https://arxiv.org/abs/2210.14896
• Wolf etal. (2019)Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, etal. 2019.Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771 (2019).
• Workshop etal. (2022)BigScience Workshop, TevenLe Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, AlexandraSasha Luccioni, François Yvon, etal. 2022.BLOOM: A 176B-parameter open-access multilingual language model.arXiv preprint arXiv:2211.05100 (2022).
• Wu etal. (2021)Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, FionaAga Behram, James Huang, Charles Bai, etal. 2021.Sustainable AI: Environmental Implications, Challenges and Opportunities.arXiv preprint arXiv:2111.00364 (2021).
• Xu etal. (2023)Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023.ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation.arXiv:2304.05977[cs.CV]
• Zhu etal. (2015)Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In The IEEE International Conference on Computer Vision (ICCV).

Appendix A Full list of task-specific models tested

Appendix B Model Evaluation