Thoughts on AI & Energy

Every action requires energy. This principle is foundational in both the physical world of atoms and the digital world of bytes.

Physical activities such as running, playing musical instruments, or even sleeping consume energy, which translates to calories for humans. Similarly, in the digital world, data centers depend on substantial amounts of energy to function, measured in kilowatts for components like semiconductors.

The energy required for any action scales with its duration. Running a marathon exemplifies this. The endurance needed to complete such a feat hinges on the energy available to the body. With enough caloric energy, finishing the race becomes possible. This relationship is quantifiable, often measured as calories burned per hour, providing a tangible metric for endurance.

Consider operating a large-scale website like Google in the digital world. Beyond the extensive infrastructure of servers and silicon, the critical requirement is energy. Large data centers, with their immense scale, consume significant amounts of electricity.

Training an AI model, particularly a large language model (LLM), is an intensive process that involves multiple vital components:

• Training Data: High-quality data is essential. It’s not just about the volume of data but the relevance and accuracy. Quality data ensures the model learns effectively to produce reliable tokens during inference.

• Specialized Silicon: GPUs (Graphics Processing Units) are preferred over CPUs (Central Processing Units) due to their superior ability to handle the massive computations required for training models. This specialized silicon accelerates the learning process.

• Energy: The training process consumes significant energy. Large-scale models require substantial kilowatt-hours to train effectively.

• Time: The duration of training impacts the model’s performance. Extended training periods, assuming high-quality data, typically result in more robust and accurate models.

Post-training, AI models are applied in real-world scenarios to perform tasks such as generating text, classifying images, or creating art. This phase is known as inference. Inference utilizes the trained model to make predictions or perform specific tasks, relying on the same two critical resources:

1. Specialized Silicon: Inference, like training, benefits from GPUs due to their computational efficiency.

2. Energy: Running inference at scale demands considerable energy, especially for complex tasks.

Performance metrics for inference are evaluated regarding tokens generated per second for individual GPUs and clusters. Efficiency in this context is measured by the ratio of tokens generated per hour to kilowatt-hours consumed.

The AI landscape is highly competitive, with companies like OpenAI, Anthropic, Google, Meta, and Microsoft vying for dominance and spending billions of dollars on computing and talent. However, recent trends indicate a convergence in model performance:

• Across the board, AI models are exhibiting similar performance levels.

• Groundbreaking advancements, akin to the leap seen with GPT-3 and GPT-4, are becoming less frequent.

This convergence in model performance can be attributed to several factors:

• Major companies are achieving parity in their approach to AI development. They draw from similar pools of resources, including energy, specialized silicon, and data sets.

• This homogeneity extends to inference, where these companies deploy comparable amounts of energy and similar silicon stacks, leading to uniformity in performance outcomes.

Looking ahead, the pursuit of capital efficiency is likely to drive significant innovations in AI:

• Increased Tokens Relative to Kilowatts per Hour: As silicon technology advances, efficiency improvements are inevitable. This progression aligns with Moore’s Law, which predicts a continual increase in computational power.

• Cheaper and More Abundant Power Supply: The cost and availability of power are critical factors. Advances in energy production, such as nuclear energy or renewable sources, could significantly reduce the cost of kilowatt-hours, enabling more extensive and more efficient model training and inference operations.

• Improved Token Quality: Access to high-quality data and substantial investments in fine-tuning models will enhance token quality. We see early indications of this with specialized models like Microsoft’s Phi, which are tailored for specific applications and demonstrate superior performance.

• Optimization of Transformer Architectures: Further innovation in transformer architecture and experimental silicon designs will yield performance improvements. This will involve optimizing software and hardware to achieve greater efficiency and capability.

Over time, advancements will diffuse from large, centralized data centers to consumer devices. The smartphone in your pocket today surpasses the capabilities of data centers from two decades ago. It's a testament to the rapid pace of technological progress and the increasing democratization of advanced computational power. It’s easy to imagine the future smartphone generating tokens per second at 1-2X faster than today’s.

However, this transition is not automatic. It will require substantial innovation in several key areas:

• Training Data: Continuously sourcing and curating high-quality training data remains a fundamental challenge. Effective data management and refinement are crucial for maintaining model accuracy and relevance. The companies with the most money will pay for the best datasets.

• State-of-the-Art Silicon: Hardware breakthroughs will be essential to support the growing demands of AI applications. Innovations in semiconductor technology, including more efficient and powerful GPUs, will play a critical role.

• Energy Consumption: As AI models and their applications scale, energy demands will increase. Sustainable and efficient energy solutions (e.g., nuclear) are necessary to meet these needs without imposing prohibitive costs or environmental impacts.

• Algorithm Development: Implementing cutting-edge algorithms will enable more efficient and powerful AI models. Advances beyond the transformer architecture will likely drive the next wave of AI capabilities.

Increasing the global tokens per hour metric will open up new use cases, which could help humanity address critical issues — from curing diseases to preempting cybersecurity threats.

If I were starting my technology career today, the intersection of AI, energy, and technology presents numerous opportunities for impactful contributions. I would try to focus on at least one of the following areas:

• Data Quality Enhancement: Improving the quality and relevance of training data can significantly enhance AI model performance.

• Silicon Technology Advancement: Innovating semiconductor technology to develop more efficient and powerful hardware.

• Energy Efficiency Improvement: Creating sustainable energy solutions to support the growing computational demands of AI.

• Algorithm Development: Pioneering new algorithms that drive efficiency and capability in AI applications.

The next few months and years will be a crucible of innovation and discovery, shaping the future of AI and its impact on society.

See you in the arena.