Scaling- Transformers, Laws and Challenges

This article was published as a part of the Data Science Blogathon.

Introduction

The other day, I was reading “Natural Language Processing with Transformers” a book authored by Lewis Tunstall, Leandro von Werra, and Thomas Wolf. In it, I came across Scaling laws and challenges associated with scaling Transformer models. This topic also included excerpts from Richard Sutton’s startling essay titled “The Bitter Lesson,” which suggested that embedding/relying on a human-knowledge approach can complicate things, making them less suited to taking advantage of general methods leveraging computation. The mention of embedding human knowledge piqued my interest to delve deeper into the matter. It reminded me of a conversation I had with a respectable AI Researcher on Twitter earlier this year.

         Source: Canva

In this post, we will be exploring:

• What are the potential drawbacks of embedding human knowledge of the domain directly into the models than leveraging the computation?

• Why is scaling required?

• What are scaling laws?

• What are the Challenges with scaling transformers?

Now let’s dive in!

Drawbacks of encoding human knowledge directly into the model

Following are the potential drawbacks of embedding human knowledge directly into the model than leveraging the computation:

1) Leveraging human knowledge leads to quicker gain. Still, in the long run, methods leveraging computation outperform:   In 2019, the researcher Richard Sutton published a startling essay titled “The Bitter Lesson,“ in which he stated general methods that leverage computation are the most effective, and by a large margin. Leveraging human knowledge of the domain can offer quick gains in the short term. However, leveraging the computation can outperform the former in the long run.  Furthermore, the human-knowledge approach often complicates methods, making them less suited to taking advantage of general methods leveraging computation.

In games like chess or Go, the concept of encoding human knowledge within AI systems was eventually outdone by greater computation. Sutton refers to this as the “bitter lesson” for the AI research community. There are signs that a similar lesson is at play with transformers. At the same time, many early BERT and GPT descendants focused on tweaking the architecture or pretraining objectives. The best-performing models like GPT-3 are basically scaled-up versions of the original models without many architectural modifications.

2) Human biases: If we decide on encoding the human knowledge directly into the model, we might also be encoding our biases. Biases could be inflicted from a person’s experience/knowledge and discretion. So, data curation is a highly subjective and sensitive matter. Furthermore, even the entire ML pipeline can be biased as well. However, utilizing a massive data pool gives us a semblance of being more unbiased than hardwiring human knowledge directly.

3). Exploration Vs. Exploitation: While studying Reinforcement Learning, I came across this concept that resonated with me. Although we are talking about the Transformers specifically, this concept can also be applied here. While exploiting our existing knowledge can provide us the semblance of quick gains, exploration by leveraging vast datasets and scaling the model could lead to long-term benefits and help us explore the unexplored.

4). Is our existing understanding of various domains absolutely correct? : The dilemma is, is our existing knowledge perfect enough that we can solely rely on that and avoid further exploration? Often, even scientists do not conclude and claim something as “absolute truth.” They often cite the results saying something like, based on these (limited set of) observations. It appears to be […….]. Furthermore, the scientific community may hold opposing views based on their different experiment outcomes and limited observations.

Why is Scaling required?

Empirical evidence suggests that large language models perform better on downstream tasks, and capabilities such as zero-shot and few-shot learning emerge in the 10 to 100 B parameter range. However, the number of parameters is not the only key factor affecting the model’s performance. The performance of language models seems to obey a power law relationship with model size and other factors. Hence, the amount of computing and training data need to be scaled in tandem to train these models.

In fig. 1, we can see a timeline of the development of the largest models since the release of the original Transformer architecture in 2017,  illustrating that model size has increased by over four orders of magnitude in just a few years!

Fig. 1 Parameter counts over time for prominent Transformer architectures

(Source: https://bit.ly/3Ovuvdy)

Scaling Laws

Scaling laws allow empirically quantifying the “bigger is better” paradigm for language models by studying their behavior with varying compute budgets C, dataset size D, and model size N. The key idea is to plot the dependence of the cross-entropy loss L on these three factors and see if a relationship emerges. For autoregressive models like those in the GPT family, the resulting loss curves are shown in Fig. 2, where each blue curve represents the training run of a single model.

We can infer a few things from these loss curves:

1. The relationship between performance and scale: Although hyperparameter optimization is typically emphasized to improve performance on a fixed set of datasets, scaling laws suggest a more productive path toward better models is to focus on increasing N, C, and D in tandem.

2. Smooth Power Laws: The test loss L has a power law relationship with each of N, C, and D across several orders of magnitude (power law relationships are linear on a log-log scale). These power-law relationships can be expressed as follows:

where, : scaling exponent

Typical values lie in the 0.05–0.095 range, and one salient feature of these power laws is that the early part of a loss curve can be extrapolated to predict the approximate loss if the training was conducted for much longer.

3. Sample efficiency: Large models can achieve the same performance as smaller models with fewer training steps. The loss curve plateaus over some training steps, indicating that one gets diminishing returns in performance compared to simply scaling up the model. Fig. 3 illustrates the scaling laws for modalities like Images, videos, mathematical problem solving, language, etc.

It is still unclear whether power-law scaling is a universal property of transformer language models. And given the cost constraint, it’s extremely desirable to have an estimate of the model’s performance in advance. Presently, scaling laws are used as a tool to extrapolate large, expensive models without explicitly training them. However, scaling isn’t as simple as it sounds. Let’s take a look at some of the challenges with scaling.