LoRA: Low-Rank Adaptation of Large Language Models

Doi, I’m Siddharth Gautam, a blockchain dev in a love/hate relationship with Rust 😞. This blog is my interpretation of Sarva Dubey’s presentation on LoRA, so proceed with caution.

Why should you care?

LLMs are trained on huge datasets that cover a wide variety of topics. Now let’s say I want a model that specializes in a particular domain, then the traditional solution is fine-tuning, where we take the trained model and continue training it on the new task.

But if we use full fine-tuning, all the trainable parameters of the trained model are updated using the new dataset. For 100 different datasets, the storage and deployment requirements become huge as we need to store all parameters for every specific task.

Due to this, we can use: Parameter-Efficient Fine-Tuning (PEFT) methods, such as Adapters (freeze the original model and insert small trainable neural network modules into it), Prefix tuning (freeze the original model and learn a trainable prefix that influences attention at multiple layers), etc that are more efficient. However, they also have certain drawbacks:

LoRA or Low-Rank Adaptation is one of the PEFT methods and says:

Keep the original model frozen and learn a much smaller low-rank update that represents how the model should change for the new task.

I’ll try to explain what this means in subsequent sections. If the blog got too boring and you decided to skip rn, just know that Shubhang Sinha will be arriving at your doorstep shortly if you do so.

The Core Hypothesis: Intrinsic Rank 

Say we have a matrix, then the maximum number of rows/columns that are (not closely related) || (linearly independent) gives us the rank of the matrix. A row or column is linearly independent if it cannot be expressed as a linear combination of other rows/colums. Also, low rank doesn’t mean that a matrix contains little information as rank does not measure the amount of information.

A model may have billions of parameters, but adapting it to a new task might require changing its weights in only a few independent directions related to the task. Intrinsic rank is a concept used to describe how many independent directions of change are needed to achieve a useful result (adapting to a new task in this scenario). 

Introducing LoRA

The idea behind LoRA is simple: instead of updating all the weights of an already trained model, we freeze the original weights and introduce two smaller, trainable matrices related to our task: A and B, to learn the required adjustments. In full fine-tuning, taking W₀ as the pretrained weight matrix and ΔW as the change to the weight matrix, we get: 

                                        W = W₀ + ΔW 

Multiplying both sides by x(the input matrix):

           

                                        y = W₀x + ΔWx

LoRA proposes that instead of training the full matrix ΔW, we use two smaller matrices:

                                       ΔW = BA

Suppose W₀ is a matrix of size 1000×1000. Updating it directly requires training one million parameters. But using smaller matrices B and A, with dimensions:

                                       A:10×1000,  B:1000×10

Choosing r=10 means that the update can have a rank of at most 10, it does not necessarily have a rank of exactly 10. Multiplying A,B gives:

                                       BA : (1000×10)(10×1000) = 1000×1000

The result has the same dimensions as W₀, but training A and B requires only 20,000 parameters instead of 1,000,000 (as A and B have 10,000 parameters each). The number 10 here is the chosen rank r used, a hyperparameter. A hyperparameter is a setting we can choose to control how a ML model is trained. In LoRA, we choose the rank r, which determines the dimensions of A and B.

The paper's full equation includes a scaling factor:

  h = W₀x + α/r * BAx

The factor α/r scales the LoRA update relative to the original weights. Here, r is the rank hyperparameter, and α is a scaling hyperparameter. The scaling helps control how strongly the learned update contributes to the output.

Inference Latency

Inference is using a trained model to generate an output. Inference latency is the time taken to produce that output, or the relevant measured portion of that process.

During training, the LoRA layer computes:

h = W₀x +  α/r * BAx    

There are two paths here, the original weight matrix which computes W₀x, and the LoRA matrices that compute BAx. We can simply add the results.

If we kept the paths separate during deployment, the LoRA branch would introduce additional computations. To avoid this, the paths are merged:

                                 W₀x + α/r*BAx = (W₀+α/r*BA)x.

Let

                                  W_merged =  W₀+ α/r*BA

Then the output becomes:

                                   h = W_mergedx

If we include the scaling factor:

                                   W_merged = W₀ + α/r * BA.

Now the system can use a single merged weight matrix instead of calculating the original and LoRA paths separately.

Why does this matter?

During training, keeping the paths separate is better because we need to learn the adapter parameters A and B, without changing the original model weights W₀. But after training, merging both paths is often helpful as otherwise the model would have to perform the computations separately and then add them during inference, leading to extra computations.

Merging also takes computation but it only needs to perform the merging computation once when preparing the model for deployment, rather than repeatedly during inference. This is helpful for reducing repetitive tasks.

It does not mean inference itself takes zero milliseconds. It means the merging operation avoids adding any extra latency. We can also keep the adapter separate and calculate its contribution during inference. This allows switching adapters easily without merging the weights again.

Switching Between Tasks

Imagine one base model needs to perform two tasks: summarization and translation. 

Instead of maintaining two complete fine-tuned models, with LoRA we just need one shared, frozen base model and we can just keep switching adapters for different tasks when needed. In this scenario, we can have one LoRA adapter for summarization and another for translation. 

Task switching can be done by switching adapters, without storing a complete copy of the model for each task.

Does LoRA Actually Deliver?

The paper evaluated LoRA on models and tasks such as RoBERTa, GPT-2, and GPT-3. It showed that LoRA can achieve performance comparable to or better than full fine-tuning while training far fewer parameters. This doesn’t mean that LoRA always outperforms full fine-tuning; results depend on the model, task and configuration.

Can a Very Small Rank Work?

Rank determines the size and learning capacity of the LoRA update.  We might think that a very small rank is computationally cheap but may not capture enough task specific information due to small size.

The paper finds that increasing rank does not necessarily improve performance. A small rank can already work well.

Which Weights Should LoRA Adapt?

We know that LoRA is awesome. But there is another question: which weight matrices in a Transformer(not Hasbro, just google if you're new to this) should we apply LoRA to? Should all of them use LoRA?

A typical decoder-only Transformer contains multiple Transformer blocks stacked on top of each other. Each block has two major components:

1. Self-attention: Helps the model determine which information from other tokens is relevant to the current token.
2. Feed-forward network (FFN): Processes and transforms the information represented by each token.

The attention mechanism uses several weight matrices in its attention mechanism like W_q(query weights), W_v(value weights), W_o(output projection weights), W_k(key weights). Applying LoRA to every matrix introduces a lot of additional parameters which would take up more memory and add more compute. To manage these costs when we are working on limited resources, we might have to reduce the rank or change the configuration.

In the WikiSQL experiment it was also seen that applying LoRA to: W_q alone with a LoRA rank 8 gave 70.4% WikiSQL accuracy, W_v alone with LoRA rank 8 gave 73.0% WikiSQL accuracy, and both W_q and W_v  with LoRA rank 4 each achieved a score of 73.7% under the tested parameter budget. When the authors adapted all four attention matrices, they obtained the same WikiSQL accuracy: 73.7%.

However, the MultiNLI results differed: the four-matrix configuration achieved 91.7%, compared with 91.3% for the W_q + W_v configuration, with a small improvement. Choosing where to apply LoRA matters. The best configurations can be found through trying more combinations and figuring out what works best, there is no universal rule.

Subspace Similarity and Feature Amplification

When LoRA adapts a pretrained model, it may learn to strengthen specific features the model already knows, rather than learning everything from scratch.

Let’s say a pretrained language model can recognise things like grammar, relationships between words, programming syntax, etc. Each of these abilities can be thought of as patterns in the model's weights. Mathematically, we can think of groups of directions in those weights as a subspace.

What is SVD and how does it help?

Singular Value Decomposition, or SVD, is a mathematical technique for analyzing matrices and identifying the directions that contribute most to a matrix's behavior. For a matrix say W, SVD gives:

                                  W=UΣVT.

The matrices U and V describe directions associated with the transformation process, while Σ contains the singular values that indicate the strength of those directions. 

After comparing the subspaces learned under different rank settings and training runs, it was seen that leading directions can overlap across different ranks. This means that increasing the rank does not necessarily uncover an entirely new collection of equally important directions. Some of the most useful structures may already be captured at a smaller rank.

The graph depicts this relationship across model layers and attention projections.

The paper says that LoRA's update can amplify certain task related directions that already exist in the pretrained weights, but are not emphasized as strongly. Again, a pretrained model has already learned many useful features. A task may need only some of them to become more influential. The adaptation can then modify the model's behavior by strengthening selected directions rather than reconstructing everything from scratch.

An example of this is when in a particular setup, an amplification factor of approximately 20 was observed for a particular rank-4 analysis, where it was detected that some direction is directly related to the task. That number will vary for different scenarios, but helps demonstrate the working.

What is the key insight?

LoRA is not just a way to store fewer parameters. The update may also reveal something about how pretrained models can be adapted efficiently.

A large model may already contain useful task relevant features, and a compact update can change how strongly those features influence its behavior.

To learn more

---