Pretrained language models (LMs) are usually fine-tuned to adapt them to new domains or tasks, a process known as micro-tuning. While refinement allows adaptation to various functions with small amounts of in-domain data, it can be prohibitively expensive for large LMs.

Parameter yield fine-tuning (PEFT) methods offer a solution by updating only a fraction of the weights, reducing memory usage and training time. Adapters, a common PEFT approach, learn processings that can be added to a subset of model weights or run alongside the frozen base model. Recent developments such as LoRA and its variants reduce the number of trainable parameters by using low-rank approaches during adapter training.

However, an important aspect of current PEFT methods is their focus on modifying weights rather than representations, despite previous research showing that representations encode rich semantic information. Representation Finetuning (ReFT) methods have been proposed as an answer to this by a team of researchers from Stanford and the Pr(Ai)2R Group.

Instead of adjusting model weights, ReFT methods train interventions to handle a small fraction of model representations, guiding model behaviors to solve top-down tasks in inference time. Their approach draws inspiration from recent work in LM hermeneutics, which intervenes in representations to identify causal mechanisms and direct model behaviors at inference time.

A notable example of the ReFT family is Low-rank Linear Subspace ReFT (LoReFT), which interpolates hidden representations in the linear subspace spanned by a low-rank projection matrix. LoReFT builds directly on existing methods such as distributed alignment search (DAS), demonstrating cutting-edge performance across several benchmarks while using significantly fewer parameters than traditional PEFT methods. Their results suggest that ReFT methods offer more efficient and effective alternatives to weight-based PEFTs, which deserve further exploration in different model families and domains.

Future research directions for ReFT include investigating its effectiveness on other model families and visual language models, and automating the search for hyperparameters. Additionally, exploring more effective task-specific interventions and investigating the power of learned orthogonal subspaces are areas of interest. ReFT advances neural network interpretation research and contributes knowledge to the field by challenging traditional approaches to interpreting single neurons.

In terms of assessment practices, it is important to establish benchmarks that allow fair comparisons of PEFT and ReFT, including comparisons of hyperparameter tuning calculation or temporal matching and no tuning or model selection based on the test set to mitigate overfitting and ensure true assessments of global performance.

check it Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us Twitter. Join us Telegram channel, Discord Channeland LinkedIn Groops.

If you like our work, you will love our work newsletter..

Don’t forget to join us 40k+ ML SubReddits

Want to get in front of 1.5 million AI audience? Work with us here