publications

2025

1. ICLR

   AdaGrad under Anisotropic Smoothness

   Yuxing Liu^*, Rui Pan^*, and Tong Zhang

   ICLR, 2025
2. ICLR

   Personalized visual instruction tuning

   Renjie Pi, Jianshu Zhang, Tianyang Han, and 3 more authors

   ICLR, 2025

2024

1. NeurIPS

   

   LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

   Rui Pan^*, Xiang Liu^*, Shizhe Diao, and 4 more authors

   NeurIPS, 2024

   Abs PDF

   The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significant roadblock to large-scale training. For instance, a 7B model typically requires at least 60 GB of GPU memory with full parameter training, which presents challenges for researchers without access to high-resource environments. Parameter Efficient Fine-Tuning techniques such as Low-Rank Adaptation (LoRA) have been proposed to alleviate this problem. However, in most large-scale fine-tuning settings, their performance does not reach the level of full parameter training because they confine the parameter search to a low-rank subspace. Attempting to complement this deficiency, we investigate the layerwise properties of LoRA on fine-tuning tasks and observe an unexpected but consistent skewness of weight norms across different layers. Utilizing this key observation, a surprisingly simple training strategy is discovered, which outperforms both LoRA and full parameter training in a wide range of settings with memory costs as low as LoRA. We name it Layerwise Importance Sampled AdamW (LISA), a promising alternative for LoRA, which applies the idea of importance sampling to different layers in LLMs and randomly freezes most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning in downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over 10%-35% in terms of MT-Bench score while achieving on-par or better performance in MMLU, AGIEval and WinoGrande. On large models, specifically LLaMA-2-70B, LISA surpasses LoRA on MT-Bench, GSM8K, and PubMedQA, demonstrating its effectiveness across different domains.

2. NeurIPS

   Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions

   Renjie Pi, Jianshu Zhang, Jipeng Zhang, and 3 more authors

   NeurIPS, 2024
3. ICLR

   Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise

   Rui Pan^*, Yuxing Liu^*, Xiaoyu Wang, and 1 more author

   ICLR, 2024
4. ECCV

   Strengthening multimodal large language model with bootstrapped preference optimization

   Renjie Pi, Tianyang Han, Wei Xiong, and 4 more authors

   ECCV, 2024
5. ACL

   Active prompting with chain-of-thought for large language models

   Shizhe Diao, Pengcheng Wang, Yong Lin, and 3 more authors

   ACL, 2024
6. ACL

   Plum: Prompt Learning using Metaheuristic

   Rui Pan, Shuo Xing, Shizhe Diao, and 6 more authors

   ACL Findings, 2024
7. EMNLP

   MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance

   Renjie Pi, Tianyang Han, Jianshu Zhang, and 6 more authors

   EMNLP, 2024
8. EMNLP

   The Instinctive Bias: Spurious Images lead to Hallucination in MLLMs

   Tianyang Han^*, Qing Lian^*, Rui Pan^*, and 5 more authors

   EMNLP, 2024
9. EMNLP

   Theoremllama: Transforming general-purpose llms into lean4 experts

   Ruida Wang, Jipeng Zhang, Yizhen Jia, and 4 more authors

   EMNLP, 2024
10. EMNLP

    Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models

    Yong Lin, Lu Tan, Hangyu Lin, and 8 more authors

    EMNLP, 2024
11. NAACL

    

    LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models

    Shizhe Diao^*, Rui Pan^*, Hanze Dong^*, and 4 more authors

    NAACL, 2024

    Best Demo Abs PDF Code

    The LMFlow team received NAACL 2024 Best Demo Award at June 16-21, Mexico City, Mexico. Please refer to the official website for more details.

    Foundation models have demonstrated a great ability to achieve general human-level intelligence far beyond traditional approaches. As the technique keeps attracting attention from the AI community, more and more foundation models have become publicly available.However, most of those models exhibit a major deficiency in specialized-domain and specialized-task applications, where the step of domain- and task-aware finetuning is still required to obtain scientific language models. As the number of available foundation models and specialized tasks keeps growing, the job of training scientific language models becomes highly nontrivial. In this paper, we take the first step to address this issue. We introduce an extensible and lightweight toolkit, LMFlow, which aims to simplify the domain- and task-aware finetuning of general foundation models.LMFlow offers a complete finetuning workflow for a foundation model to support specialized training with limited computing resources.Furthermore, it supports continuous pretraining, instruction tuning, parameter-efficient finetuning, alignment tuning, inference acceleration, long context generalization, model customization, and even multimodal finetuning, along with carefully designed and extensible APIs. This toolkit has been thoroughly tested and is available at https://github.com/OptimalScale/LMFlow.

2023

1. TMLR

   Raft: Reward ranked finetuning for generative foundation model alignment

   Hanze Dong, Wei Xiong, Deepanshu Goyal, and 7 more authors

   TMLR, 2023

2022

1. ICLR

   

   Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objects with Skewed Hessian Spectrums

   Rui Pan^*, Haishan Ye^*, and Tong Zhang^*

   ICLR, 2022

   Abs PDF

   Learning rate schedulers have been widely adopted in training deep neural networks. Despite their practical importance, there is a discrepancy between its practice and its theoretical analysis. For instance, it is not known what schedules of SGD achieve best convergence, even for simple problems such as optimizing quadratic objectives. In this paper, we propose Eigencurve, the first family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed. The condition is quite common in practice. Experimental results show that Eigencurve can significantly outperform step decay in image classification tasks on CIFAR-10, especially when the number of epochs is small. Moreover, the theory inspires two simple learning rate schedulers for practical applications that can approximate eigencurve. For some problems, the optimal shape of the proposed schedulers resembles that of cosine decay, which sheds light to the success of cosine decay for such situations. For other situations, the proposed schedulers are superior to cosine decay.