Performance Comparison of Tensor Decomposition on LLM

초록

This study explores the use of tensor decomposition techniques, specifically canonical polyadic decomposition (CPD) and Tucker decomposition, for optimizing transformer-based large language models (LLMs). By focusing on feed-forward layers (FFNs) and the final fully connected layer, we evaluate the impact of these decompositions on memory efficiency, computational performance, and overall model behavior. Through experiments on NSMC and Tiny Shakespeare datasets, we analyze the trade-offs between compression, training time, and accuracy for different rank settings. Our results reveal that Tucker decomposition provides a balanced approach to compression and computational efficiency, while CPD offers competitive performance but with higher computational costs. This work provides valuable insights into deploying tensor decomposition techniques for LLM optimization in resource-constrained environments.

키워드