Symbol-LLM
Symbol-LLM: Towards Foundational Symbol-centric Interface for Large Language Models

Fangzhi Xu1,2, Zhiyong Wu2,*, Qiushi Sun3, Siyu Ren4, Fei Yuan2, Shuai Yuan5, Qika Lin3, Yu Qiao2, Jun Liu1
1Xi'an Jiaotong University 2Shanghai Artificial Intellegence Laboratory
3National University of Singapore 4Shanghai Jiao Tong University 5Hongkong University of Science and Technology *Corresponding Author.
Paper Code arXiv 🤗 Models
symbol-llm

Figure 1: Model structure of Symbol LLM. (a) is two training stages, Injection stage and Infusion stage. (b) is the test stage with three parts of tasks, symbolic tasks, general tasks and Symbol+Delegation setting.

Large Language Models (LLMs) have greatly propelled the progress in natural language(NL)-centric tasks based on NL interface. However, the NL form is not enough for world knowledge. Current works focus on this question by injecting specific symbolic knowledge into LLM, which ignore two critical challenges: the interrelations between various symbols and the balance between symbolic-centric and NL-centric capabilities. In this work, we tackle these challenges from both a data and framework perspective and introduce Symbol-LLM series models. First, we collect 34 symbolic tasks, covering ~20 different forms, which are unified to capture symbol interrelations. Then, a two-stage tuning framework succeeds in injecting symbolic knowledge without loss of the generality ability. Extensive experiments on both symbol- and NL-centric tasks demonstrate the balanced and superior performances of Symbol-LLM series models.

Overall Performances

symbol

Figure 2: Overall performance comparisons.

Results on Symbolic Tasks

symbol

Figure 3: Results on 34 symbolic generation tasks.

Results on General Tasks

symbol

Figure 4: Results on general tasks.

Results on Symbol+Delegation Tasks

symbol

Figure 5: Math Reasoning.

BibTeX

@article{xu2023symbol,
  title={Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models},
  author={Xu, Fangzhi and Wu, Zhiyong and Sun, Qiushi and Ren, Siyu and Yuan, Fei and Yuan, Shuai and Lin, Qika and Qiao, Yu and Liu, Jun},
  journal={arXiv preprint arXiv:2311.09278},
  year={2023}
}