首页|Expert-level policy style measurement via knowledge distillation with large language model collaboration

Expert-level policy style measurement via knowledge distillation with large language model collaboration

扫码查看
原文链接
  • NETL
  • NSTL
  • Elsevier
Policy style is a crucial concept in policy science that reflects persistent patterns in the policy process across different governance settings. Despite its importance, policy style measurement faces issues of complexity, subjectivity, data sparseness, and computational cost. To overcome these obstacles, we propose KOALA, a novel KnOwledge distillation framework based on large lAnguage modeL collAboration. It transforms the weak scoring abilities of LLMs into a pairwise ranking problem, employs a small set of expert-annotated samples for non-parametric learning, and utilizes knowledge distillation to transfer insights from LLMs to a smaller, more efficient model. The framework incorporates multiple LLM-based agents (Prompter, Ranker, and Analyst) collaborating to comprehend complex measurement standards and self-explain policy style definitions. We validate KOALA on 4,572 Chinese government work reports (1954-2019) from central, provincial, and municipal levels, with a focus on the imposition dimension of policy style. Extensive experiments demonstrate KOALA's effectiveness in measuring the intensity of policy style, highlighting its superiority over state-of-the-art methods. While GPT-4 achieves only 66% accuracy in pairwise ranking of policy styles, KOALA, despite being based on GPT-3.5, achieves a remarkable 85% accuracy, highlighting significant performance improvement. This framework offers a transferable approach for quantifying complex social science concepts in textual data, bridging computational techniques with social science research.

Policy styleKnowledge distillationLLM collaboration

Yujie Zhang、Biao Huang、Weikang Yuan、Zhuoren Jiang、Longsheng Peng、Shuai Chen、Jie-Sheng Tan-Soo

展开 >

Department of Information Resources Management, School of Public Affairs, Zhejiang University, 866 Yuhangtang Rd, Xihu, Hangzhou, 310058, Zhejiang, China

School of Public Affairs, Academy of Social Governance, Zhejiang University, 866 Yuhangtang Rd, Xihu, Hangzhou, 310058, Zhejiang, China

China Academy for Rural Development (CARD) and School of Public Affairs, Zhejiang University, 866 Yuhangtang Rd, Xihu, Hangzhou, 310058, Zhejiang, China

Lee Kuan Yew School of Public Policy, National University of Singapore, 469C Bukit Timah Road, 259772, Singapore

展开 >

2025

10.1016/j.ipm.2025.104090

Information processing & management

Information processing & management

ISSN：0306-4573
年,卷(期)：2025.62(4)
  • 83