Machine learning methods have the ability to extract features from large amounts of data and dynamically optimize system performance,but there are still many challenges for machine learning to enable system software.This paper mainly surveys the research of machine learning in computer system from two aspects of learning-based operating system and learning-based compiler,discusses the challenges and potential research directions of machine learning enabling system software,and hopes to provide references for researchers.