Illumination variation,reflection inequality and limited features existing in high-speed rail surface images make the automated visual inspection task extremely difficult.In order to separate the defects from complex rail surface images in high speed motion process,according to the fact that the rail surface image has the characteristic of basically unchanged pixel value along the rail direction,the background model of the rail surface image is established.A rail surface defect detection algorithm based on background differencing is proposed.The algorithm contains four steps:extracting rail region,background modeling and differencing,threshold segmentation and image filtering.The main feature of the algorithm is extending the background differencing algorithm in video surveillance to defect image segmentation.By means of adaptive threshold segmentation and image filtering technology,the influences of image illumination variation,reflection inequality,limited features and other negative factors in the rail surface defect segmentation are decreased in certain degree.The simulation and field experiment results indicate that the proposed method can identify the block shape rail surface defects effectively.The recall and precision can reach 96% and 80.1%,respectively.