目的 柑橘是我国最常见的水果之一,目前多以人工采摘为主,成本高、效率低等问题严重制约规模化生产,因此柑橘自动采摘成为近年的研究热点.但是,柑橘生长环境复杂、枝条形态各异、枝叶和果实互遮挡严重,如何精准实时地定位采摘点成为自动采摘的关键.通过构建级联混合网络模型,提出了一种通用且高效的柑橘采摘点自动精准定位方法.方法 构建团簇框生成模型和枝条稀疏实例分割模型,对两者进行级联混合实现实时柑橘采摘点定位.首先,构建柑橘果实检测网络,提出团簇框生成模型,该模型通过特征提取、果实检测框生成和DBSCAN(density-based spatial clustering of applications with noise)果实密度聚类,实时地生成图像内果实数目最多的团簇框坐标;然后,提出融合亮度先验的枝条稀疏分割模型,该模型以团簇框内的图像作为输入,有效降低背景枝条的干扰,通过融合亮度先验的稀疏实例激活图,实时地分割出与果实相连接枝条实例;最后基于分割结果搜索果实采摘点定位坐标.结果 经过长时间户外采集制作了柑橘果实检测数据集CFDD(citrus fruit detection dataset)和柑橘枝条分割数据集CBSD(citrus branch segmentation dataset).这两个数据集由成熟果实、未成熟果实组成,包含晴天、阴天、顺光和逆光等挑战,总共37 000幅图像.在该数据集上本文方法的采摘点定位精准度达到了 95.77％,帧率(frames per second,FPS)达到了 28.21帧/s.结论 本文方法在果实采摘点定位方面取得较好进展,能够快速且准确地获取柑橘采摘点,并且提供配套的机械臂采摘设备可供该采摘点定位算法的落地使用,为柑橘产业发展提供有力支持.
Real-time citrus picking point localization guided by joint learning network
Objective Citrus is one of the most common fruits in our country.At present,it is mostly picked by hand,but issues such as high cost and low efficiency severely restrict the scale of production.Therefore,automatic citrus picking has become a research hotspot in recent years.However,the growing environment of citrus is complex,its branch has different shapes,and the branches,leaves,and fruits are seriously shielded from each other.Accurate and real-time location of the picking point becomes the crucial aspect of automated picking.Currently,research on fruit picking point localization meth-ods can be broadly categorized into two types:nondeep learning-based methods and deep learning-based methods.Non-deep learning-based methods mainly rely on digital image processing techniques,such as color space conversion,threshold segmentation,and watershed algorithm,to extract target contours and design corresponding algorithms for picking point localization.However,these methods often suffer from low accuracy and efficiency.Deep learning-based methods involve training deep learning models to perform tasks,such as detection or segmentation.The model's output is then used as an intermediate result,and specific algorithms are designed on the basis of fruit growth characteristics and task requirements to achieve fruit picking point localization.These methods offer increased accuracy and real-time capabilities,making them a recent research focus.As a result,researchers are currently more focused on the application of deep learning methods in this area.However,most of the existing picking point localization methods have limitations in practical applications,often constrained by the design of end effectors and idealized application scenarios.Therefore,this study conducted a series of research to propose a universal and efficient method for locating fruit picking points,aiming to overcome the limitations of existing methods.Method This study proposes a framework for generating cluster bounding boxes and sparse instance seg-mentation of citrus branches and combines them in a cascade model to achieve real-time localization of citrus picking points.This method is mainly implemented in three steps.In the first step,the image within the picking field of view is input into a fruit object detector based on a feature extraction module(CSPDarknet)and a path aggregation network(PAN-net).Multiple fruit object detection boxes are predicted through multilevel detection.These boxes are then clustered using a cluster box generator,and the cluster box with the most amount of citrus is selected,and its coordinates are calculated.In the second step,the image patches inside the cluster box are extracted and input to a branch sparse segmentation model.This step further focuses on segmenting the branch region,reducing background interference.Brightness priors are added to guide the weights of instance activation maps.The feature decoder learns the branch instance segmentation result.In the third step,on the basis of the branch segmentation result and the center points obtained in the first step for the cluster boxes,the branch instance masks are clustered to determine the relative position of branches to the cluster box center points.The final picking point coordinates are located by performing a pixel-wise search.Result We collected and created the citrus fruit detection dataset and citrus branch segmentation dataset through long-term outdoor collection.These two datasets consist of mature and immature citrus and include various challenges,such as sunny weather,cloudy weather,front light,and back light.The datasets have a total of 37 000 images.We conducted experimental validation on our pro-posed method using the separate datasets for citrus fruit detection and branch instance segmentation tasks.The fruit detec-tion subtask based on the YOLOv5 model achieved an accuracy of 92.82％.Additionally,our proposed branch sparse instance segmentation method(BP-SparseInst)improved the performance of the branch segmentation task by 1.15％.Fur-thermore,our pick point localization method,based on the cascaded improvement model utilizing the cluster segmentation strategy and the DBSCAN fruit density algorithm,achieved a pick point localization accuracy of 95.77％.This result repre-sents an improvement of approximately 4.1％compared with the previous method.Moreover,this method exhibited real-time performance with an FPS of 28.21 frame/s,which is an improvement of 8.07 frame/s compared with the previous method.Conclusion Through these research efforts,we have made remarkable progress in the localization of fruit picking points in citrus.Moreover,it provides a matching robotic arm picking device for the practical application of the picking point positioning algorithm.This advancement provides strong support for the development of the citrus industry.
picking robotpicking point positioning methodcluster box generatorbrightness priorbranch sparse seg-mentation model
万方数据
维普
