Deep learning technologies have accelerated the development and advancement of end-to-end autonomous driv-ing frameworks in recent years,sparking the emergence of numerous cutting-edge research topics and application deploy-ment solutions.The"divide and conquer"architecture design concept,which aims to construct multiple independent but related module components,integrate them into the developed software system in a specific semantic or geometric order,and ultimately deploy these components to the actual vehicle,is the foundation for the majority of the autonomous driving systems currently in use,also known as modular systems.However,a well-developed modular design typically comprises thousands of components,placing a considerable burden on the graphics memory and processing capacity of automotive CPUs.Furthermore,the intrinsic mistakes of each stacked module during prediction will rise with the number of stacked modules,and upstream flaws cannot be fixed in downstream modules,presenting a major risk to vehicle safety.A multi-task architecture based on the"task parallelism"principle aims to efficiently infer multiple tasks in parallel by designing various decoded heads with a shared backbone network to reduce computational consumption.However,the optimization goals for various tasks may not be consistent,and sharing features mindlessly can even degrade the overall performance of the system.In contrast to the previous two system architectures,the end-to-end technology paradigm eliminates information bottlenecks and cumulative errors due to the integration of numerous intermediate components based on rule interfaces,allowing the network to continually optimize toward a unified objective.A large model can be used to generate low-level control signals or vehicle motion planning based on inputs such as sensor data and vehicle status.With sensors serving as inputs,the early end-to-end design based on imitation and reinforcement learning directly outputs the final control com-mands for steering,braking,and acceleration.However,no explicit representation of driving scenarios in this completely"black box"network,which is also referred to as weakly interpretable end-to-end methods,is available.Thus,understand-ing the reasoning behind the decision or prediction of a vehicle is difficult for humans,making debugging,validation,and optimization challenging.Even worse,once the model malfunctions or unexpected situations occur,accurately detecting,avoiding,and repairing problems in a timely manner becomes difficult,all of which are crucial for maintaining the safe operation of intelligent vehicles.The component decoupling approach facilitates the development and optimization of indi-vidual modules in the conventional modular system,thereby guaranteeing steady representation performance and strong interpretability for each submodule.Unfortunately,this method falls short of achieving unified goals at the optimization level,that is,integrating optimization and learning toward the ultimate planning goal.A modular joint end-to-end autono-mous driving architecture,which preserves the modular driving system while allowing the differentiability of each module,is a workable solution to ensure that every module has sufficient interpretability and overall automatic optimization capabili-ties.The basic idea behind this technology lies in the creation of a unique neural network that connects all independent modules and enables the gradients from the planning modules to be fed back down to the initial sensor input for end-to-end execution.In other words,this kind of approach merely modifies the submodule connection mechanism while maintaining the classic modular technology stack;that is,this approach substitutes a new implicit interface for the previous explicit interfaces,which were rule-based and required manual creation.Modular joint end-to-end procedures offer a certain inter-pretability because of the distinct separation between modules.The explicit end-to-end system is a relative decoupling based on overall design and exhibits some degree of logic in its sequential functioning from perception to prediction,and then to planning modules during decision inference.The model can be intentionally adjusted when it encounters unknown and uncontrollable results by understanding the operational logic underlying the explicit solution.Furthermore,visualiza-tion methods,such as internal features or intermediate results of specific tasks or modules,can be utilized to analyze the decision-making operation mechanism,which can prevent potential risks caused by black box models and ensure the safe and efficient driving of intelligent vehicles.Therefore,this article conducts comprehensive analysis and research on the emerging field of end-to-end autonomous driving with promising development prospects,which summarizes the main techni-cal routes and representative research methods around the development path of end-to-end driving systems.More specifi-cally,this article,which begins with the classic modular system,analyzes the shortcomings of conventional modular and multitasking approaches while providing a brief introduction to the four functional modules of the autonomous driving sys-tem.These modules primarily include perception,prediction,planning,and decision making.Subsequently,extensive research on the emerging end-to-end autonomous driving frameworks is conducted from the perspective of input-output modality to system architecture,describing in detail the two dominant paradigms and delving into the shortcomings and drawbacks of existing research work.The existing end-to-end architecture can be categorized into two categories based on interpretable performance:weakly interpretable end-to-end,which is explored from the aspects of imitation learning,rein-forcement learning,and interpretability;or modular joint end-to-end,which is progressively investigated from bird's-eye view representation,to joint perception prediction,and ultimately,planning-oriented end-to-end methods.Afterward,a thorough discussion of the end-to-end driving system assessment is provided for closed-and open-loop evaluations,along with the corresponding situations.Finally,the research works on end-to-end autonomous driving systems are summarized,and the potential challenges and key problems that still need to be addressed are discussed from the perspectives of data mining and architecture design.
NETL
NSTL
万方数据
