An Architecture-Enhanced Performance Predictor for Transformer-Based NAS
Neural Architecture Search(NAS)represents a paradigm shift in the design of neural network architectures,transitioning from manual,intuition-based processes to automated,algorithm-driven methods.This evolution has significantly reduced design costs and improved model performance across various tasks in computer vision,natural language processing,and beyond.Despite these advances,the computational expense associated with evaluating the performance of candidate architectures remains a formidable challenge,consuming the lion's share of resources in NAS endeavors.The emergence of NAS predictors based on the Transformer architecture has offered a promising pathway to mitigating these computational demands.The Transformer's superior capability for encoding topological information of neural networks has made it an attractive foundation for developing predictors that can efficiently approximate the performance of vast numbers of architectures without the need for exhaustive training.However,the deployment of Transformer-based NAS predictors has encountered significant obstacles.The initial challenge lies in the pretreatment phase,where traditional encoding schemes like One-hot encoding fall short in capturing the full spectrum of node features within a network architecture.Such schemes can identify different types of operations but lack the granularity to describe the operational parameters with sufficient detail,such as the sizes of convolutional kernels,which are critical for accurately modeling network behavior.During the encoding stage,the reliance on the Transformer's self-attention mechanism introduces a second challenge by potentially overlooking essential structural information.The self-attention mechanism,while powerful for capturing global dependencies,may not fully preserve the hierarchical and spatial relationships inherent in neural network architectures,which are crucial for understanding their performance.The evaluation phase presents a third hurdle,where existing predictors mainly utilize Multilayer Perceptron(MLP)to estimate the performance of architectures based on forward propagation alone.This approach neglects the critical influence of back propagation gradients,an oversight that can lead to substantial inaccuracies.The dynamic interplay between forward and backward propagation in training neural networks is complex,and any predictor that fails to account for this interplay is likely to suffer from considerable predictive error,with discrepancies ranging widely between the estimated and actual performance.Addressing these challenges,our proposed solution,"An Architecture-Enhanced Performance Predictor for Transformer-based NAS,"introduces innovative strategies at each stage of the prediction process.In the pretreatment phase,we advocate for a novel encoding approach,Hyper-dimensional Embedding,which significantly expands the input feature space to provide a richer,more nuanced representation of node operations.This method allows for a more detailed characterization of architectural components,thereby enhancing the predictive model's ability to discern subtle differences between architectures.To resolve the issues identified in the encoding stage,our approach integrates the Transformer's output with additional structural information through a Graph Convolutional Network(GCN).This strategy ensures that the spatial and hierarchical relationships between architectural components are maintained and factored into the prediction process,addressing the deficiencies of the self-attention mechanism in preserving structural integrity.Finally,in the evaluation phase,we construct a more comprehensive model of network behavior by incorporating both forward and backward propagation dynamics into our predictive framework.This involves feeding a combination of dataset characteristics,structural encodings,and gradient information into the GCN predictor,thereby achieving a more accurate and holistic assessment of architectural performance.Our experimental results demonstrate the effectiveness of this approach,showing a marked improvement in the Kendall correlation coefficient by 7.45％and a reduction in training time by 1.55 times compared to state-of-the-art methods.This significant advancement underscores the potential of our proposed solution to enhance the efficiency and accuracy of NAS processes,paving the way for more rapid and cost-effective development of high-performing neural networks.
NETL
NSTL
万方数据
