查看更多>>摘要：Hardware implementation of neural networks represents a milestone for exploiting the advantages of neuromorphic-type data processing and for making use of the inherent parallelism associated with such structures. In this context, memristive devices with their analogue functionalities are called to be promising building blocks for the hardware realization of artificial neural networks. As an alternative to conventional crossbar architectures where memristive devices are organized with a top-down approach in a grid-like fashion, neuromorphic-type data processing and computing capabilities have been explored in networks realized according to the principle of self-organization similarity found in biological neural networks. Here, we explore structural and functional connectivity of self-organized memristive nanowire (NW) networks within the theoretical framework of graph theory. While graph metrics reveal the link of the graph theoretical approach with geometrical considerations, results show that the interplay between network structure and its capacity to transmit information is related to a phase transition process consistent with percolation theory. Also the concept of memristive distance is introduced to investigate activation patterns and the dynamic evolution of the information flow across the network represented as a memristive graph. In agreement with experimental results, the emergent short-term dynamics reveals the formation of self-selected pathways with enhanced transport characteristics connecting stimulated areas and regulating the trafficking of the information flow. The network capability to process spatiotemporal input signals can be exploited for the implementation of unconventional computing paradigms in memristive graphs that take into advantage the inherent relationship between structure and functionality as in biological systems. (C)& nbsp;2022 The Author(s). Published by Elsevier Ltd.& nbsp;
查看更多>>摘要：Graph Neural Networks (GNNs) are powerful architectures for learning on graphs. They are efficient for predicting nodes, links and graphs properties. Standard GNN variants follow a message passing schema to update nodes representations using information from higher-order neighborhoods iteratively. Consequently, deeper GNNs make it possible to define high-level nodes representations generated based on local as well as distant neighborhoods. However, deeper networks are prone to suffer from over-smoothing. To build deeper GNN architectures and avoid losing the dependency between lower (the layers closer to the input) and higher (the layers closer to the output) layers, networks can integrate residual connections to connect intermediate layers. We propose the Augmented Graph Neural Network (AGNN) model with hierarchical global-based residual connections. Using the proposed residual connections, the model generates high-level nodes representations without the need for a deeper architecture. We disclose that the nodes representations generated through our proposed AGNN model are able to define an expressive all-encompassing representation of the entire graph. As such, the graph predictions generated through the AGNN model surpass considerably state-of-the-art results. Moreover, we carry out extensive experiments to identify the best global pooling strategy and attention weights to define the adequate hierarchical and global-based residual connections for different graph property prediction tasks. Furthermore, we propose a reversible variant of the AGNN model to address the extensive memory consumption problem that typically arises from training networks on large and dense graph datasets. The proposed Reversible Augmented Graph Neural Network (R-AGNN) only stores the nodes representations acquired from the output layer as opposed to saving all representations from intermediate layers as it is conventionally done when optimizing the parameters of other GNNs. We further refine the definition of the backpropagation algorithm to fit the R-AGNN model. We evaluate the proposed models AGNN and R-AGNN on benchmark Molecular, Bioinformatics and Social Networks datasets for graph classification and achieve state-of-the-art results. For instance the AGNN model realizes improvements of +39% on IMDB-MULTI reaching 91.7% accuracy and +16% on COLLAB reaching 96.8% accuracy compared to other GNN variants. (C) 2022 Elsevier Ltd. All rights reserved.
查看更多>>摘要：Humans learn to recognize and manipulate new objects in lifelong settings without forgetting the previously gained knowledge under non-stationary and sequential conditions. In autonomous systems, the agents also need to mitigate similar behaviour to continually learn the new object categories and adapt to new environments. In most conventional deep neural networks, this is not possible due to the problem of catastrophic forgetting, where the newly gained knowledge overwrites existing representations. Furthermore, most state-of-the-art models excel either in recognizing the objects or in grasp prediction, while both tasks use visual input. The combined architecture to tackle both tasks is very limited. In this paper, we proposed a hybrid model architecture consists of a dynamically growing dual-memory recurrent neural network (GDM) and an autoencoder to tackle object recognition and grasping simultaneously. The autoencoder network is responsible to extract a compact representation for a given object, which serves as input for the GDM learning, and is responsible to predict pixel-wise antipodal grasp configurations. The GDM part is designed to recognize the object in both instances and categories levels. We address the problem of catastrophic forgetting using the intrinsic memory replay, where the episodic memory periodically replays the neural activation trajectories in the absence of external sensory information. To extensively evaluate the proposed model in a lifelong setting, we generate a synthetic dataset due to lack of sequential 3D objects dataset. Experiment results demonstrated that the proposed model can learn both object representation and grasping simultaneously in continual learning scenarios. (C)& nbsp;2022 The Author(s). Published by Elsevier Ltd.
查看更多>>摘要：In this paper, an array of discrete-time coupled complex-valued neural networks (CVNNs) with random system parameters and time-varying delays are introduced. The stochastic fluctuations of system parameters, which are characterized by a set of random variables, are considered in the individual CVNNs. Firstly, the synchronization issue is solved for the considered coupled CVNNs. By the use of the Lyapunov stability theory and the Kronecker product, a synchronization criterion is proposed to guarantee that the coupled CVNNs are asymptotically synchronized in the mean square. Subsequently, the state estimation issue is studied for the identical coupled CVNNs via available measurement output. By establishing a suitable Lyapunov functional, sufficient conditions are derived under which the mean square asymptotic stability of the estimation error system is ensured and the design scheme of desired state estimator is explicitly provided. Finally, two numerical simulation examples are shown for the purpose of illustrating the effectiveness of the proposed theory. (C)& nbsp;2022 Elsevier Ltd. All rights reserved.
查看更多>>摘要：Direct multi-task twin support vector machine (DMTSVM) is an effective algorithm to deal with multitask classification problems. However, the generated hyperplane may shift to outliers since the hinge loss is used in DMTSVM. Therefore, we propose an improved multi-task model RaMTTSVM based on ramp loss to handle noisy points more effectively. It could limit the maximal loss value distinctly and put definite restrictions on the influences of noises. But RaMTTSVM is non-convex which should be solved by CCCP, then a series of approximate convex problems need to be solved. So, it may be time-consuming. Motivated by the sparse solution of our RaMTTSVM, we further propose a safe acceleration rule MSA to accelerate the solving speed. Based on optimality conditions and convex optimization theory, MSA could delete a lot of inactive samples corresponding to 0 elements in dual solutions before solving the model. Then the computation speed can be accelerated by just solving reduced problems. The rule contains three different parts that correspond to different parameters and different iteration phases of CCCP. It can be used not only for the first approximate convex problem of CCCP but also for the successive problems during the iteration process. More importantly, our MSA is safe in the sense that the reduced problem can derive an identical optimal solution as the original problem, so the prediction accuracy will not be disturbed. Experimental results on one artificial dataset, ten Benchmark datasets, ten Image datasets and one real wine dataset confirm the generalization and acceleration ability of our proposed algorithm.(C) 2022 Elsevier Ltd. All rights reserved.
查看更多>>摘要：This paper studies the Lyapunov stability of nonlinear systems and the synchronization of complex neural networks in the framework of event-triggered delayed impulsive control (ETDIC), where the effect of time delays in impulses is fully considered. Based on the Lyapunov-based event-triggered mechanism (ETM), some sufficient conditions are presented to avoid Zeno behavior and achieve globally asymptotical stability of the addressed system. In the framework of event-triggered impulse control (ETIC), control input is only generated at state-dependent triggered instants and there is no any control input during two consecutive triggered impulse instants, which can greatly reduce resource consumption and control waste. The contributions of this paper can be summarized as follows: Firstly, compared with the classical ETIC, our results not only provide the well-designed ETM to determine the impulse time sequence, but also fully extract the information of time delays in impulses and integrate it into the dynamic analysis of the system. Secondly, it is shown that the time delays in impulses in our results exhibit positive effects, that is, it may contribute to stabilizing a system and achieve better performance. Thirdly, as an application of ETDIC strategies, we apply the proposed theoretical results to synchronization problem of complex neural networks. Some sufficient conditions to ensure the synchronization of complex neural networks are presented, where the information of time delays in impulses is fully fetched in these conditions. Finally, two numerical examples are provided to show the effectiveness and validity of the theoretical results. (C)& nbsp;2022 Elsevier Ltd. All rights reserved.
查看更多>>摘要：Humans can identify objects following various spatial transformations such as scale and viewpoint. This extends to novel objects, after a single presentation at a single pose, sometimes referred to as online invariance. CNNs have been proposed as a compelling model of human vision, but their ability to identify objects across transformations is typically tested on held-out samples of trained categories after extensive data augmentation. This paper assesses whether standard CNNs can support human-like online invariance by training models to recognize images of synthetic 3D objects that undergo several transformations: rotation, scaling, translation, brightness, contrast, and viewpoint. Through the analysis of models' internal representations, we show that standard supervised CNNs trained on transformed objects can acquire strong invariances on novel classes even when trained with as few as 50 objects taken from 10 classes. This extended to a different dataset of photographs of real objects. We also show that these invariances can be acquired in a self-supervised way, through solving the same/different task. We suggest that this latter approach may be similar to how humans acquire invariances. Crown Copyright (C) 2022 Published by Elsevier Ltd. All rights reserved.
Verduzco-Flores, SergioDorrell, WilliamDe Schutter, Erik
22页
查看更多>>摘要：In this paper we explore a neural control architecture that is both biologically plausible, and capable of fully autonomous learning. It consists of feedback controllers that learn to achieve a desired state by selecting the errors that should drive them. This selection happens through a family of differential Hebbian learning rules that, through interaction with the environment, can learn to control systems where the error responds monotonically to the control signal. We next show that in a more general case, neural reinforcement learning can be coupled with a feedback controller to reduce errors that arise non-monotonically from the control signal. The use of feedback control can reduce the complexity of the reinforcement learning problem, because only a desired value must be learned, with the controller handling the details of how it is reached. This makes the function to be learned simpler, potentially allowing learning of more complex actions. We use simple examples to illustrate our approach, and discuss how it could be extended to hierarchical architectures. (c) 2022 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
查看更多>>摘要：It has been observed that design choices of neural networks are often crucial for their successful optimization. In this article, we therefore discuss the question if it is always possible to redesign a neural network so that it trains well with gradient descent. This yields the following universality result: If, for a given network, there is any algorithm that can find good network weights for a classification task, then there exists an extension of this network that reproduces the same forward model by mere gradient descent training. The construction is not intended for practical computations, but it provides some orientation on the possibilities of pre-trained networks in meta-learning and related approaches. (C)& nbsp;2022 Elsevier Ltd. All rights reserved.
Li, HongmingDou, RanKeil, AndreasPrincipe, Jose C....
19页
查看更多>>摘要：Inspired by the human vision system and learning, we propose a novel cognitive architecture that understands the content of raw videos in terms of objects without using labels. The architecture achieves four objectives: (1) Decomposing raw frames in objects by exploiting foveal vision and memory. (2) Describing the world by projecting objects on an internal canvas. (3) Extracting relevant objects from the canvas by analyzing the causal relation between objects and rewards. (4) Exploiting the information of relevant objects to facilitate the reinforcement learning (RL) process. In order to speed up learning, and better identify objects that produce rewards, the architecture implements learning by causality from the perspective of Wiener and Granger using object trajectories stored in working memory and the time series of external rewards. A novel non-parametric estimator of directed information using Renyi's entropy is designed and tested. Experiments on three environments show that our architecture extracts most of relevant objects. It can be thought of as 'understanding' the world in an object-oriented way. As a consequence, our architecture outperforms state-of-the-art deep reinforcement learning in terms of training speed and transfer learning. (C) 2022 Elsevier Ltd. All rights reserved.
NSTL
Elsevier