查看更多>>摘要：? 2021 The Author(s)In this paper, a novel data-driven method for weight initialization of Multilayer Perceptrons and Convolutional Neural Networks based on discriminant learning is proposed. The approach relaxes some of the limitations of competing data-driven methods, including unimodality assumptions, limitations on the architectures related to limited maximal dimensionalities of the corresponding projection spaces, as well as limitations related to high computational requirements due to the need of eigendecomposition on high-dimensional data. We also consider assumptions of the method on the data and propose a way to account for them in a form of a new normalization layer. The experiments on three large-scale image datasets show improved accuracy of the trained models compared to competing random-based and data-driven weight initialization methods, as well as better convergence properties in certain cases.
查看更多>>摘要：? 2021 Elsevier LtdLOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.
查看更多>>摘要：? 2021 Elsevier LtdThis work addresses the problem of cross-modality inference (CMI), i.e., inferring missing data of unavailable perceptual modalities (e.g., sound) using data from available perceptual modalities (e.g., image). We overview single-modality variational autoencoder methods and discuss three problems of computational cross-modality inference, arising from recent developments in multimodal generative models. Inspired by neural mechanisms of human recognition, we contribute the NEXUS model, a novel hierarchical generative model that can learn a multimodal representation of an arbitrary number of modalities in an unsupervised way. By exploiting hierarchical representation levels, NEXUS is able to generate high-quality, coherent data of missing modalities given any subset of available modalities. To evaluate CMI in a natural scenario with a high number of modalities, we contribute the “Multimodal Handwritten Digit” (MHD) dataset, a novel benchmark dataset that combines image, motion, sound and label information from digit handwriting. We access the key role of hierarchy in enabling high-quality samples during cross-modality inference and discuss how a novel training scheme enables NEXUS to learn a multimodal representation robust to missing modalities at test time. Our results show that NEXUS outperforms current state-of-the-art multimodal generative models in regards to their cross-modality inference capabilities.
查看更多>>摘要：? 2021 Elsevier LtdHuman adaptive behavior requires continually learning and performing a wide variety of tasks, often with very little practice. To accomplish this, it is crucial to separate neural representations of different tasks in order to avoid interference. At the same time, sharing neural representations supports generalization and allows faster learning. Therefore, a crucial challenge is to find an optimal balance between shared versus separated representations. Typically, models of human cognition employ top-down modulatory signals to separate task representations, but there exist surprisingly little systematic computational investigations of how such modulation is best implemented. We identify and systematically evaluate two crucial features of modulatory signals. First, top-down input can be processed in an additive or multiplicative manner. Second, the modulatory signals can be adaptive (learned) or non-adaptive (random). We cross these two features, resulting in four modulation networks which are tested on a variety of input datasets and tasks with different degrees of stimulus-action mapping overlap. The multiplicative adaptive modulation network outperforms all other networks in terms of accuracy. Moreover, this network develops hidden units that optimally share representations between tasks. Specifically, different than the binary approach of currently popular latent state models, it exploits partial overlap between tasks.
查看更多>>摘要：? 2021 Elsevier LtdAlthough multi-view clustering has received widespread attention due to its far superior performance to single-view clustering, it still faces the following issues: (1) high computational cost, considering the introduction of multi-view information, reduces the clustering efficiency greatly; (2) complex noises and outliers, existed in real-world data, pose a huge challenge to the robustness of clustering algorithms. Currently, how to increase the efficiency and robustness has become two important issues of multi-view clustering. To cope with the above issues, an efficient correntropy-based multi-view clustering algorithm (ECMC) is proposed in this paper, which can not only improve clustering efficiency by constructing embedded anchor graph and utilizing nonnegative matrix factorization (NMF), but also enhance the robustness by exploring correntropy to suppress various noises and outliers. To further improve clustering efficiency, one of the factors of NMF is constrained to be an indicator matrix instead of a traditional non-negative matrix, so that the categories of samples can be obtained directly without any extra operation. Subsequently, a novel half-quadratic-based strategy is proposed to optimize the non-convex objective function of ECMC. Finally, extensive experiments on eight real-world datasets and eighteen noisy datasets show that ECMC can guarantee faster speed and better robustness than other state-of-the-art multi-view clustering algorithms.
查看更多>>摘要：? 2021 Elsevier LtdMatrix reordering is a task to permute the rows and columns of a given observed matrix such that the resulting reordered matrix shows meaningful or interpretable structural patterns. Most existing matrix reordering techniques share the common processes of extracting some feature representations from an observed matrix in a predefined manner, and applying matrix reordering based on it. However, in some practical cases, we do not always have prior knowledge about the structural pattern of an observed matrix. To address this problem, we propose a new matrix reordering method, called deep two-way matrix reordering (DeepTMR), using a neural network model. The trained network can automatically extract nonlinear row/column features from an observed matrix, which can then be used for matrix reordering. Moreover, the proposed DeepTMR provides the denoised mean matrix of a given observed matrix as an output of the trained network. This denoised mean matrix can be used to visualize the global structure of the reordered observed matrix. We demonstrate the effectiveness of the proposed DeepTMR by applying it to both synthetic and practical datasets.
查看更多>>摘要：? 2021 Elsevier LtdIn neurological and neuropsychiatric disorders neuronal oscillatory activity between basal ganglia and cortical circuits are altered, which may be useful as biomarker for adaptive deep brain stimulation. We investigated whether changes in the spectral power of oscillatory activity in the motor cortex (MCtx) and the sensorimotor cortex (SMCtx) of rats after injection of the dopamine (DA) receptor antagonist haloperidol (HALO) would be similar to those observed in Parkinson disease. Thereafter, we tested whether a convolutional neural network (CNN) model would identify brain signal alterations in this acute model of parkinsonism. A sixteen channel surface micro-electrocorticogram (ECoG) recording array was placed under the dura above the MCtx and SMCtx areas of one hemisphere under general anaesthesia in rats. Seven days after surgery, micro ECoG was recorded in individual free moving rats in three conditions: (1) basal activity, (2) after injection of HALO (0.5 mg/kg), and (3) with additional injection of apomorphine (APO) (1 mg/kg). Furthermore, a CNN-based classification consisting of 23,530 parameters was applied on the raw data. HALO injection decreased oscillatory theta band activity (4–8 Hz) and enhanced beta (12–30 Hz) and gamma (30–100 Hz) in MCtx and SMCtx, which was compensated after APO injection (P ? 0.001). Evaluation of classification performance of the CNN model provided accuracy of 92%, sensitivity of 90% and specificity of 93% on one-dimensional signals. The CNN proposed model requires a minimum of sensory hardware and may be integrated into future research on therapeutic devices for Parkinson disease, such as adaptive closed loop stimulation, thus contributing to more efficient way of treatment.
查看更多>>摘要：? 2021 Elsevier LtdRecently, deep semi-supervised graph embedding learning has drawn much attention for its appealing performance on the data with a pre-specified graph structure, which could be predefined or empirically constructed based on given data samples. However, the pre-specified graphs often contain considerable noisy/inaccurate connections and have a huge size for large datasets. Most existing embedding algorithms just take the graph off the shelf during the whole training stage and thus are easy to be misled by the inaccurate graph edges, as well as may result in large model size. In this paper, we attempt to address these issues by proposing a novel deep semi-supervised algorithm for simultaneous graph embedding and node classification, utilizing dynamic graph learning in neural network hidden layer space. Particularly, we construct an anchor graph to summarize the whole dataset using the hidden layer features of a consistency-constrained network. The anchor graph is used for sampling node neighborhood context, which is then presented together with node labels as contextual information to train an embedding network. The outputs of the consistency network and the embedding networks are finally concatenated together to pass a softmax function to perform node classification. The two networks are optimized jointly using both labeled and unlabeled data to minimize a single semi-supervised objective function, including a cross-entropy loss, a consistency loss and an embedding loss. Extensive experimental results on popular image and text datasets have shown that the proposed method is able to improve the performance of existing graph embedding and node classification methods, and outperform many state-of-the-art approaches on both types of datasets.
NSTL
Elsevier