aalto1 untyped-item.component.html
A Novel Method for Function Smoothness in Neural Networks
Loading...
Access rights
openAccess
publishedVersion
URL
Journal Title
Journal ISSN
Volume Title
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
This publication is imported from Aalto University research portal.
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
View publication in the Research portal (opens in new window)
View/Open full text file from the Research portal (opens in new window)
Unless otherwise stated, all rights belong to the author. You may download, display and print this publication for Your own personal use. Commercial use is prohibited.
Authors
Date
Department
Major/Subject
Mcode
Degree programme
Language
en
Pages
11
Series
IEEE Access, Volume 10, pp. 75354-75364
Abstract
Existing methods for function smoothness in neural networks have limitations. These methods can make training sensitive to their hyperparameters, or their smoothness constraints can limit model capacity. These methods can impose too much smoothness, even in areas without data, or they can impose non-meaningful smoothness constraints. The way these methods measure smoothness can also be computationally hard. One of the main methods for function smoothness, Lipschitz continuity, does not even imply differentiability theoretically, let alone continuous differentiability, that is smoothness. In this paper, we propose a method based on the theoretical definition of the derivative to ensure that the derivative of the parametrized function should tend toward its theoretical value for the given neural network parameters in the vicinity of training samples. The method changes the classifier and its training minimally and has no added hyperparameters. The proposed method is shown to achieve a smoother function in the vicinity of both training and testing samples for all tested datasets, as measured with decreased values of the Frobenius norm of the Jacobian with respect to inputs. Due to the correlation between function smoothness and generalization, the method makes classifiers generalize better and achieve higher accuracy than default classifiers for Restricted ImageNet, CIFAR10 and MNIST. Due to the correlation between function smoothness and adversarial robustness, the proposed method makes classifiers with high-capacity architecture more robust to adversarial samples generated with the PGD attack compared to default classifiers for the Restricted ImageNet, CIFAR10, Fashion-MNIST and MNIST datasets.
Description
Publisher Copyright: © 2013 IEEE.
Other note
Citation
Lindqvist, B 2022, 'A Novel Method for Function Smoothness in Neural Networks', IEEE Access, vol. 10, pp. 75354-75364. https://doi.org/10.1109/ACCESS.2022.3189363