Cost-forced and repeated selective information minimization and maximization for multi-layered neural networks¹

Article type: Research Article

Authors: Kamimura, Ryotaro

Affiliations: Kumamoto Drone Technology and Development Foundation, 2880 Kamimatuso Nishi-ku, Kumamoto 861-5289, Japan | E-mail: [email protected]

Note: [1] This paper has been written based on the paper presented at ISDA2021 [51].

Abstract: The present paper aims to propose a new information-theoretic method to minimize and maximize selective information repeatedly. In particular, we try to solve the incomplete information control problem, where information cannot be fully controlled due to the existence of many contradictory factors inside. For this problem, the cost in terms of the sum of absolute connection weights is introduced for neural networks to increase and decrease information against contradictory forces in learning, such as error minimization. Thus, this method is called a “cost-forced” approach to control information. The method is contrary to the conventional regularization approach, where the cost has been used passively or negatively. The present method tries to use the cost positively, meaning that the cost can be augmented if necessary. The method was applied to an artificial and symmetric data set. In the symmetric data set, we tried to show that the symmetric property of the data set could be obtained by appropriately controlling information. In the second data set, that of residents in a nursing home, obtained by the complicated procedures of natural language processing, the experimental results confirmed that the present method could control selective information to extract non-linear relations as well as linear ones in increasing interpretation and generalization performance.

Keywords: Cost forced, selective information, incomplete information, interpretation

DOI: 10.3233/HIS-220008

Journal: International Journal of Hybrid Intelligent Systems, vol. 18, no. 1-2, pp. 69-95, 2022