Keynotes
Prof. Sun-Yuan Kung
Princeton University, USA
Title: Progressive and Regressive NAS for Deep Learning: Theory and Applications
Please check the slides at here.
Abstract: It has recently become trendy to explore an architecture engineering process so that a machine can automatically learn the network structure as well as its parameters. This is collectively known as ``Neural Architecture Search" (NAS). In this talk, we shall present mathematical analysis on the layer sensitivity vital for both PNAS and RNAS - short for Progressive and Regressive NAS. To this end, a notion of Discriminant Matrix (DM), originally rooted on the classic works by Gauss, Fisher, and Shannon, is adopted to assess the information/redundancy embedded in a network layer. Both its trace norm, i.e. Discriminant Information" (DI), and eigenvalue analysis will play vital roles on assessing the network sensitivity revealing how to grow/prune a layer and which layer(s) to grow/prune, etc. Our analysis can precisely predict the theoretical range of the resultant DI-score upon growing or dropping a fixed number of neurons to/from a layer. For example:
For RNAS, the lower and upper bounds of the said range are fully determined by the (major and minor) eigenvalues of the DM, a critical knowledge in order for us to (1) arrive at the highest possible score, (2) determine how many of such neurons should be dropped; and (3) identify/remove the most dispensable neurons, i.e. the ones whose removal would suffer the least DiLOSS.
For PNAS, the bounds depend solely on the (major and minor) eigenvalues of an ``Innovative DM", which stems from the optimal projection from the input-residue to output-residue. (Residues are defined as the subspaces orthogonal to the targeted layer.) In order to to maximize DI, the optimal direction to grow the network must be in the same subspace defined by the principal eigenvectors of the ``Innovative DM".
The DM/DI analysis ultimately leads to an X-learning paradigm where deleterious neurons will be gradually trimmed so as to reach an improved network structure. We have recently been developing an autonomous reinforcement software based on X-Learning, named XNAS. (If time permits, we shall make a brief demo on a real-time learning performance of XNAS.) In this talk, we shall highlight some (rather exciting) simulation results by applying XNAS to (1) classification-type application scenarios: e.g. CIFAR or ImageNet; as well as (2) regression-type application scenarios: e.g. super-resolution (SR) hetero-encoder and seeing-in-the-dark.
Bio: S.Y. Kung, Life Fellow of IEEE, is a Professor at Department of Electrical Engineering in Princeton University. His research areas include machine learning, data mining, systematic design of (deep-learning) neural networks, statistical estimation, VLSI array processors, signal and multimedia information processing, and most recently compressive privacy. He was a founding member of several Technical Committees (TC) of the IEEE Signal Processing Society. He was elected to Fellow in 1988 and served as a Member of the Board of Governors of the IEEE Signal Processing Society (1989-1991). He was a recipient of IEEE Signal Processing Society's Technical Achievement Award for the contributions on "parallel processing and neural network algorithms for signal processing" (1992); a Distinguished Lecturer of IEEE Signal Processing Society (1994); a recipient of IEEE Signal Processing Society's Best Paper Award for his publication on principal component neural networks (1996); and a recipient of the IEEE Third Millennium Medal (2000). Since 1990, he has been the Editor-In-Chief of the Journal of VLSI Signal Processing Systems. He served as the first Associate Editor in VLSI Area (1984) and the first Associate Editor in Neural Network (1991) for the IEEE Transactions on Signal Processing. He has authored and co-authored more than 500 technical publications and numerous textbooks including "VLSI Array Processors", Prentice-Hall (1988); "Digital Neural Networks", Prentice-Hall (1993) ; "Principal Component Neural Networks", John-Wiley (1996); "Biometric Authentication: A Machine Learning Approach", Prentice-Hall (2004); and "Kernel Methods and Machine Learning”, Cambridge University Press (2014).
2nd Keynote Speaker
Prof. Witold Pedrycz
University of Alberta, Canada
Title: Rule-Based Modeling in Interpretable Environment of Federated Learning
Abstract: With the evident omnipresence of mobile devices, massive distributed data, limited communication bandwidth, and security and privacy requirements, federated learning becomes a suitable and practically viable design alternative supporting model construction. As a learning paradigm, it constitutes a radical departure from the well-known development schemes commonly encountered in machine learning. Rule-based models coming in the form of conditional if-then statements exhibit highly desirable modularity feature, which is relevant in the context of interpretability of their architectures.
In this talk, we revisit the existing schemes of federated learning and develop different averaging and gradient-based learning mechanisms by elaborating on ways of their application to the condition and conclusion parts of the rules and the main strategies of communication of the parameters of the models and gradients among clients and a server. A role of information granules and Granular Computing is discussed. We demonstrate that the quality of the rules at the level of the individual client can be characterized by introducing their granular counterparts (viz. the rules with granular rather than numeric parameters). It is also shown that a quality of the global model being formed at the level of the server can be conveniently quantified by endowing it with a granular generalization.
Furthermore, a three-tier layered topology of federated learning involving a layer of edge computing is investigated.
To make the talk self-contained, all required prerequisites are covered.
Bio: Witold Pedrycz (IEEE Life Fellow) is Professor and Canada Research Chair (CRC) in Computational Intelligence in the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada. He is also with the Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland. Dr. Pedrycz is a foreign member of the Polish Academy of Sciences and a Fellow of the Royal Society of Canada. He is a recipient of several awards including Norbert Wiener award from the IEEE Systems, Man, and Cybernetics Society, IEEE Canada Computer Engineering Medal, a Cajastur Prize for Soft Computing from the European Centre for Soft Computing, a Killam Prize, a Fuzzy Pioneer Award from the IEEE Computational Intelligence Society, and 2019 Meritorious Service Award from the IEEE Systems Man and Cybernetics Society. His main research directions involve Computational Intelligence, fuzzy modeling and Granular Computing, knowledge discovery, pattern recognition, data science, knowledge-based neural networks among others. Dr. Pedrycz is involved in editorial activities. He is an Editor-in-Chief of Information Sciences, Editor-in-Chief of WIREs Data Mining and Knowledge Discovery (Wiley), and Co-editor-in-Chief of Int. J. of Granular Computing (Springer) and J. of Data Information and Management (Springer).
3rd Keynote Speaker
Prof. Robert Kozma
University of Memphis, USA
Title: Sustainable Artificial Intelligence
Abstract: The development of increasingly powerful computing devices has been dominated by Moore's law for over half a century, which may reach an end soon demanding a drastic reformulation of existing approaches to computing. A key issue is the massive energy utilization by the electronics components and the consequent dissipation of the energy in the form of heat. Energy constraints are often ignored or have just secondary role in typical cutting-edge AI approaches. For example, Deep Learning Neural Networks often require huge amount of data/ time/ parameters/ energy/ computational power, which may not be readily available in various scenarios, such as edge computing and on-board applications. Human brains are very efficient devices using 20W power, which is many orders of magnitudes less than the power consumption of today’s supercomputers requiring MWs to innovatively solve a given machine learning task. Brain waves and spatio-temporal oscillations implement pattern-based computing principles. Analyzing brain oscillations and metabolism can help to develop computational and hardware implementations, which are energy-efficient and provide a path towards sustainable AI. Among the various potential solutions for sustainable AI, neuromorphic computing and chip designs gained prominence in recent years. Popular crossbar architectures are especially well suited for pattern-based computing, with the potential of complementing the sequential symbol manipulation paradigm of traditional Turing machines. Applications include autonomous on-board signal processing and control, distributed sensor systems, autonomous robot navigation and control, and rapid response to emergencies.
Bio: Dr. Robert Kozma (Fellow IEEE; Fellow INNS) holds a Ph.D. in Applied Physics (Delft, The Netherlands), two M.Sc. degrees (Mathematics, Hungary; and Power Engineering, Moscow MEI, Russia). He is Professor of Mathematics, funding Director of Center for Large-Scale Intelligent Optimization and Networks (CLION), FedEx Institute of Technology, University of Memphis, TN. He has been Visiting Professor of Computer Science, University of Massachusetts Amherst since 2016. Previous assignments include US Air Force Research Laboratory, Sensors Directorate; NASA Jet Propulsion Laboratory, Robotics Division; University of California at Berkeley, EECS and Division of Neurobiology; Otago University, Information Sciences, New Zealand; Tohoku University, Quantum Science and Engineering, Japan. He has over 30 years of experience in intelligent signal processing, large-scale networks and graph theory, distributed sensor systems, biomedical domains, including brain dynamics. He has 7 book volumes, over 300 articles in journals and proceedings, and 3 patent disclosures. Research funding by agencies NASA, DARPA, AFRL, AFOSR, NSF, and others. He is Editor-In-Chief of IEEE Transactions on Systems, Man, and Cybernetics - Systems, and serves on the Editorial Boards on several journals. He serves on the Governing Board of IEEE Systems, Man, and Cybernetics Society, and previously served on the AdCom of the IEEE Computational Intelligence Society. He is past President of the International Neural Networks Society (INNS), recipient of the Denis Gabor Award of INNS. He has been General Chair of the International Joint Conference on Neural Networks, IJCNN2009, and has been Program Chair/Co-Chair and TC member of several dozens of international conferences.
