Huawei Noah's Ark Lab has 6 papers accepted by NIPS 2018, including an Oral

Oct. 2018

                       Avatar


Abstract: NIPS, one of the top conferences in the field of machine learning, recently announced the list of papers accepted by NIPS 2018. Six papers from Huawei Noah's Ark Lab were accepted by NIPS, one of which was selected as an oral report (Oral, 0.6%, 30/4856). Among them, the first author of four papers is from Noah. In the industrial ranking based on the number of first author papers, Noah is the 7th in the world and the first in China.
NIPS (Conference and Workshop on Neural Information Processing Systems) is an International Conference on machine learning and computational neuroscience. It was originally founded in Canada in 1987 by connectionist neural network scholars. Later, its influence gradually expanded and now it has become the top conference in the field of machine learning. The theme of NIPS focus on machine learning, artificial intelligence and statistics. The conference is scheduled to be held every December, sponsored by the NIPS Foundation, and will be held at Montreal Convention Center, Canada, from December 3 to 8 this year.
NIPS has gone through more than 30 years. With the popularity of depth learning in recent years, NIPS has become one of the most important academic conferences concerned by academia and industry. The number of participants has increased rapidly year by year, reaching more than 8,000 in 2017. In 2018, NIPS has created a crazy record that the main conference tickets sold out in 11 minutes and 38 seconds. Half an hour later, all the tickets were sold out. In addition to the number of participants, the number of NIPS submissions in 2018 reached an unprecedented record of 4,856, an increase of about 50% compared with 3,240 in 2017, and an acceptance rate of 20.8%. A total of 1,011 papers were accepted, 168 of which were Spotlight (3.5%) and 30 Oral (0.6%). Competition was fierce.
There are six papers from Huawei Noah's Ark Lab were accepted by NIPS 2018, one of which was selected as one of the total 30 oral reports (Orals) and one as Spotlight. Among them, the first author of four papers is from Noah. In the industrial ranking based on the number of first author papers, Noah is the 7th in the world and the first in China.
Paper 1: Optimal Algorithms for Non-Smooth Distributed Optimization in Networks (Oral)
In collaboration with INRIA (Francis Bach), Microsoft Research, and University of Washington.
In this work, we consider the distributed optimization of non-smooth convex functions using a network of computing units. We investigate this problem under two regularity assumptions: (1) the Lipschitz continuity of the global objective function, and (2) the Lipschitz continuity of local individual functions. Under the local regularity assumption, we provide the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate. A notable aspect of this result is that, for non-smooth functions, the error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions. Under the global regularity assumption, we provide a simple yet efficient algorithm called distributed randomized smoothing (DRS) based on a local smoothing of the objective function, and show that DRS is within a d^{1/4} multiplicative factor of the optimal convergence rate, where d is the underlying dimension.
Paper 2: KONG: Kernels for ordered-neighborhood graphs (Spotlight)
In collaboration with London School of Economics.
We present novel graph kernels for graphs with node and edge labels that have ordered neighborhoods, i.e. when neighbor nodes follow an order. Combining convolutional sub graph kernels and string kernels, we design new scalable algorithms for generation of explicit graph feature maps using sketching techniques. We obtain precise bounds for the approximation accuracy and computational complexity of the proposed approaches and demonstrate their applicability on real datasets. In particular, our experiments demonstrate that neighborhood ordering results in more informative features.
Paper 3: Lipschitz regularity of deep neural networks: analysis and efficient estimation
Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-of-art methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks.
Paper 4: Causal Inference and Mechanism Clustering of a Mixture of Additive Noise Models
In collaboration with Dept. of Computer Science and Engineering, the Chinese University of Hong Kong.
The inference of the causal relationship between a pair of observed variables is a fundamental problem in science and approaches exploiting certain kind of independence in one single causal model are most commonly used. In practice, however, observations are often collected from multiple sources with heterogeneous causal models, which renders causal analysis results obtained by a single model skeptical. In this paper, we generalize the Additive Noise Model (ANM) to a mixture model, which consists of a finite number of ANMs, and provide the condition of its causal identifiability. To conduct model estimation, we propose a Gaussian Process Partially Observable Model (GPPOM) to learn the latent parameter associated with each observation, which enables us to not only infer the casual direction but also cluster observations according to their underlying generating mechanisms. Experiments on synthetic and real data demonstrate the effectiveness of our proposed approach.
Paper 5: Online Structure Learning for Feed-Forward and Recurrent Sum-Product Networks
In collaboration with University of Waterloo.
Sum-product networks have recently emerged as an attractive representation due to their dual view as a special type of deep neural network with clear semantics and a special type of probabilistic graphical model for which inference is always tractable. Those properties follow from some conditions (i.e., completeness and decomposability) that must be respected by the structure of the network. As a result, it is not easy to specify a valid sum-product network by hand and therefore structure learning techniques are typically used in practice. This paper describes a new online structure learning technique for feed-forward and recurrent SPNs. The algorithm is demonstrated on real-world datasets with continuous features for which it is not clear what network architecture might be best, including sequence datasets of varying length.
Paper 6: Learning Versatile Filters for Efficient Convolutional Neural Networks
In collaboration with University of Sydney and Peking University.
This paper introduces versatile filters to construct efficient convolutional neural network. Considering the demands of efficient deep learning techniques running on cost-effective hardware, a number of methods have been developed to learn compact neural networks. Most of these works aim to slim down filters in different ways, e.g. investigating small, sparse or binarized filters. In contrast, we treat filters from an additive perspective. A series of secondary filters can be derived from a primary filter. These secondary filters all inherit in the primary filter without occupying more storage, but once been unfolded in computation they could significantly enhance the capability of the filter by integrating information extracted from different receptive fields. Besides spatial versatile filters, we additionally investigate versatile filters from the channel perspective. The new techniques are general to upgrade filters in existing CNNs. Experimental results on benchmark datasets and neural networks demonstrate that CNNs constructed with our versatile filters are able to achieve comparable accuracy as that of original filters, but require less memory and FLOPs
As Huawei's AI research center, Noah's Ark Lab focuses on researching AI algorithms and building data-efficient and energy-efficient AI engines. The research fields mainly focus on six directions: Computer Vision, Speech and Language Processing, Search & Recommendation, Decision Making & Reasoning, AI Theory, and AI Human-Computer Interaction.
While vigorously promoting the research in AI theory, Noah believes that the future of AI depends on the AI applications, and focuses on the combination of theory and practice. For AI applications, Noah's research focuses on Huawei's main business, such as terminal intelligence, network intelligence, enterprise intelligence, etc., as well as new business areas such as car networking, safe city, smart healthcare, etc.
In the future, Noah will continue to cooperate extensively with the industry, focusing on AI basic technology research, focusing on the complementary of theoretical research and practical application, and promoting the sustainable development of AI technology, striving to bring digital to every person, home and organization for a fully connected, intelligent world.