With the size of training data and the number of parameters expanding for the sake of better accuracy of deep learning models, computation time is also increasing. It is not unusual to take several weeks to train a model. Getting multiple GPUs to link with one another for faster training is very important to reduce the time spent on trial and error and verification of new ideas, and produce results quickly.

On the other hand, it is generally known in parallel/distributed learning that the accuracy and learning rate of a model decrease gradually with increased GPUs, due to larger batch sizes and GPU communication overhead.

This time, we have improved learning algorithms and parallel performance to address these issues, and used one of Japan’s most powerful parallel computers with 1,024 of NVIDIA^(R)’s Tesla^(R) multi-node P100 GPUs and leverages Chainer’s distributed learning package ChainerMN ² for training.

As a result, we completed training ResNet-50 ³ for image classification on the ImageNet ⁴ dataset in 15 minutes, which is a significant improvement from the previously best known result ⁵.

The research paper on this achievement is available in the following URL under the title “Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes”.
(imagenet_in_15min.pdf)

Based on this research result, PFN will further accelerate its research and development activities in the fields of transportation systems, manufacturing, and bio/healthcare, which require large-scale deep learning.

1 One of the most powerful private supercomputer in Japan, contains 1,024 of NVIDIA^（R）’s Tesla^（R） multi-node P100 GPUs.https://www.preferred.jp/en/news/pr20170920

2 A package adding distributed learning functionality with multiple GPUs to the open source deep learning framework Chainer

3 A network frequently used in the field of image recognition

4 A dataset widely used for image classification

5 Training completed in 31 minutes using Intel^（R） Xeon^（R） Platinum 8160 x 1,600（Y. You et al. ImageNet Training in Minutes. CoRR,abs/1709.05011, 2017）

■ About the Chainer Open Source Deep Learning Framework

Chainer is a Python-based deep learning framework being developed mainly by PFN, which has unique features and powerful performance that allow for designing complex neural networks easily and intuitively, thanks to its “Define-by-Run” approach. Since it was open-sourced in June 2015, as one of the most popular frameworks, Chainer has attracted not only the academic community but also many industrial users who need a flexible framework to harness the power of deep learning in their research and real-world applications.

Chainer incorporates the results of the latest deep learning research. With additional packages such as ChainerMN (distributed learning), ChainerRL (reinforcement learning), ChainerCV (computer vision) and through the support of Chainer development partner companies, PFN aims to promote the most advanced research and development activities of researchers and practitioners in each field. （http://chainer.org/）

■ About Preferred Networks, Inc.

Founded in March 2014 with the aim of promoting business utilization of deep learning technology focused on IoT, PFN advocates Edge Heavy Computing as a way to handle the enormous amounts of data generated by devices in a distributed and collaborative manner at the edge of the network, driving innovation in three priority business areas: transportation, manufacturing and bio/healthcare. PFN promotes advanced initiatives by collaborating with world leading organizations, such as Toyota Motor Corporation, Fanuc Corporation, and the National Cancer Center. (https://www.preferred.jp/en/)

＊Chainer^（R） is the trademark or the registered trademark of Preferred Networks, Inc. in Japan and other countries.

投稿 Preferred Networks achieved the world’s fastest training time in deep learning, completed training on ImageNet in 15 minutes,using the distributed learning package ChainerMN and a large-scale parallel computer は Preferred Networks, Inc. に最初に表示されました。

TOKYO, JAPAN — This September, Preferred Networks, Inc. (PFN), a provider of IoT-centric deep learning systems, NTT Communications Corporation (NTT Com), the ICT solutions and international communications business within the NTT Group, and NTT Com subsidiary NTT PC Communications Incorporated (NTTPC) announced today the launch of a private supercomputer designed to facilitate research and development of deep learning, including autonomous driving and cancer diagnosis.

The new supercomputer is one of the most powerful to be developed by the private sector in Japan and is equipped with NTT Com and NTTPC’s Graphics Processing Unit (GPU) platform, and contains 1,024 of NVIDIA(R)’s Tesla(R) multi-node P100 GPUs. Theoretically, the processing speed of the new supercomputer can reach 4.7 PetaFLOPS—a massive 4,700 trillion floating point operations per second—the fastest levels of any computing environment in Japan.

Overview of the private supercomputer

PFN’s deep learning research demands an ultra high-speed, high capacity, state-of-the-art computing environment. Existing GPU platforms require massive electricity supplies, generate excessive heat and offer inadequate network speed. To address these issues, PFN adopted the NTT Com Group’s proven GPU platform, which boasts significantly advanced technology. They additionally leveraged the latest data center design, building a large-scale multi-node platform using ChainerMN, PFN’s technology that significantly accelerates the speed of deep learning by parallelizing calculations over multiple nodes.

NTT Com group has developed and released a multi-node GPU platform on Enterprise Cloud, Nexcenter(TM), a world-leading data center service, which incorporates the group’s extensive know-how in GPU performance maximization.

Following the supercomputer launch, PFN plans to increase the processing speed of its open source deep learning framework Chainer. They will further accelerate their research and development in the field of transportation systems, manufacturing, bio and healthcare industry which require a huge amount of computing resources. PFN will additionally consider the deployment of NVIDIA(R) Tesla(R) V100 GPUs, which are based on next-generation Volta GPU technology. NTT Com group will continue to support PFN’s research and the commercialization of their developed solutions with AI-related technologies and platforms.

“NVIDIA is excited to see the launch of Preferred Networks’ private supercomputer, built in partnership with NTT Com Group. Computing power is the source of competitive advantage for deep learning, the core technology of modern AI. We have high expectations that the new system will accelerate Preferred Networks’ business and contribute to Japan’s economic growth.”

Masataka Osaki
NVIDIA Japan Country Manager, Vice President of Corporate Sales

Related links:

Chainer
Enterprise Cloud
Nexcenter

◆ About Preferred Networks, Inc.
Founded in March 2014 with the aim of promoting business utilization of deep learning technology focused on IoT, PFN advocates Edge Heavy Computing as a way to handle the enormous amounts of data generated by devices in a distributed and collaborative manner at the edge of the network, driving innovation in three priority business areas: transportation, manufacturing and bio/healthcare. PFN develops and provides Chainer, an open source deep learning framework. PFN promotes advanced initiatives by collaborating with world leading organizations, such as Toyota Motor Corporation, Fanuc Corporation, and the National Cancer Center.
https://www.preferred.jp/en/

◆ About NTT Communications Corporation
NTT Communications provides consultancy, architecture, security and cloud services to optimize the information and communications technology (ICT) environments of enterprises. These offerings are backed by the company’s worldwide infrastructure, including the leading global tier-1 IP network, the Arcstar Universal One VPN network reaching over 190 countries/regions, and 140 secure data centers worldwide. NTT Communications’ solutions leverage the global resources of NTT Group companies including Dimension Data, NTT DOCOMO and NTT DATA.
www.ntt.com | Twitter@NTT Com | Facebook@NTT Com | LinkedIn@NTT Com

◆ NTT PC Communications Incorporated
NTTPC Communications Incorporated (NTTPC), established in 1985 is a subsidiary of NTT Communications, is a network service and communication solution provider in Japanese telco market, the company has been the most strategic technology company of the group throughout of years. NTTPC launched the 1st ISP service of the NTT group, so called “InfoSphere” at 1995, and also launched the 1st Internet Data Center and server hosting services of Japan so called “WebARENA” at 1997. NTTPC have always started something new in ICT market.
http://www.nttpc.co.jp/english/

Notes
1. Chainer(R) is the trademark or the registered trademark of Preferred Networks, Inc. in Japan and other countries.

2. Other company names and product names written in this release are the trademarks or the registered trademarks of each company.

投稿 Preferred Networks Launches one of Japan’s Most Powerful Private Sector Supercomputers は Preferred Networks, Inc. に最初に表示されました。

For practical application of machine learning and deep learning technologies, the ever-increasing complexity of neural network models, with a large number of parameters and much larger training datasets requires more and more computational power to train these models.

ChainerMN is a multi-node extension to Chainer that realizes large-scale distributed deep learning by high-speed communications both intra- and inter-node. PFN released the beta version of ChainerMN on May 9, 2017 and this is the first official release. The following features have been added to ChainerMN v1.0.0.

● Features of ChainerMN v1.0.0

1. Increased stability in core functions during data parallelization

With this improved stability, ChainerMN can be used more comfortably.

2. Compatibility with NVIDIA Collective Communications Library (NCCL) 2.0.

By supporting the latest version, it has become even faster.

3. More sample code (machine translation, DCGAN) is available.

These examples will help users learn more advanced ways of using ChainerMN.

4. Expansion of supported environments (non-CUDA-Aware MPI).

CUDA – Aware MPI implementation such as Open MPI and MVAPICH was necessary for the beta version, but ChainerMN is now compatible with non-CUDA-Aware MPI.

5. Initial implementation of model parallelism functions.

More complex distributed learning has become possible by getting multiple GPUs to work in the model parallelism method.
The conventional data parallelism approach is known to limit the possible batch size when increasing the nodes while maintaining accuracy. o overcome this, we have done the initial part of the more challenging implementation of model parallelism for greater speed than possible with data parallelism.

These features provide a more stable and faster than ever deep learning experiences with ChainerMN, as well as improved usability.

The following is the result of the performance measurement of ChainerMN using the image classification dataset of ImageNet. It is about 1.4 times faster than the first announcement in January this year, and 1.1 times faster than the beta version released in May. Please visit the following Chainer Blog to learn more about the experiment settings:

https://chainer.org/general/2017/02/08/Performance-of-Distributed-Deep-Learning-Using-ChainerMN.html

In addition, from October 2017, ChainerMN will become available on “XTREME DNA”, an unmanned  cloud-based super-computer deployment and operation service, provided by XTREME Design Inc.(Head office: Shinagawa-ku, Tokyo, CEO: Naoki Shibata)

ChainerMN will be added on the distributed parallel environment templates for GPU instances of the pay-per-load public cloud, Microsoft Azure. This not only eliminates the need to construct infrastructure required for large-scale distributed deep learning but also makes it easy to manage research-and-development costs.

ChainerMN aims to provide an environment in which deep learning researchers and developers can easily concentrate on the main parts of research and development including the design of neural networks. PFN will continue to improve ChainerMN by adding more features and expanding its usage environment.

◆ The Open Source Deep Learning Framework Chainer (http://chainer.org)
Chainer is a Python-based deep learning framework developed by PFN, which has unique features and powerful performance that enables users to easily and intuitively design complex neural networks, thanks to its “Define-by-Run” approach. Since it was open-sourced in June 2015, as one of the most popular frameworks, Chainer has attracted not only the academic community but also many industrial users who need a flexible framework to harness the power of deep learning in their research and real-world applications.

※1：MN in ChainerMN stands for Multi-Node. https://github.com/pfnet/chainermn

投稿 Preferred Networks officially released ChainerMN version 1.0.0, a multi-node distributed learning package, making it even faster with stablized data-parallel core functions は Preferred Networks, Inc. に最初に表示されました。