Hyperparameter importance graphic, showing which hyperparameters in a neural network matter most

Optuna v2.0 has the following new features:

• Hyperparameter importance evaluation

Optuna can provide feedback on how important each of the measured hyperparameters were on the overall performance of the algorithm. This valuable information can assist researchers and developers to focus on tuning the hyperparameters that matter most.

• Hyperband pruning

Pruning allows unpromising trials to be stopped early. One of the most recent and robust techniques for pruning is Hyperband, which is well-suited for deep learning and is now available in Optuna.

• Performance improvements

Optimization has been speeded up by improving the lower storage layer. Experiments show that searches can be up to ten times faster.

• Additional integrations

Additional integration modules are available for easy use with LightGBM to do efficient stepwise optimization as well as MLflow, AllenNLP, and TensorBoard.

Since open-sourced in December 2018, the interest from researchers and developers for Optuna has grown. In the last month, Optuna was downloaded over 100,000 times. Going forward, PFN plans to work on multi-objective optimization to allow multiple criteria to be optimized simultaneously, along with continuing to add integrations and improve the performance of Optuna.

About Optuna
Optuna was open-sourced by PFN in December 2018 as a hyperparameter optimization framework written in Python. Optuna automates the trial-and-error process of finding hyperparameters that deliver good performance. Optuna is used in many PFN projects and was an important factor in PFDet team’s award-winning performances in the first Kaggle Open Images object detection competition. https://optuna.org/

投稿 Preferred Networks Releases Optuna v2.0 は Preferred Networks, Inc. に最初に表示されました。

The pytorch-pfn-extras library includes several popular Chainer functions from user feedback during PFN’s transition from the Chainer deep learning framework to PyTorch.

pytorch-pfn-extras includes the following features:

• Extensions and reporter

Functions frequently used when implementing deep learning training programs, such as collecting metrics during training and visualizing training progress

• Automatic inference of parameter sizes

Easier network definitions by automatically inferring the sizes of linear or convolution layer parameters via input sizes

• Distributed snapshots

Reduce the costs of implementing distributed deep learning with automated backup, loading, and generation management of snapshots

pytorch-pfn-extras is available at: https://github.com/pfnet/pytorch-pfn-extras 

The migration guide from Chainer to PyTorch can also be found at: https://medium.com/pytorch/migration-from-chainer-to-pytorch-8ed92c12c8 

On April 6, Optuna was added to the PyTorch Ecosystem of tools that are officially endorsed by the PyTorch community for use in PyTorch-based machine learning and deep learning research and development.

PFN is discussing merging pytorch-pfn-extras features into the PyTorch base build with the PyTorch development team at Facebook, Inc. In response to strong demand from both internal and external users, PFN also aims to release a PyTorch version of the deep reinforcement learning library, ChainerRL, as open-source software by the end of June 2020.  

PFN aims to continue leveraging its software technology it has accumulated through the development of Chainer to contribute to the development of PyTorch and the open-source community.

The PyTorch team at Facebook commented:

“We appreciate PFN for contributing important Chainer functions, such as gathering metrics and managing distributed snapshots, through pytorch-pfn-extras. With this newly available library, PyTorch developers have the ability to understand their model performances and optimize training costs. We look forward to continued collaboration with PFN to bring more contributions to the community, like ChainerRL capabilities later this summer.”

投稿 Preferred Networks Deepens Collaboration with PyTorch Community は Preferred Networks, Inc. に最初に表示されました。

In machine learning and deep learning, it is critical that complex hyperparameters *1, which control the behavior of an algorithm during the training process, are optimized to deliver a trained model with better accuracy.
Optuna automates the trial-and-error process of optimizing hyperparameters. It finds hyperparameter values that enable the algorithm to give good performance. Since its beta version release as open-source software (OSS) in December 2018, Optuna has received development support from numerous contributors and added a number of new features based on feedbacks from the OSS community as well as in the company.

Main features of Optuna v1.0 include:

•  Efficient hyperparameter tuning with state of the art optimization algorithms
•  Support for various machine learning libraries including PyTorch, TensorFlow, Keras, FastAI, scikit-learn, LightGBM, and XGBoost
•  Support for parallel execution across multiple computing machines to significantly reduce the optimization time
•  Search space can be described by Python control statements
•  Various visualization techniques that allow users to conduct diverse analyses of the optimization results

Official website of Optuna: https://optuna.org/

Optuna has received many contributions from external developers. PFN will continue to quickly incorporate the results of the latest machine learning research into the development of Optuna and work with the OSS community to promote the use of Optuna.

*1：Hyperparameters include learning rate, batch size, number of training iterations, number of neural network layers, and number of channels.

About the hyperparameter optimization framework for machine learning Optuna
Optuna was open-sourced by PFN in December 2018 as a hyperparameter optimization framework written in Python. Optuna automates the trial-and-error process of finding hyperparameters that deliver good performance. Optuna is used in many PFN projects and was an important factor in PFDet team’s award-winning performances in the first Kaggle Open Images object detection competition.
https://optuna.org/

投稿 Preferred Networks Releases Optuna v1.0, Open-source Hyperparameter Optimization Framework for Machine Learning は Preferred Networks, Inc. に最初に表示されました。

In deep learning and machine learning, it is essential to tune hyperparameters since they control how an algorithm behaves. The precision of a model largely depends on tuning the hyperparameters. The number of hyperparameters tends to be high especially in deep learning. They include the numbers of training iterations, neural network layers and channels, learning rate, batch size, and others. Nevertheless, many deep learning researchers and engineers manually tune these hyperparameters and spend a significant amount of their time doing so.

Optuna automates the trial-and-error process of optimizing the hyperparameters. It automatically finds optimal hyperparameter values that enable the algorithm to give excellent performance. Optuna can be used not only with the Chainer open-source deep learning framework, but also with other machine learning software.

Main features of Optuna are:

• Define-by-Run style API

Optuna can optimize complex hyperparameters while maintaining high modularity.

• Pruning of trials based on learning curves

Optuna predicts the result of training with an iterative algorithm based on a learning curve. It halts unpromising trials to enable an efficient optimization process.

• Parallel distributed optimization

Optuna supports asynchronous distributed optimization and simultaneously performs multiple trials using multiple nodes.

Optuna is used in PFN projects and with good results. One example is the second place award in the Google AI Open Images 2018– Object Detection Track competition. PFN will continue to develop Optuna, while prototyping and implementing advanced functionalities.

• Official Website https://optuna.org
• Official Document https://optuna.readthedocs.io/en/stable/
• Tutorial https://optuna.readthedocs.io/en/stable/tutorial/index.html
• GitHub https://github.com/pfnet/optuna
• Developers Blog https://research.preferred.jp/2018/12/optuna-release/

* Chainer and Optuna are the trademarks or the registered trademarks of Preferred Networks, Inc. in Japan and elsewhere.

投稿 Preferred Networks releases the beta version of Optuna, an automatic hyperparameter optimization framework for machine learning, as open-source software は Preferred Networks, Inc. に最初に表示されました。