---Optimal Designs for Hyperparameter Optimization
On the afternoon of October 20, 2020, Dr. Huang Yimin, AI Basic Theory Researcher at Huawei Noah's Ark Laboratory, was invited to our institute to give a lecture on the industry forum with the theme of "Optimal Designs for Hyperparameter Optimization". Dr. Huang Yimin is a PhD candidate in the Department of Probability and Statistics at the School of Mathematical Sciences, Peking University, with a research focus on experimental design. Joined Noah's Laboratory in 2019, applying statistical theory to machine learning, conducting research in hyperparameter optimization, and supporting AutoML large particle technology.
Firstly, Dr. Huang Yimin started by introducing the basic concepts, including what hyperparameters are, the role of hyperparameter optimization, and the commercial value and challenges of hyperparameter optimization. He also used companies such as Google, Amazon, and Microsoft as examples to introduce the application scenarios of hyperparameter optimization. Then, Dr. Huang Yimin introduced the development of hyperparameter optimization, provided a brief overview of the excellent methods proposed in hyperparameter optimization in recent years, and elaborated on the statistical problems they represent, such as Bayesian optimization processes.
Afterwards, Dr. Huang Yimin introduced the latest work of the AI Foundation Laboratory in this field, including the optimization problem of black box sub functions based on model free methods and multi fidelity methods. He used cheaper black box variants to asymptotically measure the performance of a hyperparameter combination, provided corresponding experimental results, and summarized the two methods.
Finally, Dr. Huang Yimin introduced several open questions and future research directions. There are more and more hyperparameter optimization methods, and a fair and reasonable testing platform is needed to scientifically evaluate various methods; Research on overfitting issues, etc.
During the Q&A session, Dr. Huang Yimin provided detailed answers to the efficiency issues raised by classmates regarding the model free and multi fidelity methods. Hyperparameter optimization can greatly improve project efficiency; Regarding the importance of determining the search space and hyperparameter optimization algorithms raised by classmates, Dr. Huang Yimin used his project experience to carefully explain to classmates that these two steps are equally important in the project and there is no bias issue.
This industry forum lecture helped students understand the excellent methods of hyperparameter optimization, stimulated their interest in researching hyperparameter optimization and the statistical problems behind it, and has strong guiding significance for their future learning.
Contributors: Cao Wendong, Liu Jinwenang
Image provided by: Ye Xufeng
