Development plan

Recently, DL system optimization research has been widely studied and published by high-level conferences on machine learning and computation systems (e.g., OSDI, SOSP, SIGMOD, VLDB, MLSys, ASPLOS, ATC, EuroSys, PPoPP, etc.). Our Hetu development team has obtained many academic achievements and technological accumulations. We welcome all of you to join this challenging approach and we prepared two projects for you to select:

Project 1. Distributed Optimization

With the increasing amounts of data and the growing size of models, distributed DL is becoming more and more important. High-performance distributed training is a crucial challenge for emerging large DL models, such as GPT-3 and PanGu, which consist of trillions of parameters and require thousands of GPUs for distributed training. This project targets: 1) integrating data/model/pipeline parallelism to improve the training efficiency; 2) re-design the distributed system interfaces and architectures to support automatically distributed training over Hetu. (either 1) or 2))

Project 2. Certain Algorithm Optimization

In recent years, many ML researchers have focused their attention intensely on certain DL models due to their powerful performance. Making system optimization on these models is significant when scaling them to large-scale and high-dimensional data inputs. This project targets certain DL models (i.e., GNN, Transformer, Knowledge Graph Embedding) and developing individual open-source modules with GPU or distributed acceleration over Hetu.

If you have interests in these problems and want to join us, welcome to contact: xupeng.miao@pku.edu.cn