Automatic Distributed TrainingΒΆ
Supporting mulitple parallel training schemas
Data parallelism
Model parallelism
Parameter parallelism
Operator parallelism
Pipeline parallelism
Subgraph parallelism
Optimizer parallelism
Supporting popular distributed communication architectures
Parameter server
Collective communication
Supporting topology-aware automatic distributed training
Device placement
Cost estimation
Dynamic programming
