[3D-parallelism] Hybrid Model Parallelism · Pull Request !136 · PaddlePaddle/Paddle

### PR types
New features

### PR changes
APIs

### Describe
* new features
  * Hybrid Model Parallelism： 
       * Combine 3 single Model parallelism strategies (megatron, sharding, pipline) together as a Hybrid Parallelism strategy
       * Uniform switch to turn on/off each single parallelism strategy (temporally use Sharding & Pipeline as the uniform API)
* performance optimization：
    * speed
        *  remove potential un-necessary sync_calc & sync_calc in hybrid model parallelism
    * Memory usage
        * optimizer offload
        * optimizer temporary vars inplacement  [commit withdraw, update in next pr]

* **performance-related**
    *  the order of parallelism from inner to outer is : mp --> sharidng --> pp
    *  mp (megatron) and sharding parallelism will introduce large communication and is recommend: **to be used within node. (mp_degree * sharding_degree = number of gpu per node)**
    * pp parallelism have less communication load compared with above two, which make them **more suitable to use across the nodes  (pp_degree = number node)**

example
* assume we have 4 nodes with 8 gpus per node:

*  mp-sharding-pp 3D parallelism
```python
        dist_strategy.sharding = True
        dist_strategy.pipeline = True
        dist_strategy.sharding_configs = {"segment_broadcast_MB": 32,
                                            "pp_degree": 4,
                                            "sharding_degree":4,
                                            "mp_degree": 2,
                                            "optimize_offload": True,
                                            }
        dist_strategy.pipeline_configs = {"schedule_mode": "1F1B",
                                            "micro_batch_size": 1,
                                            "accumulate_steps": 4,
                                            }

```

*  mp-pp 2D parallelism
```python
        dist_strategy.sharding = True
        dist_strategy.pipeline = True
        dist_strategy.sharding_configs = {"segment_broadcast_MB": 32,
                                            "pp_degree": 4,
                                            "sharding_degree":1,
                                            "mp_degree": 8,
                                            "optimize_offload": True,
                                            }
        dist_strategy.pipeline_configs = {"schedule_mode": "1F1B",
                                            "micro_batch_size": 1,
                                            "accumulate_steps": 4,
                                            }
```

* shardign-pp 2D parallelism
```python
        dist_strategy.sharding = True
        dist_strategy.pipeline = True
        dist_strategy.sharding_configs = {"segment_broadcast_MB": 32,
                                            "pp_degree": 1,
                                            "sharding_degree":4,
                                            "mp_degree": 8,
                                            "optimize_offload": True,
                                            }
        dist_strategy.pipeline_configs = {"schedule_mode": "1F1B",
                                            "micro_batch_size": 1,
                                            "accumulate_steps": 4,
                                            }

```

*  mp-sharding 2D parallelism
```python
        dist_strategy.sharding = True
        dist_strategy.sharding_configs = {"segment_broadcast_MB": 32,
                                            "pp_degree": 1,
                                            "sharding_degree":4,
                                            "mp_degree": 8,
                                            "optimize_offload": False,
                                            "gradient_merge_acc_step": 4,
                                            }

```

GVPPaddlePaddle/Paddle

搜索帮助

GVP PaddlePaddle/Paddle