https://gitee.com/mindspore/mindspore/tree/r1.3/model_zoo/official/cv/resnet_thor

【操作步骤&问题现象】

1、拉取mindspore 1.3的镜像：

docker pull swr.cn-south-1.myhuaweicloud.com/mindspore/mindspore-gpu:1.3.0

2、运行镜像

docker run -it -v /dev/shm:/dev/shm -v /path/to/dataset:/path/to/dataset --runtime=nvidia --privileged=true swr.cn-south-1.myhuaweicloud.com/mindspore/mindspore-gpu:1.3.0

3、在8卡v100机器上训练

sh run_distribute_train_gpu.sh /path/to/dataset 8

【日志信息】（可选，上传日志内容或者附件）

epoch: 32 step: 5004, loss is 2.2936077
epoch: 32 step: 5004, loss is 2.403049
epoch: 32 step: 5004, loss is 1.8597395
epoch: 32 step: 5004, loss is 2.1784513
epoch: 32 step: 5004, loss is 2.1430013
epoch: 32 step: 5004, loss is 1.8287767
epoch: 32 step: 5004, loss is 1.927578
epoch: 32 step: 5004, loss is 1.8246645
epoch time: 5372604.995 ms, per step time: 1073.662 ms
epoch time: 5372619.719 ms, per step time: 1073.665 ms
epoch time: 5372814.251 ms, per step time: 1073.704 ms
epoch time: 5372513.237 ms, per step time: 1073.644 ms
epoch time: 5372558.263 ms, per step time: 1073.653 ms
epoch time: 5372484.428 ms, per step time: 1073.638 ms
epoch time: 5372676.158 ms, per step time: 1073.676 ms
epoch time: 5372801.586 ms, per step time: 1073.701 ms

可以看到一个epoch需要一个多小时，非常慢

解答：

目前发现原因是数据集放在NAS中导致数据IO时间异常缓慢，将数据放到本地后训练速度正常。

epoch time: 578507.771 ms, per step time: 115.609 ms
epoch time: 578507.287 ms, per step time: 115.609 ms
epoch time: 578551.361 ms, per step time: 115.618 ms
epoch time: 578576.957 ms, per step time: 115.623 ms
epoch time: 578575.861 ms, per step time: 115.623 ms
epoch time: 578589.717 ms, per step time: 115.625 ms
epoch time: 578743.663 ms, per step time: 115.656 ms
epoch time: 578714.027 ms, per step time: 115.650 ms