https://gitee.com/mindspore/mindspore/tree/r1.3/model_zoo/official/cv/resnet_thor

【操作步骤&问题现象】

1、拉取mindspore 1.3的镜像：

docker pull swr.cn-south-1.myhuaweicloud.com/mindspore/mindspore-gpu:1.3.0

2、运行镜像

docker run -it -v /dev/shm:/dev/shm -v /path/to/dataset:/path/to/dataset --runtime=nvidia --privileged=true swr.cn-south-1.myhuaweicloud.com/mindspore/mindspore-gpu:1.3.0

3、在8卡v100机器上训练

sh run_distribute_train_gpu.sh /path/to/dataset 8

epoch: 32 step: 5004, loss is 2.2936077
epoch: 32 step: 5004, loss is 2.403049
epoch: 32 step: 5004, loss is 1.8597395
epoch: 32 step: 5004, loss is 2.1784513
epoch: 32 step: 5004, loss is 2.1430013
epoch: 32 step: 5004, loss is 1.8287767
epoch: 32 step: 5004, loss is 1.927578
epoch: 32 step: 5004, loss is 1.8246645
epoch time: 5372604.995 ms, per step time: 1073.662 ms
epoch time: 5372619.719 ms, per step time: 1073.665 ms
epoch time: 5372814.251 ms, per step time: 1073.704 ms
epoch time: 5372513.237 ms, per step time: 1073.644 ms
epoch time: 5372558.263 ms, per step time: 1073.653 ms
epoch time: 5372484.428 ms, per step time: 1073.638 ms
epoch time: 5372676.158 ms, per step time: 1073.676 ms
epoch time: 5372801.586 ms, per step time: 1073.701 ms

可以看到一个epoch需要一个多小时，非常慢

您好，根据您提供的信息我拉取了resnet_thor的1.3版本在8卡V100上进行测试，性能结果是正常的，平均每个step的时间是80ms左右，每个epoch不到7min。请您检查下在训练过程中，是否有其他的进程在使用CPU或者GPU，这会导致训练的性能下降。同时您可以参考MindSpore的性能调试教程：https://www.mindspore.cn/mindinsight/docs/zh-CN/r1.3/performance_profiling_gpu.html，使用profiling进一步确认问题。