============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/profiler, inifile: /home/jenkins/sault/virtual_test/virtualenv_002/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(149696,python3.7):2024-01-11-06:04:42.279.345 [trace_attr.c:105](tid:149696) platform is 1. [INFO] ATRACE(149696,python3.7):2024-01-11-06:04:42.279.490 [trace_recorder.c:114](tid:149696) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(149696,python3.7):2024-01-11-06:04:42.279.517 [trace_signal.c:133](tid:149696) register signal handler for signo 2 succeed. [INFO] ATRACE(149696,python3.7):2024-01-11-06:04:42.279.530 [trace_signal.c:133](tid:149696) register signal handler for signo 15 succeed. [INFO] RUNTIME(149696,python3.7):2024-01-11-06:04:42.684.081 [runtime.cc:1159] 149696 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(149696,python3.7):2024-01-11-06:04:42.684.126 [runtime.cc:4719] 149696 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 7 items / 3 deselected / 4 selected test_env_enable_profiler.py [INFO] ATRACE(149999,python):2024-01-11-06:04:48.429.165 [trace_attr.c:105](tid:149999) platform is 1. [INFO] ATRACE(149999,python):2024-01-11-06:04:48.429.313 [trace_recorder.c:114](tid:149999) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(149999,python):2024-01-11-06:04:48.429.343 [trace_signal.c:133](tid:149999) register signal handler for signo 2 succeed. [INFO] ATRACE(149999,python):2024-01-11-06:04:48.429.356 [trace_signal.c:133](tid:149999) register signal handler for signo 15 succeed. [INFO] RUNTIME(149999,python):2024-01-11-06:04:48.835.035 [runtime.cc:1159] 149999 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(149999,python):2024-01-11-06:04:48.835.091 [runtime.cc:4719] 149999 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set [INFO] TDT(149999,python):2024-01-11-06:04:52.806.861 [process_mode_manager.cpp:109][OpenProcess][tid:149999] [ProcessModeManager] enter into open process deviceId[1] rankSize[0] [INFO] TDT(149999,python):2024-01-11-06:04:52.809.101 [process_mode_manager.cpp:379][InitTsdClient][tid:149999] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(149999,python):2024-01-11-06:04:52.809.238 [version_verify.cpp:34][SetVersionInfo][tid:149999] VersionVerify: send client version to server [INFO] TDT(149999,python):2024-01-11-06:04:52.809.265 [version_verify.cpp:50][SetVersionInfo][tid:149999] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(149999,python):2024-01-11-06:04:52.809.278 [version_verify.cpp:50][SetVersionInfo][tid:149999] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(149999,python):2024-01-11-06:04:52.809.619 [version_verify.cpp:66][PeerVersionCheck][tid:149999] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(149999,python):2024-01-11-06:04:52.809.637 [version_verify.cpp:87][ParseVersionInfo][tid:149999] VersionVerify: pass client version info success [INFO] TDT(149999,python):2024-01-11-06:04:52.809.649 [hdc_client.cpp:276][CheckHdcConnection][tid:149999] Service[2] create hdc success [INFO] TDT(149999,python):2024-01-11-06:04:52.809.664 [version_verify.cpp:120][SpecialFeatureCheck][tid:149999] VersionVerify: new type[35], supported [INFO] TDT(149999,python):2024-01-11-06:04:52.809.714 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:149999] [TsdClient][deviceId=1] [sessionId=1] wait package info respond [INFO] TDT(149999,python):2024-01-11-06:04:52.809.848 [process_mode_manager.cpp:379][InitTsdClient][tid:149999] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(149999,python):2024-01-11-06:04:52.810.103 [version_verify.cpp:34][SetVersionInfo][tid:149999] VersionVerify: send client version to server [INFO] TDT(149999,python):2024-01-11-06:04:52.810.115 [version_verify.cpp:50][SetVersionInfo][tid:149999] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(149999,python):2024-01-11-06:04:52.810.136 [version_verify.cpp:50][SetVersionInfo][tid:149999] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(149999,python):2024-01-11-06:04:52.810.270 [version_verify.cpp:66][PeerVersionCheck][tid:149999] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(149999,python):2024-01-11-06:04:52.810.283 [version_verify.cpp:87][ParseVersionInfo][tid:149999] VersionVerify: pass client version info success [INFO] TDT(149999,python):2024-01-11-06:04:52.810.292 [hdc_client.cpp:276][CheckHdcConnection][tid:149999] Service[2] create hdc success [INFO] TDT(149999,python):2024-01-11-06:04:52.810.303 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:149999] [TsdClient] tsd get process sign successfully, procpid[149999] signSize[48] [INFO] TDT(149999,python):2024-01-11-06:04:52.810.315 [version_verify.cpp:112][SpecialFeatureCheck][tid:149999] VersionVerify: previous type[6], supported [INFO] TDT(149999,python):2024-01-11-06:04:52.810.336 [process_mode_manager.cpp:126][OpenProcess][tid:149999] [ProcessModeManager] deviceId[1] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(149999,python):2024-01-11-06:04:53.105.859 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:149999] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(149999,python):2024-01-11-06:04:53.105.898 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:149999] enter into OpenInHost deviceid[1] [INFO] TDT(149999,python):2024-01-11-06:04:53.105.912 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:149999] host cpu not support [INFO] TDT(149999,python):2024-01-11-06:04:53.105.922 [process_mode_manager.cpp:156][OpenProcess][tid:149999] [TsdClient][deviceId=1] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(149999,python):2024-01-11-06:04:53.108.624 [device.cc:340] 149999 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(149999,python):2024-01-11-06:04:53.125.411 [npu_driver.cc:5428] 150133 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(149999,python):2024-01-11-06:04:53.125.508 [atrace_api.c:28](tid:149999) AtraceCreate start [INFO] ATRACE(149999,python):2024-01-11-06:04:53.125.591 [trace_rb_log.c:84](tid:149999) [RUNTIME_ATRACE_DEV1_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(149999,python):2024-01-11-06:04:53.125.607 [atrace_api.c:32](tid:149999) AtraceCreate end [INFO] TDT(149999,python):2024-01-11-06:04:53.125.623 [client_manager.cpp:157][SetProfilingCallback][tid:149999] [TsdClient] set profiling callback success [INFO] PROFILING(149999,python):2024-01-11-06:04:53.143.460 [msprofiler_impl.cpp:156] >>> (tid:149999) ProfNotifySetDevice called, is open: 1, devId: 1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.143.515 [msprofiler_impl.cpp:289] >>> (tid:149999) Get system free ram: 557890473984 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.143.532 [prof_cann_plugin.cpp:75] >>> (tid:149999) Init report buffer size: 131072 bytes, buffer name: api_event [INFO] PROFILING(149999,python):2024-01-11-06:04:53.147.085 [prof_cann_plugin.cpp:75] >>> (tid:149999) Init report buffer size: 131072 bytes, buffer name: compact [INFO] PROFILING(149999,python):2024-01-11-06:04:53.151.417 [prof_cann_plugin.cpp:75] >>> (tid:149999) Init report buffer size: 262144 bytes, buffer name: additional [INFO] PROFILING(149999,python):2024-01-11-06:04:53.177.421 [platform.cpp:38] >>> (tid:149999) Profiling platform version: 1.0. [INFO] PROFILING(149999,python):2024-01-11-06:04:53.177.455 [ai_drv_dev_api.cpp:384] >>> (tid:149999) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.177.582 [prof_acl_mgr.cpp:286] >>> (tid:149999) Received ProfAclInit request from acl [INFO] PROFILING(149999,python):2024-01-11-06:04:53.177.645 [msprof_reporter.cpp:98] >>> (tid:149999) Init all reporters [INFO] PROFILING(149999,python):2024-01-11-06:04:53.179.026 [prof_acl_mgr.cpp:350] >>> (tid:149999) Received ProfAclStart request from acl [INFO] PROFILING(149999,python):2024-01-11-06:04:53.179.148 [ai_drv_dev_api.cpp:384] >>> (tid:149999) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.181.044 [hdc_api.cpp:112] >>> (tid:149999) logDevId 1 create HDC server successfully [INFO] PROFILING(149999,python):2024-01-11-06:04:53.183.205 [prof_manager.cpp:384] >>> (tid:149999) Received libmsprof message to start profiling, job_id:1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.183.246 [prof_manager.cpp:152] >>> (tid:149999) Check device profiling status [INFO] PROFILING(149999,python):2024-01-11-06:04:53.183.272 [prof_manager.cpp:272] >>> (tid:149999) Begin to launch task, jobId:1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.183.286 [prof_acl_mgr.cpp:95] >>> (tid:150141) Device 1 started to wait for response [INFO] PROFILING(149999,python):2024-01-11-06:04:53.183.346 [prof_acl_mgr.cpp:2258] >>> (tid:149999) Init profiling for msproftx [INFO] PROFILING(149999,python):2024-01-11-06:04:53.183.362 [prof_acl_mgr.cpp:2376] >>> (tid:149999) MsprofSetDeviceImpl, devId:64 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.184.806 [ai_drv_prof_api.cpp:33] >>> (tid:150143) Begin to get channels, deviceId=1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.139 [prof_manager.cpp:384] >>> (tid:149999) Received libmsprof message to start profiling, job_id:64 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.163 [prof_acl_mgr.cpp:95] >>> (tid:150155) Device 64 started to wait for response [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.171 [prof_manager.cpp:152] >>> (tid:149999) Check device profiling status [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.200 [prof_manager.cpp:272] >>> (tid:149999) Begin to launch task, jobId:64 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.703 [ai_drv_prof_api.cpp:66] >>> (tid:150143) End to get channels[17], deviceId=1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.760 [prof_acl_mgr.cpp:85] >>> (tid:150157) Device 64 finished starting [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.788 [prof_acl_mgr.cpp:97] >>> (tid:150155) Device 64 finished waiting for response [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.907 [prof_acl_mgr.cpp:1422] >>> (tid:149999) Device:64 finished waiting [INFO] PROFILING(149999,python):2024-01-11-06:04:53.185.926 [ai_drv_dev_api.cpp:384] >>> (tid:149999) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.187.360 [ai_drv_prof_api.cpp:436] >>> (tid:150143) Begin to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.208.780 [ai_drv_prof_api.cpp:454] >>> (tid:150143) Succeeded to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.210.476 [ai_drv_prof_api.cpp:296] >>> (tid:150143) Begin to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43, configSize:56bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.210.503 [ai_drv_prof_api.cpp:298] >>> (tid:150143) DrvAicoreTaskBasedStart, event_num=7, events=0x49,0x4a,0x4b,0x4c,0x4d,0x4e,0x4f,, tag=0 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.211.645 [ai_drv_prof_api.cpp:319] >>> (tid:150143) Succeeded to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.213.199 [ai_drv_prof_api.cpp:591] >>> (tid:150143) Begin to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45, tag=0 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.213.794 [ai_drv_prof_api.cpp:606] >>> (tid:150143) Succeeded to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.215.300 [ai_drv_prof_api.cpp:618] >>> (tid:150143) Begin to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.215.827 [ai_drv_dev_api.cpp:384] >>> (tid:150142) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.215.894 [ai_drv_prof_api.cpp:633] >>> (tid:150143) Succeeded to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.215.901 [ai_drv_dev_api.cpp:384] >>> (tid:150142) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.215.962 [prof_acl_mgr.cpp:85] >>> (tid:150143) Device 1 finished starting [INFO] PROFILING(149999,python):2024-01-11-06:04:53.215.985 [prof_acl_mgr.cpp:97] >>> (tid:150141) Device 1 finished waiting for response [INFO] PROFILING(149999,python):2024-01-11-06:04:53.216.039 [ai_drv_dev_api.cpp:384] >>> (tid:150156) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.216.104 [prof_task.cpp:285] >>> (tid:150142) ProfTask 1 started to wait for task stop cv [INFO] PROFILING(149999,python):2024-01-11-06:04:53.216.148 [prof_acl_mgr.cpp:1413] >>> (tid:149999) All devices finished waiting [INFO] PROFILING(149999,python):2024-01-11-06:04:53.216.189 [prof_task.cpp:285] >>> (tid:150156) ProfTask 64 started to wait for task stop cv [INFO] TDT(149999,python):2024-01-11-06:04:53.216.771 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:149999] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=7] [INFO] TDT(149999,python):2024-01-11-06:04:53.216.797 [version_verify.cpp:112][SpecialFeatureCheck][tid:149999] VersionVerify: previous type[30], supported [INFO] TDT(149999,python):2024-01-11-06:04:53.216.885 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:149999] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] TDT(149999,python):2024-01-11-06:04:53.278.569 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:149999] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=6] [INFO] TDT(149999,python):2024-01-11-06:04:53.278.632 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:149999] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(149999,python):2024-01-11-06:04:53.278.995 [prof_reporter_mgr.cpp:226] >>> (tid:149999) total_size_type_info[5000], save type info length: 4544 bytes, type info size: 183 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.279.023 [prof_reporter_mgr.cpp:226] >>> (tid:149999) total_size_type_info[5500], save type info length: 35 bytes, type info size: 2 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.279.056 [prof_reporter_mgr.cpp:226] >>> (tid:149999) total_size_type_info[10000], save type info length: 404 bytes, type info size: 15 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.279.076 [prof_reporter_mgr.cpp:226] >>> (tid:149999) total_size_type_info[15000], save type info length: 67 bytes, type info size: 3 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.279.091 [uploader_dumper.cpp:178] >>> (tid:149999) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.137 [uploader_dumper.cpp:182] >>> (tid:149999) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.150 [uploader_dumper.cpp:178] >>> (tid:149999) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.179 [file_slice.cpp:356] >>> (tid:149999) [FileSliceFlush]file:aging.compact.task_track.slice_, total_size_file:768 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.350 [uploader_dumper.cpp:182] >>> (tid:149999) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.365 [uploader_dumper.cpp:178] >>> (tid:149999) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.378 [uploader_dumper.cpp:182] >>> (tid:149999) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.393 [prof_acl_mgr.cpp:435] >>> (tid:149999) Received ProfAclStop request from acl [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.416 [prof_manager.cpp:382] >>> (tid:149999) Received libmsprof message to stop profiling, job_id:1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.447 [prof_manager.cpp:305] >>> (tid:149999) Begin to stop task, jobId:1 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.463 [prof_task.cpp:344] >>> (tid:149999) Task send finished cv [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.482 [prof_task.cpp:287] >>> (tid:150142) ProfTask 1 finished waiting for task stop cv [INFO] PROFILING(149999,python):2024-01-11-06:04:53.280.568 [ai_drv_prof_api.cpp:642] >>> (tid:150143) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.281.179 [ai_drv_prof_api.cpp:650] >>> (tid:150143) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.281.200 [prof_channel.cpp:89] >>> (tid:150143) device id 1, channel: 44, total_size_channel: 80 bytes, file:data/ts_track.data, job_id:1,drvChannelReadCont:4 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.281.281 [ai_drv_prof_api.cpp:642] >>> (tid:150143) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.281.877 [ai_drv_prof_api.cpp:650] >>> (tid:150143) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.281.898 [prof_channel.cpp:89] >>> (tid:150143) device id 1, channel: 43, total_size_channel: 0 bytes, file:data/aicore.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.282.332 [ai_drv_prof_api.cpp:642] >>> (tid:150143) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.282.968 [ai_drv_prof_api.cpp:650] >>> (tid:150143) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.282.982 [prof_channel.cpp:89] >>> (tid:150143) device id 1, channel: 45, total_size_channel: 325120 bytes, file:data/hwts.data, job_id:1,drvChannelReadCont:16 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.283.026 [ai_drv_prof_api.cpp:642] >>> (tid:150143) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.283.797 [ai_drv_prof_api.cpp:650] >>> (tid:150143) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.283.813 [prof_channel.cpp:89] >>> (tid:150143) device id 1, channel: 46, total_size_channel: 0 bytes, file:data/training_trace.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.285.880 [prof_channel.cpp:407] >>> (tid:150143) ChannelPoll count: 36, Sleep count: 24, Dispatch count: 12, DispatchChannel count: 11 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.149 [file_slice.cpp:356] >>> (tid:150142) [FileSliceFlush]file:hwts.data.1.slice_, total_size_file:325120 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.326 [file_slice.cpp:356] >>> (tid:150142) [FileSliceFlush]file:ts_track.data.1.slice_, total_size_file:80 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.724 [prof_task.cpp:76] >>> (tid:150142) Uninit ProfTask succesfully [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.742 [prof_task.cpp:339] >>> (tid:150142) Task 1 finished [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.876 [prof_manager.cpp:382] >>> (tid:149999) Received libmsprof message to stop profiling, job_id:64 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.892 [prof_manager.cpp:305] >>> (tid:149999) Begin to stop task, jobId:64 [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.904 [prof_task.cpp:344] >>> (tid:149999) Task send finished cv [INFO] PROFILING(149999,python):2024-01-11-06:04:53.286.922 [prof_task.cpp:287] >>> (tid:150156) ProfTask 64 finished waiting for task stop cv [INFO] PROFILING(149999,python):2024-01-11-06:04:53.287.226 [prof_task.cpp:76] >>> (tid:150156) Uninit ProfTask succesfully [INFO] PROFILING(149999,python):2024-01-11-06:04:53.287.245 [prof_task.cpp:339] >>> (tid:150156) Task 64 finished [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.262 [uploader_dumper.cpp:178] >>> (tid:149999) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.289 [uploader_dumper.cpp:182] >>> (tid:149999) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.656 [receive_data.cpp:353] >>> (tid:149999) total_size_report module:api_event, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.684 [uploader_dumper.cpp:178] >>> (tid:149999) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.698 [uploader_dumper.cpp:182] >>> (tid:149999) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.910 [receive_data.cpp:353] >>> (tid:149999) total_size_report module:compact, push count:0, pop count:12, push size:0 bytes, pop size:768 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.930 [uploader_dumper.cpp:178] >>> (tid:149999) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(149999,python):2024-01-11-06:04:53.288.942 [uploader_dumper.cpp:182] >>> (tid:149999) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(149999,python):2024-01-11-06:04:53.289.831 [receive_data.cpp:353] >>> (tid:149999) total_size_report module:additional, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.289.939 [prof_acl_mgr.cpp:496] >>> (tid:149999) Received ProfAclFinalize request from acl [INFO] PROFILING(149999,python):2024-01-11-06:04:53.290.497 [prof_inner_api.cpp:101] >>> (tid:149999) total_size_report [api_event] read size: 0 bytes, write size: 0 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.291.241 [prof_inner_api.cpp:101] >>> (tid:149999) total_size_report [compact] read size: 12 bytes, write size: 12 bytes [INFO] PROFILING(149999,python):2024-01-11-06:04:53.296.536 [prof_inner_api.cpp:101] >>> (tid:149999) total_size_report [additional] read size: 0 bytes, write size: 0 bytes [ERROR] PIPELINE(149999,ffff89874010,python):2024-01-11-06:04:56.440.753 [mindspore/ccsrc/pipeline/jit/ps/init.cc:524] operator()] Failed to parse profiler data.RuntimeError: Read op summary failed. The file is missing basic fields. At: /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(133): _read_op_summary /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(101): parse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(310): _ascend_graph_msprof_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1293): _ascend_graph_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1040): _ascend_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(659): _analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(607): analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/envprofiling.py(230): analyse Traceback (most recent call last): File "./run_net.py", line 136, in train_with_profiler() File "./run_net.py", line 125, in train_with_profiler lenet = LeNet5() File "./run_net.py", line 55, in __init__ super(LeNet5, self).__init__() File "/home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py", line 134, in __init__ init_pipeline() RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. ---------------------------------------------------- - Framework Error Message: ---------------------------------------------------- Malloc device memory failed, free memory size is less than half of total memory size.Device 1 Device HBM total size:34359738368 Device HBM free size:1602940928 may be other processes occupying this card, check as: ps -ef|grep python ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize [INFO] RUNTIME(149999,python):2024-01-11-06:04:57.477.305 [runtime.cc:1737] 149999 ~Runtime: deconstruct runtime. [INFO] ATRACE(149999,python):2024-01-11-06:04:57.578.325 [atrace_api.c:93](tid:149999) AtraceDestroy start [INFO] ATRACE(149999,python):2024-01-11-06:04:57.578.365 [atrace_api.c:95](tid:149999) AtraceDestroy end F[INFO] ATRACE(150365,python):2024-01-11-06:05:00.973.750 [trace_attr.c:105](tid:150365) platform is 1. [INFO] ATRACE(150365,python):2024-01-11-06:05:00.973.898 [trace_recorder.c:114](tid:150365) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(150365,python):2024-01-11-06:05:00.973.929 [trace_signal.c:133](tid:150365) register signal handler for signo 2 succeed. [INFO] ATRACE(150365,python):2024-01-11-06:05:00.973.942 [trace_signal.c:133](tid:150365) register signal handler for signo 15 succeed. [INFO] RUNTIME(150365,python):2024-01-11-06:05:01.372.910 [runtime.cc:1159] 150365 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(150365,python):2024-01-11-06:05:01.372.958 [runtime.cc:4719] 150365 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set [INFO] TDT(150365,python):2024-01-11-06:05:05.270.081 [process_mode_manager.cpp:109][OpenProcess][tid:150365] [ProcessModeManager] enter into open process deviceId[1] rankSize[0] [INFO] TDT(150365,python):2024-01-11-06:05:05.272.337 [process_mode_manager.cpp:379][InitTsdClient][tid:150365] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(150365,python):2024-01-11-06:05:05.272.468 [version_verify.cpp:34][SetVersionInfo][tid:150365] VersionVerify: send client version to server [INFO] TDT(150365,python):2024-01-11-06:05:05.272.497 [version_verify.cpp:50][SetVersionInfo][tid:150365] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(150365,python):2024-01-11-06:05:05.272.511 [version_verify.cpp:50][SetVersionInfo][tid:150365] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(150365,python):2024-01-11-06:05:05.273.013 [version_verify.cpp:66][PeerVersionCheck][tid:150365] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(150365,python):2024-01-11-06:05:05.273.032 [version_verify.cpp:87][ParseVersionInfo][tid:150365] VersionVerify: pass client version info success [INFO] TDT(150365,python):2024-01-11-06:05:05.273.040 [hdc_client.cpp:276][CheckHdcConnection][tid:150365] Service[2] create hdc success [INFO] TDT(150365,python):2024-01-11-06:05:05.273.057 [version_verify.cpp:120][SpecialFeatureCheck][tid:150365] VersionVerify: new type[35], supported [INFO] TDT(150365,python):2024-01-11-06:05:05.273.107 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:150365] [TsdClient][deviceId=1] [sessionId=1] wait package info respond [INFO] TDT(150365,python):2024-01-11-06:05:05.273.248 [process_mode_manager.cpp:379][InitTsdClient][tid:150365] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(150365,python):2024-01-11-06:05:05.273.342 [version_verify.cpp:34][SetVersionInfo][tid:150365] VersionVerify: send client version to server [INFO] TDT(150365,python):2024-01-11-06:05:05.273.355 [version_verify.cpp:50][SetVersionInfo][tid:150365] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(150365,python):2024-01-11-06:05:05.273.366 [version_verify.cpp:50][SetVersionInfo][tid:150365] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(150365,python):2024-01-11-06:05:05.273.642 [version_verify.cpp:66][PeerVersionCheck][tid:150365] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(150365,python):2024-01-11-06:05:05.273.658 [version_verify.cpp:87][ParseVersionInfo][tid:150365] VersionVerify: pass client version info success [INFO] TDT(150365,python):2024-01-11-06:05:05.273.668 [hdc_client.cpp:276][CheckHdcConnection][tid:150365] Service[2] create hdc success [INFO] TDT(150365,python):2024-01-11-06:05:05.273.681 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:150365] [TsdClient] tsd get process sign successfully, procpid[150365] signSize[48] [INFO] TDT(150365,python):2024-01-11-06:05:05.273.693 [version_verify.cpp:112][SpecialFeatureCheck][tid:150365] VersionVerify: previous type[6], supported [INFO] TDT(150365,python):2024-01-11-06:05:05.273.715 [process_mode_manager.cpp:126][OpenProcess][tid:150365] [ProcessModeManager] deviceId[1] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(150365,python):2024-01-11-06:05:05.549.554 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:150365] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(150365,python):2024-01-11-06:05:05.549.585 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:150365] enter into OpenInHost deviceid[1] [INFO] TDT(150365,python):2024-01-11-06:05:05.549.597 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:150365] host cpu not support [INFO] TDT(150365,python):2024-01-11-06:05:05.549.606 [process_mode_manager.cpp:156][OpenProcess][tid:150365] [TsdClient][deviceId=1] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(150365,python):2024-01-11-06:05:05.552.361 [device.cc:340] 150365 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(150365,python):2024-01-11-06:05:05.568.994 [npu_driver.cc:5428] 150446 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(150365,python):2024-01-11-06:05:05.569.081 [atrace_api.c:28](tid:150365) AtraceCreate start [INFO] ATRACE(150365,python):2024-01-11-06:05:05.569.172 [trace_rb_log.c:84](tid:150365) [RUNTIME_ATRACE_DEV1_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(150365,python):2024-01-11-06:05:05.569.188 [atrace_api.c:32](tid:150365) AtraceCreate end [INFO] TDT(150365,python):2024-01-11-06:05:05.569.204 [client_manager.cpp:157][SetProfilingCallback][tid:150365] [TsdClient] set profiling callback success [INFO] PROFILING(150365,python):2024-01-11-06:05:05.586.941 [msprofiler_impl.cpp:156] >>> (tid:150365) ProfNotifySetDevice called, is open: 1, devId: 1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.586.993 [msprofiler_impl.cpp:289] >>> (tid:150365) Get system free ram: 556770803712 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.587.010 [prof_cann_plugin.cpp:75] >>> (tid:150365) Init report buffer size: 131072 bytes, buffer name: api_event [INFO] PROFILING(150365,python):2024-01-11-06:05:05.590.493 [prof_cann_plugin.cpp:75] >>> (tid:150365) Init report buffer size: 131072 bytes, buffer name: compact [INFO] PROFILING(150365,python):2024-01-11-06:05:05.594.893 [prof_cann_plugin.cpp:75] >>> (tid:150365) Init report buffer size: 262144 bytes, buffer name: additional [INFO] PROFILING(150365,python):2024-01-11-06:05:05.621.057 [platform.cpp:38] >>> (tid:150365) Profiling platform version: 1.0. [INFO] PROFILING(150365,python):2024-01-11-06:05:05.621.090 [ai_drv_dev_api.cpp:384] >>> (tid:150365) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.621.218 [prof_acl_mgr.cpp:286] >>> (tid:150365) Received ProfAclInit request from acl [INFO] PROFILING(150365,python):2024-01-11-06:05:05.621.279 [msprof_reporter.cpp:98] >>> (tid:150365) Init all reporters [INFO] PROFILING(150365,python):2024-01-11-06:05:05.622.640 [prof_acl_mgr.cpp:350] >>> (tid:150365) Received ProfAclStart request from acl [INFO] PROFILING(150365,python):2024-01-11-06:05:05.622.750 [ai_drv_dev_api.cpp:384] >>> (tid:150365) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.624.625 [hdc_api.cpp:112] >>> (tid:150365) logDevId 1 create HDC server successfully [INFO] PROFILING(150365,python):2024-01-11-06:05:05.626.842 [prof_manager.cpp:384] >>> (tid:150365) Received libmsprof message to start profiling, job_id:1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.626.885 [prof_manager.cpp:152] >>> (tid:150365) Check device profiling status [INFO] PROFILING(150365,python):2024-01-11-06:05:05.626.910 [prof_manager.cpp:272] >>> (tid:150365) Begin to launch task, jobId:1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.626.928 [prof_acl_mgr.cpp:95] >>> (tid:150454) Device 1 started to wait for response [INFO] PROFILING(150365,python):2024-01-11-06:05:05.626.983 [prof_acl_mgr.cpp:2258] >>> (tid:150365) Init profiling for msproftx [INFO] PROFILING(150365,python):2024-01-11-06:05:05.626.998 [prof_acl_mgr.cpp:2376] >>> (tid:150365) MsprofSetDeviceImpl, devId:64 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.628.482 [ai_drv_prof_api.cpp:33] >>> (tid:150456) Begin to get channels, deviceId=1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.628.788 [prof_manager.cpp:384] >>> (tid:150365) Received libmsprof message to start profiling, job_id:64 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.628.814 [prof_manager.cpp:152] >>> (tid:150365) Check device profiling status [INFO] PROFILING(150365,python):2024-01-11-06:05:05.628.815 [prof_acl_mgr.cpp:95] >>> (tid:150468) Device 64 started to wait for response [INFO] PROFILING(150365,python):2024-01-11-06:05:05.628.835 [prof_manager.cpp:272] >>> (tid:150365) Begin to launch task, jobId:64 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.629.357 [ai_drv_prof_api.cpp:66] >>> (tid:150456) End to get channels[17], deviceId=1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.629.498 [prof_acl_mgr.cpp:85] >>> (tid:150470) Device 64 finished starting [INFO] PROFILING(150365,python):2024-01-11-06:05:05.629.534 [prof_acl_mgr.cpp:97] >>> (tid:150468) Device 64 finished waiting for response [INFO] PROFILING(150365,python):2024-01-11-06:05:05.629.669 [prof_acl_mgr.cpp:1422] >>> (tid:150365) Device:64 finished waiting [INFO] PROFILING(150365,python):2024-01-11-06:05:05.629.687 [ai_drv_dev_api.cpp:384] >>> (tid:150365) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.631.026 [ai_drv_prof_api.cpp:436] >>> (tid:150456) Begin to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.652.601 [ai_drv_prof_api.cpp:454] >>> (tid:150456) Succeeded to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.654.211 [ai_drv_prof_api.cpp:296] >>> (tid:150456) Begin to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43, configSize:56bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.654.237 [ai_drv_prof_api.cpp:298] >>> (tid:150456) DrvAicoreTaskBasedStart, event_num=7, events=0x49,0x4a,0x4b,0x4c,0x4d,0x4e,0x4f,, tag=0 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.654.824 [ai_drv_prof_api.cpp:319] >>> (tid:150456) Succeeded to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.656.337 [ai_drv_prof_api.cpp:591] >>> (tid:150456) Begin to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45, tag=0 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.656.928 [ai_drv_prof_api.cpp:606] >>> (tid:150456) Succeeded to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.658.430 [ai_drv_prof_api.cpp:618] >>> (tid:150456) Begin to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.659.024 [ai_drv_prof_api.cpp:633] >>> (tid:150456) Succeeded to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.659.092 [prof_acl_mgr.cpp:85] >>> (tid:150456) Device 1 finished starting [INFO] PROFILING(150365,python):2024-01-11-06:05:05.659.122 [prof_acl_mgr.cpp:97] >>> (tid:150454) Device 1 finished waiting for response [INFO] PROFILING(150365,python):2024-01-11-06:05:05.659.272 [prof_acl_mgr.cpp:1413] >>> (tid:150365) All devices finished waiting [INFO] TDT(150365,python):2024-01-11-06:05:05.659.879 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:150365] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=7] [INFO] TDT(150365,python):2024-01-11-06:05:05.659.903 [version_verify.cpp:112][SpecialFeatureCheck][tid:150365] VersionVerify: previous type[30], supported [INFO] PROFILING(150365,python):2024-01-11-06:05:05.659.913 [ai_drv_dev_api.cpp:384] >>> (tid:150455) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] TDT(150365,python):2024-01-11-06:05:05.659.946 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:150365] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(150365,python):2024-01-11-06:05:05.659.981 [ai_drv_dev_api.cpp:384] >>> (tid:150455) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.660.038 [ai_drv_dev_api.cpp:384] >>> (tid:150469) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.660.180 [prof_task.cpp:285] >>> (tid:150455) ProfTask 1 started to wait for task stop cv [INFO] PROFILING(150365,python):2024-01-11-06:05:05.660.181 [prof_task.cpp:285] >>> (tid:150469) ProfTask 64 started to wait for task stop cv [INFO] TDT(150365,python):2024-01-11-06:05:05.722.097 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:150365] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=6] [INFO] TDT(150365,python):2024-01-11-06:05:05.722.161 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:150365] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(150365,python):2024-01-11-06:05:05.722.551 [prof_reporter_mgr.cpp:226] >>> (tid:150365) total_size_type_info[5000], save type info length: 4544 bytes, type info size: 183 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.722.579 [prof_reporter_mgr.cpp:226] >>> (tid:150365) total_size_type_info[5500], save type info length: 35 bytes, type info size: 2 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.722.613 [prof_reporter_mgr.cpp:226] >>> (tid:150365) total_size_type_info[10000], save type info length: 404 bytes, type info size: 15 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.722.633 [prof_reporter_mgr.cpp:226] >>> (tid:150365) total_size_type_info[15000], save type info length: 67 bytes, type info size: 3 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.722.648 [uploader_dumper.cpp:178] >>> (tid:150365) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.735 [uploader_dumper.cpp:182] >>> (tid:150365) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.748 [uploader_dumper.cpp:178] >>> (tid:150365) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.777 [file_slice.cpp:356] >>> (tid:150365) [FileSliceFlush]file:aging.compact.task_track.slice_, total_size_file:768 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.948 [uploader_dumper.cpp:182] >>> (tid:150365) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.965 [uploader_dumper.cpp:178] >>> (tid:150365) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.977 [uploader_dumper.cpp:182] >>> (tid:150365) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(150365,python):2024-01-11-06:05:05.723.994 [prof_acl_mgr.cpp:435] >>> (tid:150365) Received ProfAclStop request from acl [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.009 [prof_manager.cpp:382] >>> (tid:150365) Received libmsprof message to stop profiling, job_id:1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.048 [prof_manager.cpp:305] >>> (tid:150365) Begin to stop task, jobId:1 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.064 [prof_task.cpp:344] >>> (tid:150365) Task send finished cv [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.082 [prof_task.cpp:287] >>> (tid:150455) ProfTask 1 finished waiting for task stop cv [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.163 [ai_drv_prof_api.cpp:642] >>> (tid:150456) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.769 [ai_drv_prof_api.cpp:650] >>> (tid:150456) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.792 [prof_channel.cpp:89] >>> (tid:150456) device id 1, channel: 44, total_size_channel: 80 bytes, file:data/ts_track.data, job_id:1,drvChannelReadCont:4 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.724.905 [ai_drv_prof_api.cpp:642] >>> (tid:150456) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.725.514 [ai_drv_prof_api.cpp:650] >>> (tid:150456) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.725.532 [prof_channel.cpp:89] >>> (tid:150456) device id 1, channel: 43, total_size_channel: 0 bytes, file:data/aicore.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.725.932 [ai_drv_prof_api.cpp:642] >>> (tid:150456) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.726.567 [ai_drv_prof_api.cpp:650] >>> (tid:150456) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.726.585 [prof_channel.cpp:89] >>> (tid:150456) device id 1, channel: 45, total_size_channel: 270272 bytes, file:data/hwts.data, job_id:1,drvChannelReadCont:13 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.726.631 [ai_drv_prof_api.cpp:642] >>> (tid:150456) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.727.408 [ai_drv_prof_api.cpp:650] >>> (tid:150456) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.727.424 [prof_channel.cpp:89] >>> (tid:150456) device id 1, channel: 46, total_size_channel: 0 bytes, file:data/training_trace.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.729.551 [prof_channel.cpp:407] >>> (tid:150456) ChannelPoll count: 34, Sleep count: 24, Dispatch count: 10, DispatchChannel count: 9 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.729.849 [file_slice.cpp:356] >>> (tid:150455) [FileSliceFlush]file:hwts.data.1.slice_, total_size_file:270272 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.014 [file_slice.cpp:356] >>> (tid:150455) [FileSliceFlush]file:ts_track.data.1.slice_, total_size_file:80 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.404 [prof_task.cpp:76] >>> (tid:150455) Uninit ProfTask succesfully [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.421 [prof_task.cpp:339] >>> (tid:150455) Task 1 finished [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.565 [prof_manager.cpp:382] >>> (tid:150365) Received libmsprof message to stop profiling, job_id:64 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.582 [prof_manager.cpp:305] >>> (tid:150365) Begin to stop task, jobId:64 [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.594 [prof_task.cpp:344] >>> (tid:150365) Task send finished cv [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.611 [prof_task.cpp:287] >>> (tid:150469) ProfTask 64 finished waiting for task stop cv [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.904 [prof_task.cpp:76] >>> (tid:150469) Uninit ProfTask succesfully [INFO] PROFILING(150365,python):2024-01-11-06:05:05.730.932 [prof_task.cpp:339] >>> (tid:150469) Task 64 finished [INFO] PROFILING(150365,python):2024-01-11-06:05:05.731.962 [uploader_dumper.cpp:178] >>> (tid:150365) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(150365,python):2024-01-11-06:05:05.731.987 [uploader_dumper.cpp:182] >>> (tid:150365) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(150365,python):2024-01-11-06:05:05.732.255 [receive_data.cpp:353] >>> (tid:150365) total_size_report module:api_event, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.732.281 [uploader_dumper.cpp:178] >>> (tid:150365) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(150365,python):2024-01-11-06:05:05.732.295 [uploader_dumper.cpp:182] >>> (tid:150365) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(150365,python):2024-01-11-06:05:05.732.435 [receive_data.cpp:353] >>> (tid:150365) total_size_report module:compact, push count:0, pop count:12, push size:0 bytes, pop size:768 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.732.452 [uploader_dumper.cpp:178] >>> (tid:150365) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(150365,python):2024-01-11-06:05:05.732.463 [uploader_dumper.cpp:182] >>> (tid:150365) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(150365,python):2024-01-11-06:05:05.733.355 [receive_data.cpp:353] >>> (tid:150365) total_size_report module:additional, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.733.462 [prof_acl_mgr.cpp:496] >>> (tid:150365) Received ProfAclFinalize request from acl [INFO] PROFILING(150365,python):2024-01-11-06:05:05.734.000 [prof_inner_api.cpp:101] >>> (tid:150365) total_size_report [api_event] read size: 0 bytes, write size: 0 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.734.756 [prof_inner_api.cpp:101] >>> (tid:150365) total_size_report [compact] read size: 12 bytes, write size: 12 bytes [INFO] PROFILING(150365,python):2024-01-11-06:05:05.740.039 [prof_inner_api.cpp:101] >>> (tid:150365) total_size_report [additional] read size: 0 bytes, write size: 0 bytes [ERROR] PIPELINE(150365,ffff931b8010,python):2024-01-11-06:05:08.884.498 [mindspore/ccsrc/pipeline/jit/ps/init.cc:524] operator()] Failed to parse profiler data.RuntimeError: Read op summary failed. The file is missing basic fields. At: /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(133): _read_op_summary /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(101): parse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(310): _ascend_graph_msprof_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1293): _ascend_graph_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1040): _ascend_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(659): _analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(607): analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/envprofiling.py(230): analyse Traceback (most recent call last): File "./run_net.py", line 136, in train_with_profiler() File "./run_net.py", line 125, in train_with_profiler lenet = LeNet5() File "./run_net.py", line 55, in __init__ super(LeNet5, self).__init__() File "/home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py", line 134, in __init__ init_pipeline() RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. ---------------------------------------------------- - Framework Error Message: ---------------------------------------------------- Malloc device memory failed, free memory size is less than half of total memory size.Device 1 Device HBM total size:34359738368 Device HBM free size:1603166208 may be other processes occupying this card, check as: ps -ef|grep python ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize [INFO] RUNTIME(150365,python):2024-01-11-06:05:09.898.850 [runtime.cc:1737] 150365 ~Runtime: deconstruct runtime. [INFO] ATRACE(150365,python):2024-01-11-06:05:09.999.874 [atrace_api.c:93](tid:150365) AtraceDestroy start [INFO] ATRACE(150365,python):2024-01-11-06:05:09.999.913 [atrace_api.c:95](tid:150365) AtraceDestroy end F[INFO] ATRACE(150676,python):2024-01-11-06:05:13.126.927 [trace_attr.c:105](tid:150676) platform is 1. [INFO] ATRACE(150676,python):2024-01-11-06:05:13.127.128 [trace_recorder.c:114](tid:150676) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(150676,python):2024-01-11-06:05:13.127.160 [trace_signal.c:133](tid:150676) register signal handler for signo 2 succeed. [INFO] ATRACE(150676,python):2024-01-11-06:05:13.127.174 [trace_signal.c:133](tid:150676) register signal handler for signo 15 succeed. [INFO] RUNTIME(150676,python):2024-01-11-06:05:13.524.467 [runtime.cc:1159] 150676 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(150676,python):2024-01-11-06:05:13.524.513 [runtime.cc:4719] 150676 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set [INFO] TDT(150676,python):2024-01-11-06:05:17.423.152 [process_mode_manager.cpp:109][OpenProcess][tid:150676] [ProcessModeManager] enter into open process deviceId[1] rankSize[0] [INFO] TDT(150676,python):2024-01-11-06:05:17.425.591 [process_mode_manager.cpp:379][InitTsdClient][tid:150676] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(150676,python):2024-01-11-06:05:17.425.736 [version_verify.cpp:34][SetVersionInfo][tid:150676] VersionVerify: send client version to server [INFO] TDT(150676,python):2024-01-11-06:05:17.425.767 [version_verify.cpp:50][SetVersionInfo][tid:150676] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(150676,python):2024-01-11-06:05:17.425.782 [version_verify.cpp:50][SetVersionInfo][tid:150676] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(150676,python):2024-01-11-06:05:17.426.105 [version_verify.cpp:66][PeerVersionCheck][tid:150676] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(150676,python):2024-01-11-06:05:17.426.122 [version_verify.cpp:87][ParseVersionInfo][tid:150676] VersionVerify: pass client version info success [INFO] TDT(150676,python):2024-01-11-06:05:17.426.136 [hdc_client.cpp:276][CheckHdcConnection][tid:150676] Service[2] create hdc success [INFO] TDT(150676,python):2024-01-11-06:05:17.426.153 [version_verify.cpp:120][SpecialFeatureCheck][tid:150676] VersionVerify: new type[35], supported [INFO] TDT(150676,python):2024-01-11-06:05:17.426.208 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:150676] [TsdClient][deviceId=1] [sessionId=1] wait package info respond [INFO] TDT(150676,python):2024-01-11-06:05:17.426.345 [process_mode_manager.cpp:379][InitTsdClient][tid:150676] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(150676,python):2024-01-11-06:05:17.426.617 [version_verify.cpp:34][SetVersionInfo][tid:150676] VersionVerify: send client version to server [INFO] TDT(150676,python):2024-01-11-06:05:17.426.630 [version_verify.cpp:50][SetVersionInfo][tid:150676] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(150676,python):2024-01-11-06:05:17.426.642 [version_verify.cpp:50][SetVersionInfo][tid:150676] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(150676,python):2024-01-11-06:05:17.426.796 [version_verify.cpp:66][PeerVersionCheck][tid:150676] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(150676,python):2024-01-11-06:05:17.426.823 [version_verify.cpp:87][ParseVersionInfo][tid:150676] VersionVerify: pass client version info success [INFO] TDT(150676,python):2024-01-11-06:05:17.426.834 [hdc_client.cpp:276][CheckHdcConnection][tid:150676] Service[2] create hdc success [INFO] TDT(150676,python):2024-01-11-06:05:17.426.848 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:150676] [TsdClient] tsd get process sign successfully, procpid[150676] signSize[48] [INFO] TDT(150676,python):2024-01-11-06:05:17.426.861 [version_verify.cpp:112][SpecialFeatureCheck][tid:150676] VersionVerify: previous type[6], supported [INFO] TDT(150676,python):2024-01-11-06:05:17.426.884 [process_mode_manager.cpp:126][OpenProcess][tid:150676] [ProcessModeManager] deviceId[1] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(150676,python):2024-01-11-06:05:17.634.694 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:150676] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(150676,python):2024-01-11-06:05:17.634.730 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:150676] enter into OpenInHost deviceid[1] [INFO] TDT(150676,python):2024-01-11-06:05:17.634.743 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:150676] host cpu not support [INFO] TDT(150676,python):2024-01-11-06:05:17.634.752 [process_mode_manager.cpp:156][OpenProcess][tid:150676] [TsdClient][deviceId=1] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(150676,python):2024-01-11-06:05:17.637.437 [device.cc:340] 150676 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(150676,python):2024-01-11-06:05:17.653.915 [npu_driver.cc:5428] 150754 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(150676,python):2024-01-11-06:05:17.654.014 [atrace_api.c:28](tid:150676) AtraceCreate start [INFO] ATRACE(150676,python):2024-01-11-06:05:17.654.119 [trace_rb_log.c:84](tid:150676) [RUNTIME_ATRACE_DEV1_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(150676,python):2024-01-11-06:05:17.654.138 [atrace_api.c:32](tid:150676) AtraceCreate end [INFO] TDT(150676,python):2024-01-11-06:05:17.654.160 [client_manager.cpp:157][SetProfilingCallback][tid:150676] [TsdClient] set profiling callback success [INFO] PROFILING(150676,python):2024-01-11-06:05:17.671.935 [msprofiler_impl.cpp:156] >>> (tid:150676) ProfNotifySetDevice called, is open: 1, devId: 1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.671.999 [msprofiler_impl.cpp:289] >>> (tid:150676) Get system free ram: 555712000000 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.672.016 [prof_cann_plugin.cpp:75] >>> (tid:150676) Init report buffer size: 131072 bytes, buffer name: api_event [INFO] PROFILING(150676,python):2024-01-11-06:05:17.675.517 [prof_cann_plugin.cpp:75] >>> (tid:150676) Init report buffer size: 131072 bytes, buffer name: compact [INFO] PROFILING(150676,python):2024-01-11-06:05:17.679.945 [prof_cann_plugin.cpp:75] >>> (tid:150676) Init report buffer size: 262144 bytes, buffer name: additional [INFO] PROFILING(150676,python):2024-01-11-06:05:17.705.740 [platform.cpp:38] >>> (tid:150676) Profiling platform version: 1.0. [INFO] PROFILING(150676,python):2024-01-11-06:05:17.705.778 [ai_drv_dev_api.cpp:384] >>> (tid:150676) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.705.908 [prof_acl_mgr.cpp:286] >>> (tid:150676) Received ProfAclInit request from acl [INFO] PROFILING(150676,python):2024-01-11-06:05:17.705.972 [msprof_reporter.cpp:98] >>> (tid:150676) Init all reporters [INFO] PROFILING(150676,python):2024-01-11-06:05:17.707.378 [prof_acl_mgr.cpp:350] >>> (tid:150676) Received ProfAclStart request from acl [INFO] PROFILING(150676,python):2024-01-11-06:05:17.707.500 [ai_drv_dev_api.cpp:384] >>> (tid:150676) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.709.361 [hdc_api.cpp:112] >>> (tid:150676) logDevId 1 create HDC server successfully [INFO] PROFILING(150676,python):2024-01-11-06:05:17.711.547 [prof_manager.cpp:384] >>> (tid:150676) Received libmsprof message to start profiling, job_id:1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.711.591 [prof_manager.cpp:152] >>> (tid:150676) Check device profiling status [INFO] PROFILING(150676,python):2024-01-11-06:05:17.711.616 [prof_manager.cpp:272] >>> (tid:150676) Begin to launch task, jobId:1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.711.632 [prof_acl_mgr.cpp:95] >>> (tid:150762) Device 1 started to wait for response [INFO] PROFILING(150676,python):2024-01-11-06:05:17.711.691 [prof_acl_mgr.cpp:2258] >>> (tid:150676) Init profiling for msproftx [INFO] PROFILING(150676,python):2024-01-11-06:05:17.711.707 [prof_acl_mgr.cpp:2376] >>> (tid:150676) MsprofSetDeviceImpl, devId:64 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.713.194 [ai_drv_prof_api.cpp:33] >>> (tid:150764) Begin to get channels, deviceId=1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.713.534 [prof_manager.cpp:384] >>> (tid:150676) Received libmsprof message to start profiling, job_id:64 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.713.563 [prof_manager.cpp:152] >>> (tid:150676) Check device profiling status [INFO] PROFILING(150676,python):2024-01-11-06:05:17.713.556 [prof_acl_mgr.cpp:95] >>> (tid:150776) Device 64 started to wait for response [INFO] PROFILING(150676,python):2024-01-11-06:05:17.713.586 [prof_manager.cpp:272] >>> (tid:150676) Begin to launch task, jobId:64 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.714.032 [ai_drv_prof_api.cpp:66] >>> (tid:150764) End to get channels[17], deviceId=1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.714.175 [prof_acl_mgr.cpp:85] >>> (tid:150778) Device 64 finished starting [INFO] PROFILING(150676,python):2024-01-11-06:05:17.714.209 [prof_acl_mgr.cpp:97] >>> (tid:150776) Device 64 finished waiting for response [INFO] PROFILING(150676,python):2024-01-11-06:05:17.714.319 [prof_acl_mgr.cpp:1422] >>> (tid:150676) Device:64 finished waiting [INFO] PROFILING(150676,python):2024-01-11-06:05:17.714.340 [ai_drv_dev_api.cpp:384] >>> (tid:150676) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.715.690 [ai_drv_prof_api.cpp:436] >>> (tid:150764) Begin to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.736.488 [ai_drv_prof_api.cpp:454] >>> (tid:150764) Succeeded to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.738.146 [ai_drv_prof_api.cpp:296] >>> (tid:150764) Begin to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43, configSize:56bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.738.169 [ai_drv_prof_api.cpp:298] >>> (tid:150764) DrvAicoreTaskBasedStart, event_num=7, events=0x49,0x4a,0x4b,0x4c,0x4d,0x4e,0x4f,, tag=0 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.738.761 [ai_drv_prof_api.cpp:319] >>> (tid:150764) Succeeded to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.740.280 [ai_drv_prof_api.cpp:591] >>> (tid:150764) Begin to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45, tag=0 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.740.875 [ai_drv_prof_api.cpp:606] >>> (tid:150764) Succeeded to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.742.382 [ai_drv_prof_api.cpp:618] >>> (tid:150764) Begin to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.742.976 [ai_drv_prof_api.cpp:633] >>> (tid:150764) Succeeded to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.743.044 [prof_acl_mgr.cpp:85] >>> (tid:150764) Device 1 finished starting [INFO] PROFILING(150676,python):2024-01-11-06:05:17.743.067 [prof_acl_mgr.cpp:97] >>> (tid:150762) Device 1 finished waiting for response [INFO] PROFILING(150676,python):2024-01-11-06:05:17.743.220 [prof_acl_mgr.cpp:1413] >>> (tid:150676) All devices finished waiting [INFO] TDT(150676,python):2024-01-11-06:05:17.743.838 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:150676] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=7] [INFO] TDT(150676,python):2024-01-11-06:05:17.743.863 [version_verify.cpp:112][SpecialFeatureCheck][tid:150676] VersionVerify: previous type[30], supported [INFO] TDT(150676,python):2024-01-11-06:05:17.743.904 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:150676] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(150676,python):2024-01-11-06:05:17.744.244 [ai_drv_dev_api.cpp:384] >>> (tid:150763) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.744.317 [ai_drv_dev_api.cpp:384] >>> (tid:150763) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.744.446 [ai_drv_dev_api.cpp:384] >>> (tid:150777) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.744.510 [prof_task.cpp:285] >>> (tid:150763) ProfTask 1 started to wait for task stop cv [INFO] PROFILING(150676,python):2024-01-11-06:05:17.744.587 [prof_task.cpp:285] >>> (tid:150777) ProfTask 64 started to wait for task stop cv [INFO] TDT(150676,python):2024-01-11-06:05:17.807.449 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:150676] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=6] [INFO] TDT(150676,python):2024-01-11-06:05:17.807.510 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:150676] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.855 [prof_reporter_mgr.cpp:226] >>> (tid:150676) total_size_type_info[5000], save type info length: 4544 bytes, type info size: 183 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.880 [prof_reporter_mgr.cpp:226] >>> (tid:150676) total_size_type_info[5500], save type info length: 35 bytes, type info size: 2 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.913 [prof_reporter_mgr.cpp:226] >>> (tid:150676) total_size_type_info[10000], save type info length: 404 bytes, type info size: 15 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.934 [prof_reporter_mgr.cpp:226] >>> (tid:150676) total_size_type_info[15000], save type info length: 67 bytes, type info size: 3 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.952 [uploader_dumper.cpp:178] >>> (tid:150676) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.969 [uploader_dumper.cpp:182] >>> (tid:150676) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(150676,python):2024-01-11-06:05:17.807.980 [uploader_dumper.cpp:178] >>> (tid:150676) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.104 [file_slice.cpp:356] >>> (tid:150676) [FileSliceFlush]file:aging.compact.task_track.slice_, total_size_file:704 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.248 [uploader_dumper.cpp:182] >>> (tid:150676) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.265 [uploader_dumper.cpp:178] >>> (tid:150676) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.278 [uploader_dumper.cpp:182] >>> (tid:150676) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.294 [prof_acl_mgr.cpp:435] >>> (tid:150676) Received ProfAclStop request from acl [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.310 [prof_manager.cpp:382] >>> (tid:150676) Received libmsprof message to stop profiling, job_id:1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.343 [prof_manager.cpp:305] >>> (tid:150676) Begin to stop task, jobId:1 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.376 [prof_task.cpp:344] >>> (tid:150676) Task send finished cv [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.395 [prof_task.cpp:287] >>> (tid:150763) ProfTask 1 finished waiting for task stop cv [INFO] PROFILING(150676,python):2024-01-11-06:05:17.808.478 [ai_drv_prof_api.cpp:642] >>> (tid:150764) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.809.087 [ai_drv_prof_api.cpp:650] >>> (tid:150764) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.809.109 [prof_channel.cpp:89] >>> (tid:150764) device id 1, channel: 44, total_size_channel: 80 bytes, file:data/ts_track.data, job_id:1,drvChannelReadCont:4 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.809.184 [ai_drv_prof_api.cpp:642] >>> (tid:150764) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.809.782 [ai_drv_prof_api.cpp:650] >>> (tid:150764) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.809.801 [prof_channel.cpp:89] >>> (tid:150764) device id 1, channel: 43, total_size_channel: 0 bytes, file:data/aicore.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.810.201 [ai_drv_prof_api.cpp:642] >>> (tid:150764) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.810.829 [ai_drv_prof_api.cpp:650] >>> (tid:150764) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.810.845 [prof_channel.cpp:89] >>> (tid:150764) device id 1, channel: 45, total_size_channel: 289152 bytes, file:data/hwts.data, job_id:1,drvChannelReadCont:16 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.810.889 [ai_drv_prof_api.cpp:642] >>> (tid:150764) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.811.640 [ai_drv_prof_api.cpp:650] >>> (tid:150764) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.811.658 [prof_channel.cpp:89] >>> (tid:150764) device id 1, channel: 46, total_size_channel: 0 bytes, file:data/training_trace.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.813.635 [prof_channel.cpp:407] >>> (tid:150764) ChannelPoll count: 36, Sleep count: 24, Dispatch count: 12, DispatchChannel count: 11 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.813.890 [file_slice.cpp:356] >>> (tid:150763) [FileSliceFlush]file:hwts.data.1.slice_, total_size_file:289152 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.040 [file_slice.cpp:356] >>> (tid:150763) [FileSliceFlush]file:ts_track.data.1.slice_, total_size_file:80 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.407 [prof_task.cpp:76] >>> (tid:150763) Uninit ProfTask succesfully [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.422 [prof_task.cpp:339] >>> (tid:150763) Task 1 finished [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.562 [prof_manager.cpp:382] >>> (tid:150676) Received libmsprof message to stop profiling, job_id:64 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.579 [prof_manager.cpp:305] >>> (tid:150676) Begin to stop task, jobId:64 [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.592 [prof_task.cpp:344] >>> (tid:150676) Task send finished cv [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.609 [prof_task.cpp:287] >>> (tid:150777) ProfTask 64 finished waiting for task stop cv [INFO] PROFILING(150676,python):2024-01-11-06:05:17.814.880 [file_slice.cpp:356] >>> (tid:150777) [FileSliceFlush]file:aging.compact.task_track.slice_, total_size_file:768 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.815.045 [prof_task.cpp:76] >>> (tid:150777) Uninit ProfTask succesfully [INFO] PROFILING(150676,python):2024-01-11-06:05:17.815.069 [prof_task.cpp:339] >>> (tid:150777) Task 64 finished [INFO] PROFILING(150676,python):2024-01-11-06:05:17.816.106 [uploader_dumper.cpp:178] >>> (tid:150676) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(150676,python):2024-01-11-06:05:17.816.130 [uploader_dumper.cpp:182] >>> (tid:150676) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(150676,python):2024-01-11-06:05:17.816.973 [receive_data.cpp:353] >>> (tid:150676) total_size_report module:api_event, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.817.005 [uploader_dumper.cpp:178] >>> (tid:150676) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(150676,python):2024-01-11-06:05:17.817.021 [uploader_dumper.cpp:182] >>> (tid:150676) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(150676,python):2024-01-11-06:05:17.817.546 [receive_data.cpp:353] >>> (tid:150676) total_size_report module:compact, push count:0, pop count:12, push size:0 bytes, pop size:768 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.817.563 [uploader_dumper.cpp:178] >>> (tid:150676) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(150676,python):2024-01-11-06:05:17.817.576 [uploader_dumper.cpp:182] >>> (tid:150676) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(150676,python):2024-01-11-06:05:17.818.124 [receive_data.cpp:353] >>> (tid:150676) total_size_report module:additional, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.818.227 [prof_acl_mgr.cpp:496] >>> (tid:150676) Received ProfAclFinalize request from acl [INFO] PROFILING(150676,python):2024-01-11-06:05:17.818.783 [prof_inner_api.cpp:101] >>> (tid:150676) total_size_report [api_event] read size: 0 bytes, write size: 0 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.819.526 [prof_inner_api.cpp:101] >>> (tid:150676) total_size_report [compact] read size: 12 bytes, write size: 12 bytes [INFO] PROFILING(150676,python):2024-01-11-06:05:17.824.822 [prof_inner_api.cpp:101] >>> (tid:150676) total_size_report [additional] read size: 0 bytes, write size: 0 bytes [ERROR] PIPELINE(150676,ffffbcf8e010,python):2024-01-11-06:05:20.969.360 [mindspore/ccsrc/pipeline/jit/ps/init.cc:524] operator()] Failed to parse profiler data.RuntimeError: Read op summary failed. The file is missing basic fields. At: /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(133): _read_op_summary /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(101): parse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(310): _ascend_graph_msprof_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1293): _ascend_graph_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1040): _ascend_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(659): _analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(607): analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/envprofiling.py(230): analyse Traceback (most recent call last): File "./run_net.py", line 136, in train_with_profiler() File "./run_net.py", line 125, in train_with_profiler lenet = LeNet5() File "./run_net.py", line 55, in __init__ super(LeNet5, self).__init__() File "/home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py", line 134, in __init__ init_pipeline() RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. ---------------------------------------------------- - Framework Error Message: ---------------------------------------------------- Malloc device memory failed, free memory size is less than half of total memory size.Device 1 Device HBM total size:34359738368 Device HBM free size:1603133440 may be other processes occupying this card, check as: ps -ef|grep python ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize [INFO] RUNTIME(150676,python):2024-01-11-06:05:21.988.518 [runtime.cc:1737] 150676 ~Runtime: deconstruct runtime. [INFO] ATRACE(150676,python):2024-01-11-06:05:22.091.671 [atrace_api.c:93](tid:150676) AtraceDestroy start [INFO] ATRACE(150676,python):2024-01-11-06:05:22.091.712 [atrace_api.c:95](tid:150676) AtraceDestroy end F[INFO] ATRACE(150984,python):2024-01-11-06:05:25.167.332 [trace_attr.c:105](tid:150984) platform is 1. [INFO] ATRACE(150984,python):2024-01-11-06:05:25.167.469 [trace_recorder.c:114](tid:150984) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(150984,python):2024-01-11-06:05:25.167.497 [trace_signal.c:133](tid:150984) register signal handler for signo 2 succeed. [INFO] ATRACE(150984,python):2024-01-11-06:05:25.167.509 [trace_signal.c:133](tid:150984) register signal handler for signo 15 succeed. [INFO] RUNTIME(150984,python):2024-01-11-06:05:25.566.879 [runtime.cc:1159] 150984 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(150984,python):2024-01-11-06:05:25.566.926 [runtime.cc:4719] 150984 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set [INFO] TDT(150984,python):2024-01-11-06:05:29.584.003 [process_mode_manager.cpp:109][OpenProcess][tid:150984] [ProcessModeManager] enter into open process deviceId[1] rankSize[0] [INFO] TDT(150984,python):2024-01-11-06:05:29.586.482 [process_mode_manager.cpp:379][InitTsdClient][tid:150984] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(150984,python):2024-01-11-06:05:29.586.611 [version_verify.cpp:34][SetVersionInfo][tid:150984] VersionVerify: send client version to server [INFO] TDT(150984,python):2024-01-11-06:05:29.586.639 [version_verify.cpp:50][SetVersionInfo][tid:150984] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(150984,python):2024-01-11-06:05:29.586.653 [version_verify.cpp:50][SetVersionInfo][tid:150984] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(150984,python):2024-01-11-06:05:29.587.011 [version_verify.cpp:66][PeerVersionCheck][tid:150984] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(150984,python):2024-01-11-06:05:29.587.028 [version_verify.cpp:87][ParseVersionInfo][tid:150984] VersionVerify: pass client version info success [INFO] TDT(150984,python):2024-01-11-06:05:29.587.039 [hdc_client.cpp:276][CheckHdcConnection][tid:150984] Service[2] create hdc success [INFO] TDT(150984,python):2024-01-11-06:05:29.587.056 [version_verify.cpp:120][SpecialFeatureCheck][tid:150984] VersionVerify: new type[35], supported [INFO] TDT(150984,python):2024-01-11-06:05:29.587.152 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:150984] [TsdClient][deviceId=1] [sessionId=1] wait package info respond [INFO] TDT(150984,python):2024-01-11-06:05:29.587.297 [process_mode_manager.cpp:379][InitTsdClient][tid:150984] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(150984,python):2024-01-11-06:05:29.587.621 [version_verify.cpp:34][SetVersionInfo][tid:150984] VersionVerify: send client version to server [INFO] TDT(150984,python):2024-01-11-06:05:29.587.634 [version_verify.cpp:50][SetVersionInfo][tid:150984] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(150984,python):2024-01-11-06:05:29.587.645 [version_verify.cpp:50][SetVersionInfo][tid:150984] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(150984,python):2024-01-11-06:05:29.587.820 [version_verify.cpp:66][PeerVersionCheck][tid:150984] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(150984,python):2024-01-11-06:05:29.587.844 [version_verify.cpp:87][ParseVersionInfo][tid:150984] VersionVerify: pass client version info success [INFO] TDT(150984,python):2024-01-11-06:05:29.587.855 [hdc_client.cpp:276][CheckHdcConnection][tid:150984] Service[2] create hdc success [INFO] TDT(150984,python):2024-01-11-06:05:29.587.868 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:150984] [TsdClient] tsd get process sign successfully, procpid[150984] signSize[48] [INFO] TDT(150984,python):2024-01-11-06:05:29.587.881 [version_verify.cpp:112][SpecialFeatureCheck][tid:150984] VersionVerify: previous type[6], supported [INFO] TDT(150984,python):2024-01-11-06:05:29.587.904 [process_mode_manager.cpp:126][OpenProcess][tid:150984] [ProcessModeManager] deviceId[1] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(150984,python):2024-01-11-06:05:29.808.564 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:150984] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(150984,python):2024-01-11-06:05:29.808.594 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:150984] enter into OpenInHost deviceid[1] [INFO] TDT(150984,python):2024-01-11-06:05:29.808.605 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:150984] host cpu not support [INFO] TDT(150984,python):2024-01-11-06:05:29.808.613 [process_mode_manager.cpp:156][OpenProcess][tid:150984] [TsdClient][deviceId=1] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(150984,python):2024-01-11-06:05:29.811.348 [device.cc:340] 150984 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(150984,python):2024-01-11-06:05:29.828.046 [npu_driver.cc:5428] 151054 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(150984,python):2024-01-11-06:05:29.828.125 [atrace_api.c:28](tid:150984) AtraceCreate start [INFO] ATRACE(150984,python):2024-01-11-06:05:29.828.211 [trace_rb_log.c:84](tid:150984) [RUNTIME_ATRACE_DEV1_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(150984,python):2024-01-11-06:05:29.828.225 [atrace_api.c:32](tid:150984) AtraceCreate end [INFO] TDT(150984,python):2024-01-11-06:05:29.828.242 [client_manager.cpp:157][SetProfilingCallback][tid:150984] [TsdClient] set profiling callback success [INFO] PROFILING(150984,python):2024-01-11-06:05:29.845.963 [msprofiler_impl.cpp:156] >>> (tid:150984) ProfNotifySetDevice called, is open: 1, devId: 1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.846.015 [msprofiler_impl.cpp:289] >>> (tid:150984) Get system free ram: 554669887488 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.846.032 [prof_cann_plugin.cpp:75] >>> (tid:150984) Init report buffer size: 131072 bytes, buffer name: api_event [INFO] PROFILING(150984,python):2024-01-11-06:05:29.849.471 [prof_cann_plugin.cpp:75] >>> (tid:150984) Init report buffer size: 131072 bytes, buffer name: compact [INFO] PROFILING(150984,python):2024-01-11-06:05:29.853.874 [prof_cann_plugin.cpp:75] >>> (tid:150984) Init report buffer size: 262144 bytes, buffer name: additional [INFO] PROFILING(150984,python):2024-01-11-06:05:29.880.699 [platform.cpp:38] >>> (tid:150984) Profiling platform version: 1.0. [INFO] PROFILING(150984,python):2024-01-11-06:05:29.880.730 [ai_drv_dev_api.cpp:384] >>> (tid:150984) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.880.905 [prof_acl_mgr.cpp:286] >>> (tid:150984) Received ProfAclInit request from acl [INFO] PROFILING(150984,python):2024-01-11-06:05:29.880.969 [msprof_reporter.cpp:98] >>> (tid:150984) Init all reporters [INFO] PROFILING(150984,python):2024-01-11-06:05:29.882.362 [prof_acl_mgr.cpp:350] >>> (tid:150984) Received ProfAclStart request from acl [INFO] PROFILING(150984,python):2024-01-11-06:05:29.882.473 [ai_drv_dev_api.cpp:384] >>> (tid:150984) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.884.340 [hdc_api.cpp:112] >>> (tid:150984) logDevId 1 create HDC server successfully [INFO] PROFILING(150984,python):2024-01-11-06:05:29.886.552 [prof_manager.cpp:384] >>> (tid:150984) Received libmsprof message to start profiling, job_id:1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.886.593 [prof_manager.cpp:152] >>> (tid:150984) Check device profiling status [INFO] PROFILING(150984,python):2024-01-11-06:05:29.886.620 [prof_manager.cpp:272] >>> (tid:150984) Begin to launch task, jobId:1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.886.622 [prof_acl_mgr.cpp:95] >>> (tid:151062) Device 1 started to wait for response [INFO] PROFILING(150984,python):2024-01-11-06:05:29.886.691 [prof_acl_mgr.cpp:2258] >>> (tid:150984) Init profiling for msproftx [INFO] PROFILING(150984,python):2024-01-11-06:05:29.886.705 [prof_acl_mgr.cpp:2376] >>> (tid:150984) MsprofSetDeviceImpl, devId:64 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.888.356 [prof_manager.cpp:384] >>> (tid:150984) Received libmsprof message to start profiling, job_id:64 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.888.380 [prof_manager.cpp:152] >>> (tid:150984) Check device profiling status [INFO] PROFILING(150984,python):2024-01-11-06:05:29.888.398 [prof_manager.cpp:272] >>> (tid:150984) Begin to launch task, jobId:64 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.888.367 [ai_drv_prof_api.cpp:33] >>> (tid:151064) Begin to get channels, deviceId=1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.888.469 [prof_acl_mgr.cpp:95] >>> (tid:151074) Device 64 started to wait for response [INFO] PROFILING(150984,python):2024-01-11-06:05:29.889.350 [prof_acl_mgr.cpp:85] >>> (tid:151078) Device 64 finished starting [INFO] PROFILING(150984,python):2024-01-11-06:05:29.889.386 [prof_acl_mgr.cpp:97] >>> (tid:151074) Device 64 finished waiting for response [INFO] PROFILING(150984,python):2024-01-11-06:05:29.889.445 [ai_drv_prof_api.cpp:66] >>> (tid:151064) End to get channels[17], deviceId=1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.889.626 [prof_acl_mgr.cpp:1422] >>> (tid:150984) Device:64 finished waiting [INFO] PROFILING(150984,python):2024-01-11-06:05:29.889.648 [ai_drv_dev_api.cpp:384] >>> (tid:150984) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.891.224 [ai_drv_prof_api.cpp:436] >>> (tid:151064) Begin to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.912.306 [ai_drv_prof_api.cpp:454] >>> (tid:151064) Succeeded to start profiling DrvTsFwStart, profDeviceId=1, profChannel=44 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.914.042 [ai_drv_prof_api.cpp:296] >>> (tid:151064) Begin to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43, configSize:56bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.914.067 [ai_drv_prof_api.cpp:298] >>> (tid:151064) DrvAicoreTaskBasedStart, event_num=7, events=0x49,0x4a,0x4b,0x4c,0x4d,0x4e,0x4f,, tag=0 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.914.654 [ai_drv_prof_api.cpp:319] >>> (tid:151064) Succeeded to start profiling DrvAicoreTaskBasedStart, profDeviceId=1, profChannel=43 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.916.376 [ai_drv_prof_api.cpp:591] >>> (tid:151064) Begin to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45, tag=0 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.916.972 [ai_drv_prof_api.cpp:606] >>> (tid:151064) Succeeded to start profiling DrvHwtsLogStart, profDeviceId=1, profChannel=45 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.918.671 [ai_drv_prof_api.cpp:618] >>> (tid:151064) Begin to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.198 [ai_drv_dev_api.cpp:384] >>> (tid:151063) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.266 [ai_drv_prof_api.cpp:633] >>> (tid:151064) Succeeded to start profiling DrvFmkDataStart, devId=1, profChannel=46 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.271 [ai_drv_dev_api.cpp:384] >>> (tid:151063) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.338 [prof_acl_mgr.cpp:85] >>> (tid:151064) Device 1 finished starting [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.369 [prof_acl_mgr.cpp:97] >>> (tid:151062) Device 1 finished waiting for response [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.488 [prof_task.cpp:285] >>> (tid:151063) ProfTask 1 started to wait for task stop cv [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.502 [prof_acl_mgr.cpp:1413] >>> (tid:150984) All devices finished waiting [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.527 [ai_drv_dev_api.cpp:384] >>> (tid:151076) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.919.672 [prof_task.cpp:285] >>> (tid:151076) ProfTask 64 started to wait for task stop cv [INFO] TDT(150984,python):2024-01-11-06:05:29.920.143 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:150984] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=7] [INFO] TDT(150984,python):2024-01-11-06:05:29.920.170 [version_verify.cpp:112][SpecialFeatureCheck][tid:150984] VersionVerify: previous type[30], supported [INFO] TDT(150984,python):2024-01-11-06:05:29.920.217 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:150984] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] TDT(150984,python):2024-01-11-06:05:29.986.047 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:150984] [TsdClient] Update profiling mode [deviceId=1][sessionId=1][flag=6] [INFO] TDT(150984,python):2024-01-11-06:05:29.986.111 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:150984] [TsdClient][deviceId=1] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(150984,python):2024-01-11-06:05:29.986.491 [prof_reporter_mgr.cpp:226] >>> (tid:150984) total_size_type_info[5000], save type info length: 4544 bytes, type info size: 183 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.986.521 [prof_reporter_mgr.cpp:226] >>> (tid:150984) total_size_type_info[5500], save type info length: 35 bytes, type info size: 2 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.986.555 [prof_reporter_mgr.cpp:226] >>> (tid:150984) total_size_type_info[10000], save type info length: 404 bytes, type info size: 15 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.986.576 [prof_reporter_mgr.cpp:226] >>> (tid:150984) total_size_type_info[15000], save type info length: 67 bytes, type info size: 3 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.986.593 [uploader_dumper.cpp:178] >>> (tid:150984) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.678 [uploader_dumper.cpp:182] >>> (tid:150984) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.690 [uploader_dumper.cpp:178] >>> (tid:150984) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.719 [file_slice.cpp:356] >>> (tid:150984) [FileSliceFlush]file:aging.compact.task_track.slice_, total_size_file:768 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.881 [uploader_dumper.cpp:182] >>> (tid:150984) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.898 [uploader_dumper.cpp:178] >>> (tid:150984) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.911 [uploader_dumper.cpp:182] >>> (tid:150984) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.927 [prof_acl_mgr.cpp:435] >>> (tid:150984) Received ProfAclStop request from acl [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.941 [prof_manager.cpp:382] >>> (tid:150984) Received libmsprof message to stop profiling, job_id:1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.972 [prof_manager.cpp:305] >>> (tid:150984) Begin to stop task, jobId:1 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.987.996 [prof_task.cpp:344] >>> (tid:150984) Task send finished cv [INFO] PROFILING(150984,python):2024-01-11-06:05:29.988.015 [prof_task.cpp:287] >>> (tid:151063) ProfTask 1 finished waiting for task stop cv [INFO] PROFILING(150984,python):2024-01-11-06:05:29.988.313 [ai_drv_prof_api.cpp:642] >>> (tid:151064) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.988.992 [ai_drv_prof_api.cpp:650] >>> (tid:151064) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=44 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.989.024 [prof_channel.cpp:89] >>> (tid:151064) device id 1, channel: 44, total_size_channel: 80 bytes, file:data/ts_track.data, job_id:1,drvChannelReadCont:4 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.989.171 [ai_drv_prof_api.cpp:642] >>> (tid:151064) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.989.924 [ai_drv_prof_api.cpp:650] >>> (tid:151064) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=43 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.989.949 [prof_channel.cpp:89] >>> (tid:151064) device id 1, channel: 43, total_size_channel: 0 bytes, file:data/aicore.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.990.767 [ai_drv_prof_api.cpp:642] >>> (tid:151064) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.991.426 [ai_drv_prof_api.cpp:650] >>> (tid:151064) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=45 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.991.446 [prof_channel.cpp:89] >>> (tid:151064) device id 1, channel: 45, total_size_channel: 292480 bytes, file:data/hwts.data, job_id:1,drvChannelReadCont:13 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.991.520 [ai_drv_prof_api.cpp:642] >>> (tid:151064) Begin to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.992.124 [ai_drv_prof_api.cpp:650] >>> (tid:151064) Succeeded to stop profiling prof_stop DrvStop, profDeviceId=1, profChannel=46 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.992.142 [prof_channel.cpp:89] >>> (tid:151064) device id 1, channel: 46, total_size_channel: 0 bytes, file:data/training_trace.data, job_id:1,drvChannelReadCont:0 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.995.370 [prof_channel.cpp:407] >>> (tid:151064) ChannelPoll count: 34, Sleep count: 24, Dispatch count: 10, DispatchChannel count: 9 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.995.699 [file_slice.cpp:356] >>> (tid:151063) [FileSliceFlush]file:hwts.data.1.slice_, total_size_file:292480 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.995.865 [file_slice.cpp:356] >>> (tid:151063) [FileSliceFlush]file:ts_track.data.1.slice_, total_size_file:80 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.996.364 [prof_task.cpp:76] >>> (tid:151063) Uninit ProfTask succesfully [INFO] PROFILING(150984,python):2024-01-11-06:05:29.996.380 [prof_task.cpp:339] >>> (tid:151063) Task 1 finished [INFO] PROFILING(150984,python):2024-01-11-06:05:29.996.539 [prof_manager.cpp:382] >>> (tid:150984) Received libmsprof message to stop profiling, job_id:64 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.996.557 [prof_manager.cpp:305] >>> (tid:150984) Begin to stop task, jobId:64 [INFO] PROFILING(150984,python):2024-01-11-06:05:29.996.567 [prof_task.cpp:344] >>> (tid:150984) Task send finished cv [INFO] PROFILING(150984,python):2024-01-11-06:05:29.996.593 [prof_task.cpp:287] >>> (tid:151076) ProfTask 64 finished waiting for task stop cv [INFO] PROFILING(150984,python):2024-01-11-06:05:29.997.030 [prof_task.cpp:76] >>> (tid:151076) Uninit ProfTask succesfully [INFO] PROFILING(150984,python):2024-01-11-06:05:29.997.050 [prof_task.cpp:339] >>> (tid:151076) Task 64 finished [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.132 [uploader_dumper.cpp:178] >>> (tid:150984) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.158 [uploader_dumper.cpp:182] >>> (tid:150984) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.324 [receive_data.cpp:353] >>> (tid:150984) total_size_report module:api_event, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.351 [uploader_dumper.cpp:178] >>> (tid:150984) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.365 [uploader_dumper.cpp:182] >>> (tid:150984) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.535 [receive_data.cpp:353] >>> (tid:150984) total_size_report module:compact, push count:0, pop count:12, push size:0 bytes, pop size:768 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.551 [uploader_dumper.cpp:178] >>> (tid:150984) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(150984,python):2024-01-11-06:05:29.998.563 [uploader_dumper.cpp:182] >>> (tid:150984) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(150984,python):2024-01-11-06:05:29.999.410 [receive_data.cpp:353] >>> (tid:150984) total_size_report module:additional, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:29.999.556 [prof_acl_mgr.cpp:496] >>> (tid:150984) Received ProfAclFinalize request from acl [INFO] PROFILING(150984,python):2024-01-11-06:05:30.000.096 [prof_inner_api.cpp:101] >>> (tid:150984) total_size_report [api_event] read size: 0 bytes, write size: 0 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:30.000.905 [prof_inner_api.cpp:101] >>> (tid:150984) total_size_report [compact] read size: 12 bytes, write size: 12 bytes [INFO] PROFILING(150984,python):2024-01-11-06:05:30.006.221 [prof_inner_api.cpp:101] >>> (tid:150984) total_size_report [additional] read size: 0 bytes, write size: 0 bytes [ERROR] PIPELINE(150984,ffffbde97010,python):2024-01-11-06:05:33.152.197 [mindspore/ccsrc/pipeline/jit/ps/init.cc:524] operator()] Failed to parse profiler data.RuntimeError: Read op summary failed. The file is missing basic fields. At: /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(133): _read_op_summary /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/parser/ascend_msprof_generator.py(101): parse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(310): _ascend_graph_msprof_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1293): _ascend_graph_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(1040): _ascend_analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(659): _analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py(607): analyse /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/envprofiling.py(230): analyse Traceback (most recent call last): File "./run_net.py", line 136, in train_with_profiler() File "./run_net.py", line 125, in train_with_profiler lenet = LeNet5() File "./run_net.py", line 55, in __init__ super(LeNet5, self).__init__() File "/home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py", line 134, in __init__ init_pipeline() RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. ---------------------------------------------------- - Framework Error Message: ---------------------------------------------------- Malloc device memory failed, free memory size is less than half of total memory size.Device 1 Device HBM total size:34359738368 Device HBM free size:1603100672 may be other processes occupying this card, check as: ps -ef|grep python ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize [INFO] RUNTIME(150984,python):2024-01-11-06:05:34.172.188 [runtime.cc:1737] 150984 ~Runtime: deconstruct runtime. [INFO] ATRACE(150984,python):2024-01-11-06:05:34.273.213 [atrace_api.c:93](tid:150984) AtraceDestroy start [INFO] ATRACE(150984,python):2024-01-11-06:05:34.273.250 [atrace_api.c:95](tid:150984) AtraceDestroy end F =================================== FAILURES =================================== __________________ TestEnvEnableProfiler.test_ascend_profiler __________________ self = @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap def test_ascend_profiler(self): status = os.system( """export MS_PROFILER_OPTIONS='{"start":true, "profile_memory":true}'; python ./run_net.py --target=Ascend --mode=0; """ ) > CheckProfilerFiles(self.device_id, self.rank_id, self.profiler_path, "Ascend") test_env_enable_profiler.py:191: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_env_enable_profiler.py:43: in __init__ self._check_d_profiling_file() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def _check_d_profiling_file(self): """Check Ascend profiling file.""" aicore_file = self.profiler_path + f'aicore_intermediate_{self.rank_id}_detail.csv' # step_trace_file = self.profiler_path + f'step_trace_raw_{self.rank_id}_detail_time.csv' timeline_file = self.profiler_path + f'ascend_timeline_display_{self.rank_id}.json' aicpu_file = self.profiler_path + f'aicpu_intermediate_{self.rank_id}.csv' minddata_pipeline_file = self.profiler_path + f'minddata_pipeline_raw_{self.rank_id}.csv' queue_profiling_file = self.profiler_path + f'device_queue_profiling_{self.rank_id}.txt' memory_file = self.profiler_path + f'memory_usage_{self.rank_id}.pb' d_profiler_files = (aicore_file, timeline_file, aicpu_file, minddata_pipeline_file, queue_profiling_file, memory_file) for file in d_profiler_files: > assert os.path.isfile(file) E AssertionError: assert False E + where False = ('/home/jenkins/mindspore/testcases/testcases/tests/st/profiler/data/profiler/aicore_intermediate_0_detail.csv') E + where = .isfile E + where = os.path test_env_enable_profiler.py:78: AssertionError ________________ TestEnvEnableProfiler.test_host_profiler_none _________________ self = @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap def test_host_profiler_none(self): status = os.system( """export MS_PROFILER_OPTIONS='{"start":true, "profile_memory":true, "profile_framework":null}'; python ./run_net.py --target=Ascend --mode=0; """ ) > CheckProfilerFiles(self.device_id, self.rank_id, self.profiler_path, "Ascend", None) test_env_enable_profiler.py:205: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_env_enable_profiler.py:43: in __init__ self._check_d_profiling_file() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def _check_d_profiling_file(self): """Check Ascend profiling file.""" aicore_file = self.profiler_path + f'aicore_intermediate_{self.rank_id}_detail.csv' # step_trace_file = self.profiler_path + f'step_trace_raw_{self.rank_id}_detail_time.csv' timeline_file = self.profiler_path + f'ascend_timeline_display_{self.rank_id}.json' aicpu_file = self.profiler_path + f'aicpu_intermediate_{self.rank_id}.csv' minddata_pipeline_file = self.profiler_path + f'minddata_pipeline_raw_{self.rank_id}.csv' queue_profiling_file = self.profiler_path + f'device_queue_profiling_{self.rank_id}.txt' memory_file = self.profiler_path + f'memory_usage_{self.rank_id}.pb' d_profiler_files = (aicore_file, timeline_file, aicpu_file, minddata_pipeline_file, queue_profiling_file, memory_file) for file in d_profiler_files: > assert os.path.isfile(file) E AssertionError: assert False E + where False = ('/home/jenkins/mindspore/testcases/testcases/tests/st/profiler/data/profiler/aicore_intermediate_0_detail.csv') E + where = .isfile E + where = os.path test_env_enable_profiler.py:78: AssertionError ________________ TestEnvEnableProfiler.test_host_profiler_time _________________ self = @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap def test_host_profiler_time(self): status = os.system( """export MS_PROFILER_OPTIONS='{"start":true, "profile_memory":true, "profile_framework":"time"}'; python ./run_net.py --target=Ascend --mode=0; """ ) > CheckProfilerFiles(self.device_id, self.rank_id, self.profiler_path, "Ascend", "time") test_env_enable_profiler.py:219: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_env_enable_profiler.py:43: in __init__ self._check_d_profiling_file() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def _check_d_profiling_file(self): """Check Ascend profiling file.""" aicore_file = self.profiler_path + f'aicore_intermediate_{self.rank_id}_detail.csv' # step_trace_file = self.profiler_path + f'step_trace_raw_{self.rank_id}_detail_time.csv' timeline_file = self.profiler_path + f'ascend_timeline_display_{self.rank_id}.json' aicpu_file = self.profiler_path + f'aicpu_intermediate_{self.rank_id}.csv' minddata_pipeline_file = self.profiler_path + f'minddata_pipeline_raw_{self.rank_id}.csv' queue_profiling_file = self.profiler_path + f'device_queue_profiling_{self.rank_id}.txt' memory_file = self.profiler_path + f'memory_usage_{self.rank_id}.pb' d_profiler_files = (aicore_file, timeline_file, aicpu_file, minddata_pipeline_file, queue_profiling_file, memory_file) for file in d_profiler_files: > assert os.path.isfile(file) E AssertionError: assert False E + where False = ('/home/jenkins/mindspore/testcases/testcases/tests/st/profiler/data/profiler/aicore_intermediate_0_detail.csv') E + where = .isfile E + where = os.path test_env_enable_profiler.py:78: AssertionError _______________ TestEnvEnableProfiler.test_host_profiler_memory ________________ self = @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap def test_host_profiler_memory(self): status = os.system( """export MS_PROFILER_OPTIONS='{"start":true, "profile_memory":true, "profile_framework":"memory"}'; python ./run_net.py --target=Ascend --mode=0; """ ) > CheckProfilerFiles(self.device_id, self.rank_id, self.profiler_path, "Ascend", "memory") test_env_enable_profiler.py:233: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_env_enable_profiler.py:43: in __init__ self._check_d_profiling_file() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def _check_d_profiling_file(self): """Check Ascend profiling file.""" aicore_file = self.profiler_path + f'aicore_intermediate_{self.rank_id}_detail.csv' # step_trace_file = self.profiler_path + f'step_trace_raw_{self.rank_id}_detail_time.csv' timeline_file = self.profiler_path + f'ascend_timeline_display_{self.rank_id}.json' aicpu_file = self.profiler_path + f'aicpu_intermediate_{self.rank_id}.csv' minddata_pipeline_file = self.profiler_path + f'minddata_pipeline_raw_{self.rank_id}.csv' queue_profiling_file = self.profiler_path + f'device_queue_profiling_{self.rank_id}.txt' memory_file = self.profiler_path + f'memory_usage_{self.rank_id}.pb' d_profiler_files = (aicore_file, timeline_file, aicpu_file, minddata_pipeline_file, queue_profiling_file, memory_file) for file in d_profiler_files: > assert os.path.isfile(file) E AssertionError: assert False E + where False = ('/home/jenkins/mindspore/testcases/testcases/tests/st/profiler/data/profiler/aicore_intermediate_0_detail.csv') E + where = .isfile E + where = os.path test_env_enable_profiler.py:78: AssertionError =========================== short test summary info ============================ FAILED test_env_enable_profiler.py::TestEnvEnableProfiler::test_ascend_profiler FAILED test_env_enable_profiler.py::TestEnvEnableProfiler::test_host_profiler_none FAILED test_env_enable_profiler.py::TestEnvEnableProfiler::test_host_profiler_time FAILED test_env_enable_profiler.py::TestEnvEnableProfiler::test_host_profiler_memory ======================= 4 failed, 3 deselected in 55.16s ======================= [INFO] RUNTIME(149696,python3.7):2024-01-11-06:05:37.111.051 [runtime.cc:1737] 149696 ~Runtime: deconstruct runtime.