============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/numpy_native, inifile: /home/jenkins/sault/virtual_test/virtualenv_004/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:46.356.866 [trace_attr.c:105](tid:133333) platform is 1. [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:46.357.043 [trace_recorder.c:114](tid:133333) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:46.357.066 [trace_signal.c:133](tid:133333) register signal handler for signo 2 succeed. [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:46.357.077 [trace_signal.c:133](tid:133333) register signal handler for signo 15 succeed. [INFO] RUNTIME(133333,python3.7):2024-01-11-06:01:46.749.582 [runtime.cc:1159] 133333 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(133333,python3.7):2024-01-11-06:01:46.749.618 [runtime.cc:4719] 133333 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 1 item test_math_ops.py [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.006.906 [process_mode_manager.cpp:109][OpenProcess][tid:133869] [ProcessModeManager] enter into open process deviceId[3] rankSize[0] [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.007.607 [process_mode_manager.cpp:379][InitTsdClient][tid:133869] [TsdClient] deviceId[3] begin to init hdc client [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.007.727 [version_verify.cpp:34][SetVersionInfo][tid:133869] VersionVerify: send client version to server [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.007.751 [version_verify.cpp:50][SetVersionInfo][tid:133869] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.007.765 [version_verify.cpp:50][SetVersionInfo][tid:133869] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.143 [version_verify.cpp:66][PeerVersionCheck][tid:133869] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.158 [version_verify.cpp:87][ParseVersionInfo][tid:133869] VersionVerify: pass client version info success [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.169 [hdc_client.cpp:276][CheckHdcConnection][tid:133869] Service[2] create hdc success [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.181 [version_verify.cpp:120][SpecialFeatureCheck][tid:133869] VersionVerify: new type[35], supported [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.218 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:133869] [TsdClient][deviceId=3] [sessionId=1] wait package info respond [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.318 [process_mode_manager.cpp:379][InitTsdClient][tid:133869] [TsdClient] deviceId[3] begin to init hdc client [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.398 [version_verify.cpp:34][SetVersionInfo][tid:133869] VersionVerify: send client version to server [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.412 [version_verify.cpp:50][SetVersionInfo][tid:133869] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.423 [version_verify.cpp:50][SetVersionInfo][tid:133869] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.580 [version_verify.cpp:66][PeerVersionCheck][tid:133869] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.593 [version_verify.cpp:87][ParseVersionInfo][tid:133869] VersionVerify: pass client version info success [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.603 [hdc_client.cpp:276][CheckHdcConnection][tid:133869] Service[2] create hdc success [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.616 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:133869] [TsdClient] tsd get process sign successfully, procpid[133333] signSize[48] [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.639 [version_verify.cpp:112][SpecialFeatureCheck][tid:133869] VersionVerify: previous type[6], supported [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.008.660 [process_mode_manager.cpp:126][OpenProcess][tid:133869] [ProcessModeManager] deviceId[3] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.237.224 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:133869] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.237.259 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:133869] enter into OpenInHost deviceid[3] [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.237.271 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:133869] host cpu not support [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.237.281 [process_mode_manager.cpp:156][OpenProcess][tid:133869] [TsdClient][deviceId=3] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(133333,python3.7):2024-01-11-06:01:51.240.003 [device.cc:340] 133869 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(133333,python3.7):2024-01-11-06:01:51.254.262 [npu_driver.cc:5428] 133877 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:51.255.070 [atrace_api.c:28](tid:133869) AtraceCreate start [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:51.255.153 [trace_rb_log.c:84](tid:133869) [RUNTIME_ATRACE_DEV3_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:51.255.168 [atrace_api.c:32](tid:133869) AtraceCreate end [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.255.183 [client_manager.cpp:157][SetProfilingCallback][tid:133869] [TsdClient] set profiling callback success [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.264.622 [process_mode_manager.cpp:184][Close][tid:133869] [TsdClient] Close [deviceId=3][sessionId=1] hccp and computer enter [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.264.654 [version_verify.cpp:112][SpecialFeatureCheck][tid:133869] VersionVerify: previous type[7], supported [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.264.690 [process_mode_manager.cpp:192][Close][tid:133869] [TsdClient][deviceId=3] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.286.101 [process_mode_manager.cpp:197][Close][tid:133869] [TsdClient][logicDeviceId_=3]has recv close hccp and computer process respond [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.286.118 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:133869] enter into CloseInHost deviceid[3] [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.286.130 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:133869] host cpu not support [INFO] TDT(133333,python3.7):2024-01-11-06:01:51.286.162 [process_mode_manager.cpp:208][Close][tid:133869] [TsdClient][deviceId=3] [sessionId=1] close hccp and computer process success [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:51.286.175 [atrace_api.c:93](tid:133869) AtraceDestroy start [INFO] ATRACE(133333,python3.7):2024-01-11-06:01:51.286.188 [atrace_api.c:95](tid:133869) AtraceDestroy end F =================================== FAILURES =================================== _________________________________ test_invert __________________________________ @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard def test_invert(): x = onp.random.randint(-100, 100, (2, 3)) > match_res(mnp.invert, onp.invert, x, dtype=mnp.int16) test_math_ops.py:2619: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ utils.py:96: in match_res match_all_arrays(mnp_res, onp_res, error=error) utils.py:105: in match_all_arrays match_array(mnp_res.asnumpy(), onp_res, error) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:48: in fun arg = (stub.stub_sync(),) + arg[1:] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[RuntimeError("Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'.\n\n---------...ice/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize\n") raised in repr()] StubTensor object at 0xfffecbd9ead0> def stub_sync(self): """sync real tensor.""" if self.stub: > val = self.stub.get_value() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 3 Device HBM total size:34359738368 Device HBM free size:1602842624 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:153: RuntimeError =========================== short test summary info ============================ FAILED test_math_ops.py::test_invert - RuntimeError: Ascend kernel runtime in... ============================== 1 failed in 7.00s =============================== [INFO] RUNTIME(133333,python3.7):2024-01-11-06:01:52.926.958 [runtime.cc:1737] 133333 ~Runtime: deconstruct runtime.