============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/numpy_native, inifile: /home/jenkins/sault/virtual_test/virtualenv_004/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:48.840.255 [trace_attr.c:105](tid:111903) platform is 1. [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:48.840.411 [trace_recorder.c:114](tid:111903) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:48.840.435 [trace_signal.c:133](tid:111903) register signal handler for signo 2 succeed. [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:48.840.448 [trace_signal.c:133](tid:111903) register signal handler for signo 15 succeed. [INFO] RUNTIME(111903,python3.7):2024-01-11-05:57:49.231.564 [runtime.cc:1159] 111903 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(111903,python3.7):2024-01-11-05:57:49.231.606 [runtime.cc:4719] 111903 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 1 item test_math_ops.py [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.557.168 [process_mode_manager.cpp:109][OpenProcess][tid:112425] [ProcessModeManager] enter into open process deviceId[3] rankSize[0] [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.111 [process_mode_manager.cpp:379][InitTsdClient][tid:112425] [TsdClient] deviceId[3] begin to init hdc client [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.230 [version_verify.cpp:34][SetVersionInfo][tid:112425] VersionVerify: send client version to server [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.261 [version_verify.cpp:50][SetVersionInfo][tid:112425] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.275 [version_verify.cpp:50][SetVersionInfo][tid:112425] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.731 [version_verify.cpp:66][PeerVersionCheck][tid:112425] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.747 [version_verify.cpp:87][ParseVersionInfo][tid:112425] VersionVerify: pass client version info success [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.758 [hdc_client.cpp:276][CheckHdcConnection][tid:112425] Service[2] create hdc success [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.773 [version_verify.cpp:120][SpecialFeatureCheck][tid:112425] VersionVerify: new type[35], supported [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.811 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:112425] [TsdClient][deviceId=3] [sessionId=1] wait package info respond [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.912 [process_mode_manager.cpp:379][InitTsdClient][tid:112425] [TsdClient] deviceId[3] begin to init hdc client [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.558.999 [version_verify.cpp:34][SetVersionInfo][tid:112425] VersionVerify: send client version to server [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.012 [version_verify.cpp:50][SetVersionInfo][tid:112425] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.022 [version_verify.cpp:50][SetVersionInfo][tid:112425] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.452 [version_verify.cpp:66][PeerVersionCheck][tid:112425] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.466 [version_verify.cpp:87][ParseVersionInfo][tid:112425] VersionVerify: pass client version info success [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.476 [hdc_client.cpp:276][CheckHdcConnection][tid:112425] Service[2] create hdc success [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.489 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:112425] [TsdClient] tsd get process sign successfully, procpid[111903] signSize[48] [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.514 [version_verify.cpp:112][SpecialFeatureCheck][tid:112425] VersionVerify: previous type[6], supported [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.559.532 [process_mode_manager.cpp:126][OpenProcess][tid:112425] [ProcessModeManager] deviceId[3] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.767.132 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:112425] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.767.161 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:112425] enter into OpenInHost deviceid[3] [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.767.172 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:112425] host cpu not support [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.767.182 [process_mode_manager.cpp:156][OpenProcess][tid:112425] [TsdClient][deviceId=3] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(111903,python3.7):2024-01-11-05:57:53.769.860 [device.cc:340] 112425 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(111903,python3.7):2024-01-11-05:57:53.783.861 [npu_driver.cc:5428] 112442 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:53.785.217 [atrace_api.c:28](tid:112425) AtraceCreate start [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:53.785.314 [trace_rb_log.c:84](tid:112425) [RUNTIME_ATRACE_DEV3_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:53.785.329 [atrace_api.c:32](tid:112425) AtraceCreate end [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.785.345 [client_manager.cpp:157][SetProfilingCallback][tid:112425] [TsdClient] set profiling callback success [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.796.218 [process_mode_manager.cpp:184][Close][tid:112425] [TsdClient] Close [deviceId=3][sessionId=1] hccp and computer enter [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.796.257 [version_verify.cpp:112][SpecialFeatureCheck][tid:112425] VersionVerify: previous type[7], supported [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.796.297 [process_mode_manager.cpp:192][Close][tid:112425] [TsdClient][deviceId=3] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.817.742 [process_mode_manager.cpp:197][Close][tid:112425] [TsdClient][logicDeviceId_=3]has recv close hccp and computer process respond [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.817.761 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:112425] enter into CloseInHost deviceid[3] [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.817.773 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:112425] host cpu not support [INFO] TDT(111903,python3.7):2024-01-11-05:57:53.817.807 [process_mode_manager.cpp:208][Close][tid:112425] [TsdClient][deviceId=3] [sessionId=1] close hccp and computer process success [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:53.817.821 [atrace_api.c:93](tid:112425) AtraceDestroy start [INFO] ATRACE(111903,python3.7):2024-01-11-05:57:53.817.837 [atrace_api.c:95](tid:112425) AtraceDestroy end F =================================== FAILURES =================================== _________________________________ test_cumsum __________________________________ @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_cumsum(): x = mnp.ones((16, 16), dtype="bool") > match_array(mnp.cumsum(x).asnumpy(), onp.cumsum(x.asnumpy())) test_math_ops.py:1387: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/jenkins/.local/lib/python3.7/site-packages/mindspore/numpy/math_ops.py:3152: in cumsum return a.cumsum(axis, dtype) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/tensor.py:2195: in cumsum validator.check_axis_in_range(axis, x.ndim) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:117: in ndim return len(self.shape) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[RuntimeError("Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'.\n\n---------...ice/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize\n") raised in repr()] StubTensor object at 0xfffef786ead0> @property def shape(self): """shape stub.""" if self.stub: if not hasattr(self, "stub_shape"): > self.stub_shape = self.stub.get_shape() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 3 Device HBM total size:34359738368 Device HBM free size:1602985984 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:85: RuntimeError =========================== short test summary info ============================ FAILED test_math_ops.py::test_cumsum - RuntimeError: Ascend kernel runtime in... ============================== 1 failed in 7.18s =============================== [INFO] RUNTIME(111903,python3.7):2024-01-11-05:57:55.597.280 [runtime.cc:1737] 111903 ~Runtime: deconstruct runtime.