============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/numpy_native, inifile: /home/jenkins/sault/virtual_test/virtualenv_004/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:38.625.853 [trace_attr.c:105](tid:121123) platform is 1. [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:38.626.011 [trace_recorder.c:114](tid:121123) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:38.626.035 [trace_signal.c:133](tid:121123) register signal handler for signo 2 succeed. [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:38.626.048 [trace_signal.c:133](tid:121123) register signal handler for signo 15 succeed. [INFO] RUNTIME(121123,python3.7):2024-01-11-05:59:39.016.966 [runtime.cc:1159] 121123 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(121123,python3.7):2024-01-11-05:59:39.017.003 [runtime.cc:4719] 121123 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 1 item test_array_ops.py [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.313.965 [process_mode_manager.cpp:109][OpenProcess][tid:121590] [ProcessModeManager] enter into open process deviceId[3] rankSize[0] [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.314.951 [process_mode_manager.cpp:379][InitTsdClient][tid:121590] [TsdClient] deviceId[3] begin to init hdc client [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.315.374 [version_verify.cpp:34][SetVersionInfo][tid:121590] VersionVerify: send client version to server [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.315.401 [version_verify.cpp:50][SetVersionInfo][tid:121590] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.315.414 [version_verify.cpp:50][SetVersionInfo][tid:121590] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.181 [version_verify.cpp:66][PeerVersionCheck][tid:121590] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.196 [version_verify.cpp:87][ParseVersionInfo][tid:121590] VersionVerify: pass client version info success [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.206 [hdc_client.cpp:276][CheckHdcConnection][tid:121590] Service[2] create hdc success [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.218 [version_verify.cpp:120][SpecialFeatureCheck][tid:121590] VersionVerify: new type[35], supported [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.256 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:121590] [TsdClient][deviceId=3] [sessionId=1] wait package info respond [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.393 [process_mode_manager.cpp:379][InitTsdClient][tid:121590] [TsdClient] deviceId[3] begin to init hdc client [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.959 [version_verify.cpp:34][SetVersionInfo][tid:121590] VersionVerify: send client version to server [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.974 [version_verify.cpp:50][SetVersionInfo][tid:121590] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.316.985 [version_verify.cpp:50][SetVersionInfo][tid:121590] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.317.333 [version_verify.cpp:66][PeerVersionCheck][tid:121590] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.317.347 [version_verify.cpp:87][ParseVersionInfo][tid:121590] VersionVerify: pass client version info success [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.317.357 [hdc_client.cpp:276][CheckHdcConnection][tid:121590] Service[2] create hdc success [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.317.367 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:121590] [TsdClient] tsd get process sign successfully, procpid[121123] signSize[48] [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.317.390 [version_verify.cpp:112][SpecialFeatureCheck][tid:121590] VersionVerify: previous type[6], supported [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.317.409 [process_mode_manager.cpp:126][OpenProcess][tid:121590] [ProcessModeManager] deviceId[3] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.603.984 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:121590] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.604.012 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:121590] enter into OpenInHost deviceid[3] [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.604.023 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:121590] host cpu not support [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.604.031 [process_mode_manager.cpp:156][OpenProcess][tid:121590] [TsdClient][deviceId=3] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(121123,python3.7):2024-01-11-05:59:43.606.660 [device.cc:340] 121590 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(121123,python3.7):2024-01-11-05:59:43.620.899 [npu_driver.cc:5428] 121606 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:43.621.667 [atrace_api.c:28](tid:121590) AtraceCreate start [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:43.621.753 [trace_rb_log.c:84](tid:121590) [RUNTIME_ATRACE_DEV3_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:43.621.768 [atrace_api.c:32](tid:121590) AtraceCreate end [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.621.782 [client_manager.cpp:157][SetProfilingCallback][tid:121590] [TsdClient] set profiling callback success [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.631.154 [process_mode_manager.cpp:184][Close][tid:121590] [TsdClient] Close [deviceId=3][sessionId=1] hccp and computer enter [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.631.183 [version_verify.cpp:112][SpecialFeatureCheck][tid:121590] VersionVerify: previous type[7], supported [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.631.221 [process_mode_manager.cpp:192][Close][tid:121590] [TsdClient][deviceId=3] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.652.655 [process_mode_manager.cpp:197][Close][tid:121590] [TsdClient][logicDeviceId_=3]has recv close hccp and computer process respond [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.652.671 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:121590] enter into CloseInHost deviceid[3] [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.652.682 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:121590] host cpu not support [INFO] TDT(121123,python3.7):2024-01-11-05:59:43.652.713 [process_mode_manager.cpp:208][Close][tid:121590] [TsdClient][deviceId=3] [sessionId=1] close hccp and computer process success [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:43.652.726 [atrace_api.c:93](tid:121590) AtraceDestroy start [INFO] ATRACE(121123,python3.7):2024-01-11-05:59:43.652.737 [atrace_api.c:95](tid:121590) AtraceDestroy end F =================================== FAILURES =================================== __________________________________ test_split __________________________________ @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_split(): onp_arrs = [ onp.random.randint(1, 5, size=(9, 4, 5)).astype('float32') ] mnp_arrs = [to_tensor(arr) for arr in onp_arrs] for onp_arr, mnp_arr in zip(onp_arrs, mnp_arrs): o_split = onp_split(onp_arr) > m_split = mnp_split(mnp_arr) test_array_ops.py:1019: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_array_ops.py:996: in mnp_split b = mnp.split(input_tensor, indices_or_sections=3) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/numpy/array_ops.py:1429: in split return _split(x, indices_or_sections, opname="split", axis=axis) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/numpy/array_ops.py:1444: in _split res = P.Split(axis_new, indices_or_sections)(x) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/auto_generate/gen_ops_def.py:4208: in __call__ return super().__call__(input_x, self.axis, self.output_num) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/primitive.py:324: in __call__ return _run_op(self, self.name, args) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/primitive.py:942: in _run_op return _convert_stub(stub) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ stub = def _convert_stub(stub): "convert stub to StubNode or Value" if isinstance(stub, TensorNode): return StubTensor(stub) if isinstance(stub, tuple): return tuple(_convert_stub(e) for e in stub) if isinstance(stub, SequenceNode): > elements = stub.get_elements() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 3 Device HBM total size:34359738368 Device HBM free size:1602785280 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:196: RuntimeError =========================== short test summary info ============================ FAILED test_array_ops.py::test_split - RuntimeError: Ascend kernel runtime in... ============================== 1 failed in 7.23s =============================== [INFO] RUNTIME(121123,python3.7):2024-01-11-05:59:45.393.931 [runtime.cc:1737] 121123 ~Runtime: deconstruct runtime.