============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/sparse, inifile: /home/jenkins/sault/virtual_test/virtualenv_001/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:35.642.757 [trace_attr.c:105](tid:138041) platform is 1. [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:35.642.927 [trace_recorder.c:114](tid:138041) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:35.642.957 [trace_signal.c:133](tid:138041) register signal handler for signo 2 succeed. [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:35.642.971 [trace_signal.c:133](tid:138041) register signal handler for signo 15 succeed. [INFO] RUNTIME(138041,python3.7):2024-01-11-06:02:36.048.904 [runtime.cc:1159] 138041 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(138041,python3.7):2024-01-11-06:02:36.048.961 [runtime.cc:4719] 138041 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 1 item test_csr.py [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.506.353 [process_mode_manager.cpp:109][OpenProcess][tid:138041] [ProcessModeManager] enter into open process deviceId[0] rankSize[0] [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.019 [process_mode_manager.cpp:379][InitTsdClient][tid:138041] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.186 [version_verify.cpp:34][SetVersionInfo][tid:138041] VersionVerify: send client version to server [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.233 [version_verify.cpp:50][SetVersionInfo][tid:138041] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.248 [version_verify.cpp:50][SetVersionInfo][tid:138041] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.557 [version_verify.cpp:66][PeerVersionCheck][tid:138041] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.577 [version_verify.cpp:87][ParseVersionInfo][tid:138041] VersionVerify: pass client version info success [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.589 [hdc_client.cpp:276][CheckHdcConnection][tid:138041] Service[2] create hdc success [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.609 [version_verify.cpp:120][SpecialFeatureCheck][tid:138041] VersionVerify: new type[35], supported [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.658 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:138041] [TsdClient][deviceId=0] [sessionId=1] wait package info respond [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.799 [process_mode_manager.cpp:379][InitTsdClient][tid:138041] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.919 [version_verify.cpp:34][SetVersionInfo][tid:138041] VersionVerify: send client version to server [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.936 [version_verify.cpp:50][SetVersionInfo][tid:138041] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.508.949 [version_verify.cpp:50][SetVersionInfo][tid:138041] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.509.122 [version_verify.cpp:66][PeerVersionCheck][tid:138041] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.509.136 [version_verify.cpp:87][ParseVersionInfo][tid:138041] VersionVerify: pass client version info success [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.509.148 [hdc_client.cpp:276][CheckHdcConnection][tid:138041] Service[2] create hdc success [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.509.161 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:138041] [TsdClient] tsd get process sign successfully, procpid[138041] signSize[48] [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.509.188 [version_verify.cpp:112][SpecialFeatureCheck][tid:138041] VersionVerify: previous type[6], supported [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.509.215 [process_mode_manager.cpp:126][OpenProcess][tid:138041] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.750.415 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:138041] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.750.445 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:138041] enter into OpenInHost deviceid[0] [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.750.457 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:138041] host cpu not support [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.750.468 [process_mode_manager.cpp:156][OpenProcess][tid:138041] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(138041,python3.7):2024-01-11-06:02:40.753.197 [device.cc:340] 138041 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(138041,python3.7):2024-01-11-06:02:40.768.596 [npu_driver.cc:5428] 138617 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:40.768.666 [atrace_api.c:28](tid:138041) AtraceCreate start [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:40.768.786 [trace_rb_log.c:84](tid:138041) [RUNTIME_ATRACE_DEV0_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:40.768.804 [atrace_api.c:32](tid:138041) AtraceCreate end [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.768.833 [client_manager.cpp:157][SetProfilingCallback][tid:138041] [TsdClient] set profiling callback success [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.781.060 [process_mode_manager.cpp:184][Close][tid:138041] [TsdClient] Close [deviceId=0][sessionId=1] hccp and computer enter [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.781.101 [version_verify.cpp:112][SpecialFeatureCheck][tid:138041] VersionVerify: previous type[7], supported [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.781.145 [process_mode_manager.cpp:192][Close][tid:138041] [TsdClient][deviceId=0] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.802.026 [process_mode_manager.cpp:197][Close][tid:138041] [TsdClient][logicDeviceId_=0]has recv close hccp and computer process respond [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.802.043 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:138041] enter into CloseInHost deviceid[0] [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.802.057 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:138041] host cpu not support [INFO] TDT(138041,python3.7):2024-01-11-06:02:40.802.092 [process_mode_manager.cpp:208][Close][tid:138041] [TsdClient][deviceId=0] [sessionId=1] close hccp and computer process success [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:40.802.106 [atrace_api.c:93](tid:138041) AtraceDestroy start [INFO] ATRACE(138041,python3.7):2024-01-11-06:02:40.802.123 [atrace_api.c:95](tid:138041) AtraceDestroy end F =================================== FAILURES =================================== ________________________________ test_csr_attr _________________________________ @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_csr_attr(): """ Feature: Test CSRTensor GetAttr in Graph and PyNative. Description: Test CSRTensor.indptr, CSRTensor.indices, CSRTensor.values, CSRTensor.shape. Expectation: Success. """ if get_platform() != "linux": return indptr = Tensor([0, 1, 2]) indices = Tensor([0, 1]) values = Tensor([1, 2], dtype=mstype.float32) shape = (2, 6) csr = CSRTensor(indptr, indices, values, shape) def test_pynative_1(): return csr.indptr, csr.indices def test_pynative_2(): return csr.values, csr.shape def test_pynative_3(): return csr.astype(mstype.int32) def test_pynative_4(): return csr.to_tuple() test_graph_1 = jit(test_pynative_1) test_graph_2 = jit(test_pynative_2) test_graph_3 = jit(test_pynative_3) test_graph_4 = jit(test_pynative_4) py_indptr, py_indices = test_pynative_1() py_values, py_shape = test_pynative_2() > py_csr = test_pynative_3() test_csr.py:99: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_csr.py:87: in test_pynative_3 return csr.astype(mstype.int32) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/sparse_tensor.py:771: in astype data = self.values.astype(dtype) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/tensor.py:2071: in astype return tensor_operator_registry.get('cast')(self, dtype) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/primitive.py:324: in __call__ return _run_op(self, self.name, args) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/primitive.py:941: in _run_op stub = _pynative_executor.run_op_async(obj, op_name, args) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = args = (Prim[Cast], 'Cast', (Tensor(shape=[2], dtype=Float32, value= [ 1.00000000e+00, 2.00000000e+00]), mindspore.int32)) def run_op_async(self, *args): """ Run single op async. Args: args (tuple): Op prim and input arguments. Return: StubNode, result of run op. """ > return self._executor.run_op_async(*args) E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 0 Device HBM total size:34359738368 Device HBM free size:1603063808 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:1190: RuntimeError =========================== short test summary info ============================ FAILED test_csr.py::test_csr_attr - RuntimeError: Ascend kernel runtime initi... ============================== 1 failed in 7.39s =============================== [INFO] RUNTIME(138041,python3.7):2024-01-11-06:02:42.982.721 [runtime.cc:1737] 138041 ~Runtime: deconstruct runtime.