============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/dyn_shape_dev, inifile: /home/jenkins/sault/virtual_test/virtualenv_008/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:23.321.460 [trace_attr.c:105](tid:142536) platform is 1. [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:23.321.622 [trace_recorder.c:114](tid:142536) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:23.321.646 [trace_signal.c:133](tid:142536) register signal handler for signo 2 succeed. [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:23.321.659 [trace_signal.c:133](tid:142536) register signal handler for signo 15 succeed. [INFO] RUNTIME(142536,python3.7):2024-01-11-06:03:23.713.254 [runtime.cc:1159] 142536 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(142536,python3.7):2024-01-11-06:03:23.713.297 [runtime.cc:4719] 142536 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 2 items test_add.py [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.980.782 [process_mode_manager.cpp:109][OpenProcess][tid:142536] [ProcessModeManager] enter into open process deviceId[7] rankSize[0] [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.263 [process_mode_manager.cpp:379][InitTsdClient][tid:142536] [TsdClient] deviceId[7] begin to init hdc client [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.396 [version_verify.cpp:34][SetVersionInfo][tid:142536] VersionVerify: send client version to server [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.422 [version_verify.cpp:50][SetVersionInfo][tid:142536] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.436 [version_verify.cpp:50][SetVersionInfo][tid:142536] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.743 [version_verify.cpp:66][PeerVersionCheck][tid:142536] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.759 [version_verify.cpp:87][ParseVersionInfo][tid:142536] VersionVerify: pass client version info success [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.770 [hdc_client.cpp:276][CheckHdcConnection][tid:142536] Service[2] create hdc success [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.783 [version_verify.cpp:120][SpecialFeatureCheck][tid:142536] VersionVerify: new type[35], supported [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.824 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:142536] [TsdClient][deviceId=7] [sessionId=1] wait package info respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.982.940 [process_mode_manager.cpp:379][InitTsdClient][tid:142536] [TsdClient] deviceId[7] begin to init hdc client [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.143 [version_verify.cpp:34][SetVersionInfo][tid:142536] VersionVerify: send client version to server [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.155 [version_verify.cpp:50][SetVersionInfo][tid:142536] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.166 [version_verify.cpp:50][SetVersionInfo][tid:142536] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.329 [version_verify.cpp:66][PeerVersionCheck][tid:142536] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.344 [version_verify.cpp:87][ParseVersionInfo][tid:142536] VersionVerify: pass client version info success [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.354 [hdc_client.cpp:276][CheckHdcConnection][tid:142536] Service[2] create hdc success [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.365 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:142536] [TsdClient] tsd get process sign successfully, procpid[142536] signSize[48] [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.392 [version_verify.cpp:112][SpecialFeatureCheck][tid:142536] VersionVerify: previous type[6], supported [INFO] TDT(142536,python3.7):2024-01-11-06:03:27.983.411 [process_mode_manager.cpp:126][OpenProcess][tid:142536] [ProcessModeManager] deviceId[7] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.249.001 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:142536] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.249.031 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:142536] enter into OpenInHost deviceid[7] [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.249.043 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:142536] host cpu not support [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.249.052 [process_mode_manager.cpp:156][OpenProcess][tid:142536] [TsdClient][deviceId=7] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(142536,python3.7):2024-01-11-06:03:28.251.681 [device.cc:340] 142536 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(142536,python3.7):2024-01-11-06:03:28.266.236 [npu_driver.cc:5428] 143166 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:28.266.297 [atrace_api.c:28](tid:142536) AtraceCreate start [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:28.266.380 [trace_rb_log.c:84](tid:142536) [RUNTIME_ATRACE_DEV7_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:28.266.396 [atrace_api.c:32](tid:142536) AtraceCreate end [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.266.413 [client_manager.cpp:157][SetProfilingCallback][tid:142536] [TsdClient] set profiling callback success [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.277.022 [process_mode_manager.cpp:184][Close][tid:142536] [TsdClient] Close [deviceId=7][sessionId=1] hccp and computer enter [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.277.063 [version_verify.cpp:112][SpecialFeatureCheck][tid:142536] VersionVerify: previous type[7], supported [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.277.102 [process_mode_manager.cpp:192][Close][tid:142536] [TsdClient][deviceId=7] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.300.917 [process_mode_manager.cpp:197][Close][tid:142536] [TsdClient][logicDeviceId_=7]has recv close hccp and computer process respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.300.934 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:142536] enter into CloseInHost deviceid[7] [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.300.945 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:142536] host cpu not support [INFO] TDT(142536,python3.7):2024-01-11-06:03:28.300.979 [process_mode_manager.cpp:208][Close][tid:142536] [TsdClient][deviceId=7] [sessionId=1] close hccp and computer process success [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:28.300.993 [atrace_api.c:93](tid:142536) AtraceDestroy start [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:28.301.006 [atrace_api.c:95](tid:142536) AtraceDestroy end F[INFO] TDT(142536,python3.7):2024-01-11-06:03:29.107.884 [process_mode_manager.cpp:109][OpenProcess][tid:143236] [ProcessModeManager] enter into open process deviceId[7] rankSize[0] [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.204 [process_mode_manager.cpp:705][GetDeviceCheckCode][tid:143236] [ProcessModeManager][deviceId=7] aicpu package already exist in device [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.248 [process_mode_manager.cpp:379][InitTsdClient][tid:143236] [TsdClient] deviceId[7] begin to init hdc client [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.395 [version_verify.cpp:34][SetVersionInfo][tid:143236] VersionVerify: send client version to server [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.413 [version_verify.cpp:50][SetVersionInfo][tid:143236] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.437 [version_verify.cpp:50][SetVersionInfo][tid:143236] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.883 [version_verify.cpp:66][PeerVersionCheck][tid:143236] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.900 [version_verify.cpp:87][ParseVersionInfo][tid:143236] VersionVerify: pass client version info success [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.909 [hdc_client.cpp:276][CheckHdcConnection][tid:143236] Service[2] create hdc success [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.922 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:143236] [TsdClient] tsd get process sign successfully, procpid[142536] signSize[48] [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.934 [version_verify.cpp:112][SpecialFeatureCheck][tid:143236] VersionVerify: previous type[6], supported [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.108.952 [process_mode_manager.cpp:126][OpenProcess][tid:143236] [ProcessModeManager] deviceId[7] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.436.270 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:143236] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.436.310 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:143236] enter into OpenInHost deviceid[7] [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.436.322 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:143236] host cpu not support [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.436.333 [process_mode_manager.cpp:156][OpenProcess][tid:143236] [TsdClient][deviceId=7] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(142536,python3.7):2024-01-11-06:03:29.438.938 [device.cc:340] 143236 Init: isDoubledie:0, topologytype:0 [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:29.452.634 [atrace_api.c:28](tid:143236) AtraceCreate start [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:29.452.733 [trace_rb_log.c:84](tid:143236) [RUNTIME_ATRACE_DEV7_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:29.452.750 [atrace_api.c:32](tid:143236) AtraceCreate end [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.452.763 [client_manager.cpp:157][SetProfilingCallback][tid:143236] [TsdClient] set profiling callback success [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.460.989 [process_mode_manager.cpp:184][Close][tid:143236] [TsdClient] Close [deviceId=7][sessionId=1] hccp and computer enter [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.461.020 [version_verify.cpp:112][SpecialFeatureCheck][tid:143236] VersionVerify: previous type[7], supported [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.461.058 [process_mode_manager.cpp:192][Close][tid:143236] [TsdClient][deviceId=7] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.485.037 [process_mode_manager.cpp:197][Close][tid:143236] [TsdClient][logicDeviceId_=7]has recv close hccp and computer process respond [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.485.052 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:143236] enter into CloseInHost deviceid[7] [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.485.063 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:143236] host cpu not support [INFO] TDT(142536,python3.7):2024-01-11-06:03:29.485.092 [process_mode_manager.cpp:208][Close][tid:143236] [TsdClient][deviceId=7] [sessionId=1] close hccp and computer process success [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:29.485.107 [atrace_api.c:93](tid:143236) AtraceDestroy start [INFO] ATRACE(142536,python3.7):2024-01-11-06:03:29.485.121 [atrace_api.c:95](tid:143236) AtraceDestroy end F =================================== FAILURES =================================== _________________________________ test_add[0] __________________________________ mode = 0 @pytest.mark.level1 @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard @pytest.mark.parametrize('mode', [context.GRAPH_MODE, context.PYNATIVE_MODE]) def test_add(mode): """ Feature: Test add op. Description: Test add. Expectation: the result match with expected result. """ context.set_context(mode=mode) inputs_np = get_np_data() func = ops.add input_args = get_inputs_tensor(inputs_np) # forward: > output = func(*input_args) test_add.py:58: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/auto_generate/gen_ops_def.py:4608: in add return add_op(x, y) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/auto_generate/gen_ops_def.py:192: in __call__ return super().__call__(x, y) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/primitive.py:324: in __call__ return _run_op(self, self.name, args) /home/jenkins/.local/lib/python3.7/site-packages/mindspore/ops/primitive.py:941: in _run_op stub = _pynative_executor.run_op_async(obj, op_name, args) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = args = (Prim[Add], 'Add', (Tensor(shape=[2, 4, 8, 16], dtype=Float16, value= [[[[ 1.3320e+00, 7.1533e-01, -1.5449e+00 ... 1..., 9.6143e-01, 2.4524e-01], [ 1.3350e+00, 2.2827e-01, -1.1499e-01 ... -1.3389e+00, -2.1797e+00, 2.0859e+00]]]]))) def run_op_async(self, *args): """ Run single op async. Args: args (tuple): Op prim and input arguments. Return: StubNode, result of run op. """ > return self._executor.run_op_async(*args) E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 7 Device HBM total size:34359738368 Device HBM free size:1701441536 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:1190: RuntimeError _________________________________ test_add[1] __________________________________ mode = 1 @pytest.mark.level1 @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard @pytest.mark.parametrize('mode', [context.GRAPH_MODE, context.PYNATIVE_MODE]) def test_add(mode): """ Feature: Test add op. Description: Test add. Expectation: the result match with expected result. """ context.set_context(mode=mode) inputs_np = get_np_data() func = ops.add input_args = get_inputs_tensor(inputs_np) # forward: output = func(*input_args) expect_output = np.add(*inputs_np) > assert np.allclose(output.asnumpy(), expect_output) test_add.py:60: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:48: in fun arg = (stub.stub_sync(),) + arg[1:] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[RuntimeError("Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'.\n\n---------...ice/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize\n") raised in repr()] StubTensor object at 0xffff20d4e710> def stub_sync(self): """sync real tensor.""" if self.stub: > val = self.stub.get_value() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 7 Device HBM total size:34359738368 Device HBM free size:1701408768 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/_stub_tensor.py:153: RuntimeError =========================== short test summary info ============================ FAILED test_add.py::test_add[0] - RuntimeError: Ascend kernel runtime initial... FAILED test_add.py::test_add[1] - RuntimeError: Ascend kernel runtime initial... ============================== 2 failed in 7.67s =============================== [INFO] RUNTIME(142536,python3.7):2024-01-11-06:03:30.532.037 [runtime.cc:1737] 142536 ~Runtime: deconstruct runtime.