============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/dyn_shape_dev, inifile: /home/jenkins/sault/virtual_test/virtualenv_001/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:39.270.531 [trace_attr.c:105](tid:111033) platform is 1. [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:39.270.691 [trace_recorder.c:114](tid:111033) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:39.270.721 [trace_signal.c:133](tid:111033) register signal handler for signo 2 succeed. [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:39.270.736 [trace_signal.c:133](tid:111033) register signal handler for signo 15 succeed. [INFO] RUNTIME(111033,python3.7):2024-01-11-05:57:39.684.244 [runtime.cc:1159] 111033 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(111033,python3.7):2024-01-11-05:57:39.684.301 [runtime.cc:4719] 111033 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 2 items test_hshrink.py [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.048.217 [process_mode_manager.cpp:109][OpenProcess][tid:111033] [ProcessModeManager] enter into open process deviceId[0] rankSize[0] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.071 [process_mode_manager.cpp:379][InitTsdClient][tid:111033] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.228 [version_verify.cpp:34][SetVersionInfo][tid:111033] VersionVerify: send client version to server [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.259 [version_verify.cpp:50][SetVersionInfo][tid:111033] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.274 [version_verify.cpp:50][SetVersionInfo][tid:111033] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.701 [version_verify.cpp:66][PeerVersionCheck][tid:111033] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.720 [version_verify.cpp:87][ParseVersionInfo][tid:111033] VersionVerify: pass client version info success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.731 [hdc_client.cpp:276][CheckHdcConnection][tid:111033] Service[2] create hdc success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.746 [version_verify.cpp:120][SpecialFeatureCheck][tid:111033] VersionVerify: new type[35], supported [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.791 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:111033] [TsdClient][deviceId=0] [sessionId=1] wait package info respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.049.920 [process_mode_manager.cpp:379][InitTsdClient][tid:111033] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.071 [version_verify.cpp:34][SetVersionInfo][tid:111033] VersionVerify: send client version to server [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.085 [version_verify.cpp:50][SetVersionInfo][tid:111033] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.098 [version_verify.cpp:50][SetVersionInfo][tid:111033] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.283 [version_verify.cpp:66][PeerVersionCheck][tid:111033] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.299 [version_verify.cpp:87][ParseVersionInfo][tid:111033] VersionVerify: pass client version info success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.311 [hdc_client.cpp:276][CheckHdcConnection][tid:111033] Service[2] create hdc success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.323 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:111033] [TsdClient] tsd get process sign successfully, procpid[111033] signSize[48] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.352 [version_verify.cpp:112][SpecialFeatureCheck][tid:111033] VersionVerify: previous type[6], supported [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.050.371 [process_mode_manager.cpp:126][OpenProcess][tid:111033] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.319.409 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:111033] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.319.442 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:111033] enter into OpenInHost deviceid[0] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.319.455 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:111033] host cpu not support [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.319.466 [process_mode_manager.cpp:156][OpenProcess][tid:111033] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(111033,python3.7):2024-01-11-05:57:44.322.195 [device.cc:340] 111033 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(111033,python3.7):2024-01-11-05:57:44.338.106 [npu_driver.cc:5428] 111514 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:44.338.178 [atrace_api.c:28](tid:111033) AtraceCreate start [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:44.338.275 [trace_rb_log.c:84](tid:111033) [RUNTIME_ATRACE_DEV0_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:44.338.294 [atrace_api.c:32](tid:111033) AtraceCreate end [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.338.312 [client_manager.cpp:157][SetProfilingCallback][tid:111033] [TsdClient] set profiling callback success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.352.609 [process_mode_manager.cpp:184][Close][tid:111033] [TsdClient] Close [deviceId=0][sessionId=1] hccp and computer enter [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.352.647 [version_verify.cpp:112][SpecialFeatureCheck][tid:111033] VersionVerify: previous type[7], supported [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.352.687 [process_mode_manager.cpp:192][Close][tid:111033] [TsdClient][deviceId=0] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.373.443 [process_mode_manager.cpp:197][Close][tid:111033] [TsdClient][logicDeviceId_=0]has recv close hccp and computer process respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.373.461 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:111033] enter into CloseInHost deviceid[0] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.373.473 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:111033] host cpu not support [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.373.533 [process_mode_manager.cpp:208][Close][tid:111033] [TsdClient][deviceId=0] [sessionId=1] close hccp and computer process success [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:44.373.548 [atrace_api.c:93](tid:111033) AtraceDestroy start [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:44.373.562 [atrace_api.c:95](tid:111033) AtraceDestroy end F[INFO] TDT(111033,python3.7):2024-01-11-05:57:44.959.380 [process_mode_manager.cpp:109][OpenProcess][tid:111033] [ProcessModeManager] enter into open process deviceId[0] rankSize[0] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.959.764 [process_mode_manager.cpp:705][GetDeviceCheckCode][tid:111033] [ProcessModeManager][deviceId=0] aicpu package already exist in device [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.959.816 [process_mode_manager.cpp:379][InitTsdClient][tid:111033] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.959.969 [version_verify.cpp:34][SetVersionInfo][tid:111033] VersionVerify: send client version to server [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.959.990 [version_verify.cpp:50][SetVersionInfo][tid:111033] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.016 [version_verify.cpp:50][SetVersionInfo][tid:111033] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.350 [version_verify.cpp:66][PeerVersionCheck][tid:111033] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.367 [version_verify.cpp:87][ParseVersionInfo][tid:111033] VersionVerify: pass client version info success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.378 [hdc_client.cpp:276][CheckHdcConnection][tid:111033] Service[2] create hdc success [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.394 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:111033] [TsdClient] tsd get process sign successfully, procpid[111033] signSize[48] [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.408 [version_verify.cpp:112][SpecialFeatureCheck][tid:111033] VersionVerify: previous type[6], supported [INFO] TDT(111033,python3.7):2024-01-11-05:57:44.960.430 [process_mode_manager.cpp:126][OpenProcess][tid:111033] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.246.029 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:111033] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.246.058 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:111033] enter into OpenInHost deviceid[0] [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.246.072 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:111033] host cpu not support [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.246.084 [process_mode_manager.cpp:156][OpenProcess][tid:111033] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(111033,python3.7):2024-01-11-05:57:45.248.770 [device.cc:340] 111033 Init: isDoubledie:0, topologytype:0 [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:45.259.493 [atrace_api.c:28](tid:111033) AtraceCreate start [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:45.259.546 [trace_rb_log.c:84](tid:111033) [RUNTIME_ATRACE_DEV0_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:45.259.561 [atrace_api.c:32](tid:111033) AtraceCreate end [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.259.573 [client_manager.cpp:157][SetProfilingCallback][tid:111033] [TsdClient] set profiling callback success [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.268.734 [process_mode_manager.cpp:184][Close][tid:111033] [TsdClient] Close [deviceId=0][sessionId=1] hccp and computer enter [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.268.771 [version_verify.cpp:112][SpecialFeatureCheck][tid:111033] VersionVerify: previous type[7], supported [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.268.807 [process_mode_manager.cpp:192][Close][tid:111033] [TsdClient][deviceId=0] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.299.768 [process_mode_manager.cpp:197][Close][tid:111033] [TsdClient][logicDeviceId_=0]has recv close hccp and computer process respond [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.299.786 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:111033] enter into CloseInHost deviceid[0] [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.299.797 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:111033] host cpu not support [INFO] TDT(111033,python3.7):2024-01-11-05:57:45.299.826 [process_mode_manager.cpp:208][Close][tid:111033] [TsdClient][deviceId=0] [sessionId=1] close hccp and computer process success [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:45.299.840 [atrace_api.c:93](tid:111033) AtraceDestroy start [INFO] ATRACE(111033,python3.7):2024-01-11-05:57:45.299.855 [atrace_api.c:95](tid:111033) AtraceDestroy end F =================================== FAILURES =================================== ___________________________ test_hshrink_backward[0] ___________________________ context_mode = 0 @pytest.mark.level1 @pytest.mark.env_onecard @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training @pytest.mark.parametrize("context_mode", [ms.GRAPH_MODE, ms.PYNATIVE_MODE]) def test_hshrink_backward(context_mode): """ Feature: Auto grad. Description: test auto grad of op hshrink. Expectation: expect correct result. """ ms.context.set_context(mode=context_mode) np_array = np.array( [[0.5, 1, 2.0], [0.0533, 0.0776, -2.1233]]).astype('float32') input_x = Tensor(np_array) lambd = 0.5 > grads = hshrink_backward_func(input_x, lambd) test_hshrink.py:79: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_utils.py:42: in wrapper cell_obj = Net(fn) test_utils.py:29: in __init__ super().__init__() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = Net<>, auto_prefix = True, flags = None def __init__(self, auto_prefix=True, flags=None): Cell_.__init__(self, self._cell_tag) self._params = OrderedDict() self._cells = OrderedDict() self._params_list = OrderedDict() self._tensor_list = OrderedDict() self._primitives = OrderedDict() self.training = False self.requires_grad = False self.pynative = False self._attr_synced = False self._param_prefix = '' self._auto_prefix = auto_prefix self._scope = None self._phase = 'train' self._parameter_layout_dict = {} self._parallel_parameter_name_list = () self._parallel_parameter_merge_net_dict = {} self._create_time = int(time.time() * 1e9) self.arguments_key = "" self.compile_cache = set() cells_compile_cache[id(self)] = self.compile_cache self.parameter_broadcast_done = False self._id = 1 self.exist_names = set("") self.exist_objs = set() > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 0 Device HBM total size:34359738368 Device HBM free size:1603534848 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py:134: RuntimeError ___________________________ test_hshrink_backward[1] ___________________________ context_mode = 1 @pytest.mark.level1 @pytest.mark.env_onecard @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training @pytest.mark.parametrize("context_mode", [ms.GRAPH_MODE, ms.PYNATIVE_MODE]) def test_hshrink_backward(context_mode): """ Feature: Auto grad. Description: test auto grad of op hshrink. Expectation: expect correct result. """ ms.context.set_context(mode=context_mode) np_array = np.array( [[0.5, 1, 2.0], [0.0533, 0.0776, -2.1233]]).astype('float32') input_x = Tensor(np_array) lambd = 0.5 > grads = hshrink_backward_func(input_x, lambd) test_hshrink.py:79: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_utils.py:42: in wrapper cell_obj = Net(fn) test_utils.py:29: in __init__ super().__init__() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = Net<>, auto_prefix = True, flags = None def __init__(self, auto_prefix=True, flags=None): Cell_.__init__(self, self._cell_tag) self._params = OrderedDict() self._cells = OrderedDict() self._params_list = OrderedDict() self._tensor_list = OrderedDict() self._primitives = OrderedDict() self.training = False self.requires_grad = False self.pynative = False self._attr_synced = False self._param_prefix = '' self._auto_prefix = auto_prefix self._scope = None self._phase = 'train' self._parameter_layout_dict = {} self._parallel_parameter_name_list = () self._parallel_parameter_merge_net_dict = {} self._create_time = int(time.time() * 1e9) self.arguments_key = "" self.compile_cache = set() cells_compile_cache[id(self)] = self.compile_cache self.parameter_broadcast_done = False self._id = 1 self.exist_names = set("") self.exist_objs = set() > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 0 Device HBM total size:34359738368 Device HBM free size:1602985984 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py:134: RuntimeError =========================== short test summary info ============================ FAILED test_hshrink.py::test_hshrink_backward[0] - RuntimeError: Ascend kerne... FAILED test_hshrink.py::test_hshrink_backward[1] - RuntimeError: Ascend kerne... ============================== 2 failed in 7.64s =============================== [INFO] RUNTIME(111033,python3.7):2024-01-11-05:57:46.353.010 [runtime.cc:1737] 111033 ~Runtime: deconstruct runtime.