============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/ops/ascend/test_aicpu_ops, inifile: /home/jenkins/sault/virtual_test/virtualenv_003/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:45.516.471 [trace_attr.c:105](tid:127340) platform is 1. [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:45.516.628 [trace_recorder.c:114](tid:127340) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:45.516.653 [trace_signal.c:133](tid:127340) register signal handler for signo 2 succeed. [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:45.516.666 [trace_signal.c:133](tid:127340) register signal handler for signo 15 succeed. [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:45.909.017 [runtime.cc:1159] 127340 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:45.909.052 [runtime.cc:4719] 127340 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 3 items test_gather_d.py [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.080.490 [process_mode_manager.cpp:109][OpenProcess][tid:127340] [ProcessModeManager] enter into open process deviceId[2] rankSize[0] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.081.378 [process_mode_manager.cpp:379][InitTsdClient][tid:127340] [TsdClient] deviceId[2] begin to init hdc client [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.081.498 [version_verify.cpp:34][SetVersionInfo][tid:127340] VersionVerify: send client version to server [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.081.522 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.081.535 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.004 [version_verify.cpp:66][PeerVersionCheck][tid:127340] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.019 [version_verify.cpp:87][ParseVersionInfo][tid:127340] VersionVerify: pass client version info success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.030 [hdc_client.cpp:276][CheckHdcConnection][tid:127340] Service[2] create hdc success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.042 [version_verify.cpp:120][SpecialFeatureCheck][tid:127340] VersionVerify: new type[35], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.080 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:127340] [TsdClient][deviceId=2] [sessionId=1] wait package info respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.205 [process_mode_manager.cpp:379][InitTsdClient][tid:127340] [TsdClient] deviceId[2] begin to init hdc client [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.284 [version_verify.cpp:34][SetVersionInfo][tid:127340] VersionVerify: send client version to server [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.298 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.309 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.589 [version_verify.cpp:66][PeerVersionCheck][tid:127340] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.603 [version_verify.cpp:87][ParseVersionInfo][tid:127340] VersionVerify: pass client version info success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.613 [hdc_client.cpp:276][CheckHdcConnection][tid:127340] Service[2] create hdc success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.623 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:127340] [TsdClient] tsd get process sign successfully, procpid[127340] signSize[48] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.642 [version_verify.cpp:112][SpecialFeatureCheck][tid:127340] VersionVerify: previous type[6], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.082.662 [process_mode_manager.cpp:126][OpenProcess][tid:127340] [ProcessModeManager] deviceId[2] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.317.492 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:127340] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.317.518 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:127340] enter into OpenInHost deviceid[2] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.317.528 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:127340] host cpu not support [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.317.537 [process_mode_manager.cpp:156][OpenProcess][tid:127340] [TsdClient][deviceId=2] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:50.320.175 [device.cc:340] 127340 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:50.334.896 [npu_driver.cc:5428] 128121 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:50.334.975 [atrace_api.c:28](tid:127340) AtraceCreate start [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:50.335.072 [trace_rb_log.c:84](tid:127340) [RUNTIME_ATRACE_DEV2_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:50.335.089 [atrace_api.c:32](tid:127340) AtraceCreate end [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.335.105 [client_manager.cpp:157][SetProfilingCallback][tid:127340] [TsdClient] set profiling callback success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.346.414 [process_mode_manager.cpp:184][Close][tid:127340] [TsdClient] Close [deviceId=2][sessionId=1] hccp and computer enter [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.346.443 [version_verify.cpp:112][SpecialFeatureCheck][tid:127340] VersionVerify: previous type[7], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.346.478 [process_mode_manager.cpp:192][Close][tid:127340] [TsdClient][deviceId=2] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.377.389 [process_mode_manager.cpp:197][Close][tid:127340] [TsdClient][logicDeviceId_=2]has recv close hccp and computer process respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.377.404 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:127340] enter into CloseInHost deviceid[2] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.377.416 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:127340] host cpu not support [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.377.448 [process_mode_manager.cpp:208][Close][tid:127340] [TsdClient][deviceId=2] [sessionId=1] close hccp and computer process success [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:50.377.461 [atrace_api.c:93](tid:127340) AtraceDestroy start [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:50.377.473 [atrace_api.c:95](tid:127340) AtraceDestroy end F[INFO] TDT(127340,python3.7):2024-01-11-06:00:50.945.396 [process_mode_manager.cpp:109][OpenProcess][tid:127340] [ProcessModeManager] enter into open process deviceId[2] rankSize[0] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.945.762 [process_mode_manager.cpp:705][GetDeviceCheckCode][tid:127340] [ProcessModeManager][deviceId=2] aicpu package already exist in device [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.945.801 [process_mode_manager.cpp:379][InitTsdClient][tid:127340] [TsdClient] deviceId[2] begin to init hdc client [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.946.751 [version_verify.cpp:34][SetVersionInfo][tid:127340] VersionVerify: send client version to server [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.946.769 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.946.791 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.947.033 [version_verify.cpp:66][PeerVersionCheck][tid:127340] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.947.049 [version_verify.cpp:87][ParseVersionInfo][tid:127340] VersionVerify: pass client version info success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.947.060 [hdc_client.cpp:276][CheckHdcConnection][tid:127340] Service[2] create hdc success [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.947.074 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:127340] [TsdClient] tsd get process sign successfully, procpid[127340] signSize[48] [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.947.087 [version_verify.cpp:112][SpecialFeatureCheck][tid:127340] VersionVerify: previous type[6], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:50.947.109 [process_mode_manager.cpp:126][OpenProcess][tid:127340] [ProcessModeManager] deviceId[2] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.195.407 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:127340] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.195.433 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:127340] enter into OpenInHost deviceid[2] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.195.444 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:127340] host cpu not support [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.195.453 [process_mode_manager.cpp:156][OpenProcess][tid:127340] [TsdClient][deviceId=2] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:51.198.029 [device.cc:340] 127340 Init: isDoubledie:0, topologytype:0 [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.210.754 [atrace_api.c:28](tid:127340) AtraceCreate start [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.210.791 [trace_rb_log.c:84](tid:127340) [RUNTIME_ATRACE_DEV2_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.210.804 [atrace_api.c:32](tid:127340) AtraceCreate end [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.210.816 [client_manager.cpp:157][SetProfilingCallback][tid:127340] [TsdClient] set profiling callback success [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.220.051 [process_mode_manager.cpp:184][Close][tid:127340] [TsdClient] Close [deviceId=2][sessionId=1] hccp and computer enter [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.220.078 [version_verify.cpp:112][SpecialFeatureCheck][tid:127340] VersionVerify: previous type[7], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.220.114 [process_mode_manager.cpp:192][Close][tid:127340] [TsdClient][deviceId=2] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.241.018 [process_mode_manager.cpp:197][Close][tid:127340] [TsdClient][logicDeviceId_=2]has recv close hccp and computer process respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.241.034 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:127340] enter into CloseInHost deviceid[2] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.241.045 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:127340] host cpu not support [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.241.079 [process_mode_manager.cpp:208][Close][tid:127340] [TsdClient][deviceId=2] [sessionId=1] close hccp and computer process success [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.241.091 [atrace_api.c:93](tid:127340) AtraceDestroy start [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.241.104 [atrace_api.c:95](tid:127340) AtraceDestroy end F[INFO] TDT(127340,python3.7):2024-01-11-06:00:51.424.603 [process_mode_manager.cpp:109][OpenProcess][tid:127340] [ProcessModeManager] enter into open process deviceId[2] rankSize[0] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.017 [process_mode_manager.cpp:705][GetDeviceCheckCode][tid:127340] [ProcessModeManager][deviceId=2] aicpu package already exist in device [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.057 [process_mode_manager.cpp:379][InitTsdClient][tid:127340] [TsdClient] deviceId[2] begin to init hdc client [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.153 [version_verify.cpp:34][SetVersionInfo][tid:127340] VersionVerify: send client version to server [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.168 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.180 [version_verify.cpp:50][SetVersionInfo][tid:127340] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.463 [version_verify.cpp:66][PeerVersionCheck][tid:127340] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.478 [version_verify.cpp:87][ParseVersionInfo][tid:127340] VersionVerify: pass client version info success [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.487 [hdc_client.cpp:276][CheckHdcConnection][tid:127340] Service[2] create hdc success [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.500 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:127340] [TsdClient] tsd get process sign successfully, procpid[127340] signSize[48] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.512 [version_verify.cpp:112][SpecialFeatureCheck][tid:127340] VersionVerify: previous type[6], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.425.532 [process_mode_manager.cpp:126][OpenProcess][tid:127340] [ProcessModeManager] deviceId[2] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.712.413 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:127340] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.712.450 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:127340] enter into OpenInHost deviceid[2] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.712.462 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:127340] host cpu not support [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.712.472 [process_mode_manager.cpp:156][OpenProcess][tid:127340] [TsdClient][deviceId=2] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:51.715.088 [device.cc:340] 127340 Init: isDoubledie:0, topologytype:0 [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.727.833 [atrace_api.c:28](tid:127340) AtraceCreate start [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.727.871 [trace_rb_log.c:84](tid:127340) [RUNTIME_ATRACE_DEV2_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.727.884 [atrace_api.c:32](tid:127340) AtraceCreate end [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.727.895 [client_manager.cpp:157][SetProfilingCallback][tid:127340] [TsdClient] set profiling callback success [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.737.747 [process_mode_manager.cpp:184][Close][tid:127340] [TsdClient] Close [deviceId=2][sessionId=1] hccp and computer enter [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.737.778 [version_verify.cpp:112][SpecialFeatureCheck][tid:127340] VersionVerify: previous type[7], supported [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.737.816 [process_mode_manager.cpp:192][Close][tid:127340] [TsdClient][deviceId=2] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.758.775 [process_mode_manager.cpp:197][Close][tid:127340] [TsdClient][logicDeviceId_=2]has recv close hccp and computer process respond [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.758.802 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:127340] enter into CloseInHost deviceid[2] [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.758.814 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:127340] host cpu not support [INFO] TDT(127340,python3.7):2024-01-11-06:00:51.758.847 [process_mode_manager.cpp:208][Close][tid:127340] [TsdClient][deviceId=2] [sessionId=1] close hccp and computer process success [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.758.860 [atrace_api.c:93](tid:127340) AtraceDestroy start [INFO] ATRACE(127340,python3.7):2024-01-11-06:00:51.758.874 [atrace_api.c:95](tid:127340) AtraceDestroy end F =================================== FAILURES =================================== ______________________________ test_net[ms_type0] ______________________________ ms_type = mindspore.int32 @pytest.mark.level1 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard @pytest.mark.parametrize('ms_type', [mindspore.int32, mindspore.uint32, mindspore.float32]) def test_net(ms_type): """ Feature: test GatherD static shape. Description: input x and index is static shape. Expectation: the result match with numpy result """ context.set_context(mode=context.GRAPH_MODE) x, dim, index, expect = get_data(ms_type) > net = Net(dim) test_gather_d.py:94: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_gather_d.py:34: in __init__ super(Net, self).__init__() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = Net<>, auto_prefix = True, flags = None def __init__(self, auto_prefix=True, flags=None): Cell_.__init__(self, self._cell_tag) self._params = OrderedDict() self._cells = OrderedDict() self._params_list = OrderedDict() self._tensor_list = OrderedDict() self._primitives = OrderedDict() self.training = False self.requires_grad = False self.pynative = False self._attr_synced = False self._param_prefix = '' self._auto_prefix = auto_prefix self._scope = None self._phase = 'train' self._parameter_layout_dict = {} self._parallel_parameter_name_list = () self._parallel_parameter_merge_net_dict = {} self._create_time = int(time.time() * 1e9) self.arguments_key = "" self.compile_cache = set() cells_compile_cache[id(self)] = self.compile_cache self.parameter_broadcast_done = False self._id = 1 self.exist_names = set("") self.exist_objs = set() > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 2 Device HBM total size:34359738368 Device HBM free size:1602793472 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py:134: RuntimeError ______________________________ test_net[ms_type1] ______________________________ ms_type = mindspore.uint32 @pytest.mark.level1 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard @pytest.mark.parametrize('ms_type', [mindspore.int32, mindspore.uint32, mindspore.float32]) def test_net(ms_type): """ Feature: test GatherD static shape. Description: input x and index is static shape. Expectation: the result match with numpy result """ context.set_context(mode=context.GRAPH_MODE) x, dim, index, expect = get_data(ms_type) > net = Net(dim) test_gather_d.py:94: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_gather_d.py:34: in __init__ super(Net, self).__init__() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = Net<>, auto_prefix = True, flags = None def __init__(self, auto_prefix=True, flags=None): Cell_.__init__(self, self._cell_tag) self._params = OrderedDict() self._cells = OrderedDict() self._params_list = OrderedDict() self._tensor_list = OrderedDict() self._primitives = OrderedDict() self.training = False self.requires_grad = False self.pynative = False self._attr_synced = False self._param_prefix = '' self._auto_prefix = auto_prefix self._scope = None self._phase = 'train' self._parameter_layout_dict = {} self._parallel_parameter_name_list = () self._parallel_parameter_merge_net_dict = {} self._create_time = int(time.time() * 1e9) self.arguments_key = "" self.compile_cache = set() cells_compile_cache[id(self)] = self.compile_cache self.parameter_broadcast_done = False self._id = 1 self.exist_names = set("") self.exist_objs = set() > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 2 Device HBM total size:34359738368 Device HBM free size:1603018752 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py:134: RuntimeError ______________________________ test_net[ms_type2] ______________________________ ms_type = mindspore.float32 @pytest.mark.level1 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard @pytest.mark.parametrize('ms_type', [mindspore.int32, mindspore.uint32, mindspore.float32]) def test_net(ms_type): """ Feature: test GatherD static shape. Description: input x and index is static shape. Expectation: the result match with numpy result """ context.set_context(mode=context.GRAPH_MODE) x, dim, index, expect = get_data(ms_type) > net = Net(dim) test_gather_d.py:94: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_gather_d.py:34: in __init__ super(Net, self).__init__() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = Net<>, auto_prefix = True, flags = None def __init__(self, auto_prefix=True, flags=None): Cell_.__init__(self, self._cell_tag) self._params = OrderedDict() self._cells = OrderedDict() self._params_list = OrderedDict() self._tensor_list = OrderedDict() self._primitives = OrderedDict() self.training = False self.requires_grad = False self.pynative = False self._attr_synced = False self._param_prefix = '' self._auto_prefix = auto_prefix self._scope = None self._phase = 'train' self._parameter_layout_dict = {} self._parallel_parameter_name_list = () self._parallel_parameter_merge_net_dict = {} self._create_time = int(time.time() * 1e9) self.arguments_key = "" self.compile_cache = set() cells_compile_cache[id(self)] = self.compile_cache self.parameter_broadcast_done = False self._id = 1 self.exist_names = set("") self.exist_objs = set() > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 2 Device HBM total size:34359738368 Device HBM free size:1602985984 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py:134: RuntimeError =========================== short test summary info ============================ FAILED test_gather_d.py::test_net[ms_type0] - RuntimeError: Ascend kernel run... FAILED test_gather_d.py::test_net[ms_type1] - RuntimeError: Ascend kernel run... FAILED test_gather_d.py::test_net[ms_type2] - RuntimeError: Ascend kernel run... ============================== 3 failed in 7.88s =============================== [INFO] RUNTIME(127340,python3.7):2024-01-11-06:00:52.975.497 [runtime.cc:1737] 127340 ~Runtime: deconstruct runtime.