============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/nn, inifile: /home/jenkins/sault/virtual_test/virtualenv_006/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(112742,python3.7):2024-01-11-05:57:58.542.119 [trace_attr.c:105](tid:112742) platform is 1. [INFO] ATRACE(112742,python3.7):2024-01-11-05:57:58.542.288 [trace_recorder.c:114](tid:112742) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(112742,python3.7):2024-01-11-05:57:58.542.318 [trace_signal.c:133](tid:112742) register signal handler for signo 2 succeed. [INFO] ATRACE(112742,python3.7):2024-01-11-05:57:58.542.332 [trace_signal.c:133](tid:112742) register signal handler for signo 15 succeed. [INFO] RUNTIME(112742,python3.7):2024-01-11-05:57:58.950.036 [runtime.cc:1159] 112742 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(112742,python3.7):2024-01-11-05:57:58.950.086 [runtime.cc:4719] 112742 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 1 item test_greater.py [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.437.939 [process_mode_manager.cpp:109][OpenProcess][tid:112742] [ProcessModeManager] enter into open process deviceId[5] rankSize[0] [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.439.627 [process_mode_manager.cpp:379][InitTsdClient][tid:112742] [TsdClient] deviceId[5] begin to init hdc client [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.439.782 [version_verify.cpp:34][SetVersionInfo][tid:112742] VersionVerify: send client version to server [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.439.811 [version_verify.cpp:50][SetVersionInfo][tid:112742] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.439.825 [version_verify.cpp:50][SetVersionInfo][tid:112742] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.118 [version_verify.cpp:66][PeerVersionCheck][tid:112742] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.136 [version_verify.cpp:87][ParseVersionInfo][tid:112742] VersionVerify: pass client version info success [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.148 [hdc_client.cpp:276][CheckHdcConnection][tid:112742] Service[2] create hdc success [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.166 [version_verify.cpp:120][SpecialFeatureCheck][tid:112742] VersionVerify: new type[35], supported [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.210 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:112742] [TsdClient][deviceId=5] [sessionId=1] wait package info respond [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.335 [process_mode_manager.cpp:379][InitTsdClient][tid:112742] [TsdClient] deviceId[5] begin to init hdc client [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.466 [version_verify.cpp:34][SetVersionInfo][tid:112742] VersionVerify: send client version to server [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.481 [version_verify.cpp:50][SetVersionInfo][tid:112742] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.495 [version_verify.cpp:50][SetVersionInfo][tid:112742] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.636 [version_verify.cpp:66][PeerVersionCheck][tid:112742] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.650 [version_verify.cpp:87][ParseVersionInfo][tid:112742] VersionVerify: pass client version info success [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.661 [hdc_client.cpp:276][CheckHdcConnection][tid:112742] Service[2] create hdc success [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.672 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:112742] [TsdClient] tsd get process sign successfully, procpid[112742] signSize[48] [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.699 [version_verify.cpp:112][SpecialFeatureCheck][tid:112742] VersionVerify: previous type[6], supported [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.440.719 [process_mode_manager.cpp:126][OpenProcess][tid:112742] [ProcessModeManager] deviceId[5] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.724.465 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:112742] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.724.500 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:112742] enter into OpenInHost deviceid[5] [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.724.513 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:112742] host cpu not support [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.724.524 [process_mode_manager.cpp:156][OpenProcess][tid:112742] [TsdClient][deviceId=5] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(112742,python3.7):2024-01-11-05:58:03.727.316 [device.cc:340] 112742 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(112742,python3.7):2024-01-11-05:58:03.741.134 [npu_driver.cc:5428] 113416 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(112742,python3.7):2024-01-11-05:58:03.742.108 [atrace_api.c:28](tid:112742) AtraceCreate start [INFO] ATRACE(112742,python3.7):2024-01-11-05:58:03.742.199 [trace_rb_log.c:84](tid:112742) [RUNTIME_ATRACE_DEV5_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(112742,python3.7):2024-01-11-05:58:03.742.216 [atrace_api.c:32](tid:112742) AtraceCreate end [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.742.233 [client_manager.cpp:157][SetProfilingCallback][tid:112742] [TsdClient] set profiling callback success [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.754.210 [process_mode_manager.cpp:184][Close][tid:112742] [TsdClient] Close [deviceId=5][sessionId=1] hccp and computer enter [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.754.247 [version_verify.cpp:112][SpecialFeatureCheck][tid:112742] VersionVerify: previous type[7], supported [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.754.299 [process_mode_manager.cpp:192][Close][tid:112742] [TsdClient][deviceId=5] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.775.141 [process_mode_manager.cpp:197][Close][tid:112742] [TsdClient][logicDeviceId_=5]has recv close hccp and computer process respond [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.775.161 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:112742] enter into CloseInHost deviceid[5] [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.775.174 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:112742] host cpu not support [INFO] TDT(112742,python3.7):2024-01-11-05:58:03.775.210 [process_mode_manager.cpp:208][Close][tid:112742] [TsdClient][deviceId=5] [sessionId=1] close hccp and computer process success [INFO] ATRACE(112742,python3.7):2024-01-11-05:58:03.775.225 [atrace_api.c:93](tid:112742) AtraceDestroy start [INFO] ATRACE(112742,python3.7):2024-01-11-05:58:03.775.242 [atrace_api.c:95](tid:112742) AtraceDestroy end F =================================== FAILURES =================================== ____________________________ test_greater_backward _____________________________ @pytest.mark.level1 @pytest.mark.env_onecard @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training def test_greater_backward(): """ Feature: Auto grad. Description: test auto grad of op greater. Expectation: expect correct result. """ x = ms.Tensor(np.array([1, 2, 3]), ms.int32) y = ms.Tensor(np.array([1, 1, 4]), ms.int32) expect_out = np.array([0, 0, 0]) > grads = greater_backward_func(x, y) test_greater.py:65: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:722: in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = fn = ms_create_time = 1704923883417181696, input_signature = None, obj = None jit_config = None def __init__(self, fn, ms_create_time, input_signature=None, obj=None, jit_config=None): > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 5 Device HBM total size:34359738368 Device HBM free size:1701068800 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:319: RuntimeError =========================== short test summary info ============================ FAILED test_greater.py::test_greater_backward - RuntimeError: Ascend kernel r... ============================== 1 failed in 7.22s =============================== [INFO] RUNTIME(112742,python3.7):2024-01-11-05:58:05.417.064 [runtime.cc:1737] 112742 ~Runtime: deconstruct runtime.