============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.11.0, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/dyn_shape_dev, inifile: /home/jenkins/sault/virtual_test/virtualenv_002/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:25.952.418 [trace_attr.c:105](tid:137111) platform is 1. [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:25.952.574 [trace_recorder.c:114](tid:137111) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:25.952.606 [trace_signal.c:133](tid:137111) register signal handler for signo 2 succeed. [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:25.952.619 [trace_signal.c:133](tid:137111) register signal handler for signo 15 succeed. [INFO] RUNTIME(137111,python3.7):2024-01-11-06:02:26.365.712 [runtime.cc:1159] 137111 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(137111,python3.7):2024-01-11-06:02:26.365.770 [runtime.cc:4719] 137111 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 2 items test_round.py [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.740.046 [process_mode_manager.cpp:109][OpenProcess][tid:137111] [ProcessModeManager] enter into open process deviceId[1] rankSize[0] [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.209 [process_mode_manager.cpp:379][InitTsdClient][tid:137111] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.354 [version_verify.cpp:34][SetVersionInfo][tid:137111] VersionVerify: send client version to server [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.386 [version_verify.cpp:50][SetVersionInfo][tid:137111] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.401 [version_verify.cpp:50][SetVersionInfo][tid:137111] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.689 [version_verify.cpp:66][PeerVersionCheck][tid:137111] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.709 [version_verify.cpp:87][ParseVersionInfo][tid:137111] VersionVerify: pass client version info success [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.721 [hdc_client.cpp:276][CheckHdcConnection][tid:137111] Service[2] create hdc success [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.737 [version_verify.cpp:120][SpecialFeatureCheck][tid:137111] VersionVerify: new type[35], supported [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.785 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:137111] [TsdClient][deviceId=1] [sessionId=1] wait package info respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.741.940 [process_mode_manager.cpp:379][InitTsdClient][tid:137111] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.139 [version_verify.cpp:34][SetVersionInfo][tid:137111] VersionVerify: send client version to server [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.154 [version_verify.cpp:50][SetVersionInfo][tid:137111] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.168 [version_verify.cpp:50][SetVersionInfo][tid:137111] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.332 [version_verify.cpp:66][PeerVersionCheck][tid:137111] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.347 [version_verify.cpp:87][ParseVersionInfo][tid:137111] VersionVerify: pass client version info success [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.358 [hdc_client.cpp:276][CheckHdcConnection][tid:137111] Service[2] create hdc success [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.372 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:137111] [TsdClient] tsd get process sign successfully, procpid[137111] signSize[48] [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.403 [version_verify.cpp:112][SpecialFeatureCheck][tid:137111] VersionVerify: previous type[6], supported [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.742.428 [process_mode_manager.cpp:126][OpenProcess][tid:137111] [ProcessModeManager] deviceId[1] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.975.905 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:137111] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.975.937 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:137111] enter into OpenInHost deviceid[1] [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.975.951 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:137111] host cpu not support [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.975.961 [process_mode_manager.cpp:156][OpenProcess][tid:137111] [TsdClient][deviceId=1] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(137111,python3.7):2024-01-11-06:02:30.978.700 [device.cc:340] 137111 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(137111,python3.7):2024-01-11-06:02:30.997.487 [npu_driver.cc:5428] 137676 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:30.997.559 [atrace_api.c:28](tid:137111) AtraceCreate start [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:30.997.659 [trace_rb_log.c:84](tid:137111) [RUNTIME_ATRACE_DEV1_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:30.997.677 [atrace_api.c:32](tid:137111) AtraceCreate end [INFO] TDT(137111,python3.7):2024-01-11-06:02:30.997.695 [client_manager.cpp:157][SetProfilingCallback][tid:137111] [TsdClient] set profiling callback success [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.009.585 [process_mode_manager.cpp:184][Close][tid:137111] [TsdClient] Close [deviceId=1][sessionId=1] hccp and computer enter [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.009.621 [version_verify.cpp:112][SpecialFeatureCheck][tid:137111] VersionVerify: previous type[7], supported [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.009.669 [process_mode_manager.cpp:192][Close][tid:137111] [TsdClient][deviceId=1] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.030.627 [process_mode_manager.cpp:197][Close][tid:137111] [TsdClient][logicDeviceId_=1]has recv close hccp and computer process respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.030.649 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:137111] enter into CloseInHost deviceid[1] [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.030.662 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:137111] host cpu not support [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.030.703 [process_mode_manager.cpp:208][Close][tid:137111] [TsdClient][deviceId=1] [sessionId=1] close hccp and computer process success [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.030.718 [atrace_api.c:93](tid:137111) AtraceDestroy start [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.030.738 [atrace_api.c:95](tid:137111) AtraceDestroy end F[INFO] TDT(137111,python3.7):2024-01-11-06:02:31.599.506 [process_mode_manager.cpp:109][OpenProcess][tid:137111] [ProcessModeManager] enter into open process deviceId[1] rankSize[0] [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.599.900 [process_mode_manager.cpp:705][GetDeviceCheckCode][tid:137111] [ProcessModeManager][deviceId=1] aicpu package already exist in device [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.599.948 [process_mode_manager.cpp:379][InitTsdClient][tid:137111] [TsdClient] deviceId[1] begin to init hdc client [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.065 [version_verify.cpp:34][SetVersionInfo][tid:137111] VersionVerify: send client version to server [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.088 [version_verify.cpp:50][SetVersionInfo][tid:137111] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.113 [version_verify.cpp:50][SetVersionInfo][tid:137111] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.379 [version_verify.cpp:66][PeerVersionCheck][tid:137111] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.396 [version_verify.cpp:87][ParseVersionInfo][tid:137111] VersionVerify: pass client version info success [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.407 [hdc_client.cpp:276][CheckHdcConnection][tid:137111] Service[2] create hdc success [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.423 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:137111] [TsdClient] tsd get process sign successfully, procpid[137111] signSize[48] [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.438 [version_verify.cpp:112][SpecialFeatureCheck][tid:137111] VersionVerify: previous type[6], supported [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.600.462 [process_mode_manager.cpp:126][OpenProcess][tid:137111] [ProcessModeManager] deviceId[1] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.823.417 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:137111] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.823.446 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:137111] enter into OpenInHost deviceid[1] [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.823.460 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:137111] host cpu not support [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.823.473 [process_mode_manager.cpp:156][OpenProcess][tid:137111] [TsdClient][deviceId=1] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(137111,python3.7):2024-01-11-06:02:31.826.141 [device.cc:340] 137111 Init: isDoubledie:0, topologytype:0 [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.839.737 [atrace_api.c:28](tid:137111) AtraceCreate start [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.839.781 [trace_rb_log.c:84](tid:137111) [RUNTIME_ATRACE_DEV1_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.839.797 [atrace_api.c:32](tid:137111) AtraceCreate end [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.839.811 [client_manager.cpp:157][SetProfilingCallback][tid:137111] [TsdClient] set profiling callback success [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.848.795 [process_mode_manager.cpp:184][Close][tid:137111] [TsdClient] Close [deviceId=1][sessionId=1] hccp and computer enter [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.848.825 [version_verify.cpp:112][SpecialFeatureCheck][tid:137111] VersionVerify: previous type[7], supported [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.848.928 [process_mode_manager.cpp:192][Close][tid:137111] [TsdClient][deviceId=1] [sessionId=1] wait hccp and computer process close respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.869.880 [process_mode_manager.cpp:197][Close][tid:137111] [TsdClient][logicDeviceId_=1]has recv close hccp and computer process respond [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.869.898 [stub_process_mode_nowin.cpp:151][CloseInHost][tid:137111] enter into CloseInHost deviceid[1] [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.869.911 [stub_process_mode_nowin.cpp:154][CloseInHost][tid:137111] host cpu not support [INFO] TDT(137111,python3.7):2024-01-11-06:02:31.869.949 [process_mode_manager.cpp:208][Close][tid:137111] [TsdClient][deviceId=1] [sessionId=1] close hccp and computer process success [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.869.962 [atrace_api.c:93](tid:137111) AtraceDestroy start [INFO] ATRACE(137111,python3.7):2024-01-11-06:02:31.869.983 [atrace_api.c:95](tid:137111) AtraceDestroy end F =================================== FAILURES =================================== ______________________________ test_round_vmap[0] ______________________________ mode = 0 @pytest.mark.level1 @pytest.mark.env_onecard @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training @pytest.mark.parametrize('mode', [ms.GRAPH_MODE, ms.PYNATIVE_MODE]) @test_utils.run_test_func def test_round_vmap(mode): """ Feature: test vmap function. Description: test round op vmap. Expectation: expect correct result. """ ms.context.set_context(mode=mode) axes = -1 x = ms.Tensor(np.random.uniform(low=-255, high=255, size=(4, 3, 2)).astype(np.float32)) net_vmap = ops.vmap(ops.vmap(round_forward_func, in_axes=axes, out_axes=axes), in_axes=axes, out_axes=axes) > out = net_vmap(x) test_round.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:722: in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = fn = .after_vmap at 0xffff3348b9e0> ms_create_time = 1704924150728117504, input_signature = None, obj = None jit_config = None def __init__(self, fn, ms_create_time, input_signature=None, obj=None, jit_config=None): > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 1 Device HBM total size:34359738368 Device HBM free size:1602883584 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:319: RuntimeError ______________________________ test_round_vmap[1] ______________________________ mode = 1 @pytest.mark.level1 @pytest.mark.env_onecard @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training @pytest.mark.platform_arm_ascend_training @pytest.mark.parametrize('mode', [ms.GRAPH_MODE, ms.PYNATIVE_MODE]) @test_utils.run_test_func def test_round_vmap(mode): """ Feature: test vmap function. Description: test round op vmap. Expectation: expect correct result. """ ms.context.set_context(mode=mode) axes = -1 x = ms.Tensor(np.random.uniform(low=-255, high=255, size=(4, 3, 2)).astype(np.float32)) net_vmap = ops.vmap(ops.vmap(round_forward_func, in_axes=axes, out_axes=axes), in_axes=axes, out_axes=axes) > out = net_vmap(x) test_round.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:722: in staging_specialize out = _MindsporeFunctionExecutor(func, hash_obj, input_signature, process_obj, jit_config)(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = fn = .after_vmap at 0xffff312caef0> ms_create_time = 1704924151598959616, input_signature = None, obj = None jit_config = None def __init__(self, fn, ms_create_time, input_signature=None, obj=None, jit_config=None): > init_pipeline() E RuntimeError: Ascend kernel runtime initialization failed. The details refer to 'Ascend Error Message'. E E ---------------------------------------------------- E - Framework Error Message: E ---------------------------------------------------- E Malloc device memory failed, free memory size is less than half of total memory size.Device 1 Device HBM total size:34359738368 Device HBM free size:1602850816 may be other processes occupying this card, check as: ps -ef|grep python E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:357 Init E mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:73 Initialize /home/jenkins/.local/lib/python3.7/site-packages/mindspore/common/api.py:319: RuntimeError =========================== short test summary info ============================ FAILED test_round.py::test_round_vmap[0] - RuntimeError: Ascend kernel runtim... FAILED test_round.py::test_round_vmap[1] - RuntimeError: Ascend kernel runtim... ============================== 2 failed in 7.45s =============================== [INFO] RUNTIME(137111,python3.7):2024-01-11-06:02:32.930.330 [runtime.cc:1737] 137111 ~Runtime: deconstruct runtime.