============================= test session starts ============================== platform linux -- Python 3.7.5, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 rootdir: /home/jenkins/mindspore/testcases/testcases/tests/st/profiler, inifile: /home/jenkins/sault/virtual_test/virtualenv_006/sault/config/pytest.ini plugins: anyio-3.7.1, forked-1.1.3, xdist-1.32.0 [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:22.757.687 [trace_attr.c:105](tid:153465) platform is 1. [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:22.757.836 [trace_recorder.c:114](tid:153465) use root path: /home/jenkins/ascend/atrace [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:22.757.862 [trace_signal.c:133](tid:153465) register signal handler for signo 2 succeed. [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:22.757.873 [trace_signal.c:133](tid:153465) register signal handler for signo 15 succeed. [INFO] RUNTIME(153465,python3.7):2024-01-11-05:27:23.163.668 [runtime.cc:1159] 153465 GetAicoreNumByLevel: workingDev_=0 [INFO] RUNTIME(153465,python3.7):2024-01-11-05:27:23.163.723 [runtime.cc:4719] 153465 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set collected 8 items / 4 deselected / 4 selected test_profiler.py [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.497.498 [process_mode_manager.cpp:109][OpenProcess][tid:153465] [ProcessModeManager] enter into open process deviceId[5] rankSize[0] [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.499.776 [process_mode_manager.cpp:379][InitTsdClient][tid:153465] [TsdClient] deviceId[5] begin to init hdc client [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.505.878 [version_verify.cpp:34][SetVersionInfo][tid:153465] VersionVerify: send client version to server [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.505.926 [version_verify.cpp:50][SetVersionInfo][tid:153465] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.505.939 [version_verify.cpp:50][SetVersionInfo][tid:153465] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.343 [version_verify.cpp:66][PeerVersionCheck][tid:153465] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.358 [version_verify.cpp:87][ParseVersionInfo][tid:153465] VersionVerify: pass client version info success [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.366 [hdc_client.cpp:276][CheckHdcConnection][tid:153465] Service[2] create hdc success [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.378 [version_verify.cpp:120][SpecialFeatureCheck][tid:153465] VersionVerify: new type[35], supported [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.419 [process_mode_manager.cpp:748][GetDeviceCheckCode][tid:153465] [TsdClient][deviceId=5] [sessionId=1] wait package info respond [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.531 [process_mode_manager.cpp:379][InitTsdClient][tid:153465] [TsdClient] deviceId[5] begin to init hdc client [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.691 [version_verify.cpp:34][SetVersionInfo][tid:153465] VersionVerify: send client version to server [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.704 [version_verify.cpp:50][SetVersionInfo][tid:153465] send feature_info:{msg_type:35, features:{check before send aicpu package,}} [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.713 [version_verify.cpp:50][SetVersionInfo][tid:153465] send feature_info:{msg_type:37, features:{check before send open qs message,}} [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.970 [version_verify.cpp:66][PeerVersionCheck][tid:153465] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.982 [version_verify.cpp:87][ParseVersionInfo][tid:153465] VersionVerify: pass client version info success [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.506.990 [hdc_client.cpp:276][CheckHdcConnection][tid:153465] Service[2] create hdc success [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.507.001 [process_mode_manager.cpp:426][ConstructOpenMsg][tid:153465] [TsdClient] tsd get process sign successfully, procpid[153465] signSize[48] [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.507.033 [version_verify.cpp:112][SpecialFeatureCheck][tid:153465] VersionVerify: previous type[6], supported [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.507.053 [process_mode_manager.cpp:126][OpenProcess][tid:153465] [ProcessModeManager] deviceId[5] sessionId[1] rankSize[0], wait sub process start respond [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.713.350 [stub_process_mode_nowin.cpp:63][ProcessQueueForMdc][tid:153465] [TsdClient] it is unnecessary of current mode[0] chiptype[1] to grant queue auth to aicpusd [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.713.384 [stub_process_mode_nowin.cpp:101][OpenInHost][tid:153465] enter into OpenInHost deviceid[5] [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.713.394 [stub_process_mode_nowin.cpp:105][OpenInHost][tid:153465] host cpu not support [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.713.402 [process_mode_manager.cpp:156][OpenProcess][tid:153465] [TsdClient][deviceId=5] [sessionId=1] start hccp and computer process success [INFO] RUNTIME(153465,python3.7):2024-01-11-05:27:27.716.095 [device.cc:340] 153465 Init: isDoubledie:0, topologytype:0 [INFO] RUNTIME(153465,python3.7):2024-01-11-05:27:27.730.576 [npu_driver.cc:5428] 154996 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:27.730.621 [atrace_api.c:28](tid:153465) AtraceCreate start [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:27.730.730 [trace_rb_log.c:84](tid:153465) [RUNTIME_ATRACE_DEV5_TS0] create ring buffer success, buffer size : 131152. [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:27.730.746 [atrace_api.c:32](tid:153465) AtraceCreate end [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.730.771 [client_manager.cpp:157][SetProfilingCallback][tid:153465] [TsdClient] set profiling callback success [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.748.575 [msprofiler_impl.cpp:156] >>> (tid:153465) ProfNotifySetDevice called, is open: 1, devId: 5 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.748.669 [msprofiler_impl.cpp:289] >>> (tid:153465) Get system free ram: 586156023808 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.748.692 [prof_cann_plugin.cpp:75] >>> (tid:153465) Init report buffer size: 131072 bytes, buffer name: api_event [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.752.114 [prof_cann_plugin.cpp:75] >>> (tid:153465) Init report buffer size: 131072 bytes, buffer name: compact [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.756.491 [prof_cann_plugin.cpp:75] >>> (tid:153465) Init report buffer size: 262144 bytes, buffer name: additional [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.782.402 [platform.cpp:38] >>> (tid:153465) Profiling platform version: 1.0. [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.782.463 [ai_drv_dev_api.cpp:384] >>> (tid:153465) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.782.623 [prof_acl_mgr.cpp:286] >>> (tid:153465) Received ProfAclInit request from acl [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.782.673 [msprof_reporter.cpp:98] >>> (tid:153465) Init all reporters [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.783.969 [prof_acl_mgr.cpp:350] >>> (tid:153465) Received ProfAclStart request from acl [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.785.945 [prof_manager.cpp:384] >>> (tid:153465) Received libmsprof message to start profiling, job_id:5 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.785.990 [prof_manager.cpp:152] >>> (tid:153465) Check device profiling status [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.785.985 [prof_acl_mgr.cpp:95] >>> (tid:155004) Device 5 started to wait for response [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.786.014 [prof_manager.cpp:272] >>> (tid:153465) Begin to launch task, jobId:5 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.786.097 [prof_acl_mgr.cpp:2258] >>> (tid:153465) Init profiling for msproftx [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.786.130 [prof_acl_mgr.cpp:2376] >>> (tid:153465) MsprofSetDeviceImpl, devId:64 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.787.456 [ai_drv_prof_api.cpp:33] >>> (tid:155007) Begin to get channels, deviceId=5 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.787.664 [prof_manager.cpp:384] >>> (tid:153465) Received libmsprof message to start profiling, job_id:64 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.787.696 [prof_manager.cpp:152] >>> (tid:153465) Check device profiling status [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.787.707 [prof_acl_mgr.cpp:95] >>> (tid:155017) Device 64 started to wait for response [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.787.715 [prof_manager.cpp:272] >>> (tid:153465) Begin to launch task, jobId:64 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.788.270 [prof_acl_mgr.cpp:85] >>> (tid:155020) Device 64 finished starting [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.788.308 [prof_acl_mgr.cpp:97] >>> (tid:155017) Device 64 finished waiting for response [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.788.509 [prof_acl_mgr.cpp:1422] >>> (tid:153465) Device:64 finished waiting [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.788.565 [ai_drv_prof_api.cpp:66] >>> (tid:155007) End to get channels[17], deviceId=5 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.790.284 [ai_drv_prof_api.cpp:436] >>> (tid:155007) Begin to start profiling DrvTsFwStart, profDeviceId=5, profChannel=44 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.812.441 [ai_drv_prof_api.cpp:454] >>> (tid:155007) Succeeded to start profiling DrvTsFwStart, profDeviceId=5, profChannel=44 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.812.518 [prof_acl_mgr.cpp:85] >>> (tid:155007) Device 5 finished starting [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.812.540 [prof_acl_mgr.cpp:97] >>> (tid:155004) Device 5 finished waiting for response [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.812.682 [prof_acl_mgr.cpp:1413] >>> (tid:153465) All devices finished waiting [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.813.022 [process_mode_manager.cpp:495][UpdateProfilingConf][tid:153465] [TsdClient] Update profiling mode [deviceId=5][sessionId=1][flag=5] [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.813.040 [version_verify.cpp:112][SpecialFeatureCheck][tid:153465] VersionVerify: previous type[30], supported [INFO] TDT(153465,python3.7):2024-01-11-05:27:27.813.070 [process_mode_manager.cpp:509][UpdateProfilingConf][tid:153465] [TsdClient][deviceId=5] [sessionId=1] wait update profiling msg respond [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.819.234 [ai_drv_dev_api.cpp:384] >>> (tid:155005) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.819.297 [ai_drv_dev_api.cpp:384] >>> (tid:155019) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.819.305 [ai_drv_dev_api.cpp:384] >>> (tid:155005) Succeeded to DrvGetApiVersion version: 0x72313 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.819.441 [prof_task.cpp:285] >>> (tid:155019) ProfTask 64 started to wait for task stop cv [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:27.819.492 [prof_task.cpp:285] >>> (tid:155005) ProfTask 5 started to wait for task stop cv [TRACE] GE(153465,python3.7):2024-01-11-05:27:27.963.701 [status:INIT] [ge_api.cc:144]153465 GEInitializeImpl:GEInitialize start [ERROR] GE(153465,python3.7):2024-01-11-05:27:28.181.712 [profiling_init.cc:167]153465 ParseOptions: ErrorNo: 1343225857(Parameter's invalid!) [INIT][OPS_PRO][Check][Param]Training trace param:off is invalid. [ERROR] GE(153465,python3.7):2024-01-11-05:27:28.181.824 [profiling_init.cc:128]153465 InitProfOptions: ErrorNo: 1343225857(Parameter's invalid!) [INIT][OPS_PRO][Parse][Options]Parse training trace param {"output": "/tmp/profiler_datak74sbrlc/profiler", "fp_point": "", "bp_point": "", "training_trace": "off", "task_trace": "off", "aic_metrics": "None", "aicpu": "off", "profile_memory": "off", "hccl": "off", "l2_cache": "off", "parallel_strategy": "off", "op_time": "off", "profile_framework": "all"} failed, error_code 1343225857 [ERROR] GE(153465,python3.7):2024-01-11-05:27:28.181.875 [profiling_init.cc:50]153465 Init: ErrorNo: 1343225857(Parameter's invalid!) [INIT][OPS_PRO][Init][Profiling]Failed, error_code 1343225857 [ERROR] GE(153465,python3.7):2024-01-11-05:27:28.181.895 [ge_api.cc:87]153465 InitProfiling: ErrorNo: 4294967295(failed) [INIT][OPS_PRO][Init][Profiling] Profiling init failed. [ERROR] GE(153465,python3.7):2024-01-11-05:27:28.181.908 [ge_api.cc:196]153465 GEInitializeImpl: ErrorNo: 1343229953(GEInitialize Failed.) [INIT][OPS_PRO][Init][Profiling]Failed, error code = 4294967295 F[INFO] PROFILING(153465,python3.7):2024-01-11-05:27:28.783.376 [prof_reporter_mgr.cpp:226] >>> (tid:155001) total_size_type_info[5500], save type info length: 35 bytes, type info size: 2 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:28.783.490 [prof_reporter_mgr.cpp:226] >>> (tid:155001) total_size_type_info[10000], save type info length: 356 bytes, type info size: 13 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:28.783.513 [prof_reporter_mgr.cpp:226] >>> (tid:155001) total_size_type_info[15000], save type info length: 67 bytes, type info size: 3 FFF =================================== FAILURES =================================== _____________________ TestProfiler.test_host_profiler[all] _____________________ self = profile_framework = 'all' @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap @pytest.mark.parametrize("profile_framework", ['all', 'time', 'memory', None]) def test_host_profiler(self, profile_framework): self._train_with_profiler(device_target="Ascend", profile_memory=False, only_profile_host=True, > profile_framework=profile_framework) test_profiler.py:197: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_profiler.py:215: in _train_with_profiler lenet = LeNet5() test_profiler.py:61: in __init__ super(LeNet5, self).__init__() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = LeNet5<>, auto_prefix = True, flags = None def __init__(self, auto_prefix=True, flags=None): Cell_.__init__(self, self._cell_tag) self._params = OrderedDict() self._cells = OrderedDict() self._params_list = OrderedDict() self._tensor_list = OrderedDict() self._primitives = OrderedDict() self.training = False self.requires_grad = False self.pynative = False self._attr_synced = False self._param_prefix = '' self._auto_prefix = auto_prefix self._scope = None self._phase = 'train' self._parameter_layout_dict = {} self._parallel_parameter_name_list = () self._parallel_parameter_merge_net_dict = {} self._create_time = int(time.time() * 1e9) self.arguments_key = "" self.compile_cache = set() cells_compile_cache[id(self)] = self.compile_cache self.parameter_broadcast_done = False self._id = 1 self.exist_names = set("") self.exist_objs = set() > init_pipeline() E RuntimeError: Initialize GE failed! E E ---------------------------------------------------- E - Ascend Error Message: E ---------------------------------------------------- E E19999: Inner Error! E E19999 Training trace param:off is invalid.[FUNC:ParseOptions][FILE:profiling_init.cc][LINE:168] E TraceBack (most recent call last): E Parse training trace param {"output": "/tmp/profiler_datak74sbrlc/profiler", "fp_point": "", "bp_point": "", "training_trace": "off", "task_trace": "off", "aic_metrics": "None", "aicpu": "off", "profile_memory": "off", "hccl": "off", "l2_cache": "off", "parallel_strategy": "off", "op_time": "off", "profile_framework": "all"} failed, error_code 1343225857[FUNC:InitProfOptions][FILE:profiling_init.cc][LINE:129] E Init profiling failed, error_code 1343225857[FUNC:Init][FILE:profiling_init.cc][LINE:51] E E (Please search "Ascend Error Message" at https://www.mindspore.cn for error code description) E E ---------------------------------------------------- E - C++ Call Stack: (For framework developers) E ---------------------------------------------------- E mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_context.cc:224 InitGe /home/jenkins/.local/lib/python3.7/site-packages/mindspore/nn/cell.py:134: RuntimeError ____________________ TestProfiler.test_host_profiler[time] _____________________ self = profile_framework = 'time' @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap @pytest.mark.parametrize("profile_framework", ['all', 'time', 'memory', None]) def test_host_profiler(self, profile_framework): self._train_with_profiler(device_target="Ascend", profile_memory=False, only_profile_host=True, > profile_framework=profile_framework) test_profiler.py:197: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_profiler.py:209: in _train_with_profiler profile_framework=profile_framework) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = kwargs = {'aicore_metrics': -1, 'data_process': False, 'op_time': False, 'output_path': '/tmp/profiler_datakjl8qam3', ...} msg = 'Do not init twice in the profiler.' def __init__(self, **kwargs): self._dev_id = None self._cpu_profiler = None self._gpu_profiler = None self._md_profiler = None self._is_heterogeneous = False self._profiler_manager = None self._timeline_meta = [] self._init_time = None self._ascend_job_id = '' self._job_id_env = None self._filt_optype_names = '' self._output_path = '' self._rank_size = 1 self._rank_id = 0 self._ascend_profiler = None self._timeline_size_limit_byte = 500 * 1024 * 1024 # 500MB self._parallel_strategy = True self._model_iteration_dict = None _environment_check() # default aicore_metrics type is ArithmeticUtilization self._aicore_metrics_id = 0 self._l2_cache = "off" self._data_process = True self._op_time = True self._profile_communication = False self._has_started = False self._has_started_twice = False self.start_profile = True self._profile_memory = False self._sync_enable = True self._stop_time = 0 self._dynamic_status = False self._profile_framework = "all" self._msprof_enable = os.getenv("PROFILER_SAMPLECONFIG") if self._msprof_enable: return self._start_time = int(time.time() * 1000000) logger.info("Profiling: start time: %d", self._start_time) if kwargs.get("env_enable"): self._profiler_init(kwargs) return if Profiler._has_initialized: msg = "Do not init twice in the profiler." > raise RuntimeError(msg) E RuntimeError: Do not init twice in the profiler. /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py:473: RuntimeError ___________________ TestProfiler.test_host_profiler[memory] ____________________ self = profile_framework = 'memory' @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap @pytest.mark.parametrize("profile_framework", ['all', 'time', 'memory', None]) def test_host_profiler(self, profile_framework): self._train_with_profiler(device_target="Ascend", profile_memory=False, only_profile_host=True, > profile_framework=profile_framework) test_profiler.py:197: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_profiler.py:209: in _train_with_profiler profile_framework=profile_framework) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = kwargs = {'aicore_metrics': -1, 'data_process': False, 'op_time': False, 'output_path': '/tmp/profiler_data1rwaukvq', ...} msg = 'Do not init twice in the profiler.' def __init__(self, **kwargs): self._dev_id = None self._cpu_profiler = None self._gpu_profiler = None self._md_profiler = None self._is_heterogeneous = False self._profiler_manager = None self._timeline_meta = [] self._init_time = None self._ascend_job_id = '' self._job_id_env = None self._filt_optype_names = '' self._output_path = '' self._rank_size = 1 self._rank_id = 0 self._ascend_profiler = None self._timeline_size_limit_byte = 500 * 1024 * 1024 # 500MB self._parallel_strategy = True self._model_iteration_dict = None _environment_check() # default aicore_metrics type is ArithmeticUtilization self._aicore_metrics_id = 0 self._l2_cache = "off" self._data_process = True self._op_time = True self._profile_communication = False self._has_started = False self._has_started_twice = False self.start_profile = True self._profile_memory = False self._sync_enable = True self._stop_time = 0 self._dynamic_status = False self._profile_framework = "all" self._msprof_enable = os.getenv("PROFILER_SAMPLECONFIG") if self._msprof_enable: return self._start_time = int(time.time() * 1000000) logger.info("Profiling: start time: %d", self._start_time) if kwargs.get("env_enable"): self._profiler_init(kwargs) return if Profiler._has_initialized: msg = "Do not init twice in the profiler." > raise RuntimeError(msg) E RuntimeError: Do not init twice in the profiler. /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py:473: RuntimeError ____________________ TestProfiler.test_host_profiler[None] _____________________ self = profile_framework = None @pytest.mark.level1 @pytest.mark.platform_arm_ascend_training @pytest.mark.platform_x86_ascend_training @pytest.mark.env_onecard @security_off_wrap @pytest.mark.parametrize("profile_framework", ['all', 'time', 'memory', None]) def test_host_profiler(self, profile_framework): self._train_with_profiler(device_target="Ascend", profile_memory=False, only_profile_host=True, > profile_framework=profile_framework) test_profiler.py:197: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_profiler.py:209: in _train_with_profiler profile_framework=profile_framework) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = kwargs = {'aicore_metrics': -1, 'data_process': False, 'op_time': False, 'output_path': '/tmp/profiler_data7unx8lvt', ...} msg = 'Do not init twice in the profiler.' def __init__(self, **kwargs): self._dev_id = None self._cpu_profiler = None self._gpu_profiler = None self._md_profiler = None self._is_heterogeneous = False self._profiler_manager = None self._timeline_meta = [] self._init_time = None self._ascend_job_id = '' self._job_id_env = None self._filt_optype_names = '' self._output_path = '' self._rank_size = 1 self._rank_id = 0 self._ascend_profiler = None self._timeline_size_limit_byte = 500 * 1024 * 1024 # 500MB self._parallel_strategy = True self._model_iteration_dict = None _environment_check() # default aicore_metrics type is ArithmeticUtilization self._aicore_metrics_id = 0 self._l2_cache = "off" self._data_process = True self._op_time = True self._profile_communication = False self._has_started = False self._has_started_twice = False self.start_profile = True self._profile_memory = False self._sync_enable = True self._stop_time = 0 self._dynamic_status = False self._profile_framework = "all" self._msprof_enable = os.getenv("PROFILER_SAMPLECONFIG") if self._msprof_enable: return self._start_time = int(time.time() * 1000000) logger.info("Profiling: start time: %d", self._start_time) if kwargs.get("env_enable"): self._profiler_init(kwargs) return if Profiler._has_initialized: msg = "Do not init twice in the profiler." > raise RuntimeError(msg) E RuntimeError: Do not init twice in the profiler. /home/jenkins/.local/lib/python3.7/site-packages/mindspore/profiler/profiling.py:473: RuntimeError =========================== short test summary info ============================ FAILED test_profiler.py::TestProfiler::test_host_profiler[all] - RuntimeError... FAILED test_profiler.py::TestProfiler::test_host_profiler[time] - RuntimeErro... FAILED test_profiler.py::TestProfiler::test_host_profiler[memory] - RuntimeEr... FAILED test_profiler.py::TestProfiler::test_host_profiler[None] - RuntimeErro... ======================= 4 failed, 4 deselected in 7.56s ======================== [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.483.003 [prof_channel.cpp:407] >>> (tid:153465) ChannelPoll count: 29, Sleep count: 24, Dispatch count: 5, DispatchChannel count: 2 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.503.733 [uploader_dumper.cpp:178] >>> (tid:153465) [UploaderDumper::Flush]Begin to flush data, module:api_event [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.503.768 [uploader_dumper.cpp:182] >>> (tid:153465) [UploaderDumper::Flush]End to flush data, module:api_event [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.504.293 [receive_data.cpp:353] >>> (tid:153465) total_size_report module:api_event, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.504.323 [uploader_dumper.cpp:178] >>> (tid:153465) [UploaderDumper::Flush]Begin to flush data, module:compact [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.504.334 [uploader_dumper.cpp:182] >>> (tid:153465) [UploaderDumper::Flush]End to flush data, module:compact [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.505.325 [receive_data.cpp:353] >>> (tid:153465) total_size_report module:compact, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.505.342 [uploader_dumper.cpp:178] >>> (tid:153465) [UploaderDumper::Flush]Begin to flush data, module:additional [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.505.352 [uploader_dumper.cpp:182] >>> (tid:153465) [UploaderDumper::Flush]End to flush data, module:additional [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.506.365 [receive_data.cpp:353] >>> (tid:153465) total_size_report module:additional, push count:0, pop count:0, push size:0 bytes, pop size:0 bytes [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.506.655 [prof_task.cpp:287] >>> (tid:155019) ProfTask 64 finished waiting for task stop cv [ERROR] PROFILING(153465,python3.7):2024-01-11-05:27:31.506.840 [uploader_mgr.cpp:166] >>> (tid:155019) Get id[64] uploader failed [ERROR] PROFILING(153465,python3.7):2024-01-11-05:27:31.506.906 [prof_task.cpp:162] >>> (tid:155019) Failed to upload data for end_info [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.128 [prof_task.cpp:76] >>> (tid:155019) Uninit ProfTask succesfully [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.142 [prof_task.cpp:339] >>> (tid:155019) Task 64 finished [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.384 [prof_task.cpp:287] >>> (tid:155005) ProfTask 5 finished waiting for task stop cv [ERROR] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.485 [uploader_mgr.cpp:166] >>> (tid:155005) Get id[5] uploader failed [ERROR] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.524 [prof_task.cpp:162] >>> (tid:155005) Failed to upload data for end_info.5 [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.677 [prof_task.cpp:76] >>> (tid:155005) Uninit ProfTask succesfully [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:31.507.688 [prof_task.cpp:339] >>> (tid:155005) Task 5 finished [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:32.188.020 [prof_inner_api.cpp:101] >>> (tid:153465) total_size_report [api_event] read size: 0 bytes, write size: 0 bytes [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:32.188.808 [prof_inner_api.cpp:101] >>> (tid:153465) total_size_report [compact] read size: 0 bytes, write size: 0 bytes [INFO] PROFILING(153465,python3.7):2024-01-11-05:27:32.194.061 [prof_inner_api.cpp:101] >>> (tid:153465) total_size_report [additional] read size: 0 bytes, write size: 0 bytes [INFO] RUNTIME(153465,python3.7):2024-01-11-05:27:32.213.200 [runtime.cc:1737] 153465 ~Runtime: deconstruct runtime. [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:32.314.325 [atrace_api.c:93](tid:153465) AtraceDestroy start [INFO] ATRACE(153465,python3.7):2024-01-11-05:27:32.314.354 [atrace_api.c:95](tid:153465) AtraceDestroy end