Coverage for  / home / jenkins / .local / lib / python3.10 / site-packages / hyper_parallel / data / __init__.py: 100%

1# Copyright 2025-2026 Huawei Technologies Co., Ltd 

2# 

3# Licensed under the Apache License, Version 2.0 (the "License"); 

4# you may not use this file except in compliance with the License. 

5# You may obtain a copy of the License at 

6# 

7# http://www.apache.org/licenses/LICENSE-2.0 

8# 

9# Unless required by applicable law or agreed to in writing, software 

10# distributed under the License is distributed on an "AS IS" BASIS, 

11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

12# See the License for the specific language governing permissions and 

13# limitations under the License. 

14# ============================================================================ 

15 

16"""data: dataset building, batching, and parallel batch primitives. 

17 

18Stage 6 (05 §15.10) final layout: ``text`` (LLM builders/transforms/chat 

19templates), ``online`` (mapping/iterable raw sources), ``indexed`` 

20(``.idx``/``.bin`` datasets and the native helpers), ``batching`` 

21(collators, dataloaders, get-batch), 

22``parallel`` (DP samplers and CP/TP batch distribution), 

23``omni`` (image/video/audio data with its own facade), and 

24``tools`` (offline preparation). Shared constants live in 

25``data.constants``; dataset logging in ``data.dataset_logging``. 

26 

27The package root deliberately has no flat re-exports — import the owning 

28subpackage (multimodal symbols only via ``hyper_parallel.data.omni``). 

29"""