Diff Coverage

Diff: origin/master...HEAD, staged and unstaged changes

Source File Diff Coverage (%) Missing Lines
hyper_parallel/components/modules/attention/grouped_query_attention/cp_grouped_query_attention.py 85.7% 173
hyper_parallel/components/modules/attention/grouped_query_attention/grouped_query_attention_family.py 100%  
hyper_parallel/components/modules/attention/grouped_query_attention/cp_grouped_query_attention.py
169
170
171
172
173
174
175
176
        self.ulysses_degree = degree if self.ulysses_degree is None else self.ulysses_degree
        if self.ulysses_degree < 1 or degree % self.ulysses_degree:
            raise ValueError("GQA Ulysses degree must divide CP degree")
        if self.num_heads % self.ulysses_degree:
            raise ValueError("GQA Ulysses degree must divide TP-local Q heads")
        self.kv_head_repeats = self.ulysses_degree // gcd(self.ulysses_degree, self.num_key_value_heads)
        self.ulysses_context, self.query_context = self.cp_context.split(self.ulysses_degree)
        self.all_gather_degree = degree // self.ulysses_degree