Praha.pm logo

Inter-Rater Agreement

E. Choroba

Praha.pm (Prague)

Charles University

PerlMonks, CPAN, StackOverflow …

choroba@matfyz.cz

Inter-Rater Agreement or Inter-Annotator Reliability

Typical examples

No correct answer (no “gold data”).

Simple Percent Agreement

100 patients

A\B Yes No
Yes 45 15
No 25 15

60% agreement (45 + 15)

But as Yes is more common in both raters, some agreement might occur by chance.

Agreement by Chance

100 patients

A\B Yes No
Yes 45 15
No 25 15
A\B Yes No
Yes 25 35
No 5 35

60% agreement (45 + 15 = 25 + 35)

Yes is more common in both raters if the left table. In the right table, No is more common for rater B.

Cohen’s κ

Simplest agreement measure

κ = (po - pe) / (1 - pe)

po
observed relative agreement
pe
hypothetical probability of chance agreement

Interpretation

Back to the Example

100 patients

A\B Yes No
Yes 45 15
No 25 15
Agreement (45 + 15) / 100 0.6
κ (0.6 - 0.54) / (1 - 0.54) 0.1304
A\B Yes No
Yes 25 35
No 5 35
Agreement (45 + 15) / 100 0.6
κ (0.6 - 0.46) / (1 - 0.46) 0.2593

Back to the Example

100 patients

A\B Yes No
Yes 45 15
No 25 15
Agreement (45 + 15) / 100 0.6
κ (0.6 - 0.54) / (1 - 0.54) 0.1304
A\B Yes No
Yes 25 35
No 5 35
Agreement (45 + 15) / 100 0.6
κ (0.6 - 0.46) / (1 - 0.46) 0.2593

Back to the Example

100 patients

A\B Yes No
Yes 45 15
No 25 15
Agreement (45 + 15) / 100 0.6
κ (0.6 - 0.54) / (1 - 0.54) 0.1304
A\B Yes No
Yes 25 35
No 5 35
Agreement (45 + 15) / 100 0.6
κ (0.6 - 0.46) / (1 - 0.46) 0.2593

Implementation

Variants

Krippendorff’s coefficient α

The Story

Pilot phase: 4 annotators, 900 sentences

Additional funding: 16 annotators, 200K sentences (about 3 annotators per sentence)

Calculating Krippendorff’s α took 6 hours

Using AI to Optimise

Devel::NYTProf identified the culprit: the _build_coincidence method.

sub _build_coincidence($self) {
    my %coinc;
    my @s = $self->vals;
    for my $v (@s) {
        for my $v_ (@s) {
            $coinc{$v}{$v_} = sum(map {
                my $unit = $_;
                my @k = keys %$unit;
                sum(0,
                    map {
                        my $i = $_;
                        scalar grep $unit->{$_} eq $v_,     
                        grep $i ne $_, @k
                    } grep $unit->{$_} eq $v, @k
                ) / (@k - 1)
            } @{ $self->units });
        }
    }
    return \%coinc
}

Using AI to Optimise (2)

Five nested loops (for, for, map, map, grep).

sub _build_coincidence($self) {
    my %coinc;
    my @s = $self->vals;
    for my $v (@s) {                                     # 5
        for my $v_ (@s) {                                # 4
            $coinc{$v}{$v_} = sum(map {                  # 3
                my $unit = $_;
                my @k = keys %$unit;
                sum(0,                                   # 2
                    map {                                # 2
                        my $i = $_;
                        scalar grep $unit->{$_} eq $v_,  # 1
                        grep $i ne $_, @k                # 1
                    } grep $unit->{$_} eq $v, @k         # 2
                ) / (@k - 1)
            } @{ $self->units });
        }
    }
    return \%coinc
}

Using AI to Optimise (3)

Using AI to Optimise (4)

sub _build_coincidence($self) {
    my %coinc;
    my @vals = $self->vals;

    # Initialize coincidence counts
    for my $v (@vals) {
        for my $v_ (@vals) {
            $coinc{$v}{$v_} = 0;
        }
    }

    # Iterate over each unit
    for my $unit (@{ $self->units }) {
        my @keys = keys %$unit;
        my $unit_count = @keys;

        # Count occurrences of each value in the current unit
        my %value_count;
        $value_count{ $unit->{$_} }++ for @keys;

        # Calculate coincidences based on the value counts
        for my $v (@vals) {
            for my $v_ (@vals) {
                if (exists $value_count{$v} && exists $value_count{$v_}) {
                    my $coinc_count = 0;

                    # Count pairs of keys that match the values
                    for my $key1 (@keys) {
                        for my $key2 (@keys) {
                            next if $key1 eq $key2;  # Skip same keys
                            if ($unit->{$key1} eq $v && $unit->{$key2} eq $v_) {
                                $coinc_count++;
                            }
                        }
                    }

                    # Update the coincidence count
                    $coinc{$v}{$v_} += $coinc_count / ($unit_count - 1) if $unit_count > 1;
                }
            }
        }
    }
    return \%coinc
}

 

Using AI to Optimise (5)

        $value_count{ $unit->{$_} }++ for @keys;

# ...

                if (exists $value_count{$v} && exists $value_count{$v_}) {

Using AI to Optimise (6)

sub _build_coincidence($self) {
    my @vals = $self->vals;
    my %coinc;
    @{ $coinc{$_} }{@vals} = (0) x @vals for @vals;

    for my $unit (@{ $self->units }) {
        my %is_value;
        @is_value{ values %$unit } = ();
        my @values = keys %is_value;
        my @keys   = keys %$unit;

        for my $v (@values) {
            for my $v_ (@values) {
                my $coinc_count = 0;
                for my $key1 (@keys) {
                    for my $key2 (@keys) {
                        next if $key1 eq $key2;

                        ++$coinc_count
                            if $unit->{$key1} eq $v
                            && $unit->{$key2} eq $v_;
                    }
                }
                $coinc{$v}{$v_} += $coinc_count / (@keys - 1);
            }
        }
    }
    return \%coinc
}

 

Using AI to Optimise (7)

Conclusion

Thank you

https://e-choroba.eu/26-iaa

Credits