CAC from Raw Ratings

The raw ratings, or wide data format, is a way to organize your data in a table format where each row represents a subject, each column a rater and each data point at the junction of the row and column represents the rating the rater assigned to the subject. Its main advantage is the completeness of the information it presents. With this format, there is no loss of information as it shows what rater rated what subject and the specific rating assigned to every subject. A secondary advantage of this format is its ability to use categorical ratings as well as quantitative measurements.

Such datasets are the ones with the raw_ prefix in the datasets module. One example dataset is raw_4raters.

[1]:

from irrCAC.datasets import raw_4raters

data = raw_4raters()
data

[1]:

	Rater1	Rater2	Rater3	Rater4
Units
1	1.0	1.0	NaN	1.0
2	2.0	2.0	3.0	2.0
3	3.0	3.0	3.0	3.0
4	3.0	3.0	3.0	3.0
5	2.0	2.0	2.0	2.0
6	1.0	2.0	3.0	4.0
7	4.0	4.0	4.0	4.0
8	1.0	1.0	2.0	1.0
9	2.0	2.0	2.0	2.0
10	NaN	5.0	5.0	5.0
11	NaN	NaN	1.0	1.0
12	NaN	NaN	3.0	NaN

As you can see, a dataset of raw ratings is merely a listing of ratings that the raters assigned to the subjects. Each row is associated with a single subject. Typically, the same subject would be rated by all or some of the raters. The dataset raw_4raters contains some missing ratings represented by the symbol NaN, suggesting that some raters did not rate all subjects. As a matter of fact, in this particular case, no rater rated all subjects.

Note

The categories appears as floating numbers because of the NaN values.

Initialize CAC

To compute the various agreement coefficients using the raw ratings, first initialize a CAC object. By initializing the object, it has information about the subjects, raters, categories, and weights.

[2]:

from irrCAC.raw import CAC

cac_4raters = CAC(data)
print(cac_4raters)

<irrCAC.raw.CAC Subjects: 12, Raters: 4, Categories: [1.0, 2.0, 3.0, 4.0, 5.0], Weights: "identity">

To calculate the agreement coefficients, you call the appropriate methods.

Fleiss’ Coefficient

To calculate the Fleiss’ coefficient, call the fleiss() method.

[3]:

fleiss = cac_4raters.fleiss()
fleiss

[3]:

{'est': {'coefficient_value': 0.76117,
  'coefficient_name': "Fleiss' kappa",
  'confidence_interval': (0.42438, 1),
  'p_value': 0.00041917303853056254,
  'z': 4.97434,
  'se': 0.15302,
  'pa': 0.81818,
  'pe': 0.23872},
 'weights': array([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

Gwet’s Coefficient

To calculate Gwet’s coefficient, call the gwet() method.

[4]:

gwet = cac_4raters.gwet()
gwet

[4]:

{'est': {'coefficient_value': 0.77544,
  'coefficient_name': 'AC1',
  'confidence_interval': (0.46081, 1),
  'p_value': 0.0002087209840633264,
  'z': 5.42458,
  'se': 0.14295,
  'pa': 0.81818,
  'pe': 0.19032},
 'weights': array([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

Krippendorff’s Alpha

To calculate the Krippendorff’s Alpha, call the krippendorff() method.

[5]:

alpha = cac_4raters.krippendorff()
alpha

[5]:

{'est': {'coefficient_value': 0.74342,
  'coefficient_name': "Krippendorff's Alpha",
  'confidence_interval': (0.41906, 1),
  'p_value': 0.000459425698154714,
  'z': 5.10682,
  'se': 0.14557,
  'pa': 0.805,
  'pe': 0.24},
 'weights': array([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

Conger

To calculate the Conger’s coefficient, call the conger() method.

[6]:

conger = cac_4raters.conger()
conger

[6]:

{'est': {'coefficient_value': 0.76282,
  'coefficient_name': "Conger's kappa",
  'confidence_interval': (0.4345, 1),
  'p_value': 0.00033670657720907826,
  'z': 5.11381,
  'se': 0.14917,
  'pa': 0.81818,
  'pe': 0.23343},
 'weights': array([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

Brennar-Prediger

To calculate the Brennar-Prediger coefficient, call the bp() method.

[7]:

bp = cac_4raters.bp()
bp

[7]:

{'est': {'coefficient_value': 0.77273,
  'coefficient_name': 'Brennan-Prediger',
  'confidence_interval': (0.45421, 1),
  'p_value': 0.00011878043481217126,
  'z': 5.33959,
  'se': 0.14472,
  'pa': 0.81818,
  'pe': 0.2},
 'weights': array([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

Weighted Analysis

We can use custom weights or predefined weight types initializing the CAC objects. For the available weight types see the Weights module.

In the following example, we initialize a new object on the same data using quadratic weights.

[8]:

cac_4raters_quadratic = CAC(data, weights='quadratic')
cac_4raters_quadratic

[8]:

<irrCAC.raw.CAC Subjects: 12, Raters: 4, Categories: [1.0, 2.0, 3.0, 4.0, 5.0], Weights: "quadratic">

To see the weights’ matrix we can print the weights_mat attribute of the object.

[9]:

cac_4raters_quadratic.weights_mat

[9]:

array([[1.    , 0.9375, 0.75  , 0.4375, 0.    ],
       [0.9375, 1.    , 0.9375, 0.75  , 0.4375],
       [0.75  , 0.9375, 1.    , 0.9375, 0.75  ],
       [0.4375, 0.75  , 0.9375, 1.    , 0.9375],
       [0.    , 0.4375, 0.75  , 0.9375, 1.    ]])

Next, we simply call the method of the coefficient we want the calculation. Here for example we show the weighted Gwet coefficient.

[10]:

gwet_quadratic = cac_4raters_quadratic.gwet()
gwet_quadratic

[10]:

{'est': {'coefficient_value': 0.914,
  'coefficient_name': 'AC2',
  'confidence_interval': (0.68518, 1),
  'p_value': 2.6344384658205655e-06,
  'z': 8.79166,
  'se': 0.10396,
  'pa': 0.97538,
  'pe': 0.7137},
 'weights': array([[1.    , 0.9375, 0.75  , 0.4375, 0.    ],
        [0.9375, 1.    , 0.9375, 0.75  , 0.4375],
        [0.75  , 0.9375, 1.    , 0.9375, 0.75  ],
        [0.4375, 0.75  , 0.9375, 1.    , 0.9375],
        [0.    , 0.4375, 0.75  , 0.9375, 1.    ]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

To use custom weights pass a list of values or an array.

[11]:

import numpy as np


weights = np.array(
    [
        [1.00, 0.75, 0.50, 0.25, 0.00],
        [0.00, 1.00, 0.75, 0.50, 0.25],
        [0.25, 0.00, 1.00, 0.75, 0.50],
        [0.50, 0.25, 0.00, 1.00, 0.75],
        [0.75, 0.50, 0.25, 0.00, 1.00],
    ]
)
cac_4raters_custom_weights = CAC(data, weights=weights)
cac_4raters_custom_weights

[11]:

<irrCAC.raw.CAC Subjects: 12, Raters: 4, Categories: [1.0, 2.0, 3.0, 4.0, 5.0], Weights: "Custom Weights">

Verify the weights.

[12]:

cac_4raters_custom_weights.weights_mat

[12]:

array([[1.  , 0.75, 0.5 , 0.25, 0.  ],
       [0.  , 1.  , 0.75, 0.5 , 0.25],
       [0.25, 0.  , 1.  , 0.75, 0.5 ],
       [0.5 , 0.25, 0.  , 1.  , 0.75],
       [0.75, 0.5 , 0.25, 0.  , 1.  ]])

Calculate a coefficient using the custom weights.

[13]:

gwet_custom_weights = cac_4raters_custom_weights.gwet()
gwet_custom_weights

[13]:

{'est': {'coefficient_value': 0.78322,
  'coefficient_name': 'AC2',
  'confidence_interval': (0.47891, 1),
  'p_value': 0.00014556862670245252,
  'z': 5.66487,
  'se': 0.13826,
  'pa': 0.88636,
  'pe': 0.4758},
 'weights': array([[1.  , 0.75, 0.5 , 0.25, 0.  ],
        [0.  , 1.  , 0.75, 0.5 , 0.25],
        [0.25, 0.  , 1.  , 0.75, 0.5 ],
        [0.5 , 0.25, 0.  , 1.  , 0.75],
        [0.75, 0.5 , 0.25, 0.  , 1.  ]]),
 'categories': [1.0, 2.0, 3.0, 4.0, 5.0]}

To compare the results of the identity weights, the calculation with the quadratic weights, and the custom weights, we display the results side by side.

[14]:

import pandas as pd

df = pd.DataFrame(
    zip(gwet['est'].items(),
        gwet_quadratic['est'].items(),
        gwet_custom_weights['est'].items()),
    columns=['Identity Weights', 'Quadratic Weights', 'Custom Weights'])
df

[14]:

	Identity Weights	Quadratic Weights	Custom Weights
0	(coefficient_value, 0.77544)	(coefficient_value, 0.914)	(coefficient_value, 0.78322)
1	(coefficient_name, AC1)	(coefficient_name, AC2)	(coefficient_name, AC2)
2	(confidence_interval, (0.46081, 1))	(confidence_interval, (0.68518, 1))	(confidence_interval, (0.47891, 1))
3	(p_value, 0.0002087209840633264)	(p_value, 2.6344384658205655e-06)	(p_value, 0.00014556862670245252)
4	(z, 5.42458)	(z, 8.79166)	(z, 5.66487)
5	(se, 0.14295)	(se, 0.10396)	(se, 0.13826)
6	(pa, 0.81818)	(pa, 0.97538)	(pa, 0.88636)
7	(pe, 0.19032)	(pe, 0.7137)	(pe, 0.4758)