{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "# Compute benchmark scale membership probabilities\n", "\n", "An elaborate approach to interpret a kappa value is based on the notion of\n", "cumulative interval membership probability (CIMP). The interval probability\n", "represents the Normality-based probability that the \"true\" agreement\n", "coefficient kappa belongs to the interval in question. The general rule\n", "consists of retaining the highest interval whose CIMP equals or exceeds the\n", "threshold of 0.95. For more details see [Inter-rater reliability among multiple\n", "raters when subjects are rated by different pairs of subjects blog post](\n", "https://inter-rater-reliability.blogspot.com/2018/02/).\n", "\n", "We can use one of the available scales or define a custom one. The available\n", "ones are:\n", "\n", "* Altman\n", "* Cicchetti-Sparrow\n", "* Fleiss\n", "* Landis-Koch\n", "* Regier\n" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Initialize Benchmark\n", "\n", "First, we have to initialize a [Benchmark](\n", "../irrCAC.rst#module-irrCAC.benchmark) object." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/plain": "" }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "from irrCAC.benchmark import Benchmark\n", "\n", "benchmark = Benchmark(coeff=0.67, se=0.15)\n", "benchmark" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Altman\n", "\n", "To interpret coefficient using the Altman scale, use the `altman()` method." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleAltmanCumProb
0(0.8, 1.0)Very Good0.18168
1(0.6, 0.8)Good0.67511
2(0.4, 0.6)Moderate0.96356
3(0.2, 0.4)Fair0.99912
4(-1.0, 0.2)Poor1.00000
\n
", "text/plain": " scale Altman CumProb\n0 (0.8, 1.0) Very Good 0.18168\n1 (0.6, 0.8) Good 0.67511\n2 (0.4, 0.6) Moderate 0.96356\n3 (0.2, 0.4) Fair 0.99912\n4 (-1.0, 0.2) Poor 1.00000" }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "altman = benchmark.altman()\n", "altman_interp = pd.DataFrame(altman)\n", "altman_interp" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Ciccheti-Sparrow\n", "\n", "To interpret coefficient using the Ciccheti-Sparrow scale, use the `ciccheti_sparrow()` method.\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleCicchettiCumProb
0(0.75, 1.0)Excellent0.28699
1(0.6, 0.75)Good0.67511
2(0.4, 0.6)Fair0.96356
3(0.0, 0.4)Poor1.00000
\n
", "text/plain": " scale Cicchetti CumProb\n0 (0.75, 1.0) Excellent 0.28699\n1 (0.6, 0.75) Good 0.67511\n2 (0.4, 0.6) Fair 0.96356\n3 (0.0, 0.4) Poor 1.00000" }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cs = benchmark.cicchetti_sparrow()\n", "cs_interp = pd.DataFrame(cs)\n", "cs_interp" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } }, "source": [ "## Fleiss\n", "\n", "To interpret coefficient using the Fleiss scale, use the `fleiss()` method.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleFleissCumProb
0(0.75, 1.0)Excellent0.28699
1(0.4, 0.75)Intermediate to Good0.96356
2(-1.0, 0.4)Poor1.00000
\n
", "text/plain": " scale Fleiss CumProb\n0 (0.75, 1.0) Excellent 0.28699\n1 (0.4, 0.75) Intermediate to Good 0.96356\n2 (-1.0, 0.4) Poor 1.00000" }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fleiss = benchmark.fleiss()\n", "fleiss_intep = pd.DataFrame(fleiss)\n", "fleiss_intep" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Landis-Koch\n", "\n", "To interpret coefficient using the Landis-Koch scale, use the `landis_koch()` method.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleLandis-KochCumProb
0(0.8, 1.0)Almost Perfect0.18168
1(0.6, 0.8)Substantial0.67511
2(0.4, 0.6)Moderate0.96356
3(0.2, 0.4)Fair0.99912
4(0.0, 0.2)Slight1.00000
5(-1.0, 0.0)Poor1.00000
\n
", "text/plain": " scale Landis-Koch CumProb\n0 (0.8, 1.0) Almost Perfect 0.18168\n1 (0.6, 0.8) Substantial 0.67511\n2 (0.4, 0.6) Moderate 0.96356\n3 (0.2, 0.4) Fair 0.99912\n4 (0.0, 0.2) Slight 1.00000\n5 (-1.0, 0.0) Poor 1.00000" }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lk = benchmark.landis_koch()\n", "lk_interp = pd.DataFrame(lk)\n", "lk_interp" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false }, "source": [ "## Regier et al.\n", "\n", "To interpret coefficient using the Landis-Koch scale, use the `regier()` method.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleRegierCumProb
0(0.8, 1.0)Excellent0.18168
1(0.6, 0.8)Very Good0.67511
2(0.4, 0.6)Good0.96356
3(0.2, 0.4)Questionable0.99912
4(0.0, 0.2)Unacceptable1.00000
\n
", "text/plain": " scale Regier CumProb\n0 (0.8, 1.0) Excellent 0.18168\n1 (0.6, 0.8) Very Good 0.67511\n2 (0.4, 0.6) Good 0.96356\n3 (0.2, 0.4) Questionable 0.99912\n4 (0.0, 0.2) Unacceptable 1.00000" }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "regier = benchmark.regier()\n", "regier_interp = pd.DataFrame(regier)\n", "regier_interp" ] }, { "cell_type": "markdown", "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } }, "source": [ "## Custom Scale\n", "\n", "If you must use a custom scale, then you have to provide a dictionary as the one\n", "bellow. The `lb` key is for the lower bounds of the scale, the `ub` for the\n", "upper bounds, the `interp` key are the interpretations of each scale, and\n", "`scale_name` the name of your scale.\n", "\n", "To use the scale, call the `interpet()` method passing the custom scale as an\n", "argument." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } }, "outputs": [ { "data": { "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleMy ScaleCumProb
0(0.6, 1.0)Excellent0.67511
1(0.3, 0.6)Acceptable0.99308
2(0.0, 0.3)Poor1.00000
\n
", "text/plain": " scale My Scale CumProb\n0 (0.6, 1.0) Excellent 0.67511\n1 (0.3, 0.6) Acceptable 0.99308\n2 (0.0, 0.3) Poor 1.00000" }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_scale = dict(\n", " lb=[0.6, 0.3, 0.0],\n", " ub=[1.0, 0.6, 0.3],\n", " interp=['Excellent', 'Acceptable', 'Poor'],\n", " scale_name='My Scale')\n", "my_bench = benchmark.interpret(my_scale)\n", "my_bench_interp = pd.DataFrame(my_bench)\n", "my_bench_interp" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }