{ "cells": [ { "cell_type": "markdown", "source": [ "# Compute benchmark scale membership probabilities\n", "\n", "An elaborate approach to interpret a kappa value is based on the notion of\n", "cumulative interval membership probability (CIMP). The interval probability\n", "represents the Normality-based probability that the \"true\" agreement\n", "coefficient kappa belongs to the interval in question. The general rule\n", "consists of retaining the highest interval whose CIMP equals or exceeds the\n", "threshold of 0.95. For more details see [Inter-rater reliability among multiple\n", "raters when subjects are rated by different pairs of subjects blog post](\n", "https://inter-rater-reliability.blogspot.com/2018/02/).\n", "\n", "We can use one of the available scales or define a custom one. The available\n", "ones are:\n", "\n", "* Altman\n", "* Cicchetti-Sparrow\n", "* Fleiss\n", "* Landis-Koch\n", "* Regier\n" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "## Initialize Benchmark\n", "\n", "First, we have to initialize a [Benchmark](\n", "../irrCAC.rst#module-irrCAC.benchmark) object." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 1, "outputs": [ { "data": { "text/plain": "" }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "from irrCAC.benchmark import Benchmark\n", "\n", "benchmark = Benchmark(coeff=0.67, se=0.15)\n", "benchmark" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Altman\n", "\n", "To interpret coefficient using the Altman scale, use the `altman()` method." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 2, "outputs": [ { "data": { "text/plain": " scale Altman CumProb\n0 (0.8, 1.0) Very Good 0.18168\n1 (0.6, 0.8) Good 0.67511\n2 (0.4, 0.6) Moderate 0.96356\n3 (0.2, 0.4) Fair 0.99912\n4 (-1.0, 0.2) Poor 1.00000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleAltmanCumProb
0(0.8, 1.0)Very Good0.18168
1(0.6, 0.8)Good0.67511
2(0.4, 0.6)Moderate0.96356
3(0.2, 0.4)Fair0.99912
4(-1.0, 0.2)Poor1.00000
\n
" }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "altman = benchmark.altman()\n", "altman_interp = pd.DataFrame(altman)\n", "altman_interp" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Ciccheti-Sparrow\n", "\n", "To interpret coefficient using the Ciccheti-Sparrow scale, use the `ciccheti_sparrow()` method.\n" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 3, "outputs": [ { "data": { "text/plain": " scale Cicchetti CumProb\n0 (0.75, 1.0) Excellent 0.28699\n1 (0.6, 0.75) Good 0.67511\n2 (0.4, 0.6) Fair 0.96356\n3 (0.0, 0.4) Poor 1.00000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleCicchettiCumProb
0(0.75, 1.0)Excellent0.28699
1(0.6, 0.75)Good0.67511
2(0.4, 0.6)Fair0.96356
3(0.0, 0.4)Poor1.00000
\n
" }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "cs = benchmark.cicchetti_sparrow()\n", "cs_interp = pd.DataFrame(cs)\n", "cs_interp" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Fleiss\n", "\n", "To interpret coefficient using the Fleiss scale, use the `fleiss()` method.\n" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "data": { "text/plain": " scale Fleiss CumProb\n0 (0.75, 1.0) Excellent 0.28699\n1 (0.4, 0.75) Intermediate to Good 0.96356\n2 (-1.0, 0.4) Poor 1.00000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleFleissCumProb
0(0.75, 1.0)Excellent0.28699
1(0.4, 0.75)Intermediate to Good0.96356
2(-1.0, 0.4)Poor1.00000
\n
" }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "fleiss = benchmark.fleiss()\n", "fleiss_intep = pd.DataFrame(fleiss)\n", "fleiss_intep" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Landis-Koch\n", "\n", "To interpret coefficient using the Landis-Koch scale, use the `landis_koch()` method.\n" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 5, "outputs": [ { "data": { "text/plain": " scale Landis-Koch CumProb\n0 (0.8, 1.0) Almost Perfect 0.18168\n1 (0.6, 0.8) Substantial 0.67511\n2 (0.4, 0.6) Moderate 0.96356\n3 (0.2, 0.4) Fair 0.99912\n4 (0.0, 0.2) Slight 1.00000\n5 (-1.0, 0.0) Poor 1.00000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleLandis-KochCumProb
0(0.8, 1.0)Almost Perfect0.18168
1(0.6, 0.8)Substantial0.67511
2(0.4, 0.6)Moderate0.96356
3(0.2, 0.4)Fair0.99912
4(0.0, 0.2)Slight1.00000
5(-1.0, 0.0)Poor1.00000
\n
" }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lk = benchmark.landis_koch()\n", "lk_interp = pd.DataFrame(lk)\n", "lk_interp" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Regier et al.\n", "\n", "To interpret coefficient using the Landis-Koch scale, use the `regier()` method.\n" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 6, "outputs": [ { "data": { "text/plain": " scale Regier CumProb\n0 (0.8, 1.0) Excellent 0.18168\n1 (0.6, 0.8) Very Good 0.67511\n2 (0.4, 0.6) Good 0.96356\n3 (0.2, 0.4) Questionable 0.99912\n4 (0.0, 0.2) Unacceptable 1.00000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleRegierCumProb
0(0.8, 1.0)Excellent0.18168
1(0.6, 0.8)Very Good0.67511
2(0.4, 0.6)Good0.96356
3(0.2, 0.4)Questionable0.99912
4(0.0, 0.2)Unacceptable1.00000
\n
" }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "regier = benchmark.regier()\n", "regier_interp = pd.DataFrame(regier)\n", "regier_interp" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } }, { "cell_type": "markdown", "source": [ "## Custom Scale\n", "\n", "If you must use a custom scale, then you have to provide a dictionary as the one\n", "bellow. The `lb` key is for the lower bounds of the scale, the `ub` for the\n", "upper bounds, the `interp` key are the interpretations of each scale, and\n", "`scale_name` the name of your scale.\n", "\n", "To use the scale, call the `interpet()` method passing the custom scale as an\n", "argument." ], "metadata": { "collapsed": false, "pycharm": { "name": "#%% md\n" } } }, { "cell_type": "code", "execution_count": 7, "outputs": [ { "data": { "text/plain": " scale My Scale CumProb\n0 (0.6, 1.0) Excellent 0.67511\n1 (0.3, 0.6) Acceptable 0.99308\n2 (0.0, 0.3) Poor 1.00000", "text/html": "
\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
scaleMy ScaleCumProb
0(0.6, 1.0)Excellent0.67511
1(0.3, 0.6)Acceptable0.99308
2(0.0, 0.3)Poor1.00000
\n
" }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "my_scale = dict(\n", " lb=[0.6, 0.3, 0.0],\n", " ub=[1.0, 0.6, 0.3],\n", " interp=['Excellent', 'Acceptable', 'Poor'],\n", " scale_name='My Scale')\n", "my_bench = benchmark.interpret(my_scale)\n", "my_bench_interp = pd.DataFrame(my_bench)\n", "my_bench_interp" ], "metadata": { "collapsed": false, "pycharm": { "name": "#%%\n" } } } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }