<< Click to Display Table of Contents >> Navigation: Reliability Checks > KAPPA - Inter-Rater-Reliability > Kappa Graph |
After starting the Cohen's Kappa analysis and defining the parameters, INTERACT starts comparing the files.
This basically means: trying to find matching pairs, based on the parameters you applied in the first dialog.
A detailed overview of how INTERACT applies your rules can be found in Kappa Pair Finding Routine.
When the routine finishes, the Kappa Graph appears. This graph offers you a full graphical view on the pairs found by the Kappa routine, providing you with the chance to really understand the resulting Kappa value.
The Kappa Graph shows you the matches as well as the errors found by INTERACT.
The chart shows all Codes from the first Class and the first DataSet of both documents, like this:
In this example, the name of the class is just 'a' and the numbers 1, 2, 3, 4, 5 represent different Codes. Based on the pair finding routine, this graph shows you the results of all matches and mismatches in a colorful way:
oAll found Pairs are green.
oAll errors (=different Codes) are red.
oCodes that can be linked to a matching as well as to a different Code, are colored pink.
oCodes with no link at all (no pairs) are displayed in a soft blue and hardly visible.
Note: Even if your Coding system is not suitable for calculating Kappa, you can use this graph to present a graphical comparison between two raters!
Change View
Per default, the events are drawn per DataSet, as shown in the first image. It is also possible to group them per Code:
You can select this type of visualization in the Kappa Parameter settings on the second tab named View:
▪Select the option Draw data from A and B next to each other to group the Graph per Code.
▪Clear the option Draw data from A and B next to each other to group the Graph per Set.
Adjust Parameters
If the results to not match your expectations, adjustmenting the initial parameters might improve the situation. You can return to the Kappa Parameters dialog from within the Kappa Graph dialog:
▪Click Define parameter to return to the Kappa Parameters dialog.
Kappa Result
▪Click Kappa calculation to open the results dialog.
▪Read Kappa Results for further details.
Visible Data
This Kappa Graph, is a very unique and helpful tool to inspect the results of the pair-finding routine, between the two selected documents in a visual way:
oWhat document is considered the 'leading' document can be changed using the drop down lists for Document A and Document B.
oTo keep this Chart readable, the Kappa Graph automatically shows you just one Class from one DataSet, as selected in the upper fields for Group/Set and Class, for the two selected documents.
oYou can select other Classes and/or DataSets from the drop down menus in the upper part of the graph window, to inspect the results closely.
oDragging with your mouse over a particular area zooms into that part, for easier identification.
oUse the icon to return to full view.
Time Information
oMoving your mouse over the data, shows you the corresponding time information at the bottom of the dialog.
oIn the lower part of the graph window, you see the earliest time value as well as the latest time value, found in those documents. This information is displayed to indicate the duration covered by the graph.
oDouble-click a colored Event, to jump to the corresponding Event in the original data file, allowing you to easily check what is right.