| Title: | Optimizing the Composite Reliability in Multivariate Nested Designs |
|---|---|
| Description: | The reliability of assessment tools is a crucial aspect of monitoring student performance in various educational settings. It ensures that the assessment outcomes accurately reflect a student's true level of performance. However, when assessments are combined, determining composite reliability can be challenging, especially for naturalistic and unbalanced datasets in nested design as is often the case for Workplace-Based Assessments. This package is designed to estimate composite reliability in nested designs using multivariate generalizability theory and enhance the analysis of assessment data. The package allows for the inclusion of weight per assessment type and produces extensive G- and D-study results with graphical interpretations, and options to find the set of weights that maximizes the composite reliability or minimizes the standard error of measurement (SEM). |
| Authors: | Joyce Moonen - van Loon [aut, cre]
|
| Maintainer: | Joyce Moonen - van Loon <[email protected]> |
| License: | GPL (>= 3) |
| Version: | 1.0.4.9000 |
| Built: | 2024-11-24 05:10:45 UTC |
| Source: | https://github.com/jmoonen/compositereliabilityinnesteddesigns |
calculateReliability: determine the reliability and SEM per Type
calculateReliability(mydata, n)calculateReliability(mydata, n)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
n |
A vector containing for each Type the number of score or assessments assessments, e.g. averages, requirements. |
A list containing 2 vectors; one vector with the reliability coefficient of each Type, the other vector with the SEM values for each Type
rel <- calculateReliability(mydata, n=c("A"=3,"B"=3,C="2"))rel <- calculateReliability(mydata, n=c("A"=3,"B"=3,C="2"))
calculateVarCov: Estimate variance and covariance components of assessee p S_p and mean assessment scores i nested in assessees S_iINp, and determine the error scores S_delta
calculateVarCov(mydata, n)calculateVarCov(mydata, n)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
n |
A vector containing for each Type the number of score or assessments assessments, e.g. averages, requirements. |
A list containing the observed variances, covariances and errors scores
varcov <- calculateVarCov(mydata, c("A"=3, "B"=3, "C"=2)) varcov$S_p varcov$S_iINp varcov$S_deltavarcov <- calculateVarCov(mydata, c("A"=3, "B"=3, "C"=2)) varcov$S_p varcov$S_iINp varcov$S_delta
checkDatasets: assert that the given datasets adhere to the assumptions and requirements of this package i.e. the data set 'mydata' is a dataframe with 3 columns, named "ID", "Type" and "Score", column "Score" contains numeric data, and each combination of "ID" and "Type" exists at least once, data set n contains a numerical value for each "Type", and data set weights contains a numerical value for each "Type" and the sum of all values is equal to 1.
checkDatasets(mydata, n = NULL, weights = NULL)checkDatasets(mydata, n = NULL, weights = NULL)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
n |
A vector containing for each Type the number of score or assessments assessments, e.g. averages, requirements. |
weights |
A vector containing for each Type the weight assigned to it. The sum of weights should be equal to 1. |
A list with the number of Assessments per ID per Type
checkDatasets(mydata, n=c("A"=10, "B"=5, "C"=2), weights=c("A"=1/3,"B"=1/3, "C"=1/3))checkDatasets(mydata, n=c("A"=10, "B"=5, "C"=2), weights=c("A"=1/3,"B"=1/3, "C"=1/3))
computeCompositeReliability: multivariate generalizability theory approach to estimate the composite reliability of student performance across different types of assessments.
computeCompositeReliability(mydata, n, weights, optimizeSEM)computeCompositeReliability(mydata, n, weights, optimizeSEM)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
n |
A vector containing for each Type the number of score or assessments assessments, e.g. averages, requirements. |
weights |
A vector containing for each Type the weight assigned to it. The sum of weights should be equal to 1. |
optimizeSEM |
Boolean, if TRUE, the weights are adjusted in order to minimize the Standard Error of Measurement (SEM) |
A list containing the composite reliability coefficient, the SEM and the distribution of weights. If 'optimizeSEM' is set to TRUE, the vector of weights minimizes the SEM.
compRel <- computeCompositeReliability(mydata, n=c("A"=10, "B"=5, "C"=2), weights=c("A"=1/3,"B"=1/3, "C"=1/3), optimizeSEM=TRUE) compRel$reliability compRel$SEM compRel$weightscompRel <- computeCompositeReliability(mydata, n=c("A"=10, "B"=5, "C"=2), weights=c("A"=1/3,"B"=1/3, "C"=1/3), optimizeSEM=TRUE) compRel$reliability compRel$SEM compRel$weights
computeMaxCompositeReliability: multivariate generalizability theory approach to estimate the maximum composite reliability of student performance across different types of assessments.
computeMaxCompositeReliability(mydata, n)computeMaxCompositeReliability(mydata, n)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
n |
A vector containing for each Type the number of score or assessments assessments, e.g. averages, requirements. |
A list containing the composite reliability coefficient, the SEM and the distribution of weights.
compMaxRel <- computeMaxCompositeReliability(mydata, n=c("A"=3, "B"=2, "C"=1)) compMaxRel$reliability compMaxRel$SEM compMaxRel$weightscompMaxRel <- computeMaxCompositeReliability(mydata, n=c("A"=3, "B"=2, "C"=1)) compMaxRel$reliability compMaxRel$SEM compMaxRel$weights
DStudy: the program presents the reliability coefficient and the SEM for different numbers of assessments per type. Both the reliability coefficient and the SEM are presented in graphs for differing numbers of assessments, given insight in the impact on the reliability if more or less assessments per type were required or advised.
DStudy(mydata, maxNrAssessments = 60)DStudy(mydata, maxNrAssessments = 60)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
maxNrAssessments |
The maximum (Int) number of assessments per type on with the D study is executed |
A list containing 2 plots: reliability (plotRel) and Standard Error of Measurement SEM (plotSEM)
plots <- DStudy(mydata, maxNrAssessments = 10)plots <- DStudy(mydata, maxNrAssessments = 10)
GStudy for a dataset in which every student p has a potentially differing number of scores i on each assessment type m. i.e. model i: (p x m). The output gives descriptive statistics, reliability coefficient and SEM for each assessment type.
GStudy(mydata, nrDigitsOutput = 4)GStudy(mydata, nrDigitsOutput = 4)
mydata |
A dataframe containing columns ID, Type, Score (numeric) |
nrDigitsOutput |
Integer, number of digits in the output |
Matrix with descriptive statistics for each Type of assessment
GStudy(mydata,nrDigitsOutput=4)GStudy(mydata,nrDigitsOutput=4)
GStudyPerType: This function is mainly used within calculateVarCov.R, but can be executed on its own to determine the reliability coefficient and SEM for a dataset with a single type of assessment.
GStudyPerType(dataPerAssessmentType)GStudyPerType(dataPerAssessmentType)
dataPerAssessmentType |
A dataframe containing columns ID, Type, Score (numeric), with only one value in column Type |
A matrix presenting the observerd varianced and residual, number of ID's and the percentage of the total variance for each group
A dataset that can be used as example in package CompositeReliability.
mydatamydata
mydataA data frame with 7,240 rows and 60 columns:
ID of the student
The type of assessment
The obtained score by this student on this occasion, using the type of assessment
...