1 Department of Preventive Medicine, Feinberg School of Medicine, Northwestern University, Chicago, IL.
2 Department of Epidemiology and Public Health, Faculty of Medicine, Imperial College of Science, Technology and Medicine, London, United Kingdom.
3 Akademisch Ziekenhuis St. Rafael, Leuven, Belgium.
4 Department of Health Sciences, Shiga University of Medical Science, Otsu, Japan.
Received for publication December 9, 2003; accepted for publication February 27, 2004.
![]() |
ABSTRACT |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
albumins; albuminuria; clinical laboratory techniques; potassium; research design; sodium; urine
Abbreviations: Abbreviations: CV, coefficient of variation; INTERMAP, International Study on Macronutrients and Blood Pressure; percent TE, percentage of technical error.
![]() |
INTRODUCTION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
It is standard procedure in epidemiologic studies and clinical trials to submit blind duplicate samples to laboratories for external assessment of measurement precision. The typical estimate of precision is the percentage of technical error (percent TE) (15). However, estimating measurement precision for urinary albumin concentration may be problematic given its highly skewed distribution in most populations and the likelihood of values below the detection limit of the assay. When laboratories calculate their own internal assessments of quality, they typically compute the coefficient of variation (CV) at different known levels of the analyte (e.g., low, medium, and high), not percent TE across all levels (16). Hence, an alternative approach for external assessment of measurement precision is to compute the CV for each quality control pair of samples and then use the mean or median CV across all pairs as the estimate.
In this analysis, we compared percent TE with the mean and median CVs as estimates of measurement precision for urinary albumin concentration, the albumin:creatinine ratio (which is often used as an alternative to 24-hour albumin excretion (17)), and other urinary measures utilizing 24-hour urine collections from the International Study on Macronutrients and Blood Pressure (INTERMAP). A second objective was to assess how values below the detection limit of the assay affect precision estimates and how such values should be handled in calculating estimates.
![]() |
MATERIALS AND METHODS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
From 1996 to 1999, each participant visited the local INTERMAP research center on four occasions. Two visits were made on consecutive days, with a further two visits taking place on consecutive days 36 weeks later.
Data collection
All data were collected by trained and certified staff. Data collection included a 24-hour dietary recall and measurement of blood pressure at each visit, measurement of height and weight at the first and third visits, and collection of data on demographic and other factors by interviewer-administered questionnaire.
Two timed 24-hour urine specimens were collected for measurement of urinary sodium, potassium, creatinine, urea, magnesium, calcium, and albumin. Timed collections were started at the research center on the first and third visits and were completed at the center the following day. Urine aliquots were stored frozen at 20°C before and after being shipped frozen to the Central Laboratory, where analyses were performed with strict internal and external quality control. Levels of sodium, potassium, creatinine, urea, magnesium, and calcium were analyzed within 3 years of receipt of urine aliquots at the Central Laboratory, while albumin level was analyzed in aliquots frozen for 3 or more years, with completion in 2002. Urinary sodium and potassium concentrations were measured by emission flame photometry. Standard methods were used for analyses of other urinary variables (1922).
As part of quality control procedures, 781 urine samples (approximately a 10 percent random sample) were split at the clinical center and sent to the laboratory with different identification numbers for external assessment of measurement precision. Ongoing review of these data by the INTERMAP Data Coordinating Center (London, United Kingdom) identified 13 pairs for which the within-pair CV was large for all variables initially analyzed (i.e., sodium, potassium, creatinine, urea, calcium, and magnesium). In subsequent review of all split sample results, we identified two pairs for which four of these six variables had large discrepancies and one pair with large discrepancies for both sodium and potassium. We also identified one pair with a large discrepancy for creatinine, three with large discrepancies for urea, one with a large discrepancy for calcium, and one with a large discrepancy for albumin. Precision estimates are presented with and without inclusion of these pairs.
Statistical methods
For this report, we computed percent TE and the mean and median within-pair CV for each urinary variable, as well as albumin:creatinine ratio. Technical error is defined as (d2/2N)1/2, where d is the within-pair difference and N is the number of split sample pairs. For calculation of percent TE, the technical error is multiplied by 100 and the result is divided by the mean of all split sample values. Published estimates of measurement precision for INTERMAP are based on percent TE adjusted for sample (18).
We also used Spearman rank-order correlations to examine associations of the within-pair mean with the within-pair standard deviation and within-pair CV for each urinary measure. We did this to assess whether laboratory variability was judged to be constant at all levels for each urinary measure or increased with level.
Values below the detection limit for urinary albumin concentration
Among the 781 quality control pairs, there were 300 for which both measurements of urinary albumin concentration were less than the detection limit of 1 mg/liter, including 63 for which both values were zero, 32 for which one member of the pair was greater than zero but less than 1 and the other was equal to zero, and 205 for which both values were greater than zero but less than 1. Of the remaining 481 pairs with one or both values at or above the detection limit, there were 61 for which one value was below the limit and the other above.
We considered three approaches for handling these values in estimating measurement precision for urinary albumin concentration: 1) including all values, including those below the detection limit; 2) excluding all pairs with one or both values below the limit; and 3) assigning the next available value below the detection limit for all values below the limitthat is, assigning a value of 0.9 mg/liter for urinary albumin concentration, since urinary albumin concentration was measured to the nearest 0.1 mg/liter. The third approach treats all pairs with both values below the limit as having a within-pair standard deviation and CV of zero.
Simulation studies
To gain insight into observed differences between percent TE and mean and median CV in INTERMAP, we also conducted simulation studies. In these simulations, for each variable except urinary albumin concentration and albumin:creatinine ratio and each quality control pair, we randomly generated four variables from a normal distribution with mean equal to the INTERMAP within-pair mean. The first two values were generated assuming a constant standard deviation corresponding to the INTERMAP percent TE, and the second two were generated assuming a constant within-pair CV and the same percent TE. Because none of the urinary variables in INTERMAP are normally distributed and we wanted to see how precision estimates might vary for normally distributed variables, we also created a variable with values that were normally distributed with a mean of 150 and a standard deviation of 30 (designated "normal I"), a variable with values that were normally distributed with a mean of 50 and a standard deviation of 16 (designated "normal II"), and a variable with values that had a lognormal distribution with a mean of 23.8 and a standard deviation of 14.9. For these three variables, we generated pairs of values corresponding to a constant within-pair standard deviation and a constant within-pair CV, assuming a percent TE of 2.0 for each.
![]() |
RESULTS |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
For urinary albumin concentration, percent TE is 29.6, a value strongly influenced by the single discrepant pair, since the estimate with this pair excluded is 20.6, which is similar to the mean CV with and without this pair (i.e., 20.8 percent and 20.6 percent). The median CV of 9.4 percent for urinary albumin concentration is less than half the mean CV and is within the range (18 percent) considered acceptable for urinary albumin concentration (23). The mean and median CVs for albumin:creatinine ratio differ little from the mean and median CVs for urinary albumin concentration, while percent TE is substantially larger for albumin:creatinine ratio (42.6 percent vs. 29.6 percent) for all 781 pairs.
For urinary albumin concentration, when all values are included, including those below the detection limit, the three estimates are 20.6 percent, 20.6 percent, and 9.4 percent, respectively. If values below the detection limit are set to 0.9, percent TE changes only slightly to 19.9, while the mean and median CVs are reduced from 20.6 percent and 9.4 percent, respectively, to 8.0 percent and 2.6 percent. Exclusion of pairs with one or both values below the detection limit results in estimates that are between those described above, that is, 15.4 percent, 11.4 percent, and 6.4 percent, respectively.
To assess how well percent TE and mean CV represent laboratory precision, we determined the proportion of all quality control pairs with a within-pair CV less than or equal to the computed percent TE and mean CV for each variable, with and without the inclusion of pairs with large discrepancies. Table 2 shows the results of these analyses.
|
Table 3 gives the Spearman rank-order correlations of the within-pair mean with the within-pair standard deviation and CV for each urinary variable, with discrepant pairs excluded, and for urinary albumin concentration and albumin:creatinine ratio with and without exclusion of pairs with values below the detection limit. For all eight variables, there is a positive correlation between the within-pair mean and the standard deviation. For variables other than urinary albumin concentration and albumin:creatinine ratio, the correlations range from 0.210 for urea to 0.354 for creatinine, indicating that the within-pair standard deviation is generally larger at higher concentrations. For urinary albumin concentration and albumin:creatinine ratio, the correlations are 0.577 and 0.627, respectively, without exclusion of values below the detection limit and 0.505 and 0.519 with exclusion of those values. The larger values for urinary albumin concentration and albumin:creatinine ratio are not unexpected given their highly skewed distributions. With the exception of creatinine, the correlation between the within-pair CV and the mean is negative, indicating that the higher the concentration the smaller the within-pair CV tends to be.
|
|
With a constant standard deviation, the rank-order correlation between the within-pair mean and the standard deviation is close to zero for all variables, while the correlation between the within-pair mean and the CV ranges from 0.236 to 0.529. With a constant within-pair standard deviation, the maximum CVs for some variables are quite largefor example, 54.5 percent for calcium, 24.8 percent for urea, and 91.6 percent for the normal II variable. Even though these values were generated using a constant standard deviation and hence reflect random variability, if they were seen during an ongoing review of quality control data, they would almost certainly be flagged for review at the laboratory.
For a constant within-pair CV, percent TE is larger than the mean CV for all six INTERMAP variables (3443 percent) and the three randomly generated variables (2741 percent). Median CVs are also closer to the mean CV than they are when the within-pair standard deviation is held constant, even though all median CVs in this simulation are smaller than those from the simulation with a constant standard deviation. With a constant within-pair CV, the mean and median CVs for the two normal variables are similar.
With a constant CV, the rank-order correlation between the within-pair mean and the CV is close to zero for all variables, while the correlation between the within-pair mean and the standard deviation ranges from 0.172 to 0.540. Maximum CVs are substantially smaller when the CV is assumed to be constant than when the standard deviation is assumed to be constant, with only two variables showing a maximum that exceeds 10 percent.
![]() |
DISCUSSION |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
It is also standard procedure to examine, on an ongoing basis, within-pair differences to identify large differences or pairs that appear discrepant. Differences that are larger than some predefined value are then referred back to the laboratory for review and possible reanalysis. The goal is to identify pairs for which there was mislabeling or data entry error, so that measurement precision can be accurately estimated with and without inclusion of these pairs. Estimates of precision that include all split sample pairs provide an estimate of the overall precision of the measurements in the analysis data set, whereas the estimate with discrepant pairs excluded provides an external estimate of the analytical precision of the laboratory. It is not always easy, however, to identify samples that are discrepant due to causes other than laboratory variability, and any criterion used for identifying samples as such is likely to be arbitrary. It is easier to identify potentially discrepant samples when multiple variables are being measured than when only one is being measured, since multiple large differences can be more readily ascribed to mislabeling than can a single large difference.
In INTERMAP, as in many other studies, the parameter that has been used to assess the measurement precision of laboratory results is percent TE (18). However, percent TE assumes that the within-pair standard deviation is the same throughout the range of values. For example, it assumes that laboratory measurements of urinary albumin concentration have the same within-pair standard deviation at all levels (e.g., whether 5 mg/liter, 250 mg/liter, or 1,500 mg/liter). Such an assumption is unlikely to be correct for urinary albumin concentration, as well as for many other laboratory measures. However, we might expect the same CV at these levels. For example, a CV of 5 percent would indicate a standard deviation of 0.25 mg/liter at a mean of 5 mg/liter and a standard deviation of 75 mg/liter at a mean of 1,500 mg/liter. When laboratories calculate their own internal assessments of quality, they typically compute the CV at specific known levels of the analyte (16). Hence, an alternative approach that can be used to assess precision is to compute the CV for each pair of samples and then use the within-pair mean or median CV as the estimate.
Calculating and interpreting estimates of measurement precision is problematic when there are large numbers of values below the detection limits of the assay. This is true in INTERMAP, not only for urinary albumin concentration but also for urinary amino acids. When one of a pair of values is zero and the other is nonzero, the within-pair CV is 141.4 percent, irrespective of the size of the nonzero valuethat is, whether it is 0.2, 10, or 100. Furthermore, since values below the detection limit are generally considered unreliable, it is not clear that it is appropriate to include such values when estimating the quality of the data.
Note that estimates of measurement precision for urinary variables are based on concentration rather than actual 24-hour excretion. Since both sample values would be multiplied by the same urinary volume, the within-pair CV would remain unchanged if 24-hour excretion were used rather than concentration. However, the percent TEs will not be identical for concentration and 24-hour excretion, since percent TE involves division by the overall mean of the measurements, and urinary volume varies among participants. The albumin:creatinine ratio includes two variables with laboratory variability. However, in INTERMAP, the mean and median CV for albumin:creatinine ratio differed little from the mean and median CV for urinary albumin concentration, while percent TE was substantially greater.
In this report, we used 781 quality control pairs from 24-hour urine collections in INTERMAP to compare percent TE with the mean and median within-pair CV for urinary albumin concentration, albumin:creatinine ratio, and urinary sodium, potassium, creatinine, urea, calcium, and magnesium. We included albumin:creatinine ratio in this report even though it involves two laboratory measures, since it is often used when only a spot or random collection is available (17). We also considered three approaches for computing these estimates when a variable has a highly skewed distribution and substantial numbers of values below the detection limits of the assay.
In INTERMAP, percent TE was larger than mean CV for all urinary variables examined. Some of this difference could be attributed to the initial inclusion of pairs that were discrepant because of mislabeling or possible data entry errors. With exclusion of these pairs, percent TE was generally reduced much more than the mean CV, while the median CV changed minimally. The large impact on percent TE with exclusion of samples with large differences indicates that unless investigators can accurately identify samples with large differences as resulting from mislabeling or data entry errors, percent TE may not provide an accurate assessment of the analytical precision of the laboratory.
In INTERMAP, 46.2 percent of split sample pairs had one or both values for urinary albumin concentration below the detection limit of 1 mg/liter. Including these values in estimates of measurement precision does not appear appropriate, since in defining a detection limit one is saying that values below that limit are unreliable, and (as can be seen from the results shown in table 1) their inclusion has a marked effect on the CV estimates, particularly the mean CV, due to high within-pair CVs when the sample pair includes values below the limit.
We also examined the effect of treating values below the limit as equal, that is, as having a within-pair standard deviation and CV of zero when both values are less than 1 mg/liter, thereby markedly reducing the mean and median CV while leaving percent TE little changed. When both values are below the limit, treating them as equal makes some sense, since the laboratory produced consistent findings, even if the reported values are slightly different. However, whether or not this approach is sensible depends on how values below the detection limit will be handled in data analysesthat is, whether the variable will be analyzed as a continuous variable or a categorical variable. For example, if albumin excretion is being analyzed as a categorical variable (e.g., as micro- and macroalbuminuria), then persons with values below the detection limit clearly do not have micro- or macroalbuminuria, and it matters little whether values below the limit are left unchanged or changed to 0.9 or some other value less than 1.0. However, if albumin is to be used as a continuous variable, then values below the limit pose serious problems, particularly if the variable to be used is 24-hour excretion. If the variable to be used is urinary albumin concentration, then treating all values below the limit as equal to 0.9 would not be particularly problematic, since the relative standing of persons with values below the limit would be preserved relative to persons with values at or above the limit. However, if the analytical variable is urinary albumin excretion, then the ranking will not be preserved, since someone with a concentration of 0.5 mg/liter with a urinary volume of 3.0 liters would have a higher urinary albumin excretion than someone with a concentration of 1.2 mg/liter and a urinary volume of 1.0 liters. Hence, inclusion of persons with values below the detection limit when the investigators plan to use albumin as a continuous variable in the analyses is problematic. If persons with values below the detection limit are to be excluded, then clearly the estimates of measurement precision that exclude such values are the most appropriate estimates to use, since they reflect the data that will actually be included in the analyses.
The results reported here indicate that use of percent TE as an estimate of measurement precision is problematic for highly skewed variables. This is due to the fact that for such variables, the CV for quality control pairs can be expected to increase with the within-pair mean, making percent TE much larger than the mean and median CV, as noted in the simulations. Hence, for variables in INTERMAP, the mean and median CV appear to more fairly represent measurement precision than percent TE.
It is important to note that INTERMAP urinary aliquots were frozen for 3 or more years at 20°C before being analyzed. A number of investigators have reported that long-term freezing at this temperature can affect estimates of concentration, with the impact generally being greater at higher concentrations (2430). In one study of persons with type 2 diabetes mellitus, the median albumin:creatinine ratio decreased by 40 percent in urinary samples frozen at 20°C for 2 years (27). The effect appeared to be greater at lower levels of albumin:creatinine ratio than at higher levels. Furthermore, the percentage of values below the 2.0-mg/liter detection limit of the assay increased from zero to 34 percent over the 2-year period. In INTERMAP, the percentages of values below the 1-mg/liter detection limit were high: 43.3 percent and 45.5 percent for the first and repeat collections, respectively.
In conclusion, the results of this study suggest that mean and median CV are not equivalent to percent TE, but they provide more representative estimates of laboratory precision than percent TE. Furthermore, values below the detection limit can markedly affect estimates and are therefore best excluded when estimating the precision of laboratory measurements.
![]() |
ACKNOWLEDGMENTS |
---|
The authors acknowledge the fine work of the staff of the International Study on Macronutrients and Blood Pressure. A listing of many of these colleagues is given in an article by Stamler et al. (18).
![]() |
NOTES |
---|
![]() |
REFERENCES |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|