A critical evaluation of commercial immunoassays for antineutrophil cytoplasmic antibodies directed against proteinase 3 and myeloperoxidase in Wegener's granulomatosis and microscopic polyangiitis

E. Csernok, D. Ahlquist, S. Ullrich and W. L. Gross

Department of Rheumatology, University of Lübeck and Rheumaklinik Bad Bramstedt, Germany


    Abstract
 Top
 Abstract
 Introduction
 Patients and methods
 Results
 Discussion
 References
 
Objective. To evaluate the performance of 11 commercial enzyme-linked immunosorbent assay (ELISA) kits for the detection of antineutrophil cytoplasmic antibodies (ANCA) directed against proteinase 3 (PR3) and myeloperoxidase (MPO) in patients with Wegener's granulomatosis (WG) and microscopic polyangiitis (MPA).

Methods. Serum samples were taken from 92 patients with a histological and clinical diagnosis of WG (n=50) or MPA (n=42) and from 30 disease controls (systemic lupus erythematosus, n=15; rheumatoid arthritis, n=15) and 30 healthy controls. Each of the sera was tested for the presence of ANCA directed against PR3 and MPO using 11 commercially available direct ELISA kits, our in-house PR3- and MPO-ANCA capture ELISAs, and the indirect immunofluorescence technique (IFT).

Results. In tests for WG using PR3-ANCA, the commercial direct ELISA kits differed widely in their sensitivity (from 22 to 70%) and negative predictive value (NPV) (from 43 to 70%), but only moderately in their specificity (from 93 to 100%) and positive predictive value (PPV) (from 93 to 100%). The highest sensitivity (74%) and specificity (100%) for PR3-ANCA were obtained with the in-house capture ELISA. Similar differences and trends were noted for MPO-ANCA assays. Diagnostic sensitivity was more than 60% for four and at least 50% for six of the 11 ELISA kits. The PPV varied from 84 to 100% and the NPV from 58 to 70%. In tests for MPA, the MPO-ANCA ELISA kit designated F and the in-house capture ELISA were best (both had sensitivity 62% and specificity 100%). For both WG and MPA, maximum sensitivity for ANCA was obtained with IFT (80 and 70% respectively).

Conclusion. Determination of PR3-ANCA and MPO-ANCA with the commercial direct ELISA kits achieved poor sensitivity for both WG and MPA. The in-house PR3 and MPO-ANCA capture ELISAs performed better than the commercial ELISAs, combining higher specificity with similar sensitivity. IFT remains the best method for ANCA detection in both diseases.

KEY WORDS: ANCA, Wegener's granulomatosis, Microscopic polyangiitis, Proteinase 3, Myeloperoxidase, ELISA.


    Introduction
 Top
 Abstract
 Introduction
 Patients and methods
 Results
 Discussion
 References
 
An international group of antineutrophil cytoplasmic antibody (ANCA) researchers recently published a consensus statement presenting guidelines for ANCA testing and reporting [1]. Because antinuclear antibodies (ANA) can react with neutrophils and give a false-positive ANCA result, the researchers stated that laboratories screening with the immunofluorescence technique (IFT) should never send out positive results without first confirming them with antigen-specific enzyme-linked immunosorbent assays (ELISAs) for proteinase 3 (PR3) and myeloperoxidase (MPO). For optimal sensitivity and specificity they recommended using both IFT and ELISA on all samples [1]. Many hospital/referral laboratories, however, only use commercially available ELISA kits to detect ANCA. The number of companies producing ANCA ELISA kits has increased rapidly even though the diagnostic efficiency of these assays is not well documented. Because there is no agreed international standard, these kits employ arbitrary units whose correlation with titres obtained by IFT remains unknown. When ordering and interpreting an ANCA test, therefore, the clinician must be familiar with the specific assay being used to measure ANCA and with the differences between it and the various other commercially available ANCA assays.

In the present study we compared the sensitivity, specificity and predictive values of 11 commercial direct ANCA ELISA kits with those of IFT and our in-house PR3 and MPO-ANCA capture ELISAs in patients with clinically and histologically established Wegener's granulomatosis (WG) and microscopic polyangiitis (MPA). To determine whether sera containing high concentrations of other autoantibodies [i.e. anti-double-stranded DNA (dsDNA) antibodies] would cause problems (non-specific binding) in the assays, additional tests were performed on sera from patients with systemic lupus erythematosus (SLE) or rheumatoid arthritis (RA). Finally, this study was designed with an eye to improving the routine practice of clinical laboratories and to increase the confidence of clinicians in the results they provide.


    Patients and methods
 Top
 Abstract
 Introduction
 Patients and methods
 Results
 Discussion
 References
 
Manufacturers of ANCA kits were approached for participation in a study designed to critically evaluate the performance of ELISA-based methods for detection of ANCA. They were informed that the results would be published as a comprehensive evaluation that would describe the specific performance of any individual manufacturer. Thirteen manufacturers agreed to participate initially, but two subsequently withdrew. The 11 participating manufacturers were: (UK) Shield Diagnostics; (Germany) Euroimmun, Binding Site, Orgentec Diagnostika, IBL, Hiss Diagnostics, Imtec Immundiagnostika, Milenia Biotec, Pharmacia Diagnostics; (Sweden) Wieslab AB; (USA) Sigma Diagnostics.

Patient population and diagnostic criteria
Serum samples were obtained from well-defined patients with a clinical diagnosis of WG or MPA that had been made irrespective of serology (i.e. the presence of ANCA in sera was not a criterion for the diagnosis of WG or MPA). All serum samples were taken at the time of diagnosis. Ninety-two consecutive patients diagnosed with WG (n=50) or MPA (n=42) in the Departments of Rheumatology and Immunology at Bad Bramstedt (Rheumaklinik) and Lübeck (Medical School of the University of Lübeck) between 1997 and 2000 were studied. The patients were diagnosed according to the definitions of the 1992 Chapel Hill Consensus Conference [2] and the classification criteria of the 1990 American College of Rheumatology [3], based on typical history and characteristic clinical findings and confirmed by histology. Serum samples from 30 control patients with connective tissue diseases (SLE and RA) were also analysed. At the time sera were collected from the WG and MPA patients, disease extent was assessed using the Disease Extent Index (DEI) [4] and disease activity using the Birmingham Vasculitis Activity Score (BVAS) [5]. Sera from 30 age- and sex-matched healthy volunteers were also tested in all assays.

Methods of ANCA detection
Indirect immunofluorescence technique. ANCA detection by IFT was performed on ethanol- and formaldehyde-fixed leucocytes as described previously [6]. In our laboratory, a positive ANCA is defined as a titre of antibodies higher than 1:20.

ELISA for PR3- and MPO-ANCA. Sera were tested for PR3- and MPO-ANCA using the 11 commercial assays according to the manufacturer's instructions. In-house ELISAs were performed as described earlier [7]. The results for PR3-ANCA and MPO-ANCA were evaluated using only the true positive values from each kit as given by the manufacturer.

In-house capture ELISAs. The in-house capture ELISAs for PR3 and MPO-ANCA detection were developed as follows. Monoclonal antibodies (mAb) WGM2 (anti-PR3) and MPO AG Calbiochem (anti-MPO) were coated at a concentration of 2 µg/ml onto microtitre plates for 12 h at 4°C in carbonate buffer at pH 9.6. After washing three times in phosphate-buffered saline (PBS) and 0.05% Tween 20, purified PR3 and MPO were incubated at a concentration of 1 µg/ml in PBS at room temperature for 1 h and the plates were washed. Sera diluted 1:50 in PBS with 0.1% bovine serum albumin and 0.05% Tween 20 were incubated for 1 h at room temperature. After washing, bound immunoglobulin G (IgG) was detected by alkaline phosphatase-conjugated anti-human IgG (Dako, Hamburg, Germany). To exclude non-specific binding, a control plate was coated with an unrelated mouse mAb of the same mouse Ig subclass and the absorbance values of this plate were subtracted from the absorbance values obtained for the anti-PR3 or anti-MPO-coated plates for each individual well. A serum was regarded as positive if the absorbance was more than three standard deviations higher than values obtained with sera (n=120) from healthy donors. Values were related to a set of calibrators to quantify the values in arbitrary units/ml. The cut-off value was set at 35 units/ml. Intra-assay variability was 6%, as calculated by testing one sample 10 times in the same run. Inter-assay variability was 9%, as calculated by testing the same sample in five different runs.

Detection of PR3 and MPO-ANCA with commercial kits. Serum levels of PR3- and MPO-ANCA were measured according to the manufacturers' instructions using the 11 commercial ELISA kits. The results were graded positive or negative according to the scale provided by each manufacturer. For the purposes of this study, each company was assigned a letter (A–K) by which its kits are referred to in the Results section and the tables.

Statistical methods
The diagnostic sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), the agreement of each manufacturer's kit results and receiver operating characteristics (ROC) were analysed for sera from all patients. In estimating the diagnostic specificity, we included both the disease and healthy controls. The cut-off points for calculation of sensitivity and specificity were determined by ROC curves. Pairwise comparisons of the agreement of test results were performed using the Kendall's coefficient of correlation, tau.


    Results
 Top
 Abstract
 Introduction
 Patients and methods
 Results
 Discussion
 References
 
Diagnostic value of commercial ANCA assays
Patient characteristics are outlined in Table 1Go. Patients were categorized into two groups according to disease activity at the time of sampling. Sixty-three were classified as having active disease. Their median BVAS was 9.0 (range 2–27). Twenty-nine patients had no symptoms or signs attributable to active vasculitis at the time their serum samples were obtained. Their BVAS was 0.


View this table:
[in this window]
[in a new window]
 
TABLE 1. Clinical findings and organ involvement in 92 patients with ANCA-associated vasculitides (WG and MPA)

 
Table 2Go gives the sensitivity and specificity of ANCA testing of the WG and MPA patients using the different commercial direct ELISAs, IFT and our in-house capture ELISAs. It also shows the PPV and NPV provided by the results. The sensitivity and specificity of ANCA testing varied widely among the different assays. The best sensitivity (80%) and specificity (100%) were achieved with IFT, which is regarded as the gold standard method. In patients with a clinical and histologically proven diagnosis of WG, the sensitivities of the direct PR3-ANCA ELISA kits varied widely, ranging from 70% (kit I) to 22% (kit H). The specificity was 93% for one and 100% for 10 of the 11 ELISA kits. The PPV varied from 93 to 100% and the NPV from 43 to 67%. The highest sensitivity (74%) and specificity (100%) for PR3-ANCA were obtained with the in-house capture ELISA. The percentage of WG patients with a positive ANCA as detected by IFT was somewhat higher (80%) than that detected by the other assays. Similar differences and trends were observed for the MPO-ANCA assays. Diagnostic sensitivity was greater than 60% for four ELISA kits and at least 50% for six of the 11 kits. The PPV varied from 84 to 100% and the NPV from 62 to 70%. The capture ELISA showed the same antibody positivity (62%) as one of the best direct ELISA kits (kit F).


View this table:
[in this window]
[in a new window]
 
TABLE 2. Diagnostic performance of ANCA assays

 
Fifteen SLE sera previously identified as ANA-positive, in particular exhibiting reactivity for anti-dsDNA antibodies, were examined in each of the ANCA assays. The only MPO-ANCA kits that displayed reactivity with ANA-positive SLE sera were kits A, H and J. Detection of MPO-ANCA in sera positive only for ANA varied among these kits: kit H detected MPO-ANCA in ANA-positive sera less frequently (one serum) than did the other two kits (three sera). All RA sera were negative in all ANCA assays and only two kits detected a positive MPO-ANCA in sera from healthy controls.

ROC analysis
Table 2Go also shows the relative standings of the 11 commercial ELISAs and the other ANCA assays in terms of overall sensitivities and specificities according to their ROC. The value for the area under the ROC curve (AUC) can be interpreted as follows: an area of >0.5 means that an individual selected randomly from the patient group had a >50% chance of having a test value greater than that of a randomly chosen individual from the control group. If the values of the two groups are perfectly equal, the area under the curve equals 1. The value of the data depends on two factors; both specificity and sensitivity should be high and the difference between sensitivity and specificity should be slight. The majority of the commercially available ELISA kits met these requirements relatively well except for the PR3-ANCA kit H (low sensitivity but high specificity) and the MPO-ANCA kit E (high sensitivity and low specificity). The performance characteristics of kit H would cause many false-negative results, those of the kit E many false-positive results.

Agreement between the different ANCA assays
The agreement between the assays ranged from 64 to 98%. The direct ELISA whose results were most consistent with those of the fluorescent ANCA test was the PR3-ANCA kit I. Results of the capture ELISAs correlated better with IFT than did those of the direct ELISAs. Kendall's tau revealed poor to good agreement among the different assays (data not shown).


    Discussion
 Top
 Abstract
 Introduction
 Patients and methods
 Results
 Discussion
 References
 
ANCA testing is a diagnostic tool widely used by clinicians when considering the diagnosis of WG or MPA. The major problem with the current application of ANCA assays is the lack of international standard sera and the lack of international standardization of assays. Guidelines for ANCA testing were published recently along with the recommendation that dual testing be performed on each sample with standard IFT (only alcohol-fixed neutrophils) and with PR3- and MPO-ANCA ELISAs. Unfortunately, commercially available ELISA kits have a wide range of performance characteristics.

The present study assessed the sensitivity, specificity and predictive values of 11 commercially available PR3- and MPO-ANCA direct ELISA kits as diagnostic tests for WG and MPA and compared their performance with the IFT gold standard and with our in-house PR3- and MPO-ANCA capture ELISAs. This study has the advantage over previous studies that it examined the performance of the various assays in carefully defined patient groups (WG and MPA) and that it was conducted with the full cooperation of the 11 manufacturers of ELISA-based ANCA test kits. The major conclusions to be drawn from the present results are the following: (i) the performances of commercial direct ANCA ELISAs vary significantly from one another and, in some cases, from IFT ANCA assays; (ii) one commercial direct ELISA, our in-house PR3-ANCA capture ELISA, exhibited sensitivity and specificity for WG comparable to those obtained with IFT; and (iii) measurement of ANCA is still an imperfect science.

With regard to the sensitivity and specificity of the test kits, the rate of PR3-ANCA positivity differed markedly from kit to kit. When measured with kit H, for instance, the prevalence of ANCA in the 50 WG patients was 22% but when measured with kit I it was 70%. Depending on the kit employed, this would entail major differences in the clinician's diagnostic and therapeutic approaches in more than half of WG patients. The majority of kits showed almost perfect specificity (eight of 11 kits achieved 100% specificity), but the sensitivity varied dramatically between kits. The differences in sensitivity and specificity of the assays could have resulted from differences in the cut-off values for ‘normal’ among the different assays. We did not vary the cut-off values from those indicated by the manufacturers. However, the use of ‘borderline’ (‘equivocal’) values as positive values did produce an increase in sensitivity and led to lower specificity in only two MPO-ANCA kits (data not shown). The basic differences among the ELISA kits, therefore, were due to the borderline (equivocal) sera. As there is clearly a trade-off between sensitivity and specificity, it is appropriate to describe some results as ‘borderline’ or ‘weakly positive’. This enables the informed clinician to better interpret the result on the basis of the pretest clinical probability and whether the test is being used to confirm or exclude WG/MPA. If a test has a sufficiently high sensitivity, a negative result would rule out the diagnosis. If the test has a high specificity, a positive result would support the diagnosis. Thus, the cut-off level can be set to make the test highly sensitive or highly specific depending on the target disease. The moderate agreement we found among the kits and the choice of cut-offs may partially explain this finding. It must be emphasized that the rank orderings of sensitivity and specificity do not imply a definitive overall qualitative ranking of the test kits relative to one another.

The in-house PR3 and MPO-ANCA capture ELISAs performed better than the commercial ELISAs, combining higher specificity with comparable sensitivity. The question to be asked is why the capture ELISAs performed better than conventional ELISAs. A possible explanation is that proteins are denatured during coating onto the solid phase and that this hides or destroys conformational antigens on PR3. To avoid this in capture ELISA, the plate is precoated with a monoclonal antibody to capture the antigen. Another explanation for the findings of imperfect concordance between the two assays could be that the capture assays also detect different or other PR3 epitopes.

A recent study by Pollock et al. [8] comparing the performance of various commercial and in-house ELISA kits found a greater concordance among the PR3-ANCA ELISAs than among the MPO-ANCA ELISAs. In that study the sensitivity of the different PR3-ANCA and MPO-ANCA kits was much higher than in the present study, ranging from 88 to 100% and 59 to 100% respectively. There are critical differences in the design of that study and the present study, differences affecting both the results and the clinical interpretation. The sera examined by Pollock et al. were all from patients suspected to have systemic vasculitis, but the authors provided no clinical information [8]. Only when evaluated together with clinical information do the values for sensitivity and specificity become meaningful. The 60 normal and disease control sera included in the present study were almost invariably negative in all assays. Despite this limitation, our data suggest that ELISA kits differ in their discriminative power for the diagnosis of WG and MPA. They show that there is still room for improvement of ANCA assays despite recent efforts by manufacturers. Given the large number of ANCA ELISA kits currently available, it is imperative that each laboratory maintains its own stringent quality control programme.

Information from studies such as ours is essential for both the clinician and the laboratory manager. Because the different ANCA assays vary so dramatically, it is critical that the clinician understands the performance characteristics of the specific assay being used in his or her laboratory in order to be able to interpret its results intelligently. ELISAs have many advantages over ANCA IFT: they are less time-intensive and can be automated and computerized, and thus potentially save money. However, our results suggest that the best strategy would be to use ELISA ANCA testing as a follow-up to fluorescent ANCA testing in order to identify patients with specific antibodies to PR3 or MPO. The sensitivity provided by direct ELISA alone is too low, as our data demonstrate.

Jethwa et al. [9] recently demonstrated that false-positive tests can result from the binding of DNA contained in the antigen-binding site of anti-dsDNA antibodies to highly cationic MPO used as substrate antigen in immunoassays. In the present study, we analysed the MPO binding capacity of sera from six anti-dsDNA-positive SLE patients. Some kits showed positivity for MPO-ANCA in two of these sera, a result which confirms the purported dual reactivity of SLE sera [9]. Thus, the false-positive MPO binding activity attributable to DNA/anti-DNA antibody complexes can be a source of error in the serological evaluation of MPO-ANCA.

In summary, the commercially available PR3- and MPO-ANCA direct ELISA kits differ widely in their sensitivity and specificity for WG and MPA. This finding reinforces the notion that the value of a particular result depends on the reliability of the underlying test. The gold standard for ANCA screening remains the IFT, and capture ELISA is the preferred technique for detecting PR3- and MPO-ANCA in patients with WG and MPA. Finally, individual manufacturers were informed of the performance of their respective kits so they could take measures to correct perceived deficiencies and thus improve the reliability of this group of important diagnostic assays used in the evaluation of systemic vasculitis diseases. The companies that provided kits H and E have now modified and improved their assays.


    Acknowledgments
 
The authors would like to thank Monika Backes for her excellent technical assistance. This work was supported by a grant from the Bundesministerium für Bildung und Forschung (Kompetenznetz Rheuma 01 G1 9951 C 3.3).


    Notes
 
Correspondence to: E. Csernok, Oskar-Alexander-Strasse 26, 24576 Bad Bramstedt, Germany. Back


    References
 Top
 Abstract
 Introduction
 Patients and methods
 Results
 Discussion
 References
 

  1. Savige J, Gillis D, Benson et al. International consensus statement on testing and reporting of antineutrophil cytoplasmic antibodies (ANCA). Am J Clin Pathol1999;111:507–13.[ISI][Medline]
  2. Jennette JC, Falk RJ, Andrassy K et al. Nomenclature of systematic vasculitides: The proposal of an International Consensus Conference. Arthritis Rheum1994;37:187–9.[ISI][Medline]
  3. Leavitt RY, Fauci AS, Bloch DA et al. The American College of Rheumatology 1990 criteria for the classification of Wegener's granulomatosis. Arthritis Rheum1990;33:1101–7.[ISI][Medline]
  4. deGroot K, Gross WL, Herlyn K, Reinhold-Keller E. Development and validation of a disease extent index for Wegener's granulomatosis. Clin Nephrol2001;55:31–8.[ISI][Medline]
  5. Luqmani RA, Bacon PA, Moots RJ et al. Birmingham Vasculitis Activity Score (BVAS) in systemic necrotizing vasculitis. Q J Med1994;87:671–8.[ISI]
  6. Schönermarck U, Lamprecht P, Csernok E, Gross WL. Prevalence and spectrum of rheumatic diseases associated with proteinase 3-antineutrophil cytoplasmic antibodies (ANCA) and myeloperoxidase-ANCA. Rheumatology2001;40:178–84.[Abstract/Free Full Text]
  7. Wang G, Csernok E, de Groot K, Gross WL. Comparison of eight commercial kits for quantitation of antineutrophil cytoplasmic antibodies (ANCA). J Immunol Methods1997;208:203–11.[ISI][Medline]
  8. Pollock W, Dunster K, Rolland JM, Koh H, Savige J. A comparison of commercial and in-house ELISAs for antineutrophil cytoplasmic antibodies directed against proteinase 3 and myeloperoxidase. Pathology1999;31:38–43.[ISI][Medline]
  9. Jethwa HS, Nachman PH, Falk RJ, Jennette JC. False-positive myeloperoxidase binding activity due to DNA/anti-DNA antibody complexes: a source for analytical error in serologic evaluation of anti-neutrophil cytoplasmic autoantibodies. Clin Exp Immunol2000;121:544–50.[ISI][Medline]
Submitted 25 February 2002; Accepted 13 May 2002