1 National Public Health Service for Wales, Abton House, Wedal Road, Cardiff CF14 3QX; 2 Department of Epidemiology, Statistics and Public Health, University of Wales College of Medicine, Cardiff CF14 4XW, UK
Received 2 October 2003; returned 8 December 2003; revised 7 March 2004; accepted 9 March 2004
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Materials and methods: Antibiotic resistance estimates were analysed from an all-Wales dataset for approximately 300 000 community isolates of common pathogens.
Results: Selective testing policies were often associated with markedly increased resistance, particularly for second-line testing. Site-specific testing tended to yield variant resistance estimates for eye and ear isolates. Estimates from rule-based reporting deviated markedly from test-result-based reporting. Urinary isolates reported as Escherichia coli showed greater susceptibility than those reported as undifferentiated urinary coliforms. The proportion of isolates tested for an antibiotic by a laboratory was a useful indicator of selective testing in this dataset. Selective testing policies had invariably been applied where the proportion of isolates of a species tested against an antibiotic was <90%. As this proportion fell with increasingly selective policies, divergence from pooled-all-Wales non-selective estimates tended to increase, with a bias to increased resistance.
Conclusions: Selective testing, rule-based reporting and urinary coliform identification policies all had significant effects upon resistance estimates. Triage based upon the proportion of isolates tested seemed a useful tool in assigning analysis resources. Where <20% of isolates were tested, selective policies with inherent bias to increased resistance were common, the low number of isolates gave high potential sampling errors, and little confidence could be placed in the resistance estimate. Where 2090% of isolates were tested, detailed analysis sometimes revealed resistance estimates that might be usefully retrieved. Where 90% of isolates were tested, there was no evidence of selective testing, and inter-laboratory variation in estimates appeared to be safely ascribable to other effects, e.g. methodology or real variation in resistance levels.
Keywords: community isolates, resistance estimates
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The Welsh Antibiotic Study Group is involved in a major survey of resistance in community infections. Data for approximately 300 000 routine community isolates throughout Wales from 1996 to 2001 have been collected retrospectively, to generate hypotheses, and data for 2001 to 2003 will be collected to test these hypotheses. The aims of the survey include publication of definitive information on the effects of potential biasing factors on susceptibility estimates derived from surveillance of routine isolates. The retrospective data were particularly suited to analysis for the effects of differences in laboratory policy, as it covered a period pre-dating any major shift to inter-laboratory standardization.
Susceptibility estimates should ideally reflect the expected prevalence of resistant organisms, to allow good predictivity of success of blind therapy. Selective sampling by clinicians may bias estimates from routine diagnostic data but this might be amenable to correction from sentinel studies.1 Nonetheless, estimates from routine data reflect susceptibilities for a population that can be readily identified by practitionersthose patients where a decision to seek laboratory investigations has been takenand so contain information useful in prediction. However, further effects may degrade their relevance. This paper documents analyses for three areas where laboratory-testing policies may affect community susceptibility estimatesselective testing, rule-based reporting, and species identification of urinary coliforms. We were particularly interested in the biasing effects of the various antibiotic testing/reporting policies defined below.
Selective testing policies comprised:
In addition, we were interested in the frequency of changes in testing policies, and the effects of these changes on susceptibility estimates.
Rule-based reporting involves results that are derived from a set of rules defining invariant results for specific antibiotics and species, irrespective of any laboratory test. This may be applied to all isolates of a species (e.g. some laboratories reported all H. influenzae isolates as intermediate or resistant to erythromycin), or to specific isolates (e.g. S. pyogenes isolates might be reported as flucloxacillin-susceptible under specific circumstancessee 4 and 5 above).
Inter-laboratory variation also occurs in the extent of identification. For urinary coliform isolates, all lactose fermenting isolates may be reported as coliforms, or further identification may distinguish Escherichia coli isolates from other coliforms. It is widely believed that inclusion of coliforms other than E. coli biases susceptibility estimates to increased resistance.
Several of these policies involve selection of isolates that might show resistance, or are so severely selective that the small numbers yield unreliable resistance estimates. Resistance estimates derived from such selective testing may be a poor basis for prediction of outcome in blind therapy.
![]() |
Methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Routine susceptibility data for Welsh community isolates of urinary coliforms, MSSA and methicillin-resistant S. aureus (MRSA), H. influenzae, S. pneumoniae and S. pyogenes were analysed for the 5 years from April 1996 to March 2001. The data were gathered from 14 of the 18 Welsh and Welsh border laboratories serving Welsh practices (see Acknowledgements); four could not provide data because of various IT problems. The data comprised, for each isolate: date of specimen collection; specimen type and site; patient date of birth, identifier and gender; practice code; laboratory number; and susceptibility (where tested) to 13 antibiotics (penicillin, ampicillin/amoxicillin, co-amoxiclav, cefuroxime, cefalexin, flucloxacillin, erythromycin, tetracycline, trimethoprim, nalidixic acid, ciprofloxacin, norfloxacin and nitrofurantoin). These data were obtained from the proprietary laboratory computer system used in the 14 laboratories and imported into Microsoft Excel. Visual Basic macros were used to format the data. The data were then assembled in organism groups (urinary coliforms including those identified as E. coli, MSSA, MRSA, H. influenzae, S. pneumoniae and S. pyogenes) and transferred to Statistical Package for the Social Sciences (SPSS Inc., Chicago, IL, USA) for analysis. Duplicate isolates, defined as isolates occurring less than 92 days after a prior indistinguishable organism from the same patient, were eliminated from the data before analysis.
The methods used by laboratories to test susceptibilities were based on disc diffusion. All laboratories involved participated in the United Kingdom National External Quality Assessment Service scheme and attained acceptable results. Information on the methods used for the identification of urinary coliform isolates was obtained from a telephone survey of laboratories.
Selective susceptibilities
For each species, the total number of isolates available for testing at each laboratory was noted. The number of isolates tested by the laboratory was determined for each antibiotic, and the percentage of isolates tested (PT) was recorded for that laboratory/species/antibiotic combination. No further analysis was undertaken if <30 isolates had been tested. For 30 isolates, if 15 were susceptible, the 95% confidence limits on the resistance estimate would be 33.266.9%; if none were resistant, the limits would be 011.4%. It was felt that levels of uncertainty beyond this would remove any confidence in the estimates. The 95% confidence intervals for proportions and the difference between proportions were calculated by the method of Wilson.24
For the analysable data, details of isolates that had been tested against the antibiotic were compared with those that had not. This comparison sought consistent patterns in: date interval for specimen submission (indicating a change in testing policy); specimen site differences (indicating site-specific testing); and multiple-resistance to other antibiotics (indicating second-line testing of resistant isolates). Laboratories were contacted to confirm the deductions, obtain information on rule-based reporting, and to resolve less obvious patterns of selective testing. The latter were generally due to either prescribing-specific testing, or matched susceptibility reporting.
Pooled non-selective all-Wales susceptibility estimates were calculated for comparison with individual laboratory estimates. For the purposes of this paper, these were taken from pooled data for those laboratories testing for 90% of isolates for the organism/antibiotic, provided there was more than one such laboratory and that the results from these laboratories were test-based rather than rule-based. Analyses were also carried out for the population subdivisions recognized in the health services organization for Wales, five Health Authorities, which are further divided into a total of 22 Local Health Groups that are coterminous with unitary local authorities.
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The 14 laboratories, six organisms and 13 antibiotics gave 1092 possible laboratory/organism/antibiotic combinations. No data were available for 16 combinations. For 457, the number of isolates tested was <30; these minimal susceptibility estimates comprised clearly inappropriate combinations, e.g. urinary coliforms were rarely tested for penicillin susceptibility. For 323 combinations, the proportion tested was 90%. These non-selective susceptibility estimates comprised clearly appropriate combinations, e.g. most pneumococcal isolates were tested for penicillin susceptibility. The remaining 296 combinations comprised selective susceptibility estimates with proportions tested <90%. Details for all combinations and the full analyses are available from the authors; examples where effects were due to a single policy are shown in Table 1.
|
Site-specific testing. For the 82 instances of selective estimates involving site-specific testing, susceptibility estimates tended to differ from pooled non-selective estimates when isolates from topically treated sites (eye, ear) were selectively tested or omitted. All were readily recognized from comparison of specimen site distributions for tested and untested isolates. As an example, the pooled non-selective estimate for ciprofloxacin resistance in H. influenzae (classing intermediate results as resistant) was 0.2%, whereas a laboratory that selectively tested eye and ear isolates, with a PT of 29%, yielded a resistance estimate of 1.8%. Statistical analysis suggested that this difference was unlikely to be due to chance (difference 1.6%, 95% CI 4.3% to 0.5%). Site-specific testing gave a large range of values for the proportion tested (mean 59%, range 0.889.5%), but differences from non-selective estimates were rarely more than two-fold.
Direct susceptibilities. These were involved in 106 instances of selective testing. The percentage tested was rarely >10%, and the instances invariably involved S. aureus from urine.
Second-line testing. These accounted for six instances of selective testing; all examples involved urinary coliforms, and cefuroxime or co-amoxiclav. The pooled non-selective estimate of co-amoxiclav resistance was 9.1%, compared to 70.3% for a laboratory with second-line testing (PT = 4.2%). The mean PT was 2% (range 0.64.2%), and a strong bias to high resistance was evident, with resistance estimates about 7- to 13-fold greater than non-selective estimates. Laboratories appeared to apply the policy with varying stringency, yielding a wide range of PT values, and a wide spread of selective resistance estimates. For cefuroxime, although no laboratory had tested more than 90% of isolates throughout the trial, one had tested at this level for approximately 4 years, and resistance was 5.1% for this period. This figure was taken as the best available non-selective estimate of urinary coliform resistance to cefuroxime.
Prescribing-specific testing. These accounted for 12 instances of selective testing. For ciprofloxacin, the pooled non-selective estimate for urinary coliforms was 1.8%, compared to 9.4% for prescribing-specific selective testing (PT = 1.3%) at one laboratory. Small numbers of isolates were involved in most instances; the percentage tested was <10%; resistance estimates were often high, more than three-fold greater than non-selective estimates in some instances; and results were sometimes rule-based rather than test-based.
Matched-susceptibility reporting. Of the nine instances of matched-susceptibility reporting, most involved penicillin-susceptible S. pneumoniae, isolated in mixed culture with H. influenzae, being reported as susceptible to ampicillin and/or co-amoxiclav. Equally, S. pyogenes isolated in mixed culture with penicillin-resistant MSSA were reported as flucloxacillin-susceptible by some laboratories. The PT was usually 1020%, and deviation of more than two-fold from non-selective estimates was rare. Again, results were rule-based for some instances and test-based in others.
Test policy changes. Policy changes probably represent the most difficult area of selective testing, as the effects on susceptibility estimates depended on both the nature of the policy change and the date at which the change occurred relative to the period of data collection. The 81 instances were distributed between the species: 11 coliforms, 19 MRSA, 14 MSSA, 16 H. influenzae, 16 S. pneumoniae and five S. pyogenes.
Effect of selective testing on regional and local susceptibility estimates. All-Wales aggregate estimates that included or excluded contributions from laboratories with selective testing were compared (Tables 1 and 2). For most relevant species/antibiotic combinations, the aggregate proportion tested exceeded 50% and differences were minimal. However, for a few species/antibiotic combinations, the aggregate proportion tested was low because of widespread selective testing policies, and differences were large (e.g. for urinary coliform susceptibility to cefuroxime). In contrast, larger differences were found for estimates at Health Authority, Local Health Group or laboratory catchment level, where selective testing often made a large proportional contribution, and so tended to produce large perturbations.
|
|
Five of the 14 laboratories incorporated rule-based reporting for H. influenzae susceptibility to erythromycin as intermediate or resistant, irrespective of laboratory test results. For the remaining nine laboratories, which reported test-based results, resistance estimates varied between 10.0% and 48.3%. Other instances were rule-based reporting of ciprofloxacin resistance in MSSA, S. pneumoniae and S. pyogenes where rule-based resistance estimates were 100%, compared to examples of individual laboratory test-based estimates of, respectively: 14.1% (95% CI 13.414.7%; PT = 95.4%); 10.6% (8.712.9%; PT = 77.9%); and 4.5% (3.55.7%; PT = 74.2%). These policies were distinct in their effects from others discussed above, as they affected non-selective estimates, as well as selective estimates (see, for example, matched susceptibilities above). For those laboratories applying rule-based reporting, non-selective estimates for the organism/antibiotic combinations mentioned above were close to 100% resistance. It was considered that no valid comparison of susceptibilities could be made between rule-based and test-based results.
Identification of urinary coliform isolates
Seven of the 14 laboratories reported urinary lactose-fermenting Gram-negative bacilli as coliform or LF almost exclusively. Three laboratories reported a proportion of urinary isolates (99.6%, 54.3% and 17.4%, respectively) as E. coli. Two laboratories ceased to report community urinary coliforms as E. coli during the study (originally reporting 32.6% and 82.9% E. coli, respectively) and two laboratories began to report urinary isolates as E. coli (65.7% and 95.7%, respectively). For two of the laboratories, identification was based on an indole test at minimum, one identified mainly by colony morphology, and one sometimes identified by morphology and sometimes biochemically. Urinary isolates reported as E. coli were significantly less resistant than those that were reported as coliforms for all antibiotics tested (Table 3).
|
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The results indicate that details of a laboratorys policies on antibiotic testing can affect estimates of resistance based on routine diagnostic data. Selective policies often gave estimates that differed markedly from those based on non-selective testing in the same laboratory (where a policy change had occurred), other laboratories and pooled non-selective all-Wales figures. Second-line testing was the policy with the greatest impact on resistance estimates, similar but lesser impacts arose from other selective testing policies. Rule-based susceptibility estimates for erythromycin resistance in H. influenzae and ciprofloxacin resistance in Gram-positive cocci diverged markedly from test-based estimates. Smaller, but significant differences were found in susceptibility estimates for undifferentiated urinary coliforms and urinary isolates identified as E. coli.
For local surveillance (laboratory catchment, Local Health Group and Area Health Authority), the effects of selective testing were often large but at the all-Wales level, the effects were usually small. Selective testing inherently involves reduced isolate numbers and non-selective testing results from other laboratories often swamped the effect of selective testing. However, if regional standardization of policies was to occur, similar selective testing effects would be present in data from all laboratories and might remain unnoticed. With the current trend to inter-laboratory standardization, the all-Wales community resistance dataset may represent the last opportunity in the UK to analyse policy effects across a widely varied mix, and to untangle their effects on surveillance.
It is not our intention to suggest that any of these policies are correct or incorrect, or to suggest that policies should be changed to cater for surveillance of resistance. The primary purpose of a laboratory is to report bacteriology findings as accurately and helpfully as possible for individual patients, and this must not be compromised for any secondary purpose such as surveillance. Our intention was to document the effects of policies on local and regional surveillance and to determine whether any problems that they introduce into surveillance at local and regional levels can be detected and eliminated.
Testing policies are not the only source of aberrant data, and methodology effects must be considered. Current BSAC guidelines for urinary isolates of S. aureus recommend testing with systemic strength discs,7 so direct urine susceptibility results for these organisms would probably be discarded now. Resistance estimates for urinary and systemic isolates cannot be validly pooled or compared where they are based on distinct disc strengths or breakpoints. Aberrant non-selective estimates were mostly for one antibiotic (trimethoprim), affecting susceptibility estimates for Gram-positive cocci and results from three laboratories. This suggests methodological problems, which must be considered in analysis, but are a distinct issue from policy effects.
Economic and staffing problems may have constrained laboratories to selective testing policies, particularly in the case of site-specific testing. Susceptibility tests on a single plate, with a single set of discs covering both hospital and community isolates, have clear advantages in cost-constrained, understaffed laboratories. Pressure for selective testing may well have increased since the study, especially in laboratories that previously utilized eight discs per plate, with the change to the six-disc per plate BSAC protocol.7 Decisions on selective testing policies involve a host of factors. Testing a limited antibiotic set has advantages in laboratory economics, staffing and skill mix. Testing a broader set may have advantages in patient treatment and, from the results of this study, might well yield surveillance that is more comprehensive and reliable. UK laboratories test far fewer antibiotics per isolate than many other Western nations, but it is unlikely that the advantages to surveillance will alter this situation without financial input. However, we note the advantages of commercial automated systems that test large antibiotic sets and carry out biochemical identification, and that the effect of selective testing on surveillance should be considered when testing policies are reviewed.
Rule-based reporting brings a different class of problem, less amenable to simple detection. We consider that no valid overall resistance estimate can be generated from mixed majority rule- and test-based estimates (exemplified here for erythromycin-resistance in H. influenzae, and ciprofloxacin-resistance in Gram-positive cocci). The estimates derive from fundamentally distinct beliefs and methodology on prediction of clinical efficacy in these instances. There are no easy solutions here, other than for microbiologists to adopt a consensus view and practice based on clinical evidence in each specific case where the problem occurs. Detection of rule-based reporting relied on thorough familiarity with the testing policies of the contributing laboratories. A strategy of reporting rule-based susceptibilities as comments, rather than results would avoid confusion of surveillance data, and could probably be implemented with the automated comment functions available on many modern laboratory information systems.
The effects of regarding heterogeneous groups of organisms as a single class were clear in the analyses of urinary coliforms identified as E. coli. Species-level identification is becoming a prerequisite for interpretation of susceptibility tests for disc sensitivities in the BSAC scheme7 and in many automated systems. This has advantages for surveillance, and for individual patients, where identification of a urinary isolate as a Gram-negative bacillus other than E. coli carries implications about the underlying pathology. However, failure to identify organisms to species level will have little effect on empirical antibiotic selection for the treatment of urinary tract infection and is only an issue for surveillance when comparing areas with different practices or when practice in one area changes over time.
We suggest that laboratories should keep a log of changes in rule-bases, testing policies and methods, specifying the nature of the change, the antibiotics and species affected and the date of change. This could save considerable time in tracing discordant data during local surveillance analysis, and might well provide useful information for those involved in regional and broader levels of surveillance.
Figure 1 clearly illustrates the progressively greater spread of resistance estimates as the proportion of isolates tested by a laboratory decreases, with a concomitant increase in the range of the 95% confidence limits, and a clear trend to higher resistance estimates for more stringently selective testing. There are good theoretical reasons to suppose that many policies with stringent isolate selection (e.g. second-line testing, prescribing-specific testing) may be biased towards selective testing of resistant isolates, and the results outlined here confirm this. Further, as the proportion of isolates tested decreases, the number of isolates contributing to the selective estimate falls, and uncertainty due to potential chance sampling variation increases. There is no clear break in this progression, but examination of the dataset as presented in Figure 1 suggests that resistance estimates derived for a laboratory testing less than about 20% of isolates for the species/antibiotic combination should be treated with profound suspicion. The exception to this would be large surveys with a formally randomized isolate testing design.
Where the proportion of isolates tested by a laboratory ranges from about 20% to 90%, further detailed analysis may sometimes indicate that a portion of the data is valid. As an example, several instances of policy changes from minimal to non-selective testing were found in this dataset, allowing resistance estimates to be calculated over the period of non-selective testing. Resistance estimates where the proportion of isolates tested is 90% appear unlikely to be affected by selective testing, but need to be assessed for other effects, e.g. laboratory methodology problems, before being accepted as evidence of real inter-laboratory variation.
This triage is based on arbitrary break values for the proportion tested, and is subject to confirmation in further studies, but may be useful to others who wish to direct limited analysis resources into those areas that are likely to yield the greatest return. Note that the proportion tested can only be applied in this way for data from a single laboratory on a single antibiotic/species (or species subgroup) combination, where the laboratory policy is uniform for all isolates of the species. It would make little sense to triage data based on the proportion tested for pooled data from several laboratories with distinct policies, or for pooled species subgroup data where policies differ between the subgroupsas encountered here with MSSA and MRSA.
Overall, the study has shown that the intricacies of laboratory testing policies do affect estimates of community resistance, but that many of the resulting problems can be detected during initial data processing. Surveillance provides a context for local figures by allowing comparison with broad-based resistance data. This can be used to detect incipient problems (atypical trends, or in quality control), to inform decisions on antibiotic usage, and to monitor the effects of changes in practice. These functions are important, particularly in the context of the worrying worldwide trend to increased resistance, and rely heavily on confidence in surveillance data. It may be that many putative biasing effects are illusory, or sufficiently consistent to allow correction, or can be eliminated by application of simple rules. Resolution of the reality from the mythology of bias in resistance estimates is an important step in instituting reliable surveillance.
![]() |
Acknowledgements |
---|
The Welsh Antibiotic Study Group comprises: Dr K. Al Shafi, Royal Gwent Hospital, Newport; Dr C. Banerjee, Princess of Wales Hospital, Bridgend; Dr N. J. B. Carbarns, Nevill Hall Hospital, Abergavenny; Dr C. Cefai, Wrexham Maelor Hospital; Dr K. El-Bouri, Prince Charles Hospital, Merthyr Tydfil; Dr K. A. Fitzgerald, National Public Health Service for Wales, Cardiff; Dr G. Harrison, Prince Phillip Hospital, NPHS for Wales, Microbiology Carmarthenshire; Dr D. N. Looker, NPHS, Microbiology Rhyl; Dr L. Macfarlane, NPHS for Wales, Microbiology Aberystwyth; Dr P. Thomas, NPHS for Wales, Microbiology Swansea; Dr R. Salmon, CDSC Wales; Dr M. Sheppard, Withybush Hospital, Haverfordwest; Dr M. D. Simmons, Welsh Assembly; Dr M. Walker, NPHS for Wales, Microbiology Bangor; Dr D. White, Royal Glamorgan Hospital, Pontypridd; and the authors of this paper.
![]() |
Footnotes |
---|
Members of the Welsh Antibiotic Study Group are listed in the Acknowledgements.
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
2 . Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22, 20912.
3 . Newcombe, R. G. (1998). Two-sided confidence intervals for the single proportion: comparison of seven methods. Statistics in Medicine 17, 85772.[CrossRef][ISI][Medline]
4 . Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine 17, 87390.[CrossRef][ISI][Medline]
5
.
Lewis, D. (2002). Antimicrobial resistance surveillance: methods will depend on objectives. Journal of Antimicrobial Chemotherapy 49, 35.
6
.
Hunter, P. A. & Reeves, D. S. (2002). The current status of surveillance of resistance to antimicrobial agents: report on a meeting. Journal of Antimicrobial Chemotherapy 49, 1723.
7 . Andrews, J. for the BSAC Working Party on Susceptibility Testing. (2001). BSAC standardized disc susceptibility testing method. Journal of Antimicrobial Chemotherapy 48, Suppl. S1, S4357.[CrossRef]