1 Department of Public Health, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands; 2 Medical Research Council, Clinical Trials Unit, London, UK
* Correspondence to: Dr M. R. van Dijk, Erasmus MC-University Medical Center Rotterdam, Department of Public Health, PO Box 1738, 3000 DR Rotterdam, The Netherlands. Tel: +31-10-408-7124; Fax: +31-10-408-9449; Email: m.vandijk{at}erasmusmc.nl
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Patients and methods: The previously developed tree was applied to 456 patients with a poor prognosis as defined by the International Germ Cell Cancer Collaborative Group (IGCCCG). Next, we developed a new tree to evaluate whether a similar structure to the previous tree was found. We assessed the internal validity of the new tree, and compared the 2-year survival estimates of each subgroup together with the discriminative ability for both the previously developed and the new tree. Discriminative ability was measured by a concordance (c) statistic, which varies between 0.5 (no discrimination) and 1.0 (perfect discrimination).
Results: The 2-year survival estimates in the IGCCCG data ranged from 33% to 63%. The ordering of the subgroups was different and discriminative ability was lower than originally found (c = 0.56 in the IGCCCG data versus 0.63 originally). The new tree differed considerably from the original tree, and identified poor prognosis subgroups with 2-year survival estimates from 38% to 73%. Internal validation showed similar discriminative ability for the new tree and the original tree (c = 0.59 versus 0.56).
Conclusions: The previously developed tree showed poor validity with respect to discriminative ability and the stability of its structure. The performance of the new tree was also unsatisfactory. Given the low proportion of patients categorised as poor prognosis, it seems that the potential to identify further subgroups with the currently available patient characteristics is limited.
Key words: germ cell cancer, poor prognosis, tree modelling, validation
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Studies conducted by the Memorial-Sloan Kettering Cancer Center [12, 13
] and the German Testicular Cancer Group [7
, 14
] showed beneficial effects for high-dose chemotherapy with stem-cell support and high-dose chemotherapy, respectively. However these results were based on non-randomised trials and comparisons with historical controls. To confirm these results, two randomised, multicentre trials are currently being conducted in the USA and Europe [15
]. Furthermore, the identification of subgroups among poor prognosis patients would allow for a more accurate estimate of the individual patients' chances of survival, and increase the comparability of results from clinical trials [16
].
Kollmannsberger et al. [16] used tree modelling as an explorative method to identify important risk factors within a group of poor prognosis patients, as defined by the IGCC classification, and to find subsets of patients differing in prognosis. They developed a regression tree based on data of 332 poor prognosis patients as defined by the IGCC classification (Kollmannsberger tree). The risk factors visceral metastases, primary site and abdominal mass were used. This resulted in a tree with five poor prognosis subgroups (Figure 1A). The subgroups differed in 2-year survival, ranging from 49% to 84% [16
].
|
Kollmannsberger et al. [16] recognise these problems and the limitations of their tree. Some subgroups only had a small number of patients, and their identification may be the result of pure chance. Such subgroups may not be present when new data are considered. Furthermore, survival estimates of small groups are often unreliable. This was illustrated by the group of patients with visceral metastases and primary site testis, in which patients with an abdominal mass had a higher 2-year survival [72%; 95% confidence interval (CI) 6480%] than patients without (52%; 95% CI 2777%). Kollmannsberger et al. [16
] therefore propose that further confirmatory studies in other poor prognosis patients cohorts are needed, before the tree can be used in practice.
The aim of the present study was to evaluate the validity of the Kollmannsberger tree. We consider two aspects of validity: performance and the structure of the tree model. We evaluated the performance of the Kollmannsberger tree by applying the tree to the poor prognosis patients in the IGCCCG database [9]. Furthermore, we developed a new tree for the poor prognosis patients in the IGCCCG database to study whether its structure, that is the selection and hierarchy of risk factors, was similar to the Kollmannsberger tree.
![]() |
Patients and methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
To validate the Kollmannsberger tree, we used the 495 poor prognosis patients with nonseminomatous germ cell cancer from the IGCCCG database, which consists of 5202 adult male patients. Poor prognosis was defined by the presence of any of the poor risk factors mediastinal primary site, (non-pulmonary) visceral metastases, -fetoprotein (AFP) poor (>10 000 ng/ml), human chorionic gonadotrophin (HCG) poor (>10 000 ng/ml) and lactate dehydrogenase (LDH) poor (>10 times the upper limit of normal) [9
]. Patients were treated between 1975 and 1990 with cisplatin-based chemotherapy. Data were collected on age, primary site, date of diagnosis, levels of serum AFP, HCG and LDH, nodal disease in the abdomen, mediastinum, neck, lung metastases, spread to other visceral sites such as liver, bone and brain, and on treatment details such as previous therapy [9
]. Data for 39 patients were excluded because of missing values on the risk factors age, and lung, liver, bone and brain metastases. The endpoint was overall survival, calculated from the beginning of chemotherapy.
Statistical analyses
We assessed the performance of the Kollmannsberger tree by applying it to the 456 poor prognosis patients in the IGCCCG database. Two-year overall survival was calculated using the KaplanMeier method.
IGCC tree
Furthermore, a new tree was developed in the poor prognosis patients in the IGCCCG database and the result compared with the Kollmannsberger tree to evaluate its structure, that is the selection and hierarchy of the risk factors. We will refer to this new tree as the IGCC tree. For the development of this tree we used the same candidate risk factors and coding as Kollmannsberger et al. [16]. We first determined the risk factor that best split the data into two subgroups, leading to the largest decrease in prediction error. Splitting continued until a group reached a minimum size of five, or until no further improvement in discrimination could be made, based on the loss in exponential log-likelihood. The full tree, which might be too complex and overfit, was pruned using 10-fold cross-validation. All trees within one standard error of the lowest cross-validated prediction error were considered as equivalent. A final tree was selected from these equivalent trees [19
, 20
, 25
].
Modelling was performed with S-plus version 2000 using the RPART library, which contains a recursive partitioning method for survival data. The RPART library (rpart2.zip) and manual (rpart2doc.zip) can be found at http://www.stats.ox.ac.uk/pub/SWin
For comparison we fitted a Cox regression model using the same risk factors in the poor prognosis patients in the IGCCCG database. All risk factors were entered in the model and the final model was obtained with a backward stepwise selection procedure using a P-value of 0.05.
Predictive performance
We determined the discriminative ability, to indicate the predictive performance of the Kollmannsberger tree, the IGCC tree and the Cox regression model. The discriminative ability indicates how well a model can distinguish between patients with different survival expectations and was measured by a concordance statistic (c-statistic). For binary outcomes, the c-statistic is identical to the area under the ROC curve [26, 27
]. The c-statistic for survival data estimates the probability that for a randomly chosen pair of patients, the one having the higher predicted survival is the one who survives longer [26
]. The C-statistic varies between 0.5 and 1.0 for sensible models. A predictive model with a c of 0.5 has no predictive value, while a model with a c of 1.0 discriminates perfectly between patients differing in survival. c-statistics were computed for the Kollmannsberger tree, when applied to the Kollmannsberger patients or the IGCCCG patients, for the IGCC tree and for the Cox regression model. The steps taken in the development of the IGCC tree were internally validated by taking 100 random bootstrap samples. The development process of the tree was repeated for every bootstrap sample and the resulting tree tested on the original sample, to estimate and correct for the optimism in discriminative ability [28
, 29
]. The original sample hence served as the test sample for models developed in the bootstrap samples. The Cox regression model was validated according to the same procedure. The standard error of the corrected c-statistic was taken from the empirical distribution of the c-statistics in the test sample. This standard error was used to calculate the 95% CI of the optimism-corrected c-statistic.
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
Both branches were split further according to primary site, resulting in two final groups in the no visceral metastases branch, with 2-year survival of 63% and 50%, and in the visceral metastases branch in one final group with 2-year survival of 33% and one further subgroup with 2-year survival of 56%; the last final groups from this branch were defined by the presence of abdominal mass, with similar 2-year survival (56% and 53%). As can be seen by comparing Figure 1B and A, respectively, the 2-year survival estimates for the five final groups identified in the poor prognosis patients in the IGCCCG database by the Kollmannsberger tree were less extreme than in the original Kollmannsberger tree. This was reflected by the lower discriminative ability in the IGCCCG poor prognosis data (c = 0.56; 95% CI 0.490.64) compared with the original data (c = 0.63; 95% CI 0.560.70).
The newly developed IGCC tree is presented in Figure 1C. Trees with two to six groups gave equivalent performance based on the loss in exponantial log-likelihood. However, a tree with five final groups was chosen for fair comparability with the Kollmannsberger tree. The 2-year survival ranged from 38% to 73%. The principal determinant of survival was the total number of metastases, where three or fewer metastases resulted in a subgroup with a 2-year survival of 61% and the presence of more than three metastases in a final group with a 2-year survival of 38%. The next split was made by primary site, resulting in a subgroup of patients with testis as primary site and a 2-year survival of 64% and a final group of patients with mediastinal primary site and a 2-year survival of 49%. A further distinction was made by the size of an abdominal mass, resulting in a 2-year survival of 73% for patients with a mass 10 cm, and 56% for a mass >10 cm. Finally, the presence of lung metastases resulted in two final groups with a 2-year survival of 67% for patients without lung metastases and 42% for patients with lung metastases.
Thus, both the Kollmannsberger tree and the new IGCC tree selected primary site and abdominal mass as important risk factors, although the Kollmannsberger tree selected the presence of abdominal mass rather than size. Furthermore, the presence of lung metastases was used in the IGCC tree, but not in the Kollmannsberger tree. The apparent discriminative ability of the new IGCC tree was similar to the discriminative ability of the Kollmannsberger tree, with a c-statistic of 0.63 (95% CI 0.620.72). Internal validation revealed an optimism in c-statistic of 0.04, leading to an optimism-corrected estimate of c = 0.59 (95% CI 0.540.63) for the new IGCC tree when applied to future patients similar to those included in the IGCCCG database.
The Cox regression model selected the risk factors primary site, presence of abdominal mass, size of abdominal mass, total number of metastases, AFP and tumour markers combined. The discriminative ability of the Cox regression model was slightly higher, with a c-statistic of 0.64, but decreased to 0.61 after correction for optimism.
![]() |
Discussion |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
In our case, the selected risk factors were rather similar in the trees developed with the Kollmannsberger data and the IGCCCG poor prognosis data (presence versus size of abdominal mass and presence versus number of metastases). The structure of the trees, however, was very different. Since primary site was an important risk factor in patients with and without visceral metastases in the Kollmannsberger tree, both risk factors can be interpreted statistically as main effects. In the IGCC tree, primary site occurred only once, and no main effects were present except for the number of metastases. Furthermore, the trees fitted in the bootstrap samples varied in size, the smallest tree having only two groups and the largest tree 20. A tree size of four was most prevalent (14% of cases). The flexibility in structure and tree size led to optimism in the performance of the tree developed on the IGCCCG poor prognosis data, where the c-statistic was expected to decrease from 0.63 to 0.59 according to a bootstrap validation technique. Hence, the flexibility of tree modelling may have more cons than pros in small data sets, that is datasets with relatively few deaths. Larger datasets are required for reliable application of tree modelling.
The discriminative ability of a Cox regression model was slightly higher than the regression tree models with a c-statistic of 0.61 at internal validation. A Cox regression model was not available for the Kollmannsberger data. Should such a model be applied to the Kollmannsberger data, the selection and the number of risk factors might differ. However, the structure is likely to be similar for a given selection of risk factors, since we usually fit main effects only in a regression model. In relatively small datasets, this may be the most sensible, since we have insufficient statistical power to identify important interaction terms [30].
To assess whether the Kollmannsberger tree can be generalised to other patients, we applied it to poor prognosis patients from the IGCCCG dataset. The comparison of the Kollmannsberger and the IGCC tree was, however, limited by the differences in the two datasets. Although the distribution of risk factors in the Kollmannsberger and IGCCCG poor prognosis data was largely similar, there was a difference in 2-year survival (72% and 56%, respectively). These differences may reflect the different time periods in which the data were collected (Kollmannsberger 19841997, IGCCCG 19751990). Owing to improved treatment strategies, survival has increased over time [4, 31
]. Patients in the Kollmannsberger study were treated with regimens of either PEB or PEI. Although all poor prognosis patients in the IGCCCG database were treated with cisplatin-based chemotherapy, patients from the late 1970s and early 1980s were probably treated with the cisplatinvinblastinebleocymin regimen rather than PEB. The differences between the populations suggest that a more recent population of poor prognosis patients might be more suitable to assess the generalisability of the Kollmannsberger tree. Ideally, to assess the differences between the Kollmannsberger and the IGCCCG poor prognosis data and to make an honest comparison between both trees, the new IGCC tree should be applied to the Kollmannsberger data.
Better performance might be achieved by adding stronger risk factors that have not been used before in the classification of patients with germ cell cancer. Besides pre-treatment characteristics, the rate of tumour marker decline during the first two cycles of chemotherapy has been identified as an important risk factor. Rate of tumour marker decline predicted outcome in 189 patients, independent of risk status as defined by the IGCC classification, especially in poor prognosis patients [33]. Similar results were found in 139 poor prognosis patients in a previously conducted study [32
]. These results will be validated with data from a multicentre, randomised clinical trial [33
]. Furthermore, promising research is being carried out on the prognostic value of molecular and genetic markers. Knowledge on the role of such markers will not only allow for a better understanding of the development and progression of testicular gem cell cancer, but may also lead to a more refined assessment of prognosis and better management of germ cell tumours [34
, 35
].
In conclusion, survival of IGCC poor risk patients in the present day may have improved compared with the historical IGCCCG data. This justifies the investigation of poorer risk subgroups, although the difficulties in evaluating new treatment approaches through randomised trials in these small groups must be acknowledged. The performance of the current regression trees was unsatisfactory. The currently available risk factors are not strong enough to clearly identify subgroups among poor prognosis patients with nonseminomatous germ cell cancer, who comprise a small subgroup (15%) of metastatic germ cell tumour patients. A new model might incorporate molecular and genetic markers in addition to the risk factors currently incorporated in the IGCC classification. We suggest the use of Cox regression for the construction of such a new model, rather than tree modelling. This method is proven to be more stable and gives less optimistic results, especially in smaller datasets. Tree modelling can give insight into possible interactions between risk factors provided sufficient data is available, but should be restricted to exploratory analyses.
![]() |
Acknowledgements |
---|
Received for publication January 19, 2004. Revision received April 22, 2004. Accepted for publication April 23, 2004.
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
2. Hartmann JT, Kanz L, Bokemeyer C. Diagnosis and treatment of patients with testicular germ cell cancer. Drugs 1999; 58: 257281.[ISI][Medline]
3. Sonneveld DJ, Hoekstra HJ, van der Graaf WT et al. Improved long term survival of patients with metastatic nonseminomatous testicular germ cell carcinoma in relation to prognostic classification systems during the cisplatin era. Cancer 2001; 91: 13041315.[CrossRef][ISI][Medline]
4. Steele GS, Richie JP, Stewart AK, Menck HR. The National Cancer Data Base report on patterns of care for testicular carcinoma, 19851996. Cancer 1999; 86: 21712183.[CrossRef][ISI][Medline]
5. de Wit R, Roberts JT, Wilkinson PM et al. Equivalence of three or four cycles of bleomycin, etoposide, and cisplatin chemotherapy and of a 3- or 5-day schedule in good-prognosis germ cell cancer: a randomized study of the European Organization for Research and Treatment of Cancer Genitourinary Tract Cancer Cooperative Group and the Medical Research Council. J Clin Oncol 2001; 19: 16291640.
6. Hinton S, Catalano PJ, Einhorn LH et al. Cisplatin, etoposide and either bleomycin or ifosfamide in the treatment of disseminated germ cell tumors: final analysis of an intergroup trial. Cancer 2003; 97: 18691875.[CrossRef][ISI][Medline]
7. Bokemeyer C, Kollmannsberger C, Meisner C et al. First-line high-dose chemotherapy compared with standard-dose PEB/VIP chemotherapy in patients with advanced germ cell tumors: A multivariate and matched-pair analysis. J Clin Oncol 1999; 17: 34503456.
8. Bokemeyer C, Harstrick A, Beyer J et al. The use of dose-intensified chemotherapy in the treatment of metastatic nonseminomatous testicular germ cell tumors. German Testicular Cancer Study Group. Semin Oncol 1998; 25: 2432; discussion 4528.
9. IGCCCG. International Germ Cell Consensus Classification: a prognostic factor-based staging system for metastatic germ cell cancers. International Germ Cell Cancer Collaborative Group. J Clin Oncol 1997; 15: 594603.[Abstract]
10. Kaye SB, Mead GM, Fossa S et al. Intensive induction-sequential chemotherapy with BOP/VIP-B compared with treatment with BEP/EP for poor-prognosis metastatic nonseminomatous germ cell tumor: a randomized Medical Research Council/European Organization for Research and Treatment of Cancer study. J Clin Oncol 1998; 16: 692701.[Abstract]
11. Fizazi K, Zelek L. Is one cycle every three or four weeks' obsolete? A critical review of dose-dense chemotherapy in solid neoplasms. Ann Oncol 2000; 11: 133149.[Abstract]
12. Motzer RJ, Mazumdar M, Gulati SC et al. Phase II trial of high-dose carboplatin and etoposide with autologous bone marrow transplantation in first-line therapy for patients with poor-risk germ cell tumors. J Natl Cancer Inst 1993; 85: 18281835.[Abstract]
13. Motzer RJ, Mazumdar M, Bajorin DF et al. High-dose carboplatin, etoposide, and cyclophosphamide with autologous bone marrow transplantation in first-line therapy for patients with poor-risk germ cell tumors. J Clin Oncol 1997; 15: 25462552.[Abstract]
14. Schmoll HJ, Kollmannsberger C, Metzner B et al. Long-term results of first-line sequential high-dose etoposide, ifosfamide, and cisplatin chemotherapy plus autologous stem cell support for patients with advanced metastatic germ cell cancer: an extended phase I/II study of the German Testicular Cancer Study Group. J Clin Oncol 2003; 21: 40834091.
15. Motzer RJ, Sheinfeld S. Poor-risk testicular cancer and high-dose chemotherapy. J Clin Oncol 2003; 21: 40734074.
16. Kollmannsberger C, Nichols C, Meisner C et al. Identification of prognostic subgroups among patients with metastatic IGCCCG poor-prognosis germ-cell cancer: an explorative analysis using cart modeling. Ann Oncol 2000; 11: 11151120.[Abstract]
17. Marshall RJ. The use of classification and regression trees in clinical epidemiology. J Clin Epidemiol 2001; 54: 603609.[CrossRef][ISI][Medline]
18. Radespiel-Tröger M, Rabenstein T, Schneider HT, Lausen B. Comparison of tree-based methods for prognostic stratification of survival data. Artif Intell Med 2003; 28: 323341.[CrossRef][ISI][Medline]
19. LeBlanc M, Crowley J. Relative risk trees for censored survival data. Biometrics 1992; 48: 411425.[ISI][Medline]
20. Davis RB, Anderson JR. Exponential survival trees. Stat Med 1989; 8: 947961.[ISI][Medline]
21. Crichton NJ, Hinde JP, Marchini J. Models for diagnosing chest pain: is CART helpful? Stat Med 1997; 16: 717727.[CrossRef][ISI][Medline]
22. Harrell FE Jr. Regression Modeling Strategies with Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer 2001.
23. Nichols CR, Catalano PJ, Crawford ED et al. Randomized comparison of cisplatin and etoposide and either bleomycin or ifosfamide in treatment of advanced disseminated germ cell tumors: an Eastern Cooperative Oncology Group, Southwest Oncology Group, and Cancer and Leukemia Group B Study. J Clin Oncol 1998; 16: 12871293.[Abstract]
24. Nichols CR, Williams SD, Loehrer PJ et al. Randomized study of cisplatin dose intensity in poor-risk germ cell tumors: a Southeastern Cancer Study Group and Southwest Oncology Group protocol. J Clin Oncol 1991; 9: 11631172.[Abstract]
25. Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and Regression Trees. Belmont, CA: Wadsworth 1984.
26. Harrell FE Jr, Lee KL, Califf RM et al. Regression modelling strategies for improved prognostic prediction. Stat Med 1984; 3: 143152.[ISI][Medline]
27. Grunkemeier GL, Jin R. Receiver operating characteristic curve analysis of clinical risk models. Ann Thorac Surg 2001; 72: 323326.
28. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. London: Chapman & Hall, Inc 1993.
29. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996; 15: 361387.[CrossRef][ISI][Medline]
30. Steyerberg EW, Eijkemans MJ, Harrell FE Jr, Habbema JD. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 2001; 21: 4556.
31. Bokemeyer C. Current trends in chemotherapy for metastatic nonseminomatous testicular germ cell tumors. Oncology 1998; 55: 177188.[CrossRef][ISI][Medline]
32. Fizazi K, Kramar A, Culine IC et al. The early decline of tumor markers predicts outcome in poor prognosis non-seminomatous germ-cell tumors (NSGCT): results of a multi-institution study. Proc Am Soc Clin Oncol 2002; 21: 185a (Abstr 739).
33. Mazumdar M, Bajorin DF, Bacik J et al. Predicting outcome to chemotherapy in patients with germ cell tumors: the value of the rate of decline of human chorionic gonadotrophin and alpha-fetoprotein during therapy. J Clin Oncol 2001; 19: 25342541.
34. Heidenreich A, Srivastava S, Moul JW, Hofmann R. Molecular genetic parameters in pathogenesis and prognosis of testicular germ cell tumors. Eur Urol 2000; 37: 121135.[ISI]
35. Donadio AC, Bosl GJ. The future of therapy for nonseminomatous germ cell tumors. Chest Surg Clin N Am 2002; 12: 769789.[Medline]