Evaluation of the performance of designated screening centers of the Programme québécois de dépistage du cancer du sein (PQDCS): description of the methods
The evaluation of the Programme québécois de dépistage du cancer du sein (PQDCS) is based primarily on the analysis of performance indicators as defined in the PQDCS guidelines. Methods were initially developed to measure performance indicators at the provincial level and then in individual regions. The goal of this report is to develop an approach for the measurement and comparison of performance indicators at the screening centre level.
The indicators selected to evaluate the performance of individual centres are: the detection rate, recall rate, and number of false positives per screen detected cancer. These indicators are estimated separately for initial and subsequent screens. The percentage of in situ cancers, the percentage of small invasive cancers, and the percentage of lymph nodenegative invasive cancers were also selected. To evaluate a centre's performance, these indicator measurements were compared to the measurements for Quebec as a whole, as well as with the indicator targets set out in the PQDCS guidelines.
Comparing the indicators measured for a given screening centre with those measured for Quebec as a whole raises two challenges of a methodological order. At first, women characteristics differ from centre to centre. Also, certain centres perform only a limited number of mammograms per year. It was therefore necessary to develop a method for measuring performance indicators adjusted for the characteristics of the women screened, as well as a method for calculating a confidence interval around adjusted indicators. Finally, it was also necessary to develop a method for presenting results that would provide a better understanding and analysis of screening centre performance.
Standardization is an adjustment method frequently used to account for variations in the characteristics of individuals when comparing two or more sub-populations. In the literature, two primary standardization methods are used: direct standardization and indirect standardization. Direct standardization presents significant limitations in terms of the statistical stability of estimates when the number of subjects in individual sub-populations is relatively small. However, indirect standardization does not lend itself to the direct comparison of two sub-populations and only allows for the comparison of sub-populations with the total population. Still, given the large number of potential confounding factors and the relatively small sub-populations, indirect standardization was selected. Indirect standardization consists of comparing the observed value of an indicator in a given centre (taking into account its clientele) with the expected indicator to find with the same clientele if the centre's performance were equal to that observed across the PQDCS. Indirect standardization provides a means of establishing an observed rate/expected rate ratio (X/A). Since the number of screening mammograms performed in Quebec centres is limited, several indicators need to be calculated using data from more than one year. The method presented here is based on data from three years. Also, in order to account for any random variation (statistical variation) in the number of screens performed by screening centres, a 95% confidence interval is calculated. The indicators selected are proportions (and one odds). Binomial distribution is therefore used to calculate the confidence interval of the ratio X/A.
In order to compare the performance of each individual centre with that of Quebec as a whole, each observed rate/expected rate ratio (and its confidence interval) is multiplied by the value of the indicator calculated for Quebec. For individual screening centres, this calculation produces an “adjusted” indicator value. Moreover, a means of graphically representing adjusted indicators was developed in order to facilitate the analysis and interpretation of results. The first approach is intended to demonstrate the evolution of a centre's performance over time. The second approach is designed to simultaneously present the performance of all centres in a given period and a specific indicator.
The methods presented in these pages are now being used in Quebec to evaluate the performance of screening centres. Eventually, these measures may be used to evaluate the performance of centres in which women with abnormal screening mammograms undergo further investigation.
