Proficiency tests (PTs) have a central function in the quality monitoring of laboratories. When auditing a laboratory, it can be checked whether the basics for good analytical quality are in place and whether the routine procedures in the laboratory enable this quality. However, the actual analytical performance can only be checked in comparison with other laboratories and/or with reference values. For this reason, the state of Baden-Württemberg commissioned the Institute for Sanitary Engineering, Water Quality and Waste Management in 1987 to carry out proficiency tests for the analysis of wastewater and groundwater. In the meantime, these proficiency tests are known far beyond the borders of Baden-Württemberg. Successful participation in proficiency tests is mandatory for accreditation according to the "Ordinance of the Ministry of the Environment and Transport on Expert Bodies in Water Management" and for the listing of laboratories according to §15 of the German Drinking Water Ordinance, but many laboratories also use these proficiency tests voluntarily as proof of competence for accreditation bodies and customers.
DIN 38402 - A45 / EN ISO/IEC 17043 / ISO 13528
EN ISO/IEC 17043 describes the globally recognised requirements for proficiency tests. The technical requirements relate in particular to personnel, equipment and accomodation, the development of PT schemes, performance and evaluation of PT schemes, reporting, communication with participants and confidentiality. The management requirements are largely identical to those of ISO 9001 or EN ISO/IEC 17025.
ISO 13528 describes statistical procedures for the evaluation of PTs.
DIN 38402 - A45 (June 2014) regulates details for the performance of proficiency tests in the field of water analysis. Specifications are made for the performance of the proficiency tests themselves (sample preparation, stability and homogeneity tests, measures to avoid collusion, approved analytical methods, determination of the conventional true value, sample distribution) and, last but not least, methods for the evaluation and assessment of the proficiency tests are specified (Hampel estimator, Q-method, zU-scores).
DIN 38402 - A45 is compatible with the international specifications of EN ISO/IEC 17043 and ISO 13528 and is thus only to be understood as a concretisation for the water sector.
The interlaboratory comparisons of AQS Baden-Württemberg are carried out in accordance with the standards and guidelines mentioned here.
As a rule, the Institute of Sanitary Engineering provides at least 2 proficiency tests (PTs) for wastewater analysis, 3 PTs for drinking water analysis and one PT (in German language only) for operational analysis at wastewater treatment plants every year. The PT samples are usually based on a real matrix. The choice of parameters in the wastewater sector is based on the specifications of the Water Module of the Working Group of the Federal States on Water and in the drinking water sector on the Drinking Water Ordinance (and whatever else is currently of interest in the drinking water sector). The parameters that are important for self-monitoring are tested in the PTs for operational analysis at wastewater treatment plants.
The PTs for wastewater are usually carried out as "PTs across the German Federal States". In accordance with agreements in the Working Group of the Federal States on Water, a PT is offered and recognised in an identical manner by several organisers nationwide. Execution and evaluation are based on the leaflet A-3 of the Working Group of the Federal States on Water Issues (LAWA) (AQS leaflets for water, wastewater and sludge analysis).
The drinking water PTs are carried out in cooperation with the Institute for Hygiene and Environment of the Free and Hanseatic City of Hamburg. PTs with special parameters in drinking water are organised by AQS Baden-Württemberg in cooperation with IWW Water Centre in Mülheim/Ruhr.
The PTs for operational analysis at wastewater treatment plants are commissioned by the Minsitry of the Environment, Climate Protection and the Energy Sector Baden-Württemberg.
The laboratories usually receive three samples of the same type containing the different components in different concentrations. The AQS-BW determines the necessary stabilisation measures, container materials and concentration levels, if necessary after preliminary investigations and in consultation with the other PT organisers. The lowest concentration is chosen so that it can still be reliably determined with at least one of the methods contained in the relevant standards. The concentration range is chosen so that the monitoring limit is included in the lower range and the highest concentrations correspond to what is found in monitoring samples in reality.
The PT round will then be announced to all registered laboratories by e-mail and on the AQS Baden-Württemberg website a few weeks before the planned dispatch date, with details of the relevant methods. A deadline is set for registration for this PT round.
At the Institute for Sanitary Engineering, stock solutions for all parameters and all concentration levels are prepared with great control effort. The samples are prepared from these stock solutions via a defined system of dilution steps.
The amount of sample provided is calculated in such a way that, on the one hand, it is sufficient for multiple determinations, but, on the other hand, it is so small that, if possible, an excessively large special effort for the proficiency test samples in the laboratory will not be possible.
The samples are packed with polyurethane mouldings in suitable boxes and sent by express parcel service. The samples are thus usually in the laboratories the next day, so that the examinations can begin the same week.
In addition to a covering letter, the packages are accompanied by a leaflet that draws attention to frequently made mistakes. This leaflet should serve as a checklist before submitting the results.
The results shall be submitted electronically via data entry on the Internet. When entering the data, the analytical methods used and the result values must be entered in the required unit of measurement. The specification of the analytical method offers the possibility of a specific evaluation. On a voluntary basis, the measurement uncertainty of the result can also be indicated. In this case, the laboratory is additionally provided with an evaluation of the measurement uncertainty.
It is very important for the organisation of such a proficiency testing scheme that all deadlines are met. This is especially necessary to ensure that samples can be provided in sufficient numbers. For organisational reasons, there are sometimes still samples left for individual latecomers. However, this cannot be guaranteed for obvious reasons. The time available for evaluation is also very short, so that the AQS-BW must also insist on the results being sent in on time.
Following the evaluation, result evaluation sheets are prepared by the provider for all laboratories and sent together with the evaluation documentation. These result evaluation sheets contain the values submitted by the laboratory and the assigned values. Values that are within the tolerance limits are marked as successful.
The proficiency tests of AQS Baden-Württemberg are evaluated according to the specifications of DIN 38402 - A45. The evaluation is based on the deviation from a "conventional true value" with the help of the so-called zU scores.
The first step is to calculate the reproducibility standard deviation sR for each concentration level. This is done with the help of the Q method, a method of so-called "robust statistics". Among other things, this value is necessary for the calculation of the "robust mean" with the help of the Hampel estimator. This "robust mean" is usually used as the "conventional true value", i.e. the assigned value. In justified cases, however, the PT manager may also deviate from this and specify initial sample weights or reference values determined in another way as the assigned value.
For each measured value, a z-score is first calculated according to the following formula:
z-score = (measured value - assigned value) / standard deviation for proficiency assessment
The standard deviation for proficiency assessment may differ from the calculated reproducibility standard deviation if
- upper and lower limits for the comparative standard deviation have been set in advance and these have been exceeded or fallen short of. In this case, the exceeded or fallen short of limit is set as the target standard deviation or if
- a cross-concentration evaluation is carried out with the help of the variance function described in DIN 38402 - A45. In this case, the value of the variance function at the corresponding concentration value is used as the target standard deviation.
A combination of both cases can also occur.
In order to compensate for unfairness due to the skewed distribution of the data occurring at low concentrations, zU-scores are calculated from the z-scores, as described in DIN 38402 - A45.
Each measured value is now evaluated on the basis of its zU-score. Values between -2 and +2 are considered successful by definition. Values in the range 2 < |z-score| < 3 are classified as "questionable" and values in the range |z-score|>3 as "unsatisfactory".
The valuation of each individual value is noted on the data assessment sheet.
A parameter is considered to have been successfully determined in the proficiency test if more than half of the values have been classified as "successful".