Validation, ILC and validation report

An essential part of the development phase of any OECD Test Guideline (TG) is the validation of the method(s) under consideration and, hence, the collation and/or the generation of a relevant data set demonstrating the reproducibility/reliability, applicability/ relevance and transferability of a TG in development (OECD GD 34, 2005).

Validation of a test method is the process of establishing reliability together with the relevance of a method by following scientifically sound principles. Reliability is defined as the extent of reproducibility of results from a test within and among laboratories over time when performed using the same harmonised protocol (i.e. Standard Operating Procedure SOP). The relevance of a test method describes the relationship between the test and the effect on the target species and whether the test method is meaningful and useful for a defined purpose, with the limitations identified and includes predictivity. In brief, it is the extent to which the test method correctly measures or predicts the (biological) effect of interest, as appropriate. Regulatory need, usefulness and limitations of the test method are aspects of its relevance. New and updated test methods need to be both reliable and relevant, i.e., validated.

As a first step, pre-validation represents a proof-of-principle mostly performed by one laboratory, often having developed the first SOP and using the submitted test method for a couple of different materials to show that the test method is reproducible and applicable to different materials. SOPs describe an instruction on how to perform a test method in a comparable, uniform way. It helps to facilitate the understanding of this procedure and furthers the achievement of high-quality, repeatable and reliable test results. An SOP forms the basis of pre-validation and validation of a method that all together leads to the development of a harmonised test method resulting in an OECD TG.

The purpose of the validation process is to determine the performance characteristics, usefulness, and limitations of a test method that is under consideration for use in a regulatory context, and to determine the extent to which results from the test can be used for hazard identification, and to support risk assessments or other health and safety decisions. Validation is distinct from test method development OECD GD 34 (2005).

Important factors and workflow in the validation process are shown in Figures 1 and 2 of the OECD GD 34 (2005) but brief topics such as how to choose test materials, how to define negative and positive controls, the number of chemicals preferably evaluated, test duration, dose selection, minimum requirements and how to use blind samples in this process. Often inter- and intra-laboratory comparisons (ILCs) are performed by different laboratories.

Sometimes it can be needed to include several methods in one TG and, hence, in the validation study in order to be able to directly compare the pros and cons of each method and to find consent. Depending on the outcomes and results of the validation study, methods will probably need to be excluded again based on their applicability (i.e. the classes and types of test substances that can and cannot be tested).

Sufficient statistical expertise should be available to ensure the appropriate design of the validation studies and evaluation of resulting data. This knowledge will aid in the selection of appropriate statistical procedures, and the development of appropriate decision criteria for the data generated by the test and the effect to be predicted, modelled or measured, and can help with communicating the results of the study OECD GD 34 (2005).

The results of the validation are described in detail in a validation report that accompanies the OECD TG and that is publicly available in the document series on Testing and Assessment.

• Abandon the idea of producing the best possible assay that covers all and everything. Accept limitations. Make your work understandable, as simple as possible to avoid extra rounds of comments. Identify the essential measurands (what is a priority to address in the TG and what could be considered as a lower priority), and the minimum requirements for study design to maintain the reliability and relevance of the test method.

• Select a broad variety of chemicals for the validation to identify limitations and to optimise the applicability domain of the select test method(s) while including reference chemicals and/or benchmark materials (i.e. data-rich chemicals or materials for the particular test methods).

• The validation process implies the collation and/or the generation of an important amount of data, sometimes leading to unforeseen technical difficulties and/or contradictory results. One should note that exchanges with international experts on technical difficulties, experimental data generation and data interpretation are time and resource-consuming but indispensable.

• Having participants from all over the world in the inter-laboratory comparison can help for the international acceptance of the TG.

The validation status needs to be described in the section on the maturity of science in the Standard Proposal Submission Form (SPSF). See Drafting an SPSF & Project definition phase at OECD for further information on the timeline. The validation of a test method is the main part of the development phase at OECD with the inter-laboratory comparison often being the most time-consuming part. The latter implies the recruitment of voluntary laboratories distributed across the world, and troubleshooting management of issues resulting in further harmonisation needs (e.g. insufficient information furnished in the Standard Operating procedure developed). As described in the Validation report section, the evaluation of the results of the validation study and their compilation into the report may also be time-consuming.

An intra-laboratory comparison enables the within-laboratory comparison of results obtained using a test method and its associated Standard Operating Procedure (SOP). It is associated with the initial assessment if the relevance and reliability of a test method protocol (i.e. Prevalidation).

An inter-laboratory comparison (ILC) enables the between-laboratory comparison of results, which is associated with the broad assessment of the relevance and reliability of the test method and its SOP for finalisation.

The ILC usually contains these important steps:

- Conceptualisation and design of validation based on the result from Prevalidation (e.g. minimisation of protocol variables to test).
- Call for contribution and participation of laboratories, one should note that the whole process of finding GLP laboratory participants can take time. The OECD Test Guidelines Programme secretariat and National Coordinators of the leading Member country can support finding participating laboratories by circulating the call in their networks (see Development phase at OECD).
- Harmonisation of the experimental setups in the participating laboratories.
- Material purchase.
- ILC experiments.
- Data collection and analysis.

• Plan for sufficient time (around 1 year if well prepared) for your inter-laboratory comparison (ILC).

• Regulatory acceptance is dependent on the outcome of scientific validation. The acceptance process will be greatly facilitated by the early involvement of regulatory authorities in the planning and design of the validation study so that it will address areas of interest and concern to the respective regulators (OECD GD 34, 2005). 

• The call for participation should be circulated early on and broadly distributed. It is not always easy to find enough suitable laboratories that are participating, especially since many of them are participating at their own costs. Starting with a couple of laboratories more than maybe actually needed helps a lot since performing an ILC takes time and maybe not all laboratories can continue this the entire time (e.g. due to changes in priority, lack of funding, changes of personnel).

• Be informed on interests in the different Member countries on the Test Guideline (TG) you want to develop and seek experts from many countries. Having as many different countries participating in the ILC helps to harmonise the test methods, to take local needs and challenges into account and to have a broad and open discussion. Having a broad coverage of international experts in the experiments facilitates the discussion and commenting rounds.

• Use the JRC repository for materials (relevant for nanomaterials) or check for established harmonised reference databases (e.g. sensitization).

• The material/chemical for all participating laboratories should be coming from one batch (to enable comparison).

• Often overlooked is the fact that validation cannot be done using the same materials that have been used in the pre-validation process to establish the method, i.e. training and test sets should be independent. TGs and GDs also need to contain information on when changes to an existing SOP will have to be followed by a new validation or when this can be considered a minor technical issue.

• Providing the participants with as much information as possible facilitates the performance of the experiments, the data analysis and the compilation of results.

• Start early with chemical and material distribution (even if not yet the full set of materials is available) and get the experiments started as soon as possible since once the laboratories start performing the experiments the main questions arise.

• Most of the laboratories are participating on an in-kind contribution basis. Since all of them are very busy, response times might be long and as much support as possible is needed.

• Lab training/instructions at the beginning of the ILC are helpful in reducing uncertainties and misunderstandings (of practical issues).

Around 1 year is required for an inter-laboratory comparison (ILC) if well prepared and depending on the test method under validation, but this can easily expand to more time depending on the required harmonisation efforts and the complexity of experimental setups, the availability of material supplies, changes in participating laboratories etc.

The validation report should cover among others the intention of the validation study, the parameters under which it was performed, the results of the study and the discussion and compilation of outcomes, including an assessment of whether the conclusions of the study are supported by the data, and whether the proposed protocol is adequate and appropriate. The validation report facilitates acceptance of the test method. It will be published as supporting information for the Test Guideline (TG) in the Series on Testing and Assessment.

Extracted from the NanoHarmony Training Material – From science to standards and harmonised OECD Test Guidelines
‘The process of development of OECD documents’

The following table summarises what the National Coordinators will be looking at:

What information matters?	What will the national coordinator be looking at?
Relevance of the methodology	What does the assay do/measure, and what is the assay relevant for? What is the predictive capacity?
Limitations of the assay/technology	Chemical aspect covered Types of chemicals tested, and their expected activity/MoA Areas of uncertainty
Data interpretation/prediction model	Is it well-defined? Are templates available for the reporting?
Reproducibility	How reliable results are within and between labs?

• Prepare the validation report early on. Start writing this already at the beginning of the interlaboratory comparison (ILC) in order to identify needs for validation and to keep track of all decisions and steps made towards validation. Structuring the validation report before starting the ILC helps to define the ILC.

• The report should report the design of the ILC, and describe the procedures and results of the ILC (e.g. laboratory participation, study design, chemicals tested, analysis of the results, and conclusions). All information and data generated during the validation of the test method must be archived and made available for independent review. 

• No specific template is provided by the OECD. However, validation reports from other Test Guidelines (TGs) are publicly available and can be used as a source of inspiration.

• Try to publish the scientific developments and the validation study performed in peer-reviewed journals prior to the submission of the OECD TG draft. This can facilitate the discussion at OECD level by supporting the solidity of the performed validation study.

Drafting of the validation report should be started at the beginning of the validation study. It can be time-consuming to find an agreement on the synthesis of the outputs of the validation study from the (Ad Hoc) Expert Group members supporting the project and the participants of the validation study. Consult Development phase at OECD for further information on the timeline of the overall phase. The draft validation report needs to be ready for submission with the draft Test Guideline (TG) for commenting until its approval by the Working Party of National Coordinator of the Test Guidelines Programme (WNT) during its annual meeting in April. Similarly to the TG, the validation report is endorsed by the Chemicals and Biotechnology Committee to be published in the Series on Testing and Assessment a few months after its approval by the WNT.