Development at OECD

The development phase at OECD officially starts once the Standard Project Submission Form (SPSF) of the project is formally accepted by the Working Party of National Coordinators of the TGs programme (WNT) and integrated into the OECD Test Guidelines Programme (TGP) work plan. But to some extent, this phase can run in parallel with the pre-OECD and project definition at OECD phases and is finalised after the commenting, revision and approval phase (see Timelines section). This is generally the longest phase once the project enters the OECD.

Awareness of the steps and timings of the procedure for standardisation and development is an obvious benefit for projects aiming to deliver OECD Test Guidelines (TGs) and Guidance Documents (GDs), however, there is no one strategy to move smoothly across the development phase at OECD. For example, different types of test methods may require different scrutiny in validation (OECD GD 34, 2005), e.g. a TG on physicochemical properties may be relatively straightforward, whereas a toxicity test requires both validation on toxicological endpoints as well as validation on physicochemical characterisation of the test compound inside an organism. Furthermore, GDs generally require less scrutiny in validation.

TG developments are often prolonged by missing or insufficient funding. Funding is not only needed for test development and drafting a TG, although this relies mainly on in-kind contributions. Funding is also needed for validation by inter-laboratory comparison ) For a smooth development process, it is thus important to secure funding and available expertise over the whole process. This phase is typically longer than the normal three-year research project envisioned for the development of a test method as it includes validation steps, while the workload is constant and very much like a research project. The Project Lead institutions benefit also from regular inputs from the (Ad Hoc) Expert Group supporting the project, which is not the case in research projects. 

Indeed, almost all OECD projects approved by the WNT that are related to the development of TGs and/or GDs have an associated official OECD Expert Group, typically referred to as an Ad Hoc Expert Group. Working parties of the Chemicals and Biotechnology Committee (CBC), such as the WNT and the Working Party on Manufactured Nanomaterials (WPMN), also have more general standing Expert Groups that focus on particular areas and groups of projects. It is also possible to use these existing OECD expert groups to support the development of a relevant TG/GD project.  The (Ad Hoc) Expert Group provides advice on scientific and technical aspects and broader issues, including regulatory, related to the project and supports the development of TG/GD drafts prior to wider circulation, typically by providing comments (See OECD Expert Groups & Commenting aspects).

Regular expert consultation can occur during dedicated (Ad Hoc) Expert meetings(online or in person), organised or not with the support of the OECD Secretariat. Experts coming from various geographical regions around the world will ensure the wide acceptance and applicability of the selected method(s). Other than helping to reach a consensus, they can also provide support, and can for example be used as a source for finding participants for the interlaboratory comparisons (ILCs). It is important to document the (Ad Hoc) Expert group meetings, as the decisions taken in such meetings can be used for answering comments on documents received during the later phase (i.e. commenting and refinement phase at OECD). For example, pre-validation and validation needs discussed during the initial (Ad hoc) Expert Group may be reported alongside the data gaps and the regulatory application reviewed during the project definition phase at OECD.  

Before validation, it is essential to ensure again (usually via the NCs) that regulators from OECD Member countries and countries adhering to the Mutual Acceptance of Chemical Safety Data can make a decision based on the information provided by the select test method(s). The scope and applicability domain of the TG and GD need to be defined before starting the development phase to ensure validation within the anticipated scope (See Project definition phase at OECD). This will help fine-tune the purpose of the validation study towards learning about the applicability and accuracy of a proposed experimental method while identifying factors that affect the variability of the results.

A difficult challenge during the development phase at OECD is to balance the design and complexity versus the relevance and implementation. In the context of ever-evolving science, abandon the idea of producing the best possible assay that covers all and everything. Limit the science to the essential minimum to design a method that provides the necessary information to allow regulatory decisions (including mimicking real-life scenarios), while allowing a cost-effective protocol that is easy to follow and is easily transferable to different laboratories in different geographical regions. This often implies simplification of a methodology, which will often lose a lot in the representation of real-life circumstances, but gains in making the method easier to perform and easier transferable over different laboratories, while still valid for a specific proposed use.

See the extract below of the NanoHarmony Webinar - Learning lessons from the past- and knowing what your customer in which this challenge is presented by Frank Von Der Kammer from University of Vienna, in the context of the development of the OECD TG No. 318: Dispersion Stability of Nanomaterials in Simulated Environmental Media.

Project Lead institutions and sub-groups of institutions willing to participate might thus be required to run several rounds of testing for a clear and comprehensive test method protocol, together with an SOP, to be finalised and a (more) extensive ILC can be executed. Project Lead institutions should not underestimate the efforts needed to reach a consensus on (a) harmonised SOP(s) and ensuring equal set-ups in participating laboratories. It can be valuable to include inexperienced laboratories at this stage for clarification of SOP to resolve inconsistencies or highlight questions.

Important aspects of validation are topics such as how to choose test materials, how to define negative and positive controls, the number of chemicals preferably evaluated and how to use blind samples in this process. In the case of nanomaterials, it is recommended to use the JRC repository materials, as they are well-characterised nanomaterials which have been extensively worked on in EU research projects. However, other stakeholders such as contract and research organisations testing facilities stress the importance of including industrial relevant materials (e.g. mixtures of metals) during the validation process, which may facilitate the demonstration of the regulatory applicability of the TG and its acceptance and use at later stages.

Conducting interlaboratory comparisons can be quite time-consuming. Laboratories participating in the validation study should meet minimum standards in terms of the availability of competent staff, facilities, safety, animal welfare (where applicable) and quality assurance procedures. Ideally, the participating laboratories should have demonstrated competence with the general test method or similar methods and should provide, as a minimum, information on the adequacy of their data documentation procedures, such as their compliance with Good Laboratory Practices (GLP) (OECD GD34, 2005). It is recommended to select the participating laboratories from various continents around the world to confirm the broad applicability of the standardised test method. You can contact the OECD Secretariat who has a list of the previously participating institutions in the OECD TGP.

Responses from participating laboratories on results may be slow, as most of them participate at their own costs. Some institutions may face difficulties in engaging during the entire development phase at OECD, due to funding limitations and/or staff changes. It is thus recommended to split the ILC into several chemicals and/or materials sub-groups to ensure a manageable workload and allow a learning process on the difficulties faced by participants and adequate support and modifications.

Discussions on (results from an) ILC are best performed in a trusted environment and are documented in a validation report that will accompany the TG at the end (published separately as supporting information of the TG, required for all TGs). An independent peer review of the validation provides the basis for the acceptance of the select test method(s) (i.e. Trust in the validation). It reflects all the scientific work done and requires good preparation from the start of the development phase at OECD. It is also very helpful to address comments during the commenting phase at OECD, hence a large consensus of all participants in the ILC must be reached upon the validation report to make a convincing case. It is thus recommended to circulate the draft validation report (alongside the draft TG) early in the project’s expert group to receive feedback and support.

It is also during this development phase at OECD that the drafting of the TG/GD occurs. It is recommended to publish the work associated with a TG development to cement your project with peer-reviewed publications, as did Monikh, et al. in 2018 for example. "Scientific rationale for the development of an OECD test guideline on engineered nanomaterial stability.". However, OECD TG developers need to bear in mind that a TG is not a scientific text. The text needs to be explicit, e.g. by clear wordings in terms of should, must, recommend, etc. TGs and GDs always respond to a regulatory need, the proposed methodology has to be robust and reliable for governments to base their decisions, it supports the uptake of new and/or complex methods by relevant and sufficient guidance on testing and interpretation of data and their reporting via templates. OECD TGs and GDs are mindful of animal welfare, mindful of economic implications of testing and as broadly accessible as possible. Finally, to be endorsed, the final TG and GDs will require a consensus from all the OECD Member countries of the WNT. It is recommended to submit the draft TG and its validation report to WNT between June and September to allow for two commenting rounds and possible adoption by WNT in April of the next year. However, sometimes, draft TG and its validation report will require more data and formal evaluation for experimental production of the gaps. See the next phase of commenting and approval at the OECD.

• The drafting process of a Test Guideline (TG) or a Guidance Document (GD) itself needs resources mainly in expert knowledge, time for drafting and reviewing draft TG/GD and the Validation Report. It needs additional temporal and financial resources to conduct the inter-laboratory comparison (ILC), including but not limited to scouting for usable test materials, conducting pre-tests with those materials, providing test material to ILC participants, and collecting, evaluating and discussing results from the ILC participants. Time and travel budget are needed to attend expert group meetings when held in person at the OECD, for example.

• Consult regularly the (Ad Hoc) expert group(s), at least after each key process of this phase, listed below.

• Optimise the test methods and develop appropriate Standard Operating Procedures (SOPs) by designing and conducting pre-validation studies, if no data is available, and assess the intra-laboratory repeatability and reproducibility with a variety of reference chemicals & materials (i.e. performance standards).

• Test the inter-laboratory transferability of the optimised test methods and SOPs.

• Design the formal ILC validation work, based on the outcomes of the pre-validation studies and transferability to accumulate data on the relevance and reliability of the test methods and SOPs.

• Define the test material used for validation in the ILC.

• Find partners for the ILC and ensure international ILC participation from different stakeholder groups and across the world through communication using your network, the support of your National Coordinator, the OECD secretariat and the project’s expert group.

• Write the validation report that describes the procedures and outcomes of the ILC, including the overall data evaluation and subsequent conclusions on predictive capacity (accuracy), applicability domain, and performance standards keeping in mind the requirements of regulatory authorities.

• Draft the TG/GD (with potentially several iterations) and submit it to WNT.

OECD secretariat and National Coordinators of the WNT are once again the key stakeholders that support the development of the project at OECD.

Academics, governmental, industrial and standardisation bodies invited to be part of the (Ad Hoc) Expert Group(s) after the project approval by the Working Party in the OECD Test Guidelines Programme (WNT), should get involved in the validation process. Regular expert meetings will allow leading institutions to explain the work to international experts and participants, adopt criticism, and search for agreement. It eases the commenting phase as experts nominated by OECD Member countries are made aware of the strategy developed (See Project definition phase at OECD). In particular, contract research organisations (CROs) can indicate where method description is unclear, or when methods are difficult to perform. Methods may be laborious to perform or require specific (costly) equipment, or they are otherwise difficult to fit into a CRO’s business model. Knowledge of these difficulties is essential to improve the definition of the applicability domain and the predictivity of test methods. Similarly, the implication of regulatory experts in the (Ad Hoc) Expert group will help tackle broader issues, and help fine-tune the purpose of the validation study towards learning about the applicability and the accuracy of a proposed test method or testing strategy (See Regulatory needs process page).

• Ensure regular (online) meetings with (nominated) experts; to keep OECD experts from many countries and international scientists engaged in the process, and to look for consensus and support. Document meetings and decisions; e.g. discussion on the data interpretation which are mandatory for a good science. Experts nominated by the Working group of National Coordinators of the OECD Test Guidelines programme (WNT) and other OECD Working Parties have clearly good communication channels with their National coordinators, industry (through BIAC, the Business and Industry Advisory Committee to the OECD), and NGO delegates (e.g. ICAPO, the International Council on Animal Protection in OECD Programmes), and can provide insights on on-going activities in other TG/GD documents under development. The (Ad Hoc) Expert Group composition and its meeting frequency are thus key elements to provide both consistency and avoid contradictions between TGs/GDs. In the end, it is a collaborative process between scientists, regulators and industry all around the world, so you should seek the best compromise.

• If not already done, consult OECD GD1 on drafting a Test Guideline (TG)/Guidance Document (GD) (revised in 2009) and OECD GD34 on validation (under revision). Make yourself familiar with OECD TG wording and structure. The draft T G should be consistent with the same general content and format as the current existing TG. Inform yourself of the different possible structures of a TG/GD by consulting your National Coordinator and/or the OECD Secretariat and make yourself familiar with relevant TGs/GDs in the same section.

• Start writing the TG/GD and the associated validation report at an early stage to define validation steps needed using templates furnished by the OECD secretariat.

• In the context of ever-evolving science, abandon the idea of producing the best possible assay that covers all and everything. Limit the science to the essential minimum to design a method that provides the necessary information to allow regulatory decisions (including mimicking real-life scenarios), while allowing a cost-effective protocol that is easy to follow and is easily transferable to different laboratories in different geographical regions.

• Plan for sufficient time (around 1 year if well prepared) for your interlaboratory comparison (ILC). The number of laboratories participating in the ILC and the number of substances to test depend on the TG section targeted by the project. For example, in Human Health, there are generally three labs participating in the ILC but covering a larger number of substances. For other sections, it may not be enough to have 3 participants. Thus, you will need to attract participation and make sure that the participants will come from all around the world, and will provide relevant scientific inputs (e.g. think about the best ways to get the relevant scientists in the field involved). Seek the support of experts, National coordinators and OECD Secretariat to circulate the call for ILC participation in their networks.

• After several years it is possible to lose the alignment with the scope. Think from the beginning until the end about the science that is needed to build test methods accepted by the OECD Member countries and the scientific communities. Why a Test Guidelines been agreed upon at the beginning of the process? Did the ILC results cover the scope defined in the SPSF?

• The technical development can already start in parallel with the pre-OECD phase and the project definition phase at OECD. 
• There is flexibility in the timeline of this phase depending on the validation status of the selected test method(s) or testing strategy, but generally, this takes 2 to 5 years.
• Draft Test Guidelines (TGs)/Guidance Documents (GDs) should be submitted to Working Party of National Coordinators of the OECD Test Guidelines programme (WNT) via the OECD Secretariat between June and September to allow for two commenting rounds before submission of the final document to WNT in February to allow approval by WNT in April.