On September 28, 2011, the U.S. Environmental Protection Agency (EPA) issued its final guidance on the weight-of-evidence (WoE) (Final WoE Guidance) analysis that it will use to evaluate the results of data submitted in response to test orders issued for Tier 1 screening under the Endocrine Disruptor Screening Program (EDSP). This WoE analysis, in turn, will be used to determine what substances merit further study under Tier 2 as substances that have the potential to interact with the estrogen, androgen, or thyroid (E, A, or T) hormonal pathways. When EPA issued its final policy and procedures for initial EDSP screening, it noted that its determination that a chemical does or is not likely to have the potential to interact with the endocrine system “will be made on a weight-of-evidence basis taking into account data from the Tier 1 assays and/or other scientifically relevant information.” What follows is a brief overview of the Final WoE Guidance, which is available online, Docket Number EPA-HQ-OPPT-2010-0877-0021.
According to EPA, application of WoE analysis is an integrative and interpretive process routinely used by EPA to evaluate health and ecological toxicity in a manner that takes into account all relevant scientific and technical information. The Final WoE Guidance is significantly expanded and revised from the draft version, which was soundly criticized across the board for lacking specific, thorough, scientific guidance that would provide transparency and consistency to those reviewing and interpreting the Tier 1 screening data.
In its Final WoE Guidance, EPA states that the principles and criteria for weighing and integrating different lines of evidence articulated in existing EPA documents are considered generally applicable to evaluating data from the EDSP Tier 1 battery. The Final WoE Guidance also provides additional and much needed guidance on how EPA evaluates Other Scientifically Relevant Information (OSRI), a term that refers generally to certain information other than the Tier 1 assays that are submitted to support or clarify a determination as to whether the substance may have an effect that is similar to an effect produced by a substance that interacts with the E, A, and/or T hormonal systems. While companies may have believed that OSRI could have provided sufficient information for EPA to make determinations under EDSP, the Final WoE Guidance indicates that OSRI is at best a secondary resource to the EDSP Tier 1 assays themselves.
Background
On November 4, 2010, EPA announced in the Federal Register the availability of a draft document entitled Weight-of-Evidence Guidance Document: Evaluating Results of EDSP Tier 1 Screening to Identify Candidate Chemicals for Tier 2 Testing (Draft WoE Guidance). The release of the Draft WoE Guidance was in compliance with a directive from the House Appropriations Committee FY 2010 Report requiring EPA to develop and publish criteria for evaluating results of Tier 1 screening and determining whether a chemical should undergo Tier 2 analysis. In addition, when the Office of Management and Budget’s (OMB) Office of Information and Regulatory Affairs (OIRA) completed its EDSP Tier 1 Information Collection Review (ICR), it approved the ICR subject to certain conditions, including but not limited to the fact that “EPA should ensure sufficient opportunity prior to submission of any revision to this collection for public comment and peer review of the EPA tools to be developed to guide agency decisions on whether a chemical must proceed to Tier II, including the Weight of the Evidence Approach and Standard Evaluation Procedures.” See OIRA ICR, available online.
According to EPA, the purpose of the Draft WoE Guidance is “to set forth some general principles, criteria, and considerations EPA generally believes to be relevant using a WoE approach to evaluate data submitted as part of EPA’s EDSP” Tier 1 screening assays. The Draft WoE Guidance provided little actual scientific guidance that could be used to evaluate results of Tier 1 screening assays or to identify candidate chemicals for Tier 2 testing, however. The comments submitted repeatedly criticized EPA for offering only exceedingly general, basic considerations that would factor into WoE determinations (e.g., Tier 1 in vivo results would carry greater weight than in vitro results). The criticism was well deserved. The Draft WOE Guidance was only eight pages, and over half of the information provided general information such as the history of the EDSP and the studies that comprise the Tier 1 and Tier 2 batteries.
Final WoE Guidance
In its Federal Register notice announcing its Final WoE Guidance, EPA states that it responded to public comments by providing additional detail and clarification in four areas: (1) Tier 1 battery of assays; (2) assay endpoints; (3) OSRI; and (4) WoE analysis. EPA “acknowledges the contribution that public and peer review comments have had in helping to expand the quantity and enhance the quality of guidance provided in this final guidance.”
Tier 1 Battery and Assay Endpoints
In the Final WoE Guidance, in addition to listing the 11 in vitro and in vivo screening assays that comprise the Tier 1 testing battery, EPA also provides “an overview of the distinctive characteristics of each assay and their complementary endpoints within the battery of assays.” For example, EPA lists the four Tier 1 assays capable of detecting an androgenic or anti-androgenic effect on a chemical (AR binding (rat prostate cytosol); Hersberger (rat); Pubertal male (rat); and fish short-term reproduction), what the in vitro AR binding assay examines, and how the three in vivo assays can be used to evaluate the androgen pathway. EPA states that “[n]o one assay or endpoint in or among assays is intended to be interpreted in isolation.”
OSRI
In the Draft WoE Guidance, EPA acknowledged that a WoE approach would be used for OSRI and described in very general terms what may be OSRI-types of studies, but did not discuss specific studies or other information that EPA might consider OSRI and provided no additional guidance as to how that WoE analysis would be conducted for OSRI or how that would compare to WoE analysis for Tier 1 screening data. In the Final WoE Guidance, EPA describes two sources of OSRI: “results from EPA or OECD equivalent test guideline studies and information from published or publicly available peer-reviewed studies.”
In particular, EPA describes toxicological data requirements for health (OPPTS 870 Guideline Series) and ecological (OPPTS 850 Guideline Series) effects, and how some of these data may “provide relevant scientific and technical information for consideration along with the results of Tier 1 screening in a WoE analysis.” For example, for the estrogen hormonal pathway, EPA states that two-generation and extended one-generation reproductive toxicity studies; developmental neurotoxicity studies; and subchronic, chronic, and cancer bioassays may have estrogen-influenced endpoints such as: (1) age at vaginal opening; (2) estrous cyclicity; (3) reproductive organ weights and corresponding histopathyology; and (4) fertility.
When summarizing EPA’s views on other health and ecological effect studies as scientifically relevant information, EPA states:
[A]lthough EPA regulations in 40 CFR Part 158, subparts F and G define the toxicological data requirements for health and ecological effects, respectively, they were not specifically designed to test for the potential or a chemical to interact with the E, A, or T hormonal pathways. Nonetheless, certain EPA or OECD equivalent test guideline studies may provide contributing scientific and technical information regarding effects related to the endocrine system. The potential effects of a chemical in a broader context of toxicity (e.g., dose-response relationships and adversity) that may be directly related to non-endocrine systems may provide a better understanding of potential indirect effects on the endocrine system. Thus, EPA Part 158 guideline studies and OECD equivalents may provide relevant scientific and technical information to be considered along with the results of Tier 1 screening in a WoE analysis to determine whether or not a chemical has the potential to interact directly or indirectly with the E, A, or T hormonal pathways.
EPA also notes that published or publicly available peer-reviewed studies may be used in a WoE evaluation, however, “for non-guideline, as well as guideline studies, to be considered as primary or secondary sources of information in a WoE evaluation with Tier 1 screening results, EPA would generally evaluate the quality and relevance of the information indicated in EPA Information Quality Guidelines (USEPA, 2002b).” Despite this enhanced discussion of what OSRI EPA may consider relevant, it is also clear that EPA’s determinations about whether a chemical is a candidate for Tier 2 testing will be “primarily based” on the Tier 1 assays.
Quality of Scientific and Technical Information
A section in the Final WoE Guidance that was not present in the draft pertains to how EPA will evaluate the quality of the scientific and technical information provided. EPA references its Information Quality Guidelines, which EPA states “set forth the Agency’s policy and procedural guidance for ensuring and maximizing the quality of information, regardless of the source of information.” EPA also discusses its Science Policy Council (SPC) recommendations for the use of five General Assessment Factors (GAF): (1) soundness; (2) applicability and utility; (3) clarity and completeness; (4) uncertainty and variability; and (5) evaluation and review. While EPA provides a general description for each of these GAFs and “illustrative” considerations when evaluating scientific and technical information, EPA does not describe what considerations may be given more weight than others or how these considerations will be factored into any WoE determinations.
Perhaps more useful, EPA states that standard evaluation procedures (SEP) have been developed for each test guideline study as guidance for evaluating the conduct of each study and interpretation of results. These SEPs are available online. Although not discussed in the Final WoE Guidance, also available at this website are Data Entry Spreadsheet Templates (DEST), which provide a method for reporting all raw data in electronic format for the 890 Guideline Series for the EDSP Tier 1 battery. On its website, EPA states it “encourag(es) Test Order recipients to use the DEST as a method for standardizing raw data reporting and to assist EPA reviewers in the comprehensive and consistent evaluations of the raw data.” EPA also will prepare a data evaluation record (DER) for each test guideline study that “contains a summary of how well the study was conducted and conforms to the guideline and provides the interpretation and conclusions supported by the data.” EPA does not mention preparing DERs for scientifically relevant information, however, noting only that “EPA also develops reviews of non-guideline studies submitted as additional information that are used in the WoE analysis.” EPA also states that “DERs are also available for the standard toxicity test guideline studies used to meet 40 CFR Part 158 data requirements,” although there may be some question as to whether EPA would prepare a DER for such a study if one was not already available.
WoE Approach
EPA states that its Final WoE Guidance is intended to “describe the principles, criteria, and approach used in the WoE determination.” EPA’s approach will begin by assembling and evaluating the individual studies. EPA intends to tabulate the results of individual studies by study type and by endpoint to “provide a structured and transparent approach to facilitate the WoE determination.” For EDSP Tier 1 assays, EPA states that while “the quality of the data would generally be expected to be sound and appropriate for determining whether or not a compound interacts with E, A, or T,” EPA also recognizes there are strengths and limitations of these assays, as noted and discussed in several sources, that will be considered. EPA also provides a list of questions that it will consider for all sources of data with regard to the:
- Quality/validity of the assays;
- Reliability of the results;
- Nature of the effects observed;
- Consistency and interrelationship among endpoints reported in an individual assay; and
- Relevance, specificity, and sensitivity of the endpoints measured.
As one example, with regard to the reliability of the results, the assessments of the reliability of test results are to include such factors as:
- Did the laboratory demonstrate that it can conduct Tier 1 assays reliably?
- For assays not following Tier 1 protocols, is there confidence in the measurement of the endpoints evaluated?
- Was the experimental design adequate — including such factors as the purity and stability of test material, dosing regimen, adequate numbers of tests, and appropriate species for testing?
- Did the study include appropriate positive and negative controls to evaluate the experimental design and performance?
- Was the rationale for dose selection clearly presented, and were the dose selections appropriate?
- Was the proper statistical analysis selected, and was it performed correctly?
The determination of whether Tier 2 testing is warranted for a substance is based on the “totality of the evidence.” Once all the assays and scientifically relevant information are identified and assessed, including the development of a DER or other review, EPA states that it will conduct an integrated analysis to evaluate “whether or not the toxicity database is internally consistent with the purported or hypothesized endocrine interaction.” EPA also states that “an important and relevant consideration in the WoE evaluation is to consider the potential for the alternative hypothesis and whether the alternative has evidence in its favor (i.e., Do the data truly reflect an interaction of the chemical with the E, A, or T hormonal pathways?).” EPA provides some discussion of how it will review data to determine whether effects are consistent or isolated, and states that a “consistent pattern of responses” could “support a recommendation that a chemical be considered as a candidate for Tier 2 testing.” EPA does not go so far as to state that isolated or discordant effects would ensure a chemical not be considered for Tier 2 testing, however.
EPA states that determinations regarding candidates for Tier 2 testing will be “primarily based on an integrated analysis of the results of Tier 1 screening.” EPA states that OSRI may be relevant in its WoE assessment to confirm the results of Tier 1 screening, and “may be of special importance if marginal or weak or inconsistent relationships exist within or among the assay results.” EPA also may look to OSRI to provide additional insight on observed effects or be helpful in evaluating whether a species not specifically tested will respond in a similar manner.
EPA states that it will provide a WoE narrative or characterization so that all will be able to “clearly understand the reasoning behind the conclusion as to whether or not Tier 2 testing is needed.” According to EPA, a summary statement for a WoE analysis would typically include the “main lines” of evidence for the effect of the chemical on the E, A, and T hormonal pathways, a description of uncertainties and the extent to which these uncertainties impact the conclusions, a description of inconsistent or conflicting data and whether there is an explanation for these discordant results, and EPA’s conclusion as to whether Tier 2 testing (or repeated or modified Tier 1 screening) is needed and if so, why.
Summary
On balance, it appears EPA has developed, to a much greater extent than its draft, a guidance document for conducting WoE evaluations of the EDSP Tier 1 assays that includes a description of its guiding principles, the criteria by which it will evaluate studies, and the methods for how it will weigh the evidence and reach conclusions. The development of SEPs, DESTs, DERs, and WoE narratives/characterizations should provide some opportunities to ensure that data are evaluated in a transparent and consistent basis.
There remain, however, areas where EPA could provide even greater clarity, since in many cases EPA provided the questions or criteria it will consider but not necessarily how the answers will inform any final determinations. Stakeholders will also likely be displeased with EPA’s apparent dismissal of OSRI. While many believe these data could have obviated the need for any Tier 1 screening or potential Tier 2 testing, EPA states its view plainly in its Final WoE Guidance that OSRI will only be used as a secondary source.