Best practices for assessing models and managing risk
Sound model development, model implementation, model use, and model validation is especially important as CECL models debut.
See CECL model validation requirements in this guide, "4 Elements of an effective CECL model validation."
Models are relied upon to answer financial reporting and decision-making questions, including CECL reporting and budgeting.
Even small banks or credit unions not regulated by the Federal Reserve are required to address control risks from models.
Often it is difficult for in-house teams to be both familiar with the model and objective enough to challenge it effectively.
Model governance overview
What are model risk management and model validation?
Model risk management (MRM) is a framework of systemic oversight of the models a financial institution or organization relies on for financial reporting, decision-making, and other critical purposes. As financial regulators have noted, this oversight is important because of the "possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused.” Consider the potential consequences of using incorrect or misapplied models to conduct stress testing, assess capital adequacy, estimate credit losses, or monitor and manage high-risk customers.
An effective MRM function:
- creates and maintains an inventory of models used
- ranks those models by categorical risk
- ensures that models receive an appropriate and timely effective challenge within a system of strong internal control.
“Appropriate” and “timely” are relative concepts, so for low-risk models, an analysis might occur every three years. According to Abrigo advisors, current expected credit loss (CECL) models are typically high-risk, and the material estimates the models produce are too important not to warrant an annual examination at a minimum.
Model validation is a series of tests of the critical elements of a model. Testing is performed by a separate party with no involvement in the development or day-to-day use of the model to avoid conflicts of interest.
Model validation results in identifying the weaknesses and limitations of a model, which may then be taken into consideration by bank or credit union management as the model outputs are utilized. You might often make a qualitative adjustment after identifying a limitation in your model—perhaps your model doesn’t address peaks and valleys very well, and you need to be sure they are accurately reflected. Periodic review of models is a best practice—as is getting a validation every time your financial institution or its markets undergo a change that could affect a model’s output.
An effective validation framework includes four core elements:
- Evaluation of conceptual soundness, including developmental evidence
- Evaluation of model inputs, which are your data and assumptions
- Evaluation of the key calculations of the model
- Evaluation of outcomes analysis, including back-testing and benchmarking
These elements apply equally to model validations performed in-house or by a vendor. They are outlined in the Federal Reserve’s supervisory letter SR11-7.
MRM and model validation regulations
SR 11-7, issued by the Federal Reserve and OCC in 2011, is the supervisory guidance on model risk management. All federally regulated banks are required to perform model validations, and SR 11-7 is a starting point to learning the requirements and understanding expectations. It is good for financial institutions to be familiar with it as they adopt and validate models for CECL, as it can help determine what can be done to address control risks. Even smaller banks or credit unions not regulated by the Federal Reserve or OCC are required to address control risks from any models used, and this guidance may be helpful. It can be applied as appropriate considering your organization’s size, nature, complexity, and model sophistication.
The regulatory guidance stresses that model risk management should be an ongoing monitoring process at an institution. It establishes three elements that comprise an appropriate model risk management framework:
1. Sound development, implementation, and use of models, meaning clear policies and procedures surrounding model validation and controls
2. Robust model validation procedures, performed by an objective group to ensure the model performs reliably
3. Governance such as bank or credit union board and senior management oversight policies and procedures, controls and compliance, and an appropriate organizational structure.
A guiding principle for managing model risk is the “effective challenge” of models. This means critical analysis of a model during a model validation by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes. Effective challenge is a requirement for banks regulated by the Federal Reserve or OCC.
Effective challenge does not need to be difficult or combative. It is essential to maintaining a constructive system of internal control wherever models are relied upon, especially where they support financial reporting.
Learn how to integrate CECL and other risk management models
Internal vs. third-party model validation
Much like building vs. buying CECL software, the choice between internal or third-party CECL model validations takes many factors into account. MRM and model validation functions can be accomplished internally by a skilled team of modelers and model risk management professionals separate from those who developed and use the model. You can build controls that test model validation objectives internally. But because in-house MRM or model validation teams typically haven’t spent much time observing other models— especially CECL models— the risks are more likely to be overstated or understated.
“The relative recency of CECL validation means there haven’t been a lot of models out there to be observed yet,” said Abrigo’s Managing Director of Advisory Services, Neekis Hammond, speaking at a recent ABA webinar on CECL MRM and model validation. “It can be hard for in-house teams, however objective, to see the full picture of your model’s function.”
Both MRM and model validation responsibilities could also be performed by an external third-party provider. Even large companies with established in-house MRM teams predating CECL used a third-party vendor for their CECL validations when their teams didn’t have CECL experience or a standing pool of peer data to leverage.
It is up to your institution how MRM and model validation will be done and by whom. But to stay compliant with regulatory guidance, make sure that objectivity and experience are taken into consideration.
Who is responsible for MRM and model validation?
Those directly responsible for executing MRM and model validation activities such as effective challenge should be separate from the model development and its use. Just like auditor independence, this is a matter of objectivity. If you have an internally developed model, those who developed it are probably not perfectly objective about evaluating those models. Those who use it on a day-to-day basis are also not objective enough to step back and ask if they are using the models correctly. Whoever is responsible for model validation should be separate from model development and use.
Executive management must own accountability comprehensively across the organization to understand and address the operational, financial reporting, and other risks involved in model usage. But this doesn’t mean CEOs or Chief Credit Officers should perform model validations. They often lack experience with model calculations to identify flaws, even if they meet the objectivity standard.
Whether your institution relies on an in-house team, a third-party vendor, or a combination of the two to stay on top of MRM and model validations, remember that MRM requires continuous oversight. Model inventories and risk grades are typically updated at least annually or any time your institution undergoes a change such as an acquisition or a branch expansion. And if you feel your institution lacks a thorough enough knowledge of its models to meet expectations, vendors can be a good solution. Using independent, objective, skilled validators to determine that your models are well-designed and reliable is key to peace of mind.
Learn what to expect from an independent model validation.
keep me informed Watch Webinar
As an expert in the field of model risk management and validation, I bring forth a wealth of knowledge and practical experience to discuss the best practices for assessing models and managing risks, particularly in the context of CECL (Current Expected Credit Loss) models.
Model Risk Management (MRM) Overview: Model risk management is a crucial framework for overseeing the models that financial institutions rely on for financial reporting, decision-making, and other critical purposes. The goal is to mitigate the possible adverse consequences, including financial loss, that may arise from decisions based on incorrect or misused models. This includes using models for stress testing, assessing capital adequacy, estimating credit losses, and managing high-risk customers.
An effective MRM function involves:
- Creating and maintaining an inventory of models used.
- Ranking models by categorical risk.
- Ensuring models receive appropriate and timely effective challenges within a system of strong internal control.
For models like CECL, which are considered high-risk, an annual examination is often recommended, according to Abrigo advisors.
Model Validation Framework: Model validation is a series of tests conducted by a separate party with no involvement in the model's development or day-to-day use to avoid conflicts of interest. The validation process aims to identify weaknesses and limitations in a model. An effective validation framework includes four core elements:
- Evaluation of conceptual soundness, including developmental evidence.
- Evaluation of model inputs, which include data and assumptions.
- Evaluation of key calculations of the model.
- Evaluation of outcomes analysis, including back-testing and benchmarking.
These elements apply to both in-house and third-party model validations and are outlined in the Federal Reserve's supervisory letter SR11-7.
Federal Guidance (SR 11-7): SR 11-7, issued by the Federal Reserve and OCC in 2011, serves as supervisory guidance on model risk management. It mandates that all federally regulated banks perform model validations. The guidance emphasizes the importance of ongoing monitoring and establishes three elements for an appropriate model risk management framework:
- Sound development, implementation, and use of models with clear policies and procedures.
- Robust model validation procedures conducted by an objective group.
- Governance, including oversight policies, controls, compliance, and an appropriate organizational structure.
The "effective challenge" of models is a guiding principle, requiring critical analysis by objective, informed parties during model validation.
Internal vs. Third-Party Model Validation: The choice between internal and third-party model validations depends on various factors. Internal teams can build controls to test model validation objectives, but the lack of exposure to external models, especially for newer models like CECL, may lead to overstated or understated risks. External third-party providers bring objectivity and experience.
Accountability in Model Validation: Those responsible for executing model risk management and validation activities should be separate from model development and use to ensure objectivity. Executive management must own comprehensive accountability across the organization, understanding and addressing operational, financial reporting, and other risks involved in model usage.
Continuous oversight is essential, and model inventories and risk grades should be updated regularly, especially during organizational changes. Whether using an in-house team, a third-party vendor, or a combination of both, independent, objective, skilled validators play a key role in ensuring models are well-designed and reliable.
In conclusion, adherence to regulatory guidance, ongoing monitoring, and the incorporation of an effective challenge are essential components of a robust model risk management and validation framework in the context of CECL models.