Effective model risk management and model validation in banking - Abrigo (2024)

Model governance overview

What are model risk management and model validation?

Model risk management (MRM) is a framework of systemic oversight of the models a financial institution or organization relies on for financial reporting, decision-making, and other critical purposes. As financial regulators have noted, this oversight is important because of the "possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused.” Consider the potential consequences of using incorrect or misapplied models to conduct stress testing, assess capital adequacy, estimate credit losses, or monitor and manage high-risk customers.

An effective MRM function:

• creates and maintains an inventory of models used
• ranks those models by categorical risk
• ensures that models receive an appropriate and timely effective challenge within a system of strong internal control.

“Appropriate” and “timely” are relative concepts, so for low-risk models, an analysis might occur every three years. According to Abrigo advisors, current expected credit loss (CECL) models are typically high-risk, and the material estimates the models produce are too important not to warrant an annual examination at a minimum.

Model validation is a series of tests of the critical elements of a model. Testing is performed by a separate party with no involvement in the development or day-to-day use of the model to avoid conflicts of interest.

Model validation results in identifying the weaknesses and limitations of a model, which may then be taken into consideration by bank or credit union management as the model outputs are utilized. You might often make a qualitative adjustment after identifying a limitation in your model—perhaps your model doesn’t address peaks and valleys very well, and you need to be sure they are accurately reflected. Periodic review of models is a best practice—as is getting a validation every time your financial institution or its markets undergo a change that could affect a model’s output.

An effective validation framework includes four core elements:

• Evaluation of conceptual soundness, including developmental evidence
• Evaluation of model inputs, which are your data and assumptions
• Evaluation of the key calculations of the model
• Evaluation of outcomes analysis, including back-testing and benchmarking

These elements apply equally to model validations performed in-house or by a vendor. They are outlined in the Federal Reserve’s supervisory letter SR11-7.

Federal guidance

MRM and model validation regulations

SR 11-7, issued by the Federal Reserve and OCC in 2011, is the supervisory guidance on model risk management. All federally regulated banks are required to perform model validations, and SR 11-7 is a starting point to learning the requirements and understanding expectations. It is good for financial institutions to be familiar with it as they adopt and validate models for CECL, as it can help determine what can be done to address control risks. Even smaller banks or credit unions not regulated by the Federal Reserve or OCC are required to address control risks from any models used, and this guidance may be helpful. It can be applied as appropriate considering your organization’s size, nature, complexity, and model sophistication.

The regulatory guidance stresses that model risk management should be an ongoing monitoring process at an institution. It establishes three elements that comprise an appropriate model risk management framework:

1. Sound development, implementation, and use of models, meaning clear policies and procedures surrounding model validation and controls

2. Robust model validation procedures, performed by an objective group to ensure the model performs reliably

3. Governance such as bank or credit union board and senior management oversight policies and procedures, controls and compliance, and an appropriate organizational structure.

A guiding principle for managing model risk is the “effective challenge” of models. This means critical analysis of a model during a model validation by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes. Effective challenge is a requirement for banks regulated by the Federal Reserve or OCC.

Validation teams

Internal vs. third-party model validation

Much like building vs. buying CECL software, the choice between internal or third-party CECL model validations takes many factors into account. MRM and model validation functions can be accomplished internally by a skilled team of modelers and model risk management professionals separate from those who developed and use the model. You can build controls that test model validation objectives internally. But because in-house MRM or model validation teams typically haven’t spent much time observing other models— especially CECL models— the risks are more likely to be overstated or understated.

“The relative recency of CECL validation means there haven’t been a lot of models out there to be observed yet,” said Abrigo’s Managing Director of Advisory Services, Neekis Hammond, speaking at a recent ABA webinar on CECL MRM and model validation. “It can be hard for in-house teams, however objective, to see the full picture of your model’s function.”

Both MRM and model validation responsibilities could also be performed by an external third-party provider. Even large companies with established in-house MRM teams predating CECL used a third-party vendor for their CECL validations when their teams didn’t have CECL experience or a standing pool of peer data to leverage.

It is up to your institution how MRM and model validation will be done and by whom. But to stay compliant with regulatory guidance, make sure that objectivity and experience are taken into consideration.

Accountability

Who is responsible for MRM and model validation?

Those directly responsible for executing MRM and model validation activities such as effective challenge should be separate from the model development and its use. Just like auditor independence, this is a matter of objectivity. If you have an internally developed model, those who developed it are probably not perfectly objective about evaluating those models. Those who use it on a day-to-day basis are also not objective enough to step back and ask if they are using the models correctly. Whoever is responsible for model validation should be separate from model development and use.

Executive management must own accountability comprehensively across the organization to understand and address the operational, financial reporting, and other risks involved in model usage. But this doesn’t mean CEOs or Chief Credit Officers should perform model validations. They often lack experience with model calculations to identify flaws, even if they meet the objectivity standard.

Whether your institution relies on an in-house team, a third-party vendor, or a combination of the two to stay on top of MRM and model validations, remember that MRM requires continuous oversight. Model inventories and risk grades are typically updated at least annually or any time your institution undergoes a change such as an acquisition or a branch expansion. And if you feel your institution lacks a thorough enough knowledge of its models to meet expectations, vendors can be a good solution. Using independent, objective, skilled validators to determine that your models are well-designed and reliable is key to peace of mind.