Daniel Liu

MRMD No. 12: Review of Model Risk Book completeness Model Risk Management Discussion (MRMD or “Mermaid”) >>> No. 12: Tuesday, August 13th at 8 AM ET (New York) <<< Join Zoom Meeting https://lnkd.in/e-phH8zX Agenda: Compare existing books on MRM, going back to 2011 See picture.  Can we use these past works to make sure our new textbook is complete? Overview: The FRBNY SR 2011-7 document was birth of modern MRM.  A handful of books have been produced since then, but none is comprehensive (see picture again). I feel the most useful overall is the first, by Massimo Morini and published by John Wiley. Funny story:  Our book was first submitted to Wiley. But we got crickets.  Never a response.  Asked several times.  I guess the dog ate our manuscript. 😁  They have those nice flashy covers, but maybe they are not active publishing in this area anymore.  They have never issued a second edition of Morini, even after 13 years. We went with Academic Press instead, and they have been very responsive. Highly recommended. Suggested reference:  Goldberg, M., Maheshwari, C., and Schachter, J., “Model Risk Management: a textbook for financial engineers,” Cambridge: Academic Press (forthcoming) 2026.

5

There is no perfect metric that will tell you whether a model is good, bad, good enough, or even useful. Making the determination about the usability of a model for a specific problem requires multiple metrics and analysis. In this video I cover why AUC is used in the quant finance space (specifically for credit modeling probability of defaults) and mention a few other metrics that many use as well. The key takeaway from this video is that the ROC curves themselves show model performance over the entire modeling space and not just based on the selection of a threshold for defining good and bad. The AUC is just a simple metric (area under the curve) to help compare models based on their ROC curves. As I point out in the video, the shapes of the curves also matters as they show where the model is better and worse at separating goods and bads. #quantitativefinance #creditrisk #banking #fintech #data #statistics #machinelearning https://lnkd.in/gDiNMxhX

142

7 Comments

The introduction of a capital floor in Canadian banking is a significant step towards enhancing model risk discipline and ensuring competitive balance among financial institutions. The Office of the Superintendent of Financial Institutions (OSFI) has announced a one-year delay in the full implementation of this capital floor, which is set at 72.5%. This delay allows banks additional time to adjust to the new requirements, ensuring they can meet the standards without compromising their operational stability. According to OSFI, the capital floor is designed to provide a minimum level of capital that banks must maintain, which is crucial for safeguarding the financial system. The capital floor aims to mitigate risks associated with internal models that banks use to calculate their capital requirements. By establishing a baseline, OSFI seeks to ensure that all banks operate on a level playing field, reducing competitive disadvantages that may arise from differing capital calculations. More read on https://lnkd.in/gQbkWGPZ Check out Capital floor implementation note of Big 6 (Table 3- https://lnkd.in/guqydFc8)

43

Great summary of the CPAB findings of crypto audits!

11

Still getting through FinCEN's proposed rule to Strengthen & Modernize Financial Institutions' AML/CFT Programs (it is 178 pages). Of note, the requirement for institutions to ensure that their program is not only risk-based and reasonably designed, but also effective and in-line with national AML/CFT priorities will be new for some. The word "effective*" is used 181 times in the document. And for innovative software providers like us and those looking to adopt/pilot new approaches, "innovation" is used 34 times (and is strongly supported throughout!). Also, performing risk assessments becomes a strict regulatory requirement (though you should already be doing it); with the requirement as part of your risk assessments to consider FinCEN's national priorities and to apply resources accordingly. And in their own words: "FinCEN, in consultation with the appropriate Federal functional regulators, intends for these updates to: (1) reinforce the risk-based approach for AML/CFT programs; (2) make AML/CFT programs more dynamic and responsive to evolving ML/TF risks; (3) ultimately render AML/CFT programs more effective in achieving the purposes of the BSA; and (4) reinforce the focus of AML/CFT programs toward a more risk-based, innovative, and outcomes-oriented approach to combating illicit finance activity risks and safeguarding national security, as opposed to public perceptions that such programs are focused on mere technical compliance with the requirements of the BSA." https://lnkd.in/eab-3PqY

3

If you work in risk management, compliance, audit, and cybersecurity, this question arises several times a day: how can I separate normal from unusual transactions by learning the patterns and anomalies in operational and financial data? Let me guide you to code in Python using the PyTorch library to artificially generate data on realistic scenarios mimicking real-world observations and events. With these new synthetic data, you can feed models, called generative adversarial networks, to evaluate how created scenarios are similar to the real-work data. Data that are significantly different from the generated synthetic data are flagged as unusual or fake. Traditional rule-based compliance systems face challenges in detecting anomalies within intricate, high-dimensional data sets. In contrast, adversarial neural networks possess the capability to learn and model intricate patterns inherent within the data. This ability renders them highly effective in identifying subtle anomalies that might otherwise go unnoticed by conventional methods. Adversarial neural networks have demonstrated superior accuracy in anomaly detection tasks. Picture this, you have a dataset of customer transactions that you want to use to monitor fraud, AML compliance, and credit risk. This dataset contains information like the transaction amount, date, location, due diligence, frequencies, and payments. You can use a generative adversarial network model to produce new, made-up transaction data that looks and behaves very similar to the real transactions. This synthetic data has the same statistical properties and patterns as the original data, but it has more transactions and doesn't contain any real customer information to preserve privacy. The augmented data can better detect anomalies that may not be caught by traditional rule-based systems or manual reviews. You can also ignore normal variations in data and focus on unusual patterns, reducing the number of false positives. You can generate synthetic transactions that are representative of the normal population, which can then be used to select a more risk-targeted sample for audit testing. If you learn something in this post, show some love 💛 by liking, commenting, and sharing this post. If you want to learn more about data-driven risk management, let me recommend following great colleagues in Denmark and Switzerland: Graeme Keith, Hans Læssøe, Bo Thygesen, Frederik Thygesen, Suraj Anvekar, Stefan Hunziker, PhD, CICP, Toni Villanen, Jakob Hvidberg Hansen, Christian Dønning, Camila Vicenci Witt, Martin Falck-Hansen, Jesper Sobol, Anders Søborg, Thomas Boegballe, Christian Julin, Kersi Porbunderwala, Jacob Bock Axelsen, Lars Christiansen Bagger, Joël Pastre, FICA, Elena Maran, and Giacomo Minello. #GAN #GenerativeAdversarialNetwork #GenerativeAdversarialNetworks #RiskManagement #Cybersecurity #AnomalyDetection #AICompliance #Compliance #Audit #InternalControls #Python #PyTorch #DataPrivacy

45

11 Comments

Really interesting to see the updated guidance (CPG 230) published by APRA today with a focus on proportionality and a 12 month extension on certain aspects of the standard for non SFIs, specifically business continuity and scenario analysis. As noted in the update "In offering more time, APRA is looking to give smaller entities a bit more space to get the foundations right.". Getting the foundations right is absolutely paramount to the successful implementation and achieving full compliance by the due dates. Organisations should absolutely not take their foot off the pedal on implementation. A lot of good work has been done to date but there is much more to do to continue on this journey to protecting customers and Financial Institutions from the ever growing threat of operational and non financial risks to ensuring sustainable and successful business. https://lnkd.in/gWD7dFGv

28

50% of the large banks have weak or insufficient risk management🤯 This was a finding of the OCC in a recent confidential assessment of 22 US banks that was subsequently reported on by Bloomberg News. We're in Africa...why should we care...because when the US sneezes the world catches a cold. The last global financial crisis is a very distant memory. People very quickly moved on from the global pandemic and soon they will move on from the global IT outage. Note how many times I said "global". We are not immune from the impact and ripple effect of global events and we need to be vigilant against complacency after an immediate danger has past. I for one am concerned when 50% of potentially Globally Systemically Important Financial Institutions (G-SIFIs) and Systemically Important Financial Institutions (SIFIs) are found to have poor risk management in place. It's time to get the house in order before the next thing lands....and there will be a next thing.

19

2 Comments

I'm happy the share Chapter 3 (Future values are the focus of risk measurement) in my Introduction to Quantitative Finance for Risk https://lnkd.in/gPDeBZYE 📗 The 2-minute version 📗 Risk measurement is satisfied with imprecise estimates of the tail (and variance, skew, kurtosis) in a future distribution. This agenda interrogates the methodology of our future projections and our selection of discount rate(s). Most of the traditional approaches have drawbacks (well, prediction is hard). Trees incorporate volatility and path-dependence. Risk aversion introduces subjectivity which is important because individuals have different preferences and different portfolios. Simulations are the probably the best approach given the agenda of risk measurement (and therefore risk management). P.S. GenAI did not write a *single* word of this 15 minute read (says substack) which took me many hours to write. I personally think "write for me" is a lame use case for LLMs. Also, if an LLM writes for me, then I should attribute the text to the LLM, just like I would for any text written by someone or something who is not me.

56

7 Comments

For those that missed Randy Bean and Tom Davenport's article "How Scotiabank Built an Ethical Engaged Culture" in MIT Sloan Management Review, I wanted to recognize Scotiabank's data and AI program. Firstly, congratulations to Grace Lee on receiving the DataIQ award for Most Innovative Use of AI award for its chatbot and Best Responsible AI ethics program! AI-based chatbots are common across large banks. But what differentiates chatbots? As any data practitioner will tell you, it is the quality of the underlying training data and the fine-tuning of the model supporting it. Here are a couple of key points that stood out in Randy and Tom's article: ▶ The use of AI for AI to train the chatbot. Increasingly, we are seeing this method for expediting the efficiency of AI systems. Since chatbots are dependent on documents and unstructured data for knowledge, the contact center at the bank needed to ensure that each document used to train the document was clear, unique and up to date. Uniqueness is important to ensure that duplicate copies of documents do not give a weighted bias to the model response. It was also important that each document is up to date to ensure that conflicting information was not given as a model output. As a result of improved governance on unstructured data, the accuracy of the chatbot skyrocketed from 35% to 90%! Super impressive. ▶ The effectiveness of the chatbot to assist in the contact center allowed for 40% of the questions to be answered without human intervention. The remaining questions were routed to a human agent who received an AI-generated summary of their situation which helped to improve the speed of response by 10%. These types of results supported a positive culture impact across the bank. This significant shift is largely due to the success of the contact center's chatbot and the cross-collaboration efforts of many teams to support this effort. I am seeing a shift in how companies are approaching their data management programs. Similar to Scotiabank in the past, there was an effort to focus on identifying authoritative data sources which meant that structured data governance was the focus. However, now with the enterprise scalability of GenAI, companies are now expanding their data governance to UNstructured data. Why? Because unstructured data is where the untapped knowledge and value lies in the company. Without proper unstructured governance, companies cannot successfully build out their AI models. I predict that the faster companies will consolidate their structured and unstructured data governance programs, the faster they will succeed in AI. Want to hear and learn more about governing and classifying both your structured and unstructured data for GenAI? Put a comment below or send me a direct message. #chiefdataofficer #chiefAIofficer #GenAI #AIGovernance #datagovernance #data #AI https://lnkd.in/eW-6cAXF

38

2 Comments

3 Trade-offs of Model Validation and My Take on It First, some context... Regulatory guidelines generally require credit modeling teams to validate their models annually and submit their validation to the relevant authority. (Deliverable = validation doc). Typically, the regulator publishes guidelines on what should be tested. This is generally not a small list of checks, but it is also not exhaustive or set in stone. The end result? Long validation documents with a ton of tests. Putting this together takes time, is costly, and generally involves a lot of pressure, with tight deadlines and high stakes. The question is, how much value is in it for the company going through the exercise? This question brings me to the trade-offs: 1. There's definitely value, but is it enough though? 2. What percentage of this exercise is actually useful? While some parts are crucial, many aspects can feel like a box-ticking exercise. 3. Is an overload of information once a year actually effective? My take on it? I get it, you have to create the document for the regulator, so you might as well try and make the most of it. But ... there's value in (1) having less, more precise information, (2) having it more frequently, and (3) having it in a format other than a document that is easier to digest. Basically just fancy words for monitoring. This will very likely also get you closer to bringing model validations into strategic decision-making within.

27

2 Comments

Always a great read from an extremely competent set of professionals. A PASA WG that produces really good content. The trick to enriching your data (and therefore its accuracy) is developing a procedure to decide what you can and can't accept. The data will be driven by consumer actions (no always accurate) and data sources can vary on data fields such as DoBs. A data house's probability algorithm will produce a confidence score that Trustees need to be comfortable in accepting and therefore accepting the data change. Wish more Schemes would consider cyclical enrichment - bearing mind data degrades over time.

2

TD Bank could face billions in fines following money-laundering probe TD is not alone in its potential involvement with money laundering, as CIBC and RBC have also recently been fined. Canada’s TD Bank may be facing billions of dollars in fines as Canadian and American regulators crack down on money-laundering schemes that came to light over the past year. National Bank analyst Gabriel Dechaine said total penalties for money-laundering allegations could easily hit $2 billion, according to a note issued to clients, while non-monetary penalties are expected to impact the bank’s operations for years. TD is not alone in its potential involvement with money laundering, as CIBC and RBC have also recently been fined by Canada’s financial crime regulator for failing to comply with anti-money laundering measures. But while the banks will always be at risk of unknowingly facilitating organized crime, experts say they could be doing more to prevent it. 

1

My latest article was released in June in the Joint Risk Management Section Newsletter. This article gives you a different way to think about economics from a system science perspective and introduces concepts such as complexity, emergence, and interdependence. The concepts allow better conformity to the behavior of the markets and policyholder behavior. I hope you enjoy it and find it interesting. https://lnkd.in/gZ6P-JnX

6

Ryan O'Connell, CFA, FRM won't stop publishing great videos, so I can't help but try to add value to this topic of Expected Shortfall (ES): 1. Regarding its relevance, he refers to Basel's regulatory requirement to employ ES for internal market risk models; that requirement ("In calculating ES, a bank must use a 97.5th percentile, one-tailed confidence level.") can be found at their well-designed, dynamic reference at https://lnkd.in/g4s6YNka 2. Technically, the ES can be probability-delimited or quantile-delimited (Ryan illustrates the quantile-delimited version and that's why he accurately dubs it ~ CVaR). What's the difference? Say your P&L distribution is n = 100 with this illustratively clean pattern: {-100, -99, -98, -97, -96, -95, ... , -2, -1}. The 95.0% probability-delimited ES is unequivocally 98.0 because that's the conditional average of the 5.0% tail. The 95.0% quantile-delimited ES could be 97.5 or 98.0 or even 99.0 because discrete VaR has multiple values, as a quantile. Personally, I prefer the probability-delimited ES because it's naturally the average of the (1-c)% tail, rather than the average of values worst than the "sometimes equivocating" VaR. CVaR inherits VaRs discrete ambiguity. Still, notice that the first line at BIS (link above) reads "33.1. Banks will have flexibility in devising the precise nature of their expected shortfall (ES) models". As with VaR, BIS gives banks flexibility in their model but sets quantitative standards. 3. Against its advantage of being coherent (as Ryan says), the classic disadvantage of ES was said to be that it cannot be backtested. Although MSCI says that ES can be backtested (see https://lnkd.in/gA4BaSTZ), clearly it's harder to backtest than VaR: the VaR backtest is a dead-simple binomial test (or extensions of same). I hope that adds something to Ryan's video!

53

6 Comments