PODCAST
Enforcement and Economics: Driving Better Case Outcomes Through Collaboration
Litigation economics is not something they teach in most schools, but when it comes to unique or first of their kind enforcement matters, it's an important skill to have, whether it be in establishing sampling methodologies, the creation of a benchmark or development of restitution frameworks.
On this episode, we hear from Chris Kelly, acting Head of Enforcement, and Lori Walsh, Vice President of the Office of the Chief Economist, about how FINRA's Enforcement team has been partnering up with the Office of Regulatory Economics and Market Analysis to ensure better case outcomes.
Resources mentioned in this episode:
Episode 77: How an Enforcement Action Becomes an Enforcement Action
Listen and subscribe to our podcast on Apple Podcasts, Google Podcasts, Spotify or wherever you listen to your podcasts. Below is a transcript of the episode. Transcripts are generated using a combination of speech recognition software and human editors and may contain errors. Please check the corresponding audio before quoting in print.
FULL TRANSCRIPT
00:00 - 00:24
Kaitlyn Kiernan: Litigation economics is not something they teach in most schools, but when it comes to unique or first of their kind enforcement matters, it's an important skill to have, whether it be in establishing sampling methodologies, the creation of a benchmark or development of restitution frameworks. On this episode, we hear how FINRA's Enforcement team has been partnering up with the Office of Regulatory Economics and Market Analysis to ensure better case outcomes.
00:24 – 00:34
Intro Music
00:34 - 00:53
Kaitlyn Kiernan: Welcome to FINRA Unscripted. I'm your host, Kaitlyn Kiernan. I'm pleased to welcome back two repeat guests to the show today in what is, perhaps, our most Irish-sounding lineup to date. Joining me are Chris Kelly, acting Head of Enforcement and Lori Walsh, Vice President of the Office of the Chief Economist. Lori and Chris, welcome back to the show.
00:53 - 00:54
Chris Kelly: Thanks for having us.
00:54 - 00:57
Lori Walsh: Thank you so much. I'm excited to be back.
00:57 - 01:26
Kaitlyn Kiernan: So, earlier this year on Episode 124, introducing the Office of Regulatory Economics and Market Analysis, of which the Office of the Chief Economist is one part, Lori hinted at some interesting work stemming from a partnership with FINRA Enforcement. So, today Chris and Lori are joining us to dig more into that topic. But first to kick us off, can you both quickly reintroduce yourselves for those who haven't had an opportunity to catch our prior episodes together? Chris, maybe we can start with you?
01:26 - 01:43
Chris Kelly: So, my name is Chris Kelly. I've been at FINRA for about nine years. I started at FINRA as the Chief Counsel, it was then known as the North Region of the Enforcement department. And more recently I served as the Deputy Head of Enforcement. And very recently, for the past several months, I've served as the acting Head of Enforcement here at FINRA.
01:43 - 01:44
Kaitlyn Kiernan: And Lori, how about you?
01:44 - 02:02
Lori Walsh: As you mentioned, I'm Vice President and Deputy Chief Economist in the Office of the Chief Economist. I've been at FINRA for over six years, and one of my passions is supporting Regulatory Operations, primarily Enforcement. So, I have a good time with this.
02:03 - 02:17
Kaitlyn Kiernan: Thanks, Lori. Now, as I just mentioned on Episode 124, Lori mentioned that Enforcement works with REMA when it comes to unique or first of their kind enforcement matters. Chris, to start, can you tell us what we mean here?
02:18 - 02:47
Chris Kelly: Yeah. So, I've been at FINRA for almost nine years and during that time I've seen an uptick in the complexity of the matters that we handle. First, because of the certain products and trading strategies that have become more complex. And second, because of the size of some of the matters and frankly, the sheer amount of data that we have to work through. So, REMA has become an invaluable resource to us and Enforcement in these matters to make sure that what we're doing not only makes sense in terms of the facts, the evidence of the law, but also in terms of the data, the math and the economics.
02:48 - 02:55
Kaitlyn Kiernan: And so, what do you consider when you think about whether or not to bring in REMA and at what point in the process would that happen?
02:55 - 03:33
Chris Kelly: It depends. For example, we may be early on in an investigation where we're trying to determine whether there's a violation, and if so, what is the scope of that violation? And in that case, we may bring REMA in earlier on to help us with coming up with a sample size and a sampling methodology to determine the scope of a violation across a large dataset. Conversely, it could be later in the investigation where we know there's a violation, we have a pretty good sense of the scope of the violation, but now we want to determine if there's been customer harm and, if so, how much restitution should be repaid to customers. So, in that scenario, we generally would bring REMA in later in the process when we're at the point of determining restitution.
03:34 - 03:41
Kaitlyn Kiernan: And Lori, what kind of support can REMA and the Office of the Chief Economist offer to FINRA Enforcement?
03:41 - 06:07
Lori Walsh: We offer primarily four categories of support. And as Chris said, it varies a little bit based on when we're brought in and what the need is for the Enforcement team. When we're brought in early to help provide statistical evidence of violations or understanding the scale or scope of violations, we might help in identifying an adequate sample size to determine the scale or scope of a violation. How often do we think it's actually happening, and can we create a sample in which we think is representative of the population and see how often it actually does occur? We may also provide some statistical analysis, maybe creating benchmarks to identify outlier behavior or probability of particular behaviors happening absent a violation versus with a violation. So, we provide some statistical analysis, provide statistical methods to evaluate those things. We might also be brought in to help with data set construction. We've done that a few times where creating the underlying data set to identify particular violations, to estimate customer harm is complex. We might be merging multiple data sets together in a complicated way in order to even set up the set for evaluation of restitution. So, we've done that along with general data analysis. If we're brought in in the latter parts of the case where we are now at a point where the Enforcement team has identified the violation, they've identified the particular instances of customer harm and we jump in and we provide more of a consulting group. We hear the restitution framework that's being proposed. We'll talk through some of the details of the case. We will share advice about maybe ways to improve the restitution methodology, but it's largely consultative in nature. We might do a little data analysis with it. If benchmarking is part of the restitution framework, we might help support the benchmarking, but largely it tends to be consultative in nature and just bouncing ideas off of us and asking questions, what about this and what about that?
06:08 - 06:27
Kaitlyn Kiernan: Sounds like a great comprehensive relationship that the two teams have. And Lori, prior to joining FINRA, I believe you worked with the SEC Division of Enforcement and working with Reg Ops, being close to your heart, how do you think that experience helps facilitate this type of collaboration?
06:28 - 07:39
Lori Walsh: It helps tremendously. I was at the SEC for 17 years, ten years I was in on the Economics team and even while I was there, I periodically would help support the Office of Litigation Economics in the Economics group that works with Enforcement, and I might support some cases. And so, there I got introduced to Enforcement, how Enforcement views the evidence needed to bring cases. When I moved to the Division of Enforcement, I first was the Deputy Chief of the Office of Market Intelligence, the acting Chief, and then I created my own little team, an analytical team within Enforcement, to conduct analytics. That helped me tremendously as well. I got to see on the investigative side what type of evidence is needed, what are the prongs of a particular violation that need to be proved out for an enforcement case? And what is the type of evidence that best supports those prongs of the violation? And that really helped me formulate a full picture of investigation through litigation.
07:40 - 08:00
Kaitlyn Kiernan: That's definitely a great skill set for this role you're in now too. But not everyone on your team has that background working so closely with an Enforcement team. So, how can the relationship between REMA and members of Enforcement work when the REMA team maybe isn't as familiar with that enforcement process?
08:01 - 09:12
Lori Walsh: For the first several years I worked here and worked with Enforcement, I took the lead on most of the cases primarily for that reason. Litigation economics is not taught in schools. It's a very specialized skill set and understanding the unique economic needs to build an enforcement case is not always intuitive. I worked closely with our Analyst team, with our Economist team to teach them how Enforcement's thinking about it, what they need from the advice that we provide. Recently, I hired an amazing analyst who has several years of experience in the SEC's Office of Litigation Economics and at external litigation firms. He's been a tremendous asset. He knows much of the way this process works. He has even more insight than I do and having him on the team has been game changing and really allows me to take a step back from more of the day-to-day work. We are building a team that is strong and getting stronger.
09:13 - 09:22
Kaitlyn Kiernan: That's great to hear. Now, Chris, the Enforcement team works on a lot of cases every year—about how often are you collaborating with REMA on matters?
09:23 - 09:32
Chris Kelly: Yeah, I would say in terms of the overall Enforcement docket, it is a small minority of the matters, but there isn't a time during the year where we're not collaborating with REMA on some matter.
09:32 - 09:41
Kaitlyn Kiernan: And can you walk us through some cases where REMA and Enforcement collaborated and share how the two worked together to get to the finish line on a case?
09:41 - 11:46
Chris Kelly: Absolutely. So, I think a pretty good illustrative example is we had a case recently where we were investigating a firm for a violation of Reg SHO. And in that case, the allegation was that the firm had failed to close out failure to deliver, or FTD positions, that had resulted from short sales. And one of the things we wanted to do was to get a sense of how many of those FTDs, those failures to deliver, that were in the case. But we were looking at an enormous sample size, a five-year period with literally hundreds of thousands, if not millions of transactions. And so, rather than try to go transaction by transaction through each day of the relevant period, we worked with Lori's team to try to come up with a sampling methodology and appropriate sample size, which we did. And then we went back to the firm and said, 'hey, rather than spend the time and resources, that's going to be necessary to look at every day of the relevant period, why don't we pick an appropriate sample size and use this methodology to find out the number of FTDs during that sample and then extrapolate from there?' The firm agreed. They agreed with our methodology and sample size, and they agreed and were, frankly, thankful for our willingness to work with them to avoid the cost and the resources that would have been necessary to conduct the more fulsome analysis. And so, we got to where we needed to get in terms of an idea of the scope of the violation much more quickly than we would have otherwise, certainly much more quickly than we could have without REMA. A very different but also an illustrative example of how we work with REMA, we had a case recently where we were investigating whether a representative had engaged in certain outside business activities. And to make a long story short, as part of that investigation, we were trying to determine whether or not certain crypto wallets were connected to those outside business activities. And in addition to having expertise in data analysis and economics, there are folks within REMA who have expertise in digital assets. And so, we worked with Lori's team to help us work backwards from these crypto wallets and see if we could connect them to the outsized business activity, something that we didn't have necessarily the expertise in-house to do.
11:47 - 11:50
Kaitlyn Kiernan: Thanks, Chris. And Lori, how about you? Any examples you can walk us through?
11:50 - 14:10
Lori Walsh: Yeah, I have so many interesting examples. I'm just going to highlight a couple here. The first one I'm going to talk about falls into what I mentioned, the statistical methodology bucket. So, a few years ago I was working on a cherry-picking case or a preferential allocation case. And essentially the facts of the case were there was a broker-dealer who had preferred accounts, firm accounts, whatever you might call them, and customer accounts. And in examining the profitability of the firm accounts versus the customer accounts, there was a stark disparity. The way the broker-dealer allocated trades at the end of the day was being looked into to see was there a preferential allocation of profitable trades being allocated to the preferred accounts versus to the customer accounts. So, Enforcement came to us. They had a really strong case anyway, but they said, 'hey, can you provide a statistical probability that this occurred at random as opposed to being a preferential allocation?' So, we did Monte Carlo simulations. We ran a simulation 5,000 times where we essentially randomly allocated trades to either the preferred accounts or to the customer accounts in the same proportion that they were actually allocated. We ran that simulation 5,000 times and calculated preferred account profits to customer account profits. I will say, not a single simulation showed the very high level of profits that were actually realized by the preferred accounts. What that translates to statistically is more than a 1 in 20,000,000,000,000 chance that these were not allocated preferentially. So, it was a really interesting example of how we could provide a statistical method to even more clearly, strongly make the case that it was impossible for this to have happened in any way except through preferential allocation of trades. So, that was an interesting, fun one that we had done a while back.
14:10 - 14:27
Kaitlyn Kiernan: That sounds interesting. So, the Enforcement team's looking at it, they're like, 'it seems unlikely that it would happen,' but your data analysis and running these simulations said, 'it's definitely not a coincidence, this is more than just unlikely,' but impossible that it was just an accident.
14:27 - 16:24
Lori Walsh: Exactly. Another case we worked on a few years back, it was a very large, complicated case. Essentially a firm had a glitch in their system where the analyst's recommendations were being fed to the brokers who were passing the recommendations to customers, and the wrong analyst recommendations were being passed through to customers. So, what might be a buy by an analyst might show up as a sell by an analyst to the broker, and then the recommendation was made to the customer to sell, and the trade was executed. The firm identified the algorithmic error. So, first we worked with the Enforcement team to put together a restitution framework—what are the particular trades that ended up in customer harm and how do we measure the amount of customer harm? We worked with them to put together a framework. They went back to the firm and got the framework. The firm executed the restitution framework and came back to the Enforcement team to show the results and the Enforcement team was uncomfortable with that analysis. They came to us, and they said, 'do you think you could validate the restitution calculations made by the firm?' We said, yeah—it was fairly complicated. We received the data, we applied the same calculations, and we identified a number of issues, including data that was missing. There were accounts that were missing from the restitution and there were some significant errors in the way the restitution was done. The final restitution amount ended up being about 30% higher than the firm had reported. And it was, I think, a big win for us to show our analytical chops. It was an early case that we took that was a lot of effort, but I think helped pay off for the harmed customers.
16:24 - 16:28
Kaitlyn Kiernan: Yeah, 30% more restitution back is not insignificant.
16:29 - 17:35
Lori Walsh: No, definitely not. Another type of case that we've supported is when we have multiple firms that we are bringing a case against. We need to be consistent in how we apply it across firms. How do we think about a framework that can be applied generally across firms that closely captures actual customer harm but can be applied generally? And so, we've worked with Enforcement on a few of these broader cases where we analyze the underlying economics of the violative behavior. And we might say, 'okay, if we apply this assumption to this factor, it should have little impact on the outcome.' But we need to be much more accurate when we're measuring this other piece because it can have a significant impact on the outcome. And so, we work with the Enforcement team to design a more general restitution framework that fits all of the firms in a way that gets the most accurate restitution analysis across the board.
17:36 - 17:54
Kaitlyn Kiernan: Well, thanks so much. Those are all great examples of the various ways the two groups work together. So, at the end of the day, I think I heard a few different people benefiting from this collaboration—firms, investors. But who do you see benefiting from your teams working together?
17:55 - 18:31
Chris Kelly: Well, I think everyone benefits from a process that is most likely to lead to the right result. At the end of the day, our job at FINRA and our job in Enforcement is to get to the right result, whatever that result may be. And while I'm extraordinarily proud of the staff we have in the Enforcement department and think they are among the most talented you will find anywhere, many of us are not experts in data analytics or economics or statistics. Not every case requires those skill sets, but when we do have a need for those skill sets, it is invaluable for us to have REMA standing by, ready, willing and able to assist us, to make sure that we're getting to the right result.
18:32 - 19:36
Lori Walsh: I will add that REMA benefits greatly from this collaboration as well. As I talked about in the last podcast, we have a significant role in policy as well as Regulatory Operations. This really helps us, understanding the ability to examine for and enforce the rules that we're putting in place and where it works and where it might need some improvement. It really helps us when we are evaluating potential regulatory changes, policy changes, and understanding the potential economic implications of it. And so, it helps us tie together two different sides of FINRA in a unique way that I think not many teams within FINRA are able to do, and it benefits our work on both sides of the equation, which is really exciting for me, working every day to tie these even more closely together and our knowledge and insights into both sides helps.
19:36 - 20:22
Kaitlyn Kiernan: It sounds like you're seeing the entire life cycle of regulation where you sit in REMA. Well, Lori and Chris, thank you so much for joining me to talk through how your teams are working together to the benefit of investors, the industry and FINRA. It was interesting to hear the whole scope of the collaboration. Listeners, if you don't already, be sure to subscribe to FINRA Unscripted wherever you listen to podcasts. You can also check out the show notes for links to our other recent episodes with Lori and Chris. If you have any ideas for future episodes or any thoughts on today's episode, you can email us at [email protected]. Today's episode was produced by me, Kaitlyn Kiernan, coordinated by Hannah Krobock and engineered by John Williams. Thanks for joining us.
20:22 – 20:28
Outro Music
20:28 - 20:55
Disclaimer: Please note FINRA podcasts are the sole property of FINRA, and the information provided is for informational and educational purposes only. The content of the podcast does not constitute any FINRA Rule or amendment or interpretation to such rules. Compliance with any recommended conduct presented does not mean that a firm or person has complied with the full extent of their obligations under FINRA Rules, the rules of any other SRO or securities laws. This podcast is provided as is. FINRA and its affiliates are not responsible for any human or mechanical errors or omissions. Parties may not reproduce these podcasts in any form without the express written consent of FINRA.