(Article from Registered Funds Alert, February 2017)
For more information, please visit the Registered Funds Alert Resource Center.
As summarized in a prior Alert, new reporting requirements will provide the SEC with more detailed and aggregate information about investment advisers, funds and their portfolios. This data will also be required to be filed in a XML, a structured data format that allows for sophisticated data analysis. The SEC has openly acknowledged that information from the new reporting requirements will be used to support not just rulemaking, but also examination and enforcement efforts. While the new reporting requirements will not take effect until 2018, the SEC is already developing the tools to analyze the massive amounts of new data it will receive. One of these tools, which has not been widely publicized but is worthy of note, is the Investment Company Risk Assessment (the “Risk Assessment Tool”).
The Risk Assessment Tool was developed by the Office of Risk Assessment (“ORS”) within the Division of Economic and Risk Analysis (“DERA”) and is a tool designed to help the SEC put “big data” to use. To put the term “big data” in context, Scott W. Bauguess, Interim Director and Chief Economist of DERA, has characterized it as “any data that approaches our computational limitations on analyzing it.” One example of big data is trading data, as a single day’s worth of aggregate market trading data would take much longer than one day to analyze, although Mr. Bauguess’s characterization means that the concept of big data will change over time as computing power evolves.
DERA integrates sophisticated analysis of economic, financial and legal disciplines with data analytics and quantitative methodologies, and ORS aims to centralize DERA’s ongoing work in risk assessment activities. While the majority of DERA staff is focused on the preparation of economic analysis and research in connection with the SEC’s rulemaking and policy development function, DERA increasingly facilitates the SEC’s ongoing disclosure filing review, inspections by OCIE and investigations by Enforcement. The Risk Assessment Tool is designed to be a part of those efforts. As noted in the SEC’s Agency Financial Report (Fiscal Year 2016) (“2016 Financial Report”), “[t]he Investment Company Risk Assessment was operationalized in FY 2016, and creates a system of risk rankings based on detecting anomalous investment company characteristics, allowing Enforcement and OCIE to dig deeper and determine if specific, violative conduct might be occurring at a fund … .” The Risk Assessment Tool is also noted among the SEC’s accomplishments regarding efforts to identify potential misconduct with advance analytics tools. Limited information is available about the Risk Assessment Tool, including the entire extent to which it is operational or if final metrics or factors have been determined.
In evaluating the potential impact of the Risk Assessment Tool, it is useful to examine two similar programs developed by ORS, the Corporate Issuer Risk Assessment tool (“Corporate Assessment Tool”) and the Broker-Dealer Risk Assessment tool (“Broker-Dealer Assessment Tool”). The Corporate Assessment Tool provides a comprehensive overview of the financial reporting environment and assists Enforcement in detecting aberrant patterns in financial statements that may warrant additional inquiry. The Corporate Assessment Tool grew out of an initiative focused on estimates of earnings quality and indications of inappropriate managerial discretion in the use of accruals. As explained by Mr. Bauguess in a speech, the Corporate Assessment includes “modeling measures of earnings quality as part of more than two hundred thirty (230) custom metrics provided to SEC staff. These include measures of earnings smoothing, auditor activity, tax treatments, key financial ratios, and indicators of managerial actions.” The Corporate Assessment Tool enables Enforcement to compare an issuer to its peers in order to detect abnormal results and financial reporting anomalies. The Broker-Dealer Assessment Tool was developed in conjunction with OCIE and helps identify outlier behavior that allows OCIE to prioritize inspections for inappropriate risk or fraudulent activity. As discussed in a speech by Mark Flannery, then-current Director and Chief Economist of DERA:
“The process works as follows. BDs are first classified by their type of dealing activity—for instance whether or not they carry customer securities on their books (i.e., a “carrying broker”). This allows staff to analyze how a firm’s behavior compares to its peers. We then look for predictors of potentially anomalous or concerning behavior, which include potential risks related to, for example, its operations, financing, workforce, or firm structure … A score card rates how each firm’s activity in each of these areas compares to its peer firms, and results from these score cards are used to help prioritize the sequence of BD inspections as well as areas for examiners to focus on.”
Similar to the Corporate Assessment Tool, the Broker-Dealer Assessment Tool identifies outlier activities or certain changes over time at a broker-dealer relative to its peers.
As noted above, the Risk Assessment Tool creates a system of risk rankings based on detecting anomalous investment company characteristics and is available to both Enforcement and OCIE. A presentation delivered by DERA staff on February 19, 2016 at the SEC Speaks conference, including Messrs. Flannery and Bauguess, provided some of the possible metrics for the Risk Assessment Tool. In producing a “Risk Ranking,” the Risk Assessment Tool will focuses on four sets of characteristics, as shown in the diagram below.
In looking at these characteristics, including the “Analysis of Fund Characteristics” subcategories, DERA may look to academic studies that model potential misconduct (e.g., dividend juicing or return gaps) in a way that can be recreated by DERA and applied to available data.
While the DERA presentation did not elaborate on any additional factors for investment company characteristics or activities, or give additional details on how they could be weighed, it is apparent that the information provided by the new reporting requirements could feed into several of these factors. For instance, under Form N-PORT investment companies will need to provide fund flow data for each of the preceding three months as part of new liquidity risk information, as well as monthly return and security lending data, which all tie directly to factors identified above. In addition, Form N-PORT will require that each portfolio investment have an identifier number/code. As discussed later in this Alert, valuation has become an examination and enforcement priority and keying individual investments with an identification number would allow the SEC to compare valuations of the same security across various funds and look for outliers.
Although the DERA presentation cited above did not elaborate on “cluster analysis/uniqueness,” DERA has provided some insight on this topic in its white paper titled “Mutual Funds Apart from the Crowd.” In the white paper, DERA regarded uniqueness as employing innovative and unique investment strategies. It then measured fund uniqueness based on a cluster analysis of fund returns, and examined how this measure related to other factors. As the white paper describes:
“[A]ll strategies are based on the same set of available investment vehicles such as stocks, bonds, derivatives, etc. Therefore, strategies overlap to some degree either due to similar security selection or due to similar methods of changing the selection over time, or both. The degree of overlap in strategies differ across funds. Some overlap more, and therefore, exhibit a more similar stream of returns. Others overlap less and, therefore, have a more unique return streams, which are less substitutable.”
In the white paper, DERA defined cluster analysis as a machine learning technique that divides data into groups (clusters) that are meaningful and useful. For example, in the white paper a key cluster/group was fund returns, but one can envision other categories that could be relevant to investment companies, including expenses, duration, sector allocation, etc. By using cluster analysis to measure fund uniqueness, the Risk Assessment Tool could potentially flag fund outliers across many categories. Moreover, as discussed above both Form N-PORT and Form N-CEN will provide the SEC with a tremendous amount of new data that could be used for future cluster analysis.
Although the DERA presentation noted above did not provide any additional factors regarding “text analytics on disclosure,” the presentation separately elaborated on DERA’s text analytics initiative, which aims to identify “themes” in unstructured narrative disclosure. In examining a document, the Risk Assessment Tool could apply statistical methods to look at the distribution of root words in relation to those in other documents. Also, the Risk Assessment Tool could potentially be used to identify topics in disclosure, and then look for similar themes in other documents. In addition, DERA has begun performing sentiment analysis to assess tonality in filings, which can potentially identify negative tones or tones of obfuscation. As explained by Mr. Bauguess in a speech, once topics and tones are incorporated they can be mapped:
“into known measures of risk—such as examination results or past enforcement actions—using machine learning algorithms. Once trained, the final model can be applied to new documents as they are filed by registrants, with levels of risk assigned on the basis of historical findings across all filers. This process can be applied to different types of disclosures, or to unique categories of registrants, and the results then used to help inform us on how to prioritize where investigative and examination staff should look.”
Tools with text analytics, like the Risk Assessment Tool, could be trained to learn disclosures associated with misconduct and then identify similar disclosure in other documents. Given the focus on narrative disclosure and that compliance with Form N-PORT and Form N-CEN will begin in 2018, the text analytics of the Risk Assessment Tool will likely be focused on existing reporting requirements, i.e. annual and semi-annual reports on Form N-CSR. The SEC could look to prior enforcement actions involving disclosure issues and use them as a reference point, helping the Risk Assessment Tool in developing its text analytics function and flagging similar disclosure by other registrants.
The Risk Assessment Tool could become a very powerful tool available to the SEC, similar to the Corporate Assessment Tool and Broker-Dealer Assessment Tool. Increasing sophistication of the Risk Assessment Tool could potentially, over time, decrease the need for intensive on-site examinations. It could offset, from the perspective of effectiveness of oversight, any future budgetary cuts in personnel at the SEC. Further, it could obviate the need for third-party exams, as many have called for. Those are lofty goals, but the Risk Assessment Tool seems designed to achieve such goals (whether it succeeds is of course an open question). Although there is currently very limited information regarding the Risk Assessment Tool, we intend to follow its development closely and will report any updates in future Alerts.