Intern - Year Round (Model Risk Management)

Overview

Model risk is the potential for adverse consequences (e.g., financial loss, poor business or strategic decisions, reputational damage) arising from decisions based on incorrect or misused model outputs and reports. Model risk comes down to making informed business decisions by understanding a model's weaknesses or limitations. With a variety of models designed to maintain Navy Federal's financial stability and its ability to make impactful member-related decisions, it is more important than ever to ensure those models are tested, challenged on current model usage and practices, and assessed on the effectiveness for their intended uses.

Our team serves as a 2nd line of defense, and our mission is to:
* Be a trusted partner in modeling and risk management
* Challenge models and deliver transparency for a common interest across the organization
* Deliver model risk management as a competitive advantage that benefits our members
* Manage model risk, not eliminate it
* Collectively become a modeling "Center of Excellence" for the enterprise

MRM is responsible for maintaining a governance framework that standardizes model risk activities from model development through retirement. MRM uses control processes to ensure sound model risk management across all model lifecycle activities.
Core components of Model Risk Management's role include:
* Identification
* Maintaining a full inventory of models including documentation, code, and datasets
* Assessment
* Assessing the conceptual soundness of models and their risk through analysis
* Recommending model performance enhancements and risk mitigations
* Monitoring
* Monitoring model usage, changes, performance, data drift, and production errors
* Reporting
* Providing an enterprise-level assessment of model risk
* Tracking the status of individual model risk findings and approving mitigation/remediation plans

This year round intern will support end-to-end validations of Navy Federal's machine learning models. This validation would include analyzing the model's data, conceptual soundness, key assumptions, limitations, estimation, calibration, and implementation as appropriate. Validations may also include developing a challenger model against the original model design. At the conclusion of each validation, the associate will have identified a list of model risk findings that will be presented to the Model Owner for remediation. Models selected for validation may be from a variety of possible subject domains, such as fraud detection, targeted marketing, natural language processing, or more.

1. Review model documentation and meet with business owners or experienced model validators, as needed, to gain an understanding of the model
2. Conduct a variety of independent model validation tasks, which may include, but is not limited to, assessments of the following:
a. Model theoretical framework, fitting methods, assumptions
b. Data creation process, including data preparation and data quality
c. Model specification
d. Model development and performance testing, which may include in-sample, out-of-sample, out-of-time backtesting, sensitivity testing, stress testing, and performance monitoring results
e. Ongoing monitoring plan, including proposed or existing ongoing monitoring metrics and thresholds
f. Appropriateness of the implementation testing plan scope in context of associated model risk, testing metrics, and user acceptance testing (UAT)
g. Code and calculations associated with development/estimation and/or implementation
h. Model documentation including completeness, accuracy, level of details
3. Document and quantify the materiality of each finding
4. Develop recommendations for model developers to mitigate the risks identified

Responsibilities

* Leverage technologies - including Python and R - to analyze and gain insights within large data sets
* Evaluate model design and performance and perform champion/challenger development. Analyze model input data, assumptions, and overall methodology.
* Using statistical practices, analyze current and historical data to make predictions, and identify risks, and opportunities, enabling better decisions on planned/future events
* Provide analytics insights and solutions to solve complex business problems
* Examine data from multiple sources and share insights with leadership and stakeholders
* Transform data presented in models, charts, and tables into a format that is useful to the business and aids in effective decision-making
* Develop and maintain an understanding of relevant industry standards, best practices, business processes, and technology used in modeling and within the financial services industry
* Assess the model's overall fit/suitability with its intended use and purpose
* Provide subject matter expertise to identify models' key assumptions, limitations, and weaknesses, and recommend practical solutions to mitigate model risk
* Prepare and present clear, thorough reports to model developers and model owners explaining the analysis performed, the results of the analysis, and recommendations for improvement

Qualifications

* Pursuing master's or doctoral degree in Data Science, Statistics, Mathematics, Computer Science, Engineering or another quantitative field, or related field
* Ability to understand complex business problems
* Skill in communicating actionable insights using data to technical and non-technical audiences
* Skill in descriptive, predictive, and prescriptive analytics and modeling
* Experience in using classification models to solve business problems
* Demonstrates an understanding of the modeling lifecycle
* Skill in data mining, data wrangling, and data transformation
* Skill interpreting, extrapolating, and interpolating data for statistical research and modeling
* Skill in Python or R
* Skill in SQL and querying (able to pull/transform your own data)

Desired Qualifications
* Experience developing production models in the financial services industry
* Experience with model validation under the OCC Supervisory Guidance on Model Risk Management
* Experience with Artificial Intelligence system development or testing

About Us