MLD107 · Model Evaluation, Tuning and Explainability
About Course
Evaluate models with metrics that match business costs, validate and tune them without fooling yourself, and explain their behavior to the people who rely on them.
Who it is for
Practitioners who can build scikit-learn models and need to evaluate, improve and justify them rigorously.
Prerequisites
MLD105, or equivalent experience training regression and classification models with scikit-learn.
What you will be able to do
- Select and compute regression and classification metrics, including cost-based evaluation, for a business problem
- Design validation schemes, including stratified, grouped and time-based cross-validation, that reflect how the model will be used
- Tune hyperparameters with randomized search or Optuna while keeping a clean test set
- Explain models globally and for single predictions with permutation importance and SHAP, and document them in a model card that includes a fairness check
Lessons
- Lesson 1 · Metrics that match the business
- Lesson 2 · Validation and hyperparameter tuning
- Lesson 3 · Explainability, fairness and documentation
- Lesson 4 · Lab and assignment
Lab: Evaluate, tune and explain a loan-default model
Using a small-business lending dataset from a community lender, you will compare models with a cost-based metric, tune a gradient boosting model with time-aware validation, and explain its predictions. You will finish with a model card that states performance, limits and a fairness check across groups.
Free tools: Google Colab (free) with scikit-learn, Optuna and SHAP
Assessed assignment
Take a model you built in MLD105 or the provided equipment-failure dataset and produce a full evaluation package. Include a justified metric and cost analysis, a validation scheme that matches deployment, a documented tuning process, a single final test evaluation with confidence intervals from bootstrapping, global and local explanations, and a model card with intended use, limitations and a fairness or subgroup analysis.
| Criterion | Weight | What excellent work looks like |
|---|---|---|
| Metrics and costs | 25% | Metrics and thresholds follow from business costs, and results are reported with uncertainty. |
| Validation and tuning | 30% | The validation scheme matches deployment, tuning never touches the test set and the process is reproducible. |
| Explainability | 25% | Global and local explanations are correct, interpreted carefully and connected to domain knowledge. |
| Model card and fairness | 20% | The model card is complete and honest, and subgroup performance is analyzed with sensible conclusions. |
Portfolio piece
A complete model evaluation package with tuning log, explanations and a model card.
Next step
Continue with MLD108 · Putting Models to Work: Deployment and Monitoring · 8 hours · $56.00
Course Content
MLD107 · Model Evaluation, Tuning and Explainability (8 h)
-
Lesson 1 · Metrics that match the business
-
Lesson 2 · Validation and hyperparameter tuning
-
Lesson 3 · Explainability, fairness and documentation
-
Lesson 4 · Lab and assignment
-
MLD107 Assessment · Model Evaluation, Tuning and Explainability