Advanced Machine Learning with Databricks

Participants should have:

• Basic knowledge of data science and machine learning concepts (e.g., classification and regression models).
• Familiarity with common ML evaluation metrics (e.g., F1-score).
• Experience with Python and ML libraries (e.g., scikit-learn, XGBoost).
• Intermediate-level knowledge of ML development and the use of Git for ML projects.

If you do not have one or more of the pre-requisites QA recommends:

• QADSMLP - Data Science and Machine Learning with Python
• DBASP - Apache Spark Programming with Databricks

Target Audience

This course is designed for:

• Data scientists and ML engineers who want to scale machine learning workflows with Databricks.
• MLOps practitioners aiming to streamline ML lifecycle management, testing, and deployment.
• AI/ML professionals implementing CI/CD, model monitoring, and production-ready ML systems.

By the end of this course, learners will be able to:

• Explain Apache Spark’s architecture and its role in scalable machine learning.
• Develop ML models using Spark ML and pandas APIs on Spark.
• Perform hyperparameter tuning with Optuna on Spark.
• Leverage MLflow and Unity Catalog for model tracking, packaging, and governance.
• Implement MLOps best practices, including CI/CD, pipeline management, and environment separation.
• Deploy and monitor ML models with Databricks Asset Bundles (DABs), Workflows, and Mosaic AI Model Serving.
• Use model rollout strategies, A/B testing, and drift detection for production ML.

Machine Learning at Scale

• Machine learning development with Spark
  • Overview of Spark architecture for machine learning workloads.
  • Introduction to Spark ML for model development.
  • Model tracking and packaging with MLflow and Unity Catalog.
• Model tuning with Optuna on Spark
  • Overview of hyperparameter tuning techniques.
  • Introduction to Optuna on Spark for automated model optimization.
  • Scaling Optuna with Spark for distributed tuning.

Advanced Machine Learning Operations

• Overview of MLOps on Databricks
  • Review of MLOps concepts and industry best practices.
  • Streamlining development to deployment in Databricks.
• Continuous workflows for MLOps
  • Implementing automated testing with Databricks.
  • CI/CD for ML models, including versioning and governance.
• Model rollout strategies and monitoring
  • A/B testing and shadow deployments.
  • Model quality monitoring using Lakehouse monitoring.
  • Drift detection and custom model performance metrics.
• Scaling deployments with Databricks Asset Bundles (DABs)
  • Deploying ML models as code with DABs.
  • Ensuring multiple environment consistency for production ML.

Exams and Assessments

This course does not include formal assessments.

Hands-On Learning

This course includes:

• Hands-on labs with Spark ML, Optuna tuning, and MLflow tracking.
• Practical exercises on CI/CD pipelines, model packaging, and governance.
• Instructor-led demonstrations of model deployment and monitoring techniques.