LetsDevOps: Setup CI/CD to deploy Machine learning model using Azure Databricks. Airflow to trigger the Job(Workflow). Complete guide to MLOps Setup.

Introduction

The final section of this blog series will cover the process of creating and deploying the machine learning project using Azure DevOps.

MLOps Architecture

Prerequisite

1. Azure DevOps Account

2. Databricks Workspace

Setup CI/CD pipeline

Step 1: Get the Project Template

https://github.com/sumitraj0103/Letsdevops/tree/main/mlops_demo

Step 2: Understand YAML Pipeline Structure

Step 3: Update master yaml pipeline

3.1 Get your Databricks URL and update.

Step 4: Create Azure Artifact Feed

Step 5: Update build-publish template

1. Get the created feed name and update the template

Step 6: Create the Master Pipeline

Step 7: Create variable of Databricks token

Run the Pipeline

Step 6: Udpate the Feed detail in the deploy-to-environment

Once build part is run now you can fetch the feed details since the feed update with package.

1. Instruction on how to fetch given in the troubleshooting section below.

2. Update the template wiht the created feed value.

   
   

Vaildate the Deployment

Best Practices & Tips

1. Version control

2. Deploy to Prod from Main branch only

3. Security consideration

4. Key vault to store the Secret.

Orchestrate Job run using Airflow

https://www.letsdevops.net/post/letsdevops-how-to-run-orchestrate-databricks-job-using-apache-airflow

Troubleshooting

Issue: Forbidden Error during the Artifact download

Solution: Make sure Build service has the Contibutor access assigned.

How to fetch the Azure Artifact Feed Detail

Step: 1 Make sure the Feed is created and you have Package in it.

In my case the package is availble in the Azure Artifact Feed.

Step 2: Edit the Pipeline and Select Download Artifact.

Step 3: Select the Package you want to get the Feed details.

Step: 4 Once you add the task you can find the details