top of page
Writer's pictureSumit Raj

LetsDevOps: Setup CI/CD to deploy Machine learning model using Azure Databricks. Airflow to trigger the Job(Workflow). Complete guide to MLOps Setup.

Introduction

The final section of this blog series will cover the process of creating and deploying the machine learning project using Azure DevOps.


MLOps Architecture

Prerequisite


  1. Azure DevOps Account

  2. Databricks Workspace


Setup CI/CD pipeline


Step 1: Get the Project Template



Step 2: Understand YAML Pipeline Structure


Step 3: Update master yaml pipeline

3.1 Get your Databricks URL and update.



Step 4: Create Azure Artifact Feed
Step 5: Update build-publish template
  1. Get the created feed name and update the template

Step 6: Create the Master Pipeline


Step 7: Create variable of Databricks token

Run the Pipeline


Step 6: Udpate the Feed detail in the deploy-to-environment

Once build part is run now you can fetch the feed details since the feed update with package.

  1. Instruction on how to fetch given in the troubleshooting section below.

  2. Update the template wiht the created feed value.




Vaildate the Deployment


Best Practices & Tips

  1. Version control

  2. Deploy to Prod from Main branch only

  3. Security consideration

  4. Key vault to store the Secret.


Orchestrate Job run using Airflow




Troubleshooting

Issue: Forbidden Error during the Artifact download

Solution: Make sure Build service has the Contibutor access assigned.



How to fetch the Azure Artifact Feed Detail


Step: 1 Make sure the Feed is created and you have Package in it.

In my case the package is availble in the Azure Artifact Feed.



Step 2: Edit the Pipeline and Select Download Artifact.

Step 3: Select the Package you want to get the Feed details.


Step: 4 Once you add the task you can find the details





コメント


bottom of page