Implementing an end-to-end machine learning workflow with azure data factory

Click for: original source

From collecting data to sending results, ADF constructs the right MLOps Lifecycle on one screen. By Rahulraj Singh.

Flow Diagram of the complete Machine Learning System

Source: https://towardsdatascience.com/implementing-an-end-to-end-machine-learning-workflow-with-azure-data-factory-f16cbeeffd94/

Author will walk through an entire Machine Learning Operation Cycle and show how to establish every step of the way using Azure Data Factory (ADF). Yes, it is possible, easy, and extremely reliable. As a bonus, it also automatically sets you up to receive alerts for any sort of data anomalies occurring throughout the process, so you do not have to worry about monitoring the workflow manually.

The article content is split into:

  • How would you define an End-to-End (E2E) ML Workflow?
  • What is Azure Machine Learning?
  • Walking through the workflow step-by-step
  • Azure Function to create a trigger for the pipeline
  • Azure Databricks for data preprocessing and storing to Data Lakes
  • Build and train models from the stored data using Azure Machine Learning
  • Push the final output into Azure Blob Storage
  • Access the blob storage from Power BI to build reporting dashboards

… and more. And we liked: “Storytelling is the most important part of any analytics process”. You will also find links to further reading on the topic in the article. Good job!

[Read More]

Tags big-data machine-learning data-science cloud azure