Structuring and automating AI workflows

1. The goal and the outcome of an ML workflow

The goal of a machine learning workflow is training a model from data, in order to do prediction on unseen data (a spam filter), or mine information from it (profiling customers), or automate some operation which is hard to code explicitly (NPCs in video games).

The word model here means something different from Chapter 5. In statistics and machine learning, a model is a mathematical representation of a real-world process, commonly attained by fitting a parametric function over a sample of data describing the process — for instance f(x) = β₀ + β₁x. Neural networks are a popular family of models: a feed-forward network is a cascade of layers, and many admissible architectures serve disparate purposes.

The outcome of the workflow is described in five clauses, and each one carries an operational consequence:

2. The phases of an ML workflow

The slide states the shape of the process before its content: the process of producing an ML model is not linear nor simple. There could be many iterations, up to reaching a satisfactory evaluation; the whole workflow may be re-started upon data changes; and updates in the model imply further integration and deployment efforts in downstream systems.

These steps are cyclical, not linear: one often revisits data, retrains, or refines features.

The worked example

The lecture instantiates all eight activities on one problem: forecast footfall/visits to some office by day and time, useful for staffing and opening-hours planning. Problem framing asks whether to model it as a regression task or a time-series forecasting task. Data collection gathers historical footfall data, calendar events, weather data. Data preparation cleans and preprocesses, handling missing values. Feature engineering creates relevant features such as day of week, holidays, weather conditions. Then training, evaluation, packaging — and monitoring, with a pointed observation: new offices or online services may change footfall patterns. The world moves, so the model expires.

3. Notebooks, and the pitfalls of manual work

How are ML workflows typically performed? Via notebooks, for instance Jupyter. The lecture is balanced about it — four advantages and one decisive disadvantage:

The six pitfalls

• Non-reproducibility: hidden state, out-of-order execution, forgotten seeds.
• Weak provenance: params, code version, data slice and metrics not logged.
• Human-in-the-loop gating: print accuracy → eyeball → tweak → rerun.
• Fragile artifacts: models overwritten, files named final_v3.ipynb.
• Environment drift: "works on my machine" dependencies and data paths.
• Collaboration pain: merge conflicts, opaque diffs, reviewability issues.

Why manual runs mislead

The worked failure is four lines long and every reader recognises it:

1. Run 1: random split → train → print accuracy = 0.82
2. Tweak hyperparams → rerun only the training cell → accuracy = 0.86
3. Forgot to fix the seed / re-run the split → different data, different metric
4. No record of params, code, data → the "best" model cannot be justified

Consequences: incomparable results, irreproducible models; hard to automate, schedule, or roll back; no trace from model → code → data → metrics.

4. ML projects versus ordinary software projects

And the conclusion, stated as an implication: no structured process ⇒ ML projects may fail to move from notebooks to real-world use.

5. MLOps

Machine Learning Operations (MLOps): the practice of organizing and automating the end-to-end process of building, training, deploying, and maintaining machine-learning models.

MLOps adds infrastructure + processes + automation to make each step more reliable:

• Data → version control for datasets, metadata, lineage tracking
• Training → automated pipelines that reproduce experiments on demand
• Evaluation → systematic tracking of metrics, logs and artifacts
• Deployment → continuous integration and delivery for ML models, often with model registries
• Monitoring → automated checks for performance, drift, fairness, anomalies
• Collaboration → shared repositories, environments and documentation

What may happen without MLOps

Data in ad-hoc spreadsheets or local files, with no version control. Training in personal notebooks, hard to reproduce later. Model evaluation manual and undocumented, hard to compare. Deployment as copy-paste code or manual sharing of a model file. Monitoring much harder, so models silently degrade. Collaboration as "send me your notebook by email". The consequences listed are fragile non-reproducible workflows, long delays when models need updating, difficulty scaling beyond a single researcher, and low trust from stakeholders — "why did accuracy drop?".

6. MLflow: components and set-ups

MLflow (mlflow.org) is an open-source Python framework for MLOps and, most recently, LLMOps. It is usable either in-cloud (for instance via Databricks) or on-premises (self-hosted) — the lecture uses the latter.

The three common set-ups

How MLflow works, in one loop

Two assumptions: some Python code is in place to perform ML tasks (via scikit-learn, TensorFlow, PyTorch), and the code uses MLflow's Python API to log metadata about experiments, datasets, models, metrics. Then: start the Python code; the MLflow API invoked in the code logs all relevant metadata and artifacts as the code runs; and metadata and artifacts are stored, depending on the configuration, on the local file system or on a remote backend and artifact store. Where metadata ≈ experiment id, run id, timings, data schemas, input parameters, hyper-parameters, metric values and artifact ≈ dataset, model, chart.

Two usage remarks temper the requirement. The second assumption may require additional effort from the developers, but this is kept minimal via auto-logging available for most common ML libraries. And there is no big constraint on how to organize the Python code itself — though many benefits (automation, reproducibility) come from organizing it as an MLflow Project, which implies decomposing the code into multiple scripts, thinking about the parametric aspects of the experiment and accounting for command-line arguments, and thinking about the environment where the code will run.

7. The Tracking API

Install with pip install mlflow, then consider the dummy script the lecture uses to introduce every concept at once:

import sys        # to read command-line arguments
import tempfile   # to save generated files into temporary directories
import mlflow     # to use MLflow functionalities
from random import Random

# Set the experiment name (creates it if it does not exist)
mlflow.set_experiment(experiment_name="logging_example")
# Read a seed from command-line arguments (default: 42)
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 42
rand = Random(seed)
# Start an MLflow run, naming it "example_run"
with mlflow.start_run(run_name="example_run") as run:
    print(f"Started MLflow run with ID: {run.info.run_id} in experiment ID: {run.info.experiment_id}")
    mlflow.log_param("seed", seed)
    for i in range(5):
        mlflow.log_metric(f"random_{i}", rand.random())
    mlflow.log_metric("random_4", rand.randint(1, 10))  # overwrite last metric
    with tempfile.TemporaryDirectory() as tmpdir:
        file_path = f"{tmpdir}/example.txt"
        with open(file_path, "w") as f:
            f.write("This is an example artifact.")
        mlflow.log_artifact(file_path, artifact_path="examples")
    # Simulate an error in the run if the seed parameter is odd
    if seed % 2 == 1:
        raise ValueError("Let the run fail for odd seeds!")
    print("Run completed successfully.")

Run it twice, python logging_example.py 42 and python logging_example.py 43: the first succeeds, the second raises. Note the sentence in the comments — experiments and runs are identified by their numeric IDs — and note that the seed is a command-line argument rather than a literal, which is the first step towards an MLflow Project.

Afterwards an mlruns/ folder appears next to the script, and its shape is the data model of MLflow:

mlruns
└── 931233098002846893                        <- experiment id
    ├── 378f18735f6d4abd8abeba76f4029bea      <- run id
    │   ├── artifacts/examples/example.txt
    │   ├── meta.yaml
    │   ├── metrics/{random_0 ... random_4}
    │   ├── params/seed
    │   └── tags/{mlflow.runName, mlflow.source.git.commit,
    │             mlflow.source.name, mlflow.source.type, mlflow.user}
    └── 9b52b7b7416e423ca9c878fba9b5c667      <- the failing run, recorded all the same

Start the web UI with mlflow ui and browse to http://127.0.0.1:5000. Clicking the experiment name shows the two runs, and the latest run is marked as failing while the earliest one is successful: the exit code of the run is registered automatically. The Chart view compares logged metrics across all runs; clicking a run shows its parameters, metrics and metadata — the same information logged via the Python API plus some automatically-inferred metadata, and the same data stored on the file system in mlruns/. The Artifacts tab shows example.txt inside the virtual examples/ folder, exactly as requested in the code.

Look at the tags in the tree once more: mlflow.source.git.commit, mlflow.source.name, mlflow.user. That is the trace model → code → data → metrics whose absence was the fourth consequence in §3, materialised without anyone writing it.

8. Autologging

The effort objection is answered by one line of code. Consider a script training a decision tree classifier on the Iris dataset:

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
import mlflow
import sys

mlflow.set_experiment("autologging-example")
# Enable autologging for scikit-learn (and other ML libraries in general)
mlflow.autolog(log_datasets=True, log_models=True,
               log_model_signatures=True, log_input_examples=True)
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 42
with mlflow.start_run(run_name="autologging_run"):
    X, y = load_iris(return_X_y=True)
    model = DecisionTreeClassifier(random_state=seed)
    model.fit(X, y)
    training_score = model.score(X, y)
    print("Training accuracy:", training_score)
    if training_score < 0.9:
        raise ValueError("Training accuracy is too low: " + str(training_score))

The run logs many more information automatically — dataset, model, parameters, metrics — and the recurring comment on the slides is the point: recall that we did not log any of these explicitly in the Python code. What appears:

• Automatically computed metrics on training data: accuracy, f1-score, precision, AUC ROC.
• The actual parameters of the scikit-learn class.
• The training dataset schema, that is, the input schema expected by the model.
• Artifacts: estimator.html (HTML representation of the processing pipeline), metric_info.json, training_confusion_matrix.png.
• A model id such as m-cbc72d11f1e6405bbaa77889f08b92dd, meaning the model URI is mlflow://m-cbc72d11f1e6405bbaa77889f08b92dd.
• Inside the logged model: MLmodel (YAML description), model.pkl (the serialized model, Python pickle), requirements.txt (the Python environment, pip format), serving_input_example.json (example input for serving).

9. Serving, containerization and the model registry

MLflow assists in model deployment by mediating the interaction between logged models and their clients, where clients are assumed to use the models in inference mode (prediction serving) and may be either command-line tools or Web API consumers. It also automates the creation of container images, deployable on common cloud platforms (AWS SageMaker, Azure ML, Google Cloud AI Platform) or on-premises via Docker or Kubernetes.

Three ways to obtain the same predictions

# 1. command line, against a logged model
mlflow models predict --env-manager local \
  -m "models:/m-1abbea58e1cf442ab9412b7eae572523" \
  -i path/to/serving_input_example.json

# 2. as a web service
mlflow models serve --env-manager local \
  -m "models:/m-1abbea58e1cf442ab9412b7eae572523" -p 1234
curl http://localhost:1234/invocations -H "Content-Type:application/json" --data '{
    "inputs": [[5.1, 3.5, 1.4, 0.2], [7.0, 3.2, 4.7, 1.4], [6.3, 3.3, 6.0, 2.5]]
}'

# 3. as a container image
mlflow models build-docker \
  -m "models:/m-1abbea58e1cf442ab9412b7eae572523" -n iris-classifier-dt:latest
docker run --rm -it --network host iris-classifier-dt:latest
curl http://localhost:8000/invocations -H "Content-Type:application/json" --data '{
    "inputs": [[5.1, 3.5, 1.4, 0.2], [7.0, 3.2, 4.7, 1.4], [6.3, 3.3, 6.0, 2.5]]
}'

All three answer {"predictions": [0, 1, 2]}. The build log of the third is instructive: the generated Dockerfile starts from a slim Python image, installs nginx, copies the model directory to /opt/ml/model, installs the pinned MLflow version and the model dependencies. That is the deploy automation via containerization demanded by the exam, obtained from a model URI.

The model registry

One may register a model, which means give it a human-friendly name plus versioning: in the UI, visit a logged model page and click Register. Afterwards:

• The model appears in the Models section with its symbolic name.
• Multiple versions of the same model may coexist.
• Each version is referred to via the URI models:/<model-name>/<version> — for instance models:/iris-classifier/1.
• The latest version is referred to via models:/<model-name>@latest.

10. MLflow Projects

The lecture first builds a realistic scenario — a classifier for the Adult Income (census) dataset via a scikit-learn pipeline, with train-test split, missing value imputation, categorical encoding, feature scaling, model selection via cross-validation between logistic regression and random forests, and hyper-parameter tuning via grid search — implemented as a Jupyter notebook plus MLflow, explicitly to discuss shortcomings and possible improvements. The winner in the demo is a Random Forest Classifier, registered as adult-best-random-forest.

The problems with the notebook

• Notebooks are interactive by nature, making it hard to automate the workflow execution.
• Executing the code is time-consuming and requires a human operator to start the notebook and run all cells, making it hard to schedule periodic runs (for instance for retraining).
• Pictures are commonly embedded in the notebook itself.
• No code decomposition, poor version control, making it hard to maintain and extend.
• [Critical] Parameters are hard-coded in the notebook, making it hard to tune the workflow without modifying the code, and hard to reproduce past runs with different parameter settings.

Solution: organize the code as an MLflow Project — a standard format for packaging and sharing reproducible data science code, resting on five assumptions.

With that in place, a training run and a test run are two commands, and the experiment name can carry a timestamp so that runs group themselves:

EXPERIMENT_NAME="adult-classifier-$(date +'%Y-%m-%d-%H-%M')"
mlflow run -e train --env-manager=local --experiment-name "$EXPERIMENT_NAME" . -P model_type=both

# re-run with different parameters, no code edits
mlflow run -e train --env-manager=local --experiment-name "$EXPERIMENT_NAME" . \
    -P model_type=random_forest -P test_size=0.25 -P cv_splits=5 -P random_state=123

# then test the best model (the training logs print the exact command)
mlflow run -e test --env-manager=local --experiment-name $EXPERIMENT_NAME . \
    -P run_id=<TRAINING_RUN_ID> -P model_uri=models:/<BEST_MODEL_ID>

Compare this with the notebook it replaces. The [Critical] problem — hard-coded parameters — is gone by construction: every knob is a declared parameter with a default, so a past run can be reproduced by replaying its parameters, and a new run needs no code edit. The remaining commands are exactly the kind of thing a CI pipeline or a scheduler can execute at 3 a.m.

11. Generative AI workflows

The second half of the lecture asks what changes when the model is not trained but rented. The goal of a GenAI workflow is engineering prompts, tools, vector stores, and agents to constrain and govern the behavior of pre-trained (foundation) models, in order to: generate contents (bring unstructured data into a particular format, produce summaries, reports, highlights); interpret unstructured data (extract entities, relations, sentiments, answer questions about a document); automate data-processing tasks which are hard to code explicitly; and interact with users via natural language.

The third bullet deserves its three sub-cases, because they are a criterion for when a GenAI approach is even appropriate: the task is ill-defined (write an evaluation paragraph for each student's work); the task requires mining information from unstructured data (find the parties involved in this contract); the task is complex yet too narrow to allow for general purpose coding (plan a vacation itinerary based on user preferences).

The nomenclature

The outcomes

• Foundation models are commonly not produced in-house but accessed via APIs — yet the choice of which model(s) to use is crucial: they must be available, configured, and most commonly imply costs (per call, per token).
• A set of prompt templates known to work well for the tasks at hand, commonly assessed via semi-automatic evaluations on a validation set of inputs.
• A set of tool servers implementing the MCP protocol so that tools can be invoked by PFMs — software modules somewhat similar to ordinary web services, offering one endpoint per tool.
• A set of agents implementing the orchestration logic, commonly via libraries such as LangChain or LlamaIndex.
• A set of vector stores if needed, populated with relevant data — software modules similar to a DBMS, offering CRUD operations on data chunks indexed by their embeddings.

The phases are similar to the ML workflow in the sense that the goal is to process data, but different in many details, e.g. no training is involved. There may be many iterations for PFM selection and prompt tuning; the workflow may restart upon data changes, task changes, or new PFM availability; the interplay between prompts, models, tasks and data needs continuous monitoring; and the data-flow between components must be tracked for debugging and monitoring.

The peculiar activities

• Foundation model selection: choose the most suitable pre-trained model(s) based on task requirements, performance, cost, data protection and availability — which implies trying out prompts, even manually, on different models.
• Prompt engineering: design, test and refine prompt templates — engineering variables, lengths, formats, contents.
• Evaluations: establish assertions and metrics to assess PFM responses, obtained by instantiating templates over actual data. Somewhat similar to unit tests in ordinary software, and important when automatic, because they allow quick evaluations on prompt/model combinations.
• Tracking the data-flow between components (agents, PFM, tools, vector stores) to monitor costs and latency and to debug unexpected behaviours — also useful for auditing and governance.

12. LLMOps

LLM Operations (LLMOps): the practice of organizing and automating the end-to-end process of building, evaluating, deploying, and maintaining GenAI applications. In a nutshell: MLOps for GenAI.

Compare the two definitions word by word. MLOps says building, training, deploying, maintaining; LLMOps says building, evaluating, deploying, maintaining. Training is replaced by evaluation, which is the whole difference between the two halves of this chapter.

What LLMOps adds, component by component

• Foundation models → catalogs of available models, with metadata on capabilities, costs and usage policies.
• Provider gateways → standardized APIs to access different PFM providers (OpenAI, HuggingFace) uniformly, without code rewrites.
• Prompt engineering → version control for prompt templates, systematic testing frameworks.
• Tool integration → standardized protocols (e.g. MCP) and libraries, plus gateway technologies to aggregate multiple tools.
• Agents → provider-agnostic libraries and frameworks (e.g. LangChain) to build, manage and orchestrate agents.
• Vector stores → standardized interfaces to store and retrieve chunks via embeddings, with support for multiple backend DBMS.
• Evaluation & monitoring → automated frameworks to run evaluations, track performance and monitor costs.

What happens without LLMOps

• Foundation models are hard-coded in the application, making it difficult to switch providers or models.
• Prompt templates are scattered in code or documents, making it hard to track changes or reuse them.
• Tools are manually integrated, leading to brittle connections, lack of observability, maintenance challenges.
• Agents are ad-hoc scripts mixing logic, PFM calls and tool invocations, hard to debug, extend or compose.
• Vector stores are tightly coupled with specific DBMS, making it hard to migrate or scale.
• Evaluation and monitoring are manual and sporadic, leading to undetected issues, cost overruns and loss of trust.

13. MLflow for LLM applications

MLflow may be used to track experiments involving LLMs. Its Tracking API logs prompts used for queries, responses obtained, metrics (tokens used, latency), artifacts (generated text files, images), metadata (model name, version) and parameters (temperature, max tokens). Beyond that:

• An ad-hoc API lets Python programmers express evaluation metrics for LLM responses — LLM-as-a-Judge for correctness scoring, or custom user-defined metrics as Python functions.
• Support for multiple LLM providers and their client libraries, including auto-logging for some of them (OpenAI, LangChain, HuggingFace).
• Support for annotating LLM responses with human feedback.
• Support for profiling the data-flow back and forth between clients and providers, agents using tools included, e.g. to monitor costs and latency.

LLM-as-a-Judge

The idea: use an LLM to evaluate the quality of responses generated by another LLM, possibly via custom criteria defined by the user, where the criteria are expressed in natural language. And the framing that makes it engineering rather than magic:

Think of criteria as unit tests for LLM prompt-response pairs — correctness, relevance, completeness, conciseness, formatting.

Examples given: the response must be in English; the response must contain at least 3 examples; if the user question is asking for sensitive code, then the response must kindly decline to answer.

The running example

For the final exam of a Software Engineering course, students must answer open questions about the course topics. The questions are known (a CSV with category, question text and weight/difficulty: What is computer science?, What is an algorithm?, What were software crises?, and so on) but students are missing examples of good answers. Idea: generate examples of good answers via LLMs and provide students with them — possibly enhanced with RAG over the course material to guarantee coherence with it, and possibly with search-engine tools to enrich the answers with up-to-date references. MLflow may help in (1) selecting the best models and (2) the best prompts, assuming that (3) evaluation metrics are defined for generated answers.

You are a university professor preparing model answers
for a software engineering examination.

Category: {category}

Question: {question}

Difficulty: {weight}/4

Provide a clear and accurate answer suitable for an exam context.
Be concise but comprehensive.

Provide an answer that:
1. Explains the concept clearly
2. Includes at least one concrete example or use case
3. Relates to real-world software development scenarios
4. Is easy to understand for someone learning the subject

Instructions:
- Provide a rigorous, academically sound answer
- Include relevant technical terminology
- Reference key concepts and principles where appropriate
- Structure your answer clearly with proper explanations
- Aim for a comprehensive yet focused response suitable for academic evaluation

Always use the provided web search tool to complement your answers
with relevant and up-to-date links or references.
In calling the tool, you should automatically infer the most relevant
query based on the conversation so far.

The criteria, as code

General criteria used in the example: english (the answer should be in English); software_engineering_related (the answer correctly contextualizes the question within the domain of software engineering); reference_to_definition (the answer should reference or quote relevant definitions); relevance_to_query; plus two custom scores, enough_words (more than 10 words) and not_too_many_words (less than 1000 words). Question-specific correctness criteria are added on top — for What is computer science? the answer should mention study of computation, algorithms, data structures, software, hardware and should not argue that computer science is the study of computers.

import mlflow
from mlflow.entities import Feedback
from mlflow.genai.scorers import Guidelines, scorer, RelevanceToQuery

@scorer
def enough_words(outputs: dict) -> Feedback:
    text = outputs['choices'][-1]['message']['content']
    word_count = len(text.split())
    score = word_count >= 10
    rationale = (
        f"The response has more than 10 words: {word_count}"
        if score
        else f"The response does not have enough words because it has less than 10 words: {word_count}."
    )
    return Feedback(value=score, rationale=rationale)

def guidelines_model(model: str = None):
    yield Guidelines(model=model, name="english",
        guidelines="The answer should be in English.")
    yield Guidelines(model=model, name="software_engineering_related",
        guidelines="The answer is correctly contextualizing the question within the domain of software engineering.")
    yield Guidelines(model=model, name="reference_to_definition",
        guidelines="The answer should reference and/or quote relevant definitions for the concepts mentioned in the question.")
    yield RelevanceToQuery(model=model)
    yield enough_words
    yield not_too_many_words

Two kinds of scorer sit side by side in that generator: guidelines, evaluated by a judge model from a natural-language criterion, and custom scores, ordinary Python returning a Feedback with a value and a rationale. Deterministic checks stay deterministic; only the judgments that genuinely need language go to a model.

The project

The example repository (github.com/gciatto/example-llmops) is an MLflow Project like the one in §10 — same descriptor, different domain:

example-llmops/
├── MLproject                        # MLflow Project descriptor
├── register_all_prompts.py          # register prompt templates
├── generate_answers.py              # generate answers, no agents/tools
├── generate_answers_with_agent.py   # generate answers with agents/tools
├── evaluate_responses.py            # evaluate generated responses
├── prompts/                         # academic.txt basic.txt concise.txt
│                                    # practical.txt system.txt tools.txt
├── python_env.yaml
└── questions.csv

EXPERIMENT_ID="se-answers-$(date +'%Y-%m-%d-%H-%M')"
mlflow run -e register_all_prompts --env-manager=local --experiment-name $EXPERIMENT_ID .
mlflow run -e generate_answers      --env-manager=local --experiment-name $EXPERIMENT_ID . -P max_questions=4
mlflow run -e evaluate_responses    --env-manager=local --experiment-id  $EXPERIMENT_ID . -P generation_run_id=<GENERATION_RUN_ID>

In the UI, the Prompts section lists the registered templates with their versioning and content; the Experiments section lists generation and evaluation runs; clicking a trace shows the interactions with the provider, and for the agent variant it shows which and how many tool invocations were performed, with a Details & Timeline tab profiling the entire data-flow back and forth between client and provider. Two practical notes from the slides: the runs cap the number of questions to save time and costs, and evaluation may take some time, as Guidelines evaluations are performed via further LLM queries — judging is itself a metered operation.

Check your understanding