I'm a DevOps engineer moving into MLOps.
When I started, I assumed my Python scripting background had me covered. I already wrote automation scripts, parsed YAML, called APIs, handled errors. Python is Python, right?
Wrong.
DevOps Python and ML Python look different, feel different, and break in different ways. After completing a focused Python-for-ML course, here are the 8 patterns that genuinely shifted how I work — written specifically for engineers with a DevOps background.
1. DataFrames over loops
What DevOps engineers do: process data with for loops and lists of dicts.
What ML code does: everything lives in a pandas DataFrame.
# The DevOps way
data = []
with open('data.csv') as f:
for line in f.readlines()[1:]: # skip header
parts = line.strip().split(',')
data.append({'name': parts[0], 'value': float(parts[1])})
high_values = [d for d in data if d['value'] > 100]
# The ML way
import pandas as pd
df = pd.read_csv('data.csv')
high_values = df[df['value'] > 100] # done in one line
Why it matters: Every ML library — sklearn, MLflow, XGBoost — expects DataFrames as input. Keep working with lists of dicts and you'll spend half your time converting between formats. Learn the DataFrame mental model first; everything else follows.
2. NumPy arrays for numerical work
What DevOps engineers do: use Python lists. Flexible, familiar, fine.
What ML code does: uses NumPy arrays for anything numerical.
# The DevOps way
numbers = [1, 2, 3, 4, 5]
doubled = [n * 2 for n in numbers] # loop over each element
squared = [n ** 2 for n in numbers] # another loop
# The ML way
import numpy as np
numbers = np.array([1, 2, 3, 4, 5])
doubled = numbers * 2 # vectorized — no loop
squared = numbers ** 2 # vectorized — no loop
# Operations across entire arrays in one line
mean = numbers.mean()
std = numbers.std()
normalized = (numbers - mean) / std
Why it matters: ML regularly works with 10 million data points. At that scale, a Python loop takes minutes; a vectorized NumPy operation takes seconds. The performance difference isn't academic — it determines whether your data pipeline is usable or not.
3. Dataclasses for config management
What DevOps engineers do: YAML files parsed to Python dicts. You know this pattern deeply — Kubernetes, Ansible, Docker Compose all run on it.
What ML code does: hyperparameters and model configs live in Python dataclasses.
# The DevOps way — YAML parsed to a dict
import yaml
config = yaml.safe_load(open('config.yaml'))
learning_rate = config['model']['learning_rate'] # no type checking
batch_size = config['trainign']['batch_size'] # typo fails at runtime, not at write time
# The ML way — dataclass with types and defaults
from dataclasses import dataclass
@dataclass
class ModelConfig:
learning_rate: float = 0.001
batch_size: int = 32
epochs: int = 10
model_name: str = "random_forest"
max_depth: int = 5
config = ModelConfig(learning_rate=0.01, epochs=50)
print(config.learning_rate) # IDE autocomplete works
print(config) # ModelConfig(learning_rate=0.01, batch_size=32, ...)
Why it matters: ML experiments involve running the same model with hundreds of different hyperparameter combinations. Dataclasses give you autocomplete, type checking (typos caught at write time, not runtime), and they're naturally printable and comparable. They're not a replacement for YAML — they work alongside it for the Python-layer config.
4. Context managers beyond file handling
What DevOps engineers do: you already know with open(). You understand context managers in principle.
What ML code does: extends this same pattern into experiment tracking, GPU management, and timing.
# You already know this
with open('config.yaml') as f:
config = yaml.safe_load(f)
# ML extends the exact same pattern
import mlflow
mlflow.set_experiment("housing-price-prediction")
with mlflow.start_run(run_name="random_forest_v1"):
mlflow.log_params({
"n_estimators": 100,
"max_depth": 5,
"learning_rate": 0.01
})
# ... train your model here ...
mlflow.log_metrics({
"train_accuracy": 0.94,
"val_accuracy": 0.91,
"rmse": 42300
})
mlflow.sklearn.log_model(model, "model")
# run auto-closes and saves everything, even on exception
Why it matters: You already understand the mental model — the context manager guarantees cleanup. In ML, this pattern appears constantly: experiment tracking, PyTorch GPU contexts, timed blocks, feature store connections. Recognising it means you're never starting from zero.
5. Pathlib over os.path
What DevOps engineers do: os.path.join(), os.makedirs(), os.path.exists().
What ML code does: uses pathlib.Path — the modern Python standard.
import os
# The DevOps way
data_path = os.path.join('data', 'raw', 'train.csv')
model_dir = os.path.join('models', 'experiment_1', 'v2')
os.makedirs(model_dir, exist_ok=True)
if os.path.exists(data_path):
with open(data_path) as f:
content = f.read()
from pathlib import Path
# The ML way
data_path = Path('data') / 'raw' / 'train.csv'
model_dir = Path('models') / 'experiment_1' / 'v2'
model_dir.mkdir(parents=True, exist_ok=True)
if data_path.exists():
content = data_path.read_text()
# Bonus: globbing is clean
all_csvs = list(Path('data').glob('**/*.csv'))
Why it matters: Every ML codebase and library you'll read or contribute to uses pathlib — sklearn, MLflow, PyTorch DataLoaders. Open a PR using os.path and reviewers will ask you to change it. Better to build the habit now.
6. Type hints are not optional in ML
What DevOps engineers do: skip type hints on scripts. Scripts are short-lived and single-purpose — they run, they work, they're done.
What ML code does: enforces type hints because ML code goes to production and lives for years.
# DevOps script — no type hints, fine for a one-off
def process_data(data, threshold):
return [x for x in data if x > threshold]
# ML production code — type hints are documentation
import pandas as pd
from typing import Optional
def preprocess_features(
df: pd.DataFrame,
target_column: str,
threshold: float = 0.5,
drop_nulls: bool = True
) -> tuple[pd.DataFrame, pd.Series]:
"""
Separate features and target, apply threshold filter.
Args:
df: Raw input DataFrame from data pipeline.
target_column: Name of the label column to predict.
threshold: Minimum value filter for numerical features.
drop_nulls: Whether to drop rows with missing values.
Returns:
(X, y) tuple ready for sklearn fit().
"""
if drop_nulls:
df = df.dropna()
X = df.drop(columns=[target_column])
y = df[target_column]
return X, y
Why it matters: You'll write a preprocessing function today. Someone else — or future you — will call it six months later while debugging a production model. Type hints and docstrings are the difference between "self-explanatory" and "I have to read the entire function to understand what it expects." In ML, functions get reused across experiments for months.
7. Generators for large datasets
What DevOps engineers do: load files into memory. For config files and log snippets, this is perfectly fine.
What ML code does: uses generators to stream large data without running out of RAM.
# DevOps way — loads everything into memory
def load_all_logs(filepath):
with open(filepath) as f:
return f.readlines() # entire file in RAM — fine at 10MB, fatal at 50GB
# ML way — generator, one line at a time
def stream_logs(filepath):
with open(filepath) as f:
for line in f:
yield line.strip() # constant memory regardless of file size
# The ML pattern you'll use most: batch generator for training
def batch_generator(
data: np.ndarray,
labels: np.ndarray,
batch_size: int = 32
):
"""Yields (X_batch, y_batch) pairs without loading all data at once."""
indices = np.arange(len(data))
for start in range(0, len(data), batch_size):
batch_idx = indices[start:start + batch_size]
yield data[batch_idx], labels[batch_idx]
for X_batch, y_batch in batch_generator(X_train, y_train, batch_size=32):
model.partial_fit(X_batch, y_batch)
Why it matters: ML training datasets are routinely 10GB–1TB. You cannot pd.read_csv() a 100GB file. Once you understand yield, you'll recognise the same pattern in PyTorch DataLoaders, TensorFlow tf.data pipelines, and every production data pipeline you'll ever build.
8. Experiment logging, not print statements
What DevOps engineers do: print() for debugging, logging for production scripts. You log events and errors.
What ML code does: tracks parameters, metrics, and artifacts across every run so you can compare them.
# The DevOps approach
print(f"Accuracy: {accuracy:.4f}") # lost when terminal closes
print(f"RMSE: {rmse:.2f}")
import logging
logging.info(f"Model trained. Accuracy: {accuracy}") # better, but still siloed
# The ML approach — tracked, comparable, reproducible
import mlflow
mlflow.set_experiment("california-housing")
with mlflow.start_run(run_name="xgboost_depth5"):
mlflow.log_params({
"model_type": "XGBoost",
"max_depth": 5,
"n_estimators": 200,
"learning_rate": 0.01,
"feature_set": "v2_with_distance"
})
# ... train your model ...
mlflow.log_metrics({
"train_rmse": 38400,
"val_rmse": 42300,
"train_r2": 0.96,
"val_r2": 0.94
})
mlflow.sklearn.log_model(model, "model")
# mlflow ui
# Now compare run 1 vs run 47 side-by-side — every param and metric tracked
Why it matters: You'll run the same model with 50 different parameter combinations trying to improve accuracy. Without tracking, by run 20 you'll have no idea which config produced which result. MLflow runs locally or in the cloud — it's the DevOps-familiar way into experiment tracking, built on tools you already recognise.
The mindset shift in one sentence
DevOps Python keeps systems running. ML Python transforms data and trains models.
Same language — different idioms, different patterns, different instincts.
The context manager you already know is the same pattern powering experiment tracking. The YAML config you know is the same concept as a dataclass. The log file you monitor is the same idea as an experiment run.
The bridge is shorter than it looks. You just need to know where it starts.
Top comments (0)