Conozco a muchos devs que usan UNA herramienta de IA y la tratan como un chatbot glorificado. Le piden todo al mismo lugar — desde autocompletar una línea hasta razonar sobre arquitectura — y después reclaman que “la IA no sirve para cosas complejas.”

La verdad es que estas herramientas no son intercambiables. Tienen fortalezas distintas, operan en capas distintas del trabajo, y cuando las combinas bien, el resultado es exponencial.

Después de varias semanas (y cometiendo varios errores), llegué a un setup que quiero compartir paso a paso.

Las tres capas del workflow

Antes de meternos en la configuración, hay que entender el modelo mental:

🧠 Pensar → Claude Code. Arquitectura, diseño de sistemas, razonamiento complejo, debugging cross-service. Ahora con subagentes nombrados, hooks, Computer Use y contexto de hasta 1M de tokens con Opus 4.6.

Bonus: Cuando inicio desde cero o bien hay un tópico en el cual no me siento muy cómodo. Siempre inicio con Deep Research de Gemini si cuento con fuentes las ordeno y proceso inicialmente en NotebookLM.

🔧 Ejecutar → GitHub Copilot (Editor + Cloud Agent). Completar código, refactors multi-archivo, issues convertidos en PRs de forma autónoma. Desde abril 2026, el Cloud Agent puede investigar y planificar sin abrir un PR, y tú eliges el modelo que usa.

⚡ Orquestar → Claude Cowork. Research, documentos, datos, organización de archivos. Con Dispatch puedes asignar tareas desde el celular. Con Computer Use, Claude literalmente usa tu computador. Con Scheduled Tasks, todo esto corre en automático.

NOTA: Si no te sientes cómodo dando demasiados permisos, siempre puedes configurar o ejecutar en una maquina virtual.

La clave: cada herramienta opera en su capa sin pisarse. Cuando intentas que una haga el trabajo de otra, los resultados bajan.

Paso 1: Configurar Claude Code como tu arquitecto

Qué es

Claude Code es una herramienta de terminal que te da un agente con acceso completo a tu codebase. No es un autocompletador — es un razonador que entiende el contexto completo de tu proyecto. Mientras escribo este post Claude Code es una plataforma agéntica completa: subagentes, hooks, plugins, Computer Use, Dispatch, y soporte para contexto de 1M de tokens con Opus 4.6. Siempre puedes usar Sonnet 4.6.

Instalación (Mac)

$ brew install --cask claude-code

# Iniciar en tu proyecto
cd tu-proyecto
claude

Configuración clave: los archivos de agente

Acá es donde Claude Code se pone realmente poderoso. Puedes definir agentes especializados con archivos .md que actúan como profesionales con expertise específica. Los subagentes nombrados aparecen en el typeahead de @ — los invocas como mencionarías a alguien en Slack:

.claude/
├── settings.json
├── commands/
│   ├── architect.md       # Agente de arquitectura
│   ├── security-audit.md  # Auditor de seguridad
│   ├── db-reviewer.md     # Revisor de modelos y migraciones
│   └── test-strategist.md # Estratega de testing

Ejemplo de un agente arquitecto (architect.md) ):

---
description: "Arquitecto de software senior para proyectos Rails"
---

Eres un arquitecto de software senior especializado en aplicaciones Rails.

## Tu rol
- Evaluar decisiones de diseño antes de implementar
- Proponer patrones que escalen
- Identificar acoplamiento innecesario
- Revisar la separación de responsabilidades
- Proponer mejoras practicas conocidas en seguridad preferentemente frameworks NIST, OWASP, ISO27001, PCI.

## Reglas
- Siempre considera el impacto en performance
- Prefiere composición sobre herencia
- Cada servicio debe tener una sola razón para cambiar
- Documenta las decisiones con ADRs cuando sea relevante
- Manten actualizada la documentación

Para usarlo:

# En claude
@architect ¿Cómo debería estructurar el módulo de conciliación bancaria?

Nuevos features que me volaron la cabeza

Hooks: Puedes definir hooks que se disparan antes o después de que Claude use una herramienta. Ejemplo: un PostToolUse que corre prettier automáticamente después de cada edición, o un PreToolUse que bloquea comandos peligrosos. Esto convierte a Claude Code en algo que se integra con tu pipeline, no que vive al lado.

Computer Use: Claude Code ahora puede interactuar con interfaces gráficas — abrir tu browser, clickear, llenar formularios, correr tu app y verificar visualmente que funciona. Un ciclo completo de “escribo código → lo corro → verifico la UI → corrijo”, totalmente autónomo.

Dispatch: Puedes asignar tareas desde el celular. Le mandas un prompt desde la app móvil, Claude Code trabaja en tu computador de escritorio, y te avisa cuando terminó. Perfecto para lanzar una refactorización mientras vai en el metro.

Plugin Marketplace: Plugins que extienden las capacidades de Claude Code. Ahora pueden incluir ejecutables en bin/, abriendo la puerta a integraciones más potentes con herramientas externas.

Cuándo usar Claude Code

• Diseñar un feature nuevo: “Necesito agregar multi-tenancy a la app. ¿Cuáles son las opciones y tradeoffs?”

• Debugging complejo: “Este test falla intermitentemente en CI pero nunca en local. Analiza el código y las dependencias.”

• Refactoring arquitectónico: “Quiero extraer el módulo de facturación a un engine de Rails. Propón el plan.”

• Revisión de seguridad: @security-audit revisa el flujo de autenticación OAuth completo.

• Testing visual: Con Computer Use: “Corre la app, navega al formulario de login, prueba con credenciales inválidas y reporta qué muestra la UI.”

• A/B Testing: “Dame 2 alternativas para decidir la mejor opción a implementar en el hero del landing page.”

Pro tip: el scope del proyecto lo es todo

Crea un archivo CLAUDE.md en la raíz de tu proyecto con el contexto que Claude Code necesita:

# Contexto del proyecto (ultra resumido)

## Stack
- Rails 8.0.2, Ruby 4.0.2, PostgreSQL 17
- Hotwire (Turbo + Stimulus)
- Sidekiq para jobs async

## Convenciones
- Service objects en app/services/
- Form objects para validaciones complejas
- Queries complejas en app/queries/
- No usar callbacks de ActiveRecord para lógica de negocio

## Dominio
- SaaS contable para micro y pequeña empresa chilena
- Integración con SII (Servicio de Impuestos Internos)
- Multi-tenant por cuenta de contador
- Open Core con licencia MIT para features base

Este archivo es tu multiplicador silencioso. Mientras mejor sea el contexto, mejores son las respuestas. Y ahora con contexto de hasta 1M de tokens (disponible para Max, Team y Enterprise con Opus 4.6), Claude Code puede tener en mente tu codebase completo.

Disclaimer: La version MAX tiene un valor de USD 100. En mi caso se paga totalmente. Pero depende de cada uno buscar el tier que mas le acomode. Si tienes dudas como aprovechar mejor tu plan de USD 20. Hablemos.

Paso 2: GitHub Copilot como tu ejecutor

Setup en VS Code

1. Instala la extensión GitHub Copilot y GitHub Copilot Chat

2. Asegúrate de tener un plan que incluya el Cloud Agent (Pro, Pro+, Business, Enterprise)

3. Habilita Agent Mode en la configuración

Las tres modalidades que uso

a) Inline completions (el clásico)

Copilot autocompleta mientras escribes. Funciona increíble para boilerplate repetitivo, tests unitarios siguiendo patrones existentes, migraciones de base de datos, y código que sigue convenciones claras del proyecto.

Tip: mientras mejor nombrado esté tu código, mejores son las sugerencias. Copilot se alimenta de contexto local.

Tip: si ya tienes todo armado se vuelve un poco molesto.

b) Agent Mode en el editor

Agent Mode está disponible tanto en VS Code como en JetBrains. Para tareas complejas dentro del editor:

@workspace Agrega validaciones de RUT chileno al modelo Client,
incluyendo el algoritmo de verificación y tests unitarios

Agent Mode determina qué archivos tocar, hace los cambios, corre comandos en terminal, e itera hasta completar la tarea. Ideal para implementar features bien definidos, agregar tests a código existente, y refactors multi-archivo acotados.

Dato importante: ahora puedes elegir qué modelo usa Copilot — GPT-5.4, Claude Opus 4.6, Gemini u o3 — dependiendo de la complejidad de la tarea. Mi favorito sigue siendo Sonnet 4.6.

c) Cloud Agent (el game changer — actualizado abril 2026)

Acá es donde Copilot brilla para ejecución autónoma. Lo que antes se llamaba “coding agent” ahora es Copilot Cloud Agent, y desde abril 2026 ya no está limitado a flujos de pull request.

Las novedades que cambiaron mi uso diario:

Research y planificación sin PR: Puedes pedirle a Copilot que investigue tu repo, arme un plan de implementación y espere tu aprobación antes de escribir una sola línea. Antes tenía que crear un PR sí o sí — ahora no.

Model picker: Eliges qué modelo usa para cada tarea. Modelo rápido para tests simples, modelo pesado para un refactor complejo.

Self-review con Code Review: Copilot revisa su propio código antes de taguearte. Lo he visto más de una vez corregirse solo — simplificando lógica innecesariamente compleja antes de que yo la vea.

Security scanning integrado: Code scanning, secret scanning y detección de dependencias vulnerables corren automáticamente dentro del workflow del agente. Y gratis, aunque normalmente es parte de GitHub Advanced Security.

50% más rápido: Desde marzo, el startup del agente es la mitad de rápido. El loop de feedback se siente mucho más ágil.

El flujo actualizado:

1. Asigna un issue a Copilot en GitHub o delega desde VS Code Chat

2. Copilot investiga el repo y genera un plan (nuevo: puedes revisarlo antes de que codee)

3. Codea en su propio ambiente, corre tests y linters

4. Hace self-review con Copilot Code Review + security scanning

5. Te deja un PR listo o un diff para revisar (nuevo: tú eliges cuándo crear el PR)

Lo que delego al Cloud Agent: issues de complejidad baja-media bien descritos, agregar tests a código sin cobertura, limpiar tech debt (dead code, deprecations), actualizar dependencias con cambios de API, documentación de código, y fix de vulnerabilidades detectadas por security scanning.

Lo que NO delego: cambios arquitectónicos, lógica de negocio compleja, cualquier cosa que requiera entender el “por qué” del dominio.

Configuración de agentes custom

Crea archivos en .github/agents/ para codificar las prácticas de tu equipo:

# .github/agents/rails-contributor.md
name: Rails Contributor
description: Sigue nuestras convenciones de Rails para contribuciones

instructions: |
  - Usa service objects para lógica de negocio
  - Tests con RSpec, no Minitest
  - Factories con FactoryBot, no fixtures
  - Cada PR debe incluir tests
  - Migraciones reversibles siempre que sea posible
  - Sigue las convenciones de app/services/ para nuevos servicios

Bonus: puedes usar agentes de terceros desde VS Code y GitHub — Claude de Anthropic y OpenAI Codex están disponibles como agentes alternativos en Pro+ y Enterprise.

Sí, puedes asignar un issue a Claude directamente desde GitHub. Y esto es genial cuando estas en el Gym, en el Uber o almorzando y viene la inspiración. Si tu despliegue es automatizado, puedes sincronizar en un ambiente bajo y probar los cambios entre serie y serie.

Paso 3: Claude Cowork para todo lo demás

Qué es

Cowork es el motor agéntico de Claude Code, pero accesible desde Claude Desktop sin terminal. Trabaja directo con tus archivos locales, carpetas y aplicaciones. Siento que pasó de “preview interesante” o un nice-to-have a herramienta de productividad seria con Projects, Dispatch, Computer Use y Scheduled Tasks.

Setup

1. Descarga Claude Desktop desde claude.com/download (macOS y Windows)

2. Necesitas un plan pagado (Pro, Max, Team o Enterprise)

3. Abre la app y cambia a la pestaña Cowork

4. Apunta Claude a la carpeta con la que quieres trabajar

Lo que cambió todo: Dispatch + Computer Use + Scheduled Tasks

Estas tres features juntas convirtieron a Cowork en algo que antes requería scripts, APIs y horas de setup.

Dispatch: Una conversación continua entre tu celular y tu computador de escritorio. Le mandas una tarea desde la app móvil, Claude la ejecuta en tu desktop usando tus archivos locales, conectores y plugins, y te avisa cuando terminó.

Ejemplo real: voy en el metro y le digo desde el celular “revisa las facturas que llegaron hoy a Gmail, descarga los PDFs y organízalos en /facturas/2026-04/”. Cuando llego a la casa, está listo.

Computer Use: Claude literalmente usa tu computador — mueve el mouse, clickea botones, abre apps, navega el browser. No es metafórico. Es literal. Funciona en macOS y Windows.

Esto desbloquea todo lo que antes no se podía automatizar: interactuar con apps sin API, llenar formularios web, verificar UIs, trabajar con sistemas legacy.

Scheduled Tasks: Escribes un prompt una vez y Cowork lo ejecuta automáticamente — cada hora, día, semana, o cuando tú quieras. Combinado con Dispatch y Computer Use, tienes automatización real sin escribir una línea de código.

Un prompt de 10 lineas te permite leer, clasificar y responder tu bandeja de InMails de LinkedIn. Justo antes de desayunar.

Casos de uso reales que ocupo toda la semana

Research y síntesis

Tengo estas 5 fuentes sobre regulación tributaria chilena para 2026
en la carpeta /research/tributario/. Necesito un resumen ejecutivo
con los cambios que impactan a micro y pequeña empresa, organizado
por fecha de entrada en vigencia.

Cowork lee todos los archivos, sintetiza, y te entrega un documento estructurado.

Generación de documentos

Con la data del sprint review de esta semana (en /sprints/2026-w14/),
genera un reporte para stakeholders con: velocidad del equipo,
items completados vs comprometidos, y blockers identificados.
Formato: Excel con gráficos.

Cowork genera archivos Excel con fórmulas funcionales, PowerPoints con contenido real, documentos formateados. La weá funciona.

Briefing matutino automático (Scheduled Task)

Todos los días a las 8am:
- Revisa mis emails no leídos en Gmail del último día
- Revisa mi calendario para hoy
- Revisa los PRs abiertos en GitHub que necesitan mi review
- Genera un briefing con prioridades sugeridas
- Déjalo en /briefings/YYYY-MM-DD.md

Esto corre solo. Cuando abro el computador, el briefing ya está ahí.

Organización de información

La carpeta /downloads/facturas-2026/ tiene 200+ PDFs de facturas
electrónicas. Renombra cada una con el formato
YYYY-MM-DD_RUT-emisor_monto.pdf y organízalas en subcarpetas por mes.

Algo que a mano tomaría horas, Cowork lo resuelve en minutos.

Análisis de datos

Revisa el archivo clientes.csv y dame un análisis de:
- Distribución por comuna
- Ticket promedio por segmento
- Tasa de churn por mes de incorporación
Exporta los resultados como dashboard HTML interactivo.

Conectores y Plugins

Cowork se conecta a Google Drive, Gmail, DocuSign y FactSet. Además, el Plugin Marketplace (lanzado en marzo) tiene 15+ plugins verificados cubriendo diseño, operaciones, HR y más. Puedes instalarlos desde claude.com/plugins.

Pro tip: Projects en Cowork

Organiza tu trabajo en Projects dentro de Cowork. Cada Project tiene su propio contexto, archivos, instrucciones, tareas programadas y memoria. Nada se mezcla entre proyectos distintos. Es como tener escritorios separados para cada cliente en vez de un solo escritorio lleno de papeles.

Paso 4: El workflow integrado

Acá se junta todo. Este es mi flujo real para un feature nuevo:

Fase 1: Investigación y diseño (Cowork + Claude Code)

1. Cowork: “Investiga cómo otros SaaS contables de LATAM manejan la conciliación bancaria automática. Busca en /research/ y complementa con búsqueda web.”

2. Claude Code: “Dado nuestro modelo de datos actual y esta investigación, ¿cuál es la mejor arquitectura para implementar conciliación bancaria? Considera que procesamos ~50k transacciones/mes por tenant.” — con 1M de tokens de contexto, puede tener todo el codebase en mente.

Fase 2: Planificación (Claude Code + Copilot)

3. Claude Code: “Descompón la implementación en issues de GitHub, cada uno independiente y deployable. Incluye criterios de aceptación y estimación de complejidad.”

4. Copilot Cloud Agent: Le pido que investigue el repo y genere un plan de implementación para los issues más complejos — sin crear PR todavía. Reviso el plan, ajusto, y recién ahí le doy luz verde.

Fase 3: Ejecución (Copilot Cloud Agent + Editor)

5. Copilot Cloud Agent: Asigno los issues de complejidad baja-media directamente a Copilot. Modelo rápido para los triviales, modelo pesado para los que tienen edge cases.

6. Copilot Editor (Agent Mode): Los issues más complejos los trabajo en pair con Agent Mode, donde yo guío y Copilot ejecuta.

Fase 4: Review y documentación (Claude Code + Cowork)

7. Claude Code: Reviso las PRs generadas por Copilot a nivel arquitectónico: “¿Este approach escala? ¿Hay edge cases que no estamos cubriendo?” — con @security-audit para un chequeo de seguridad.

8. Cowork: “Con los PRs mergeados esta semana, actualiza la documentación técnica en /docs/ y genera el changelog para el release.” — esto lo tengo como Scheduled Task que corre los viernes.

Paso 5: Reglas que mantengo

Después de muchos de errores, estas son las reglas que sigo:

1. Nunca delegues lo que no entiendes. Si no puedes revisar críticamente el output de la IA, no deberías estar delegándole esa tarea. Las herramientas amplifican tu capacidad, no la reemplazan. Esto aplica triple con Computer Use — Claude puede hacer cosas irreversibles en tu computador.

2. El contexto es el multiplicador. Invierte tiempo en escribir buenos CLAUDE.md, AGENTS.md, y descripciones de issues. El ROI es brutal: 30 minutos de contexto te ahorran horas de iteración con resultados mediocres.

3. Cada herramienta en su capa. Cuando me pillo usando Claude Code para algo que Copilot hace mejor (autocompletar, ejecutar un issue simple), me detengo y cambio. Y viceversa. No mezcles.

4. Revisa todo. Los PRs de Copilot Agent pasan por el mismo review que cualquier PR humano. Claude Code puede equivocarse en edge cases. Cowork puede sintetizar información incorrecta. Tu criterio es el último filtro. El self-review de Copilot y el security scanning ayudan, pero no reemplazan tus ojos.

5. Itera el setup, no solo el código. Cada semana reviso qué funcionó y qué no en el workflow. Los archivos de agente se actualizan, las instrucciones de proyecto se refinan, los prompts mejoran. Usa /powerup en Claude Code — son lecciones interactivas que te enseñan features que probablemente no estás usando.

6. Controla los permisos. Con Dispatch y Computer Use, Claude tiene acceso real a tu máquina. Revisa qué carpetas y conectores tiene habilitados. Piensa en el peor caso, no en el caso esperado.

Setup mínimo para empezar hoy

Si todo esto te parece mucho, acá va el MVP del workflow:

1. Instala Claude Code → brew install --cask claude-code (5 min)

2. Crea un CLAUDE.md con el contexto de tu proyecto (30 min)

3. Crea al menos un subagente en .claude/commands/ (10 min)

4. Habilita Copilot Agent Mode en VS Code (5 min)

5. Asigna tu primer issue a Copilot en GitHub (2 min)

6. Descarga Claude Desktop, abre Cowork y pruébalo con una tarea real (10 min)

7. Configura Dispatch conectando la app móvil con el desktop (5 min)

En poco más de una hora tienes el stack funcionando. Después vas iterando y agregando complejidad según lo necesites.

Reflexión final

No estamos en la era de “la IA me reemplaza” ni en la de “la IA no sirve”. Estamos en la era de armar tu propio equipo de agentes, donde cada uno tiene su especialidad y tú diriges la orquesta.

Claude Code piensa. Copilot ejecuta. Cowork orquesta el resto.

Y con Dispatch, Computer Use y Scheduled Tasks, ese equipo ahora trabaja aunque no estés sentado frente al computador.

La pregunta no es si usar IA para desarrollar. La pregunta es si estás usando las herramientas correctas en las capas correctas.

Y si todavía estás copiando y pegando prompts en un chatbot para todo... hay un 10x esperándote en la mesa.

🌍 En este episodio de Piloto Automático, analizamos junto a Daniela Ramírez las nuevas tendencias en sostenibilidad y cómo transformarán el mercado y la tecnología. Una charla esencial para entender los desafíos que vienen. ¡Dale play! ▶️

Building a medallion-architecture platform pipelines across multiple processing domains creates a specific engineering challenge that has nothing to do with the data itself: how do you keep architectural discipline consistent across every notebook, every pipeline, and every environment as the platform evolves?

Quality rules get forgotten. Partition keys get omitted. Configuration patterns drift between notebooks. A new developer copies an old notebook and misses the latest pattern. A production fix bypasses the MERGE pattern and introduces duplicates. These are not hypothetical risks — they are the natural entropy of any complex codebase under active development.

Traditional approaches — code reviews, style guides, wiki documentation — rely on human attention. They work until the platform reaches a scale where no single person can hold the full architectural context. Claude Code solves this by embedding the platform’s architectural knowledge directly into the development toolchain.

02 · The Technology Stack

Before discussing how Claude Code helps, it is worth mapping the stack that needs to be maintained. Each component has specific configuration requirements, failure modes, and interaction patterns that must be consistently applied across every pipeline.

• Microsoft Fabric — Unified analytics platform. Lakehouses, pipelines, notebooks, and scheduling under one capacity.

• Delta Lake — ACID transactions on the lakehouse. MERGE/UPSERT for idempotent writes. Schema enforcement at Silver layer.

• Apache Spark — All transformation logic in PySpark notebooks. Shared sessions via orchestrator pattern. Window functions for time-series continuity.

• Azure Cosmos DB — Low-latency serving layer for real-time dashboards. Partition discipline enforced by code. REST API exposure.

• Azure Key Vault — Secrets management. 3-tier config cascade: Key Vault → Spark Config → Default Value across all environments.

• Terraform — Infrastructure as Code for capacity provisioning, Cosmos DB accounts, Key Vault policies, and network rules.

The challenge is not any single component — it is the interactions between them. A Cosmos container defined in Terraform must match the partition key used in the notebook that writes to it. A Key Vault secret path must be consistent across the Terraform policy, the Spark config, and the notebook’s fallback default. These cross-cutting dependencies are exactly where drift causes silent failures.

03 · Why Claude Code for Data Platform Development

A data platform is not a web app. There is no npm run dev. The development loop involves PySpark notebooks that run on remote clusters, Delta Lake tables you can only inspect by querying, documents in a serving layer that silently misdirect if a partition key is missing, and Terraform plans that manage infrastructure across three environments.

Claude Code fits this workflow because it operates in the terminal — the same place where you run az login, terraform plan, pipe logs, and inspect tables. It reads the full project structure, understands architectural context from CLAUDE.md, and executes agentic workflows that chain multiple tools: search the codebase for all MERGE statements, validate partition key consistency across Terraform and notebooks, run a lint pass, and generate a compliance report — all in a single session.

The Strategic Value

The value of Claude Code for data platforms is not speed — it is consistency at scale. When every notebook must follow the same structure, every write must use the same pattern, and every deployment must pass the same governance checks, the question is not whether rules will be violated. It is how quickly those violations get caught. Claude Code shifts detection from code review (hours/days) to code generation (instant).

04 · CLAUDE.md — The Platform’s Persistent Memory

The CLAUDE.md file sits at the root of the repository. It encodes the architectural decisions, quality rules, naming conventions, and environment setup that every developer — human or AI — must follow. Claude Code reads it at the start of every session. No repetition, no drift.

# Data Platform — Real-Time Analytics Platform

## Project Overview
Medallion architecture on Microsoft Fabric (F4 capacity).
Bronze → Silver → Gold → Cosmos DB / Power BI serving layer.
Three processing domains: orchestrated batch, hybrid event/scheduled,
incremental replication. 15+ active pipelines.

## Architecture Rules
- Every write uses Delta Lake MERGE (UPSERT). No append-only writes.
- Every Cosmos DB document MUST include partition_date field.
- All fact-to-dimension joins MUST be LEFT JOIN with count guard.
- Every cache() must have a matching unpersist().
- Named constants only. No hardcoded literals for thresholds.
- Imports at module top level. Never inside loops or conditionals.

## Environment Configuration
3-tier cascade: Key Vault → Spark Config → Default Value.
Notebooks MUST work standalone (dev) and under orchestrator (prod).
@docs/environment-setup.md for Key Vault paths per environment.

## Notebook Structure
Every notebook follows: Config → Read → Transform → Write → Cleanup.
Orchestrator children detect shared temp views via try/except.
Fallback file discovery scans last N timestamped folders via regex.

## Validation
- spark-submit --deploy-mode local for unit testing
- Domain-specific thresholds: @docs/validation-rules.md
- Geospatial: reject coordinates outside configured bounding box
- Alert classification: auto-computed, not manually overridable

## Documentation
Every code change requires a docs update in the same commit.
User-facing docs are bilingual (Spanish/English).
See @docs/checklist-template.md for mandatory checklist.

This file is under 60 lines. It does not try to cover everything — it uses @path imports to reference deeper documentation that Claude reads only when working on related tasks. This follows the progressive disclosure principle: load context only when it is relevant.

Design Principle

A good CLAUDE.md is not a knowledge dump — it is a routing table. It tells Claude what the rules are and where to find deeper context. The architecture rules section encodes the six quality rules from the platform. The @docs/ references point to validation thresholds, environment configs, and schema definitions that change more frequently than the architectural patterns themselves.

05 · The Unified Skills System

The unified skills (.claude/skills/) system has two invocation modes:

• User-invoked skills are triggered explicitly with /skill-name. These are the workflows you call when you need them — creating a notebook, running a governance audit, validating infrastructure consistency.

• Agent-invoked skills are loaded automatically by Claude when it recognizes a matching context. If a skill’s description field matches what you are working on, Claude loads it without being asked. This means a skill describing “Spark notebook creation” can activate automatically when Claude detects you are building a new pipeline.

Every skill lives in its own directory with a SKILL.md as the entry point. The SKILL.md file has two parts: YAML frontmatter that controls when and how the skill runs, and markdown content with the instructions Claude follows.

What Skills Can Do Now

Beyond simple prompt instructions, the current skills system supports: frontmatter fields that restrict which tools Claude can use (allowed-tools), override the model (model), or spawn the skill as an isolated subagent (agent: true); supporting files like templates, example outputs, and validation scripts that live alongside SKILL.md; and dynamic context injection via !command syntax that runs a shell command and injects its output into the skill prompt before Claude sees it. This makes skills genuinely programmable, not just instructional.

06 · Pipeline Creation Skills

The most valuable skills for a data platform encode the pipeline structure itself, so every new notebook starts compliant. Here is the spark-notebook skill with proper frontmatter:

.claude/skills/spark-notebook/
├── SKILL.md                    # Main instructions (required)
├── templates/
│   └── notebook-template.py    # Skeleton Claude fills in
├── examples/
│   └── silver-telemetry.py     # Reference output showing expected format
└── scripts/
    └── validate-structure.sh   # Script to verify cell order compliance

---
name: spark-notebook
description: >
  Creates or modifies PySpark notebooks for the Data platform.
  Use when building new pipelines, adding analytical notebooks, or
  refactoring existing notebooks to match the mandatory structure.
allowed-tools:
  - Read
  - Write
  - Bash(spark-submit:*)
  - Bash(python:*)
---

# Spark Notebook Creation Skill

When creating or modifying a PySpark notebook for this platform,
follow this structure strictly.

## Mandatory Cell Order
1. Parameters cell — Fabric parameters + inj_* injected params
2. Configuration cell — 3-tier config resolution
3. Source reading cell — Check for shared temp views first
4. Transformation cells — Business logic with named constants
5. Write cell — Delta Lake MERGE with composite key
6. Serving sync cell — Include partition_date in every document
7. Cleanup cell — unpersist() all cached DataFrames

## Config Resolution Pattern
Always implement the 3-tier cascade:

    endpoint = spark.conf.get(
        "spark.dataplatform.cosmos.endpoint",
        "https://default-dev.documents.azure.com"
    )

## Orchestrator Detection

    try:
        df = spark.sql("SELECT * FROM tmp_source_data")
        running_under_orchestrator = True
    except:
        df = spark.read.parquet(fallback_path)
        running_under_orchestrator = False

## Write Pattern
ALWAYS use MERGE. Never .mode("overwrite") or .mode("append"):

    delta_table.alias("target").merge(
        df.alias("source"),
        "target.entity_id = source.entity_id AND
         target.event_date = source.event_date"
    ).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

## Anti-Patterns (reject if found)
- NEVER use .mode("overwrite") on Silver/Gold tables
- NEVER omit partition_date in serving-layer documents
- NEVER use cache() without unpersist()
- NEVER hardcode thresholds as inline literals

## Supporting Files
- See templates/notebook-template.py for the cell skeleton
- See examples/silver-telemetry.py for a complete reference
- Run scripts/validate-structure.sh to verify compliance

When a developer types /spark-notebook, Claude loads the skill, reads the supporting files for context, follows the cell structure, and produces a notebook that is already compliant with every quality rule. Because the description field mentions “building new pipelines” and “refactoring existing notebooks,” Claude can also load this skill automatically when it detects related work — no slash command needed.

The same approach works for other repeatable patterns: a telemetry-ingestion skill that encodes the hybrid event/scheduled execution modes, a tail-fetching skill that encodes the incremental replication pattern with LEAD window functions, or a serving-sync skill that enforces partition discipline and retry logic. Each gets its own directory, its own frontmatter, and its own supporting files.

07 · Governance Audit Skills

Where pipeline creation skills help create correct code, governance audit skills help verify that existing code stays correct. These are user-invoked skills — you call them explicitly when you need a cross-cutting check.

---
name: validate-serving-layer
description: >
  Scans all notebooks that write to the serving layer and verifies
  partition key presence, naming conventions, and retry logic.
  Use before commits, during PR reviews, or as a pre-deploy check.
allowed-tools:
  - Read
  - Bash(grep:*)
  - Bash(find:*)
---

# Validate Serving Layer Documents

Scan all notebooks in the repository that write to Cosmos DB.
For each one, verify:

1. Every document dictionary includes a "partition_date" key
2. The partition key value is derived from data, not hardcoded
3. The container name matches convention: "gold-{domain}-{entity}"
4. Error handling wraps the write with retry logic

Report findings as a table:
| Notebook | Container | partition_date present | Naming OK | Retry logic |

Flag any notebook that fails any check as NEEDS FIX with severity.

---
name: audit-cache
description: >
  Finds orphaned cache() calls without matching unpersist().
  Critical on constrained Spark capacity where orphaned caches
  cause OOM crashes on shared sessions.
allowed-tools:
  - Read
  - Bash(grep:*)
---

# Audit Cache/Unpersist Pairs

Search all .py and notebook files for .cache() calls.
For each cached DataFrame, verify a matching .unpersist()
exists in the same notebook.

On constrained capacity, orphaned caches cause OOM crashes
on shared Spark sessions. This is a production blocker.

List all violations with the exact line where
unpersist() should be added.

---
name: check-merge-keys
description: >
  Verifies that every Delta Lake MERGE operation uses the correct
  composite key as documented in the table schemas. Partial keys
  cause silent duplicates that compound over time.
allowed-tools:
  - Read
  - Bash(grep:*)
context:
  - docs/table-schemas.md
---

# Check MERGE Key Consistency

For every Delta Lake MERGE operation in the codebase:
1. Extract the merge condition (composite key)
2. Verify the key matches the table's documented primary key
   in @docs/table-schemas.md
3. Flag any MERGE that uses a partial key — this causes
   silent duplicates that compound over time

Report as: | Table | Expected Key | Actual Key | Match? |

Notice the frontmatter differences. The validate-serving-layer skill restricts tools to Read and basic Bash commands — it is an auditor, not a modifier. The check-merge-keys skill uses the context field to automatically load the table schemas documentation, so Claude has the reference data it needs without searching for it.

Running /validate-serving-layer takes seconds and catches the exact failure mode that cost days to debug in production: a serving database silently misdirecting documents when a partition key is missing. The skill encodes that institutional knowledge permanently — it does not depend on any individual remembering to check.

Agent Skills vs. User-Invoked Skills

Governance audits are user-invoked — you call /validate-serving-layer when you want a check. But the spark-notebook skill can work both ways: invoked explicitly with /spark-notebook, or loaded automatically when Claude detects you are creating a new pipeline. The description field in the frontmatter is what drives automatic loading. Write it like a search query for the kind of work it applies to, and Claude will load it when the context matches.

08 · Built-in Skills Worth Knowing

Claude Code ships with bundled skills that complement custom platform skills. Several are directly relevant to data platform development:

• /review — Code review with context-aware analysis. Useful for catching quality rule violations in PRs before custom audit skills run.

• /simplify — Refactors complex code for clarity. On a platform where notebook readability directly affects maintainability, this is high leverage.

• /batch — Runs a command across multiple files. Combined with audit skills, this enables bulk compliance checks: “run /audit-cache across all notebooks in the telemetry domain.”

• /loop — Executes a prompt at recurring intervals within a session. Acts as a lightweight monitoring layer during deployment windows: /loop 30m "check pipeline logs for CRITICAL errors".

• /debug — Reads the session debug log and diagnoses Claude Code issues. Useful when a skill behaves unexpectedly or a tool call fails silently.

These bundled skills work alongside custom skills. A typical governance workflow might chain /review for general code quality, then /validate-serving-layer for platform-specific checks, then /audit-cache for memory safety — all in the same session.

What’s Next

This post covered the setup: CLAUDE.md as the architectural routing table, the unified skills system with frontmatter and supporting files, pipeline creation skills that encode mandatory structures, and governance audit skills that verify compliance.

In Part III, we take these enablers into production. We cover how the six quality rules become an enforceable governance framework, how Terraform and notebook code are cross-referenced to prevent infrastructure drift, a complete Spark notebook example using the tail-fetching pattern, how auto memory compounds debugging insights across sessions, and how agentic workflows chain everything into full-repository audits that run in seconds.

Claude Code · CLAUDE.md · Unified Skills · Frontmatter · PySpark · Delta Lake · Data Governance · Platform Engineering

In Part II, we set up the enablers that embed architectural knowledge into Claude Code for a data platform:

• CLAUDE.md — the platform’s persistent memory: architecture rules, quality gates, and environment configuration loaded at every session start.

• Pipeline creation skills — skills with frontmatter and supporting files (templates, examples, validation scripts) that encode the mandatory notebook structure, so every new pipeline starts compliant. Auto-loaded by Claude when it detects relevant work, or invoked explicitly with /spark-notebook.

• Governance audit skills — user-invoked skills (/validate-serving-layer, /audit-cache, /check-merge-keys) that verify existing code against the quality rules in seconds. Each uses allowed-tools to restrict itself to read-only operations.

• Bundled skills — /review, /batch, /loop, and /debug ship with Claude Code and chain with custom skills for comprehensive governance workflows.

This post takes those enablers into production. We show how a representative set of six quality rules—illustrative rather than exhaustive—can be operationalized into an enforceable framework, how a dedicated skill cross-references Terraform and notebook code, what a complete Spark notebook looks like when generated by the skill, and how auto memory and agentic workflows compound the platform’s resilience over time.

01 · Data Quality and Governance Through Claude Code

The six quality rules from the platform architecture are not documentation — they become enforceable checks when embedded into the skills system. Each rule maps to a specific skill, a specific mechanism, and a specific moment in the development cycle:

Notice the mix of enforcement types. Pipeline creation skills (like spark-notebook) prevent violations at generation time — the code is never wrong in the first place. Governance audit skills (like /validate-serving-layer and /audit-cache) verify existing code retroactively. And CLAUDE.md rules apply to every session regardless of which skill is active. The three layers overlap by design: if a creation skill misses something, an audit skill catches it; if both miss it, auto memory (Section 04) learns from the production failure and flags it next time.

The Governance Feedback Loop

When Claude Code detects a violation — an orphaned cache() or a missing partition key — it does not just flag the problem. It explains why the rule exists (OOM on shared sessions, silent data loss in the serving layer) and suggests the exact fix. Over time, this trains the development team to internalize the rules, not just follow them. The tool becomes a teaching mechanism, not just a linting tool.

For domain-specific data quality, Claude Code validates incoming data patterns against documented thresholds. Coordinates outside valid bounding boxes, sensor readings outside physical ranges, alert classifications that must be computed consistently — all of these can be checked through a single audit skill that reads both the code and the validation rules documentation referenced via the context frontmatter field.

02 · Terraform — Infrastructure-to-Application Consistency

Infrastructure as Code is not just about provisioning. In a data platform, the Terraform definitions create the contract that application code must follow: container names, partition key paths, throughput settings, Key Vault policies. Claude Code helps enforce that contract through a dedicated cross-check skill.

# infra/modules/cosmos/main.tf

resource "azurerm_cosmosdb_account" "analytics" {
  name                = "dataplatform-${var.environment}-cosmos"
  location            = var.location
  resource_group_name = var.resource_group_name
  offer_type          = "Standard"
  kind                = "GlobalDocumentDB"

  consistency_policy {
    consistency_level = "Session"
  }
}

resource "azurerm_cosmosdb_sql_database" "analytics_db" {
  name                = "operational-analytics"
  resource_group_name = var.resource_group_name
  account_name        = azurerm_cosmosdb_account.analytics.name
}

# One container per Gold-layer entity
resource "azurerm_cosmosdb_sql_container" "batch_kpis" {
  name                = "gold-batch-kpis"
  resource_group_name = var.resource_group_name
  account_name        = azurerm_cosmosdb_account.analytics.name
  database_name       = azurerm_cosmosdb_sql_database.analytics_db.name
  partition_key_paths  = ["/partition_date"]   # ← Must match notebook code
  throughput          = 400
}

# infra/modules/fabric/main.tf

resource "azurerm_fabric_capacity" "analytics" {
  name                = "dataplatform-${var.environment}"
  resource_group_name = var.resource_group_name
  location            = var.location
  sku {
    name = "F4"     # 4 CU ceiling — budget constraint
    tier = "Fabric"
  }
}

resource "azurerm_key_vault" "dataplatform" {
  name                     = "kv-dataplatform-${var.environment}"
  location                 = var.location
  resource_group_name      = var.resource_group_name
  sku_name                 = "standard"
  tenant_id                = data.azurerm_client_config.current.tenant_id
  purge_protection_enabled = true
}

This cross-boundary check is a natural fit for the skills system. A dedicated skill can auto-load the Terraform files via the context frontmatter field and cross-reference them against notebook write targets:

---
name: check-infra-consistency
description: >
  Cross-references Terraform Cosmos DB container definitions against
  notebook write targets. Detects partition key mismatches, missing
  containers, and naming convention violations across infrastructure
  and application code.
allowed-tools:
  - Read
  - Bash(grep:*)
  - Bash(find:*)
context:
  - infra/modules/cosmos/main.tf
  - docs/table-schemas.md
---

# Check Infrastructure-to-Application Consistency

1. Parse all Cosmos container definitions in infra/modules/cosmos/
   Extract: container name, partition_key_paths, database name
2. Search all notebooks for Cosmos write operations
   Extract: target container name, partition key field in document
3. Cross-reference:
   - Does every notebook container match a Terraform definition?
   - Does every Terraform container have a notebook that writes to it?
   - Do partition key paths match between infra and application code?
4. Check naming convention: "gold-{domain}-{entity}" in both layers

Report as:
| Container | In Terraform | In Notebooks | Partition Key Match | Naming OK |

Flag mismatches as CRITICAL — these cause silent data loss.

This is the kind of drift that traditional CI/CD pipelines do not cover because it spans two different codebases — infrastructure and application. A single skill bridges both.

03 · Spark Notebook — The Tail-Fetching Pattern in Practice

Here is a simplified version of the incremental replication pattern — the technique that bridges batch boundaries for gapless time series. This is what Claude Code generates when the /spark-notebook skill is invoked. The skill loads its supporting files — the cell skeleton from templates/notebook-template.py and the reference implementation from examples/silver-telemetry.py — then produces a notebook where every cell follows the mandatory structure, every config uses the 3-tier cascade, and every quality rule is applied automatically.

# notebooks/silver_incremental_telemetry.py

# ── Cell 1: Parameters ──
inj_config = ""   # Injected by orchestrator, empty if standalone
environment = "dev"

# ── Cell 2: Configuration (3-tier cascade) ──
SOURCE_CONN = spark.conf.get(
    "spark.dataplatform.source.connection",
    "jdbc:postgresql://localhost:5432/telemetry"   # dev default
)
SILVER_TABLE = "lakehouse.silver_incremental_telemetry"
BATCH_HOURS = 2
ACTIVE_THRESHOLD = 0.50  # Named constant — not inline literal

# ── Cell 3: Read Source + Tail Fetch ──
new_batch = spark.read.jdbc(
    SOURCE_CONN, "telemetry_events",
    properties={"fetchsize": "5000"}
).filter(f"event_time >= NOW() - INTERVAL '{BATCH_HOURS} hours'")

# Tail records bridge the previous batch boundary
from pyspark.sql import functions as F
from pyspark.sql.window import Window

existing = spark.read.format("delta").table(SILVER_TABLE)
tails = existing.groupBy("entity_id").agg(
    F.max("event_time").alias("event_time")
)
tail_records = existing.join(tails, ["entity_id", "event_time"])

# ── Cell 4: Transform — LEAD window for end_time ──
combined = tail_records.unionByName(new_batch)
w = Window.partitionBy("entity_id").orderBy("event_time")
with_end = combined.withColumn(
    "end_time", F.lead("event_time").over(w)
)
result = with_end.filter(
    f"event_time >= NOW() - INTERVAL '{BATCH_HOURS} hours'"
).cache()

# ── Cell 5: Derive status from thresholds ──
result = result.withColumn("status",
    F.when(F.col("connected") == False, F.lit("Disconnected"))
     .when(F.col("idle_ratio") > ACTIVE_THRESHOLD, F.lit("Idle"))
     .when(F.col("activity_metric") > 0, F.lit("Active"))
     .otherwise(F.lit("Standby"))
)

# ── Cell 6: Write — Delta Lake MERGE ──
from delta.tables import DeltaTable
target = DeltaTable.forName(spark, SILVER_TABLE)
target.alias("t").merge(
    result.alias("s"),
    "t.entity_id = s.entity_id AND t.event_time = s.event_time"
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

# ── Cell 7: Cleanup ──
result.unpersist()

The pattern is generic. Replace entity_id with any entity key — asset IDs, device IDs, vehicle IDs, sensor IDs — and the tail-fetching technique works identically. The skill encodes the pattern; the developer provides the domain context. Claude Code handles the structural compliance.

After generating the notebook, you can run the skill’s validation script (scripts/validate-structure.sh) to confirm compliance, or invoke /audit-cache to verify that every cache() is paired with unpersist(). The creation skill and the audit skills form a closed loop.

04 · Auto Memory — Compounding Knowledge Across Sessions

Claude Code builds auto memory as it works — saving learnings like build commands, debugging insights, and project-specific patterns across sessions without explicit configuration. For a complex data platform, this creates a compounding knowledge loop:

How Memory Compounds

After debugging an OOM crash caused by an orphaned cache() on constrained capacity, Claude Code saves that insight. Next time it creates or reviews a notebook, it proactively flags unmatched cache calls — not because of a rule in CLAUDE.md or a skill’s anti-pattern list, but because it learned from a real production failure.

After encountering a serving-layer write that silently failed because of a missing partition key, that debugging pattern enters Claude’s working memory. It surfaces the warning before the same mistake happens again — across all developers on the project, not just the one who fixed it originally.

This is the strategic advantage that separates AI-assisted development from traditional tooling. A linter checks rules that were explicitly written. Skills encode patterns that were deliberately designed. Auto memory adds a third layer: patterns that were learned from experience. Every debugging session, every production incident, every code review feeds back into Claude Code’s project-specific understanding. The platform gets harder to break over time — not through more rules, but through accumulated operational wisdom.

The Three Layers of Protection

Think of it as defense in depth. Skills prevent known violations at generation time. Governance audit skills catch violations that slip through in existing code. Auto memory catches the violations that nobody thought to write a rule for yet — because they have not happened before. When a new failure mode occurs and gets debugged, it enters auto memory immediately, before anyone writes a new skill or updates CLAUDE.md. The formal rule comes later; the protection starts now.

05 · Agentic Workflows — Cross-Cutting Audits at Scale

The most powerful use of Claude Code for data platforms is not generating code — it is running cross-cutting audits that span infrastructure, application code, and documentation simultaneously. These workflows chain custom skills with bundled skills for comprehensive checks.

# Full governance audit — chains custom and bundled skills
$ claude -p "Run all governance checks:
  1. /validate-serving-layer — partition keys
  2. /audit-cache — orphaned caches
  3. /check-merge-keys — MERGE key consistency
  4. /check-infra-consistency — Terraform ↔ notebook alignment
  Then run /review on any notebook flagged as NEEDS FIX.
  Report all findings as a single markdown table with severity."

# Bulk audit across a domain using /batch
$ claude
> /batch "run /audit-cache on every notebook in notebooks/telemetry/"

# Pipe deployment logs for anomaly detection
$ tail -500 pipeline-run.log | claude -p "Analyze for:
  - Failed MERGE operations
  - Serving-layer write timeouts
  - Unexpected null counts in partition columns
  - Spark OOM warnings
  Classify each as CRITICAL / WARNING / INFO"

# Recurring monitoring during deployments
$ claude
> /loop 30m "Check the latest pipeline run logs for
  CRITICAL errors. Summarize and suggest a fix."

The workflow chains four custom governance skills, then feeds flagged notebooks into the bundled /review skill for general code quality analysis. The /batch bundled skill enables running any audit across multiple files in a single invocation — useful for domain-wide compliance checks before a release. The /loop skill acts as a lightweight monitoring layer during deployment windows, and the Unix-philosophy piping (tail | claude) integrates Claude Code into the same operational workflows that already exist in the terminal.

06 · The Full Development Stack

Each layer builds on the previous one. CLAUDE.md provides the architectural foundation. Pipeline creation skills encode the implementation patterns. Governance audit skills verify compliance — amplified by bundled skills like /review and /batch for reach. Auto memory retains the lessons from production. And agentic workflows tie everything together into a system that actively prevents regression — not just in code, but in the cross-cutting architectural decisions that make a data platform reliable.

07 · Key Takeaways

Claude Code is not a code generator — it is an architectural enforcement layer. For complex data platforms, the value is not in writing PySpark faster. It is in embedding foundational and reliable knowledge into the development workflow so that quality rules are enforced automatically, failure patterns are remembered across sessions, and cross-cutting checks run in seconds rather than hours.

CLAUDE.md is a routing table, not a knowledge dump. Keep it concise, encode the rules, and use progressive disclosure for deeper context. Under 60 lines is enough for production ready pipeline platform.

Pipeline creation skills prevent violations at generation time. Catching a missing partition key during code review costs hours. Preventing it with a skill that encodes the mandatory structure, supported by templates and examples, costs zero. Encode every repeatable pipeline pattern as a skill.

Governance audit skills make compliance auditable and repeatable. The /validate-serving-layer skill catches the exact failure mode that caused days of debugging — and it runs in seconds. Every hard-won lesson from production should become a skill. Use allowed-tools to keep audit skills read-only and context to auto-load reference documentation.

Auto memory creates a compounding advantage. Skills catch known patterns. Memory catches patterns that nobody thought to write a skill for yet. The more failures Claude Code sees, the harder the platform becomes to break.

Cross-boundary consistency is the hardest problem. Terraform container definitions, notebook partition keys, serving-layer document schemas, and documentation must all agree. A single skill with the right context fields bridges infrastructure and application code — catching drift that no single-tool CI/CD pipeline can detect.

The Series

• Part I — How to Build a Real-Time Analytics Platform on a Constrained Budget.

• Part II — Development Engine for Complex Data Platforms.

• Part III — Enforcing Data Quality and Governance with Claude Code. (You are here)

Claude Code · Terraform · PySpark · Delta Lake · Unified Skills · Data Governance · Auto Memory · Agentic Workflows · Platform Engineering

Every data-intensive operation faces the same structural problem: the systems that generate data were not designed to talk to each other. Dispatch platforms export batch files on a X-hour cadence. Telemetry systems stream sensor readings every 10 minutes. GPS trackers emit continuous position data. Environmental sensors report for regulatory compliance. Operational databases accumulate millions of rows of time-series data.

Each source has its own format, cadence, latency requirements, and failure modes. The business, however, does not care about any of that. It needs a unified analytical layer that produces consistent KPIs, feeds real-time dashboards, and supports historical analysis — all on a compute budget that is not unlimited.

This is a pattern that repeats across industries: manufacturing, logistics, energy, mining, agriculture, insurance, financial, etc. The specific data sources change, but the architectural challenge is the same — how do you turn fragmented, heterogeneous data streams into a single source of truth without building an expensive monolith?

The Real Constraint

The hardest problem is not ingesting the data. It is maintaining consistency across all consumers. When a KPI is computed in a Spark notebook, displayed in a Power BI dashboard, and served through a REST API, all three must produce the same number. Business constants, exclusion rules, and threshold logic must be identical everywhere — and they must survive team changes, late-arriving data, and pipeline failures.

02 · Architecture Strategy: The Medallion Pattern

The medallion architecture is not new. But the value is not in the pattern itself — it is in the discipline of separating concerns cleanly. Each layer has a specific responsibility, and no layer is allowed to bypass the one before it.

Bronze is a pure landing zone. Data arrives exactly as-is — no transformations, no cleaning, no schema enforcement. This is the safety net: if anything goes wrong downstream, the raw data is always available for replay.

Silver is where trust is established. Schema enforcement, data type validation, deduplication, and ACID-compliant writes (via Delta Lake MERGE) happen here. Every record that enters Silver has passed a set of validation rules. This is the layer where domain-specific checks — geographic bounding boxes, valid ranges, referential integrity — are applied at ingestion time, not after the fact.

Gold is the business layer. Aggregations, enrichment with dimension tables, KPI computation, and industry-standard metrics are calculated here. This is the only layer that consumers should ever query directly.

Serving is the delivery mechanism. Cosmos DB provides sub-second latency for real-time dashboards. Power BI connects directly to Gold tables for historical reporting. REST APIs expose KPIs to external systems.

Why This Matters Strategically

The medallion pattern is fundamentally a risk management architecture. Each layer isolates a different kind of failure. Bronze absorbs source-system instability. Silver prevents bad data from contaminating analytics. Gold ensures business logic is computed once and shared everywhere. When something breaks — and in a 24/7 operation, it will — the blast radius is contained.

03 · Three Ingestion Patterns for Different Data Behaviors

Not all data behaves the same way. A shift report that arrives every 12 hours requires a fundamentally different ingestion strategy than a sensor reading that arrives every 10 minutes, or a database that accumulates millions of rows of time-series data. Trying to force all three through the same pipeline creates fragile, overloaded systems.

I want to share three distinct ingestion patterns proved by my self in large data solutions, each designed for a specific data behavior:

Pattern 1 — Orchestrated Batch

For periodic, structured exports: an orchestrator notebook acts as the single entry point. It resolves all secrets once, reads all source files once into shared Spark temp views, then runs child notebooks sequentially within a single Spark session. This reduces cold start overhead by approximately 60% and guarantees that all child notebooks consume identical input data. Each child can still run independently in development mode by falling back to its own file discovery logic.

Pattern 2 — Hybrid Event-Driven + Scheduled

For multiple independent streams with different formats and frequencies: each stream gets its own isolated notebook with three execution modes. Event-driven (triggered by new files, sub-5-minute latency) handles 90% of normal operations. Scheduled (every 1–4 hours, using time-based path generation instead of recursive directory listing) provides self-healing recovery. Backfill (full recursive scan, manual trigger) supports initial loads. The scheduled mode uses generated path patterns rather than listing thousands of folders, so performance stays constant regardless of total data volume.

Pattern 3 — Incremental Replication with Tail Fetching

For continuously growing source databases: the pipeline fetches new records from the source, but also retrieves the tail record per entity from the existing Delta Lake table. These tails are unioned with the new batch before applying window functions (LEAD) to compute derived fields like end times. This bridges batch boundaries without reprocessing historical data, keeping each run’s scope minimal while producing a complete, gapless time series.

The key insight: these three patterns are transferable. They are not specific to any industry or vendor. An orchestrated batch pattern works the same way whether it processes manufacturing production runs or logistics delivery cycles. The hybrid event/scheduled pattern applies anywhere you have multiple independent data sources with different cadences. And tail fetching solves the incremental time-series problem regardless of the source database technology.

04 · Data Governance as Engineering Discipline

Governance in a real-time analytics platform cannot be a separate workstream. It has to be embedded in every write operation, every pipeline structure, and every deployment. Here is the framework that makes that practical:

Idempotency everywhere. Every write operation uses Delta Lake MERGE (UPSERT). Re-running any pipeline for any time window produces identical results without creating duplicates. When pipelines run on schedules, they will occasionally overlap, retry, or be manually re-triggered. Every write must be safe to repeat. This is the foundation of data trust in any batch or hybrid processing system.

Validation at ingestion, not downstream. Bad data that enters the Silver layer will propagate to every consumer. Geographic coordinates outside valid ranges are rejected on arrival. Geometry data is validated and centroids are extracted during ingestion. Alert thresholds are computed at write time with classification logic that cannot be manually overridden. The principle: if a record passes Silver, it is trustworthy.

Configuration hierarchy, not hardcoded values. All pipelines use a 3-tier cascade: Key Vault → Spark Config → Default Value. This supports multiple environments (dev/UAT/prod) without code changes and makes standalone notebook development possible without infrastructure access.

Six Quality Rules (at least) — Enforced Repo-Wide

These are not guidelines. They are deployment blockers:

1. Every environment config must have a default value.

2. Every document in the serving layer must carry its partition key.

3. All fact-to-dimension joins must be LEFT joins with count guards (depends on use case).

4. Every cache() must be paired with unpersist() to prevent memory exhaustion on shared sessions.

5. Named constants replace hardcoded literals everywhere.

6. Imports stay at the top of each module.

Documentation as engineering. Every code change requires a documentation update in the same commit. User-facing documentation must consider all the roles involved (Data engineers don’t speak data scientists’ language). Technical context files include mandatory checklists. Incomplete documentation equals incomplete work — not a nice-to-have that gets deferred.

05 · Budget Strategy: Running Pipelines on Minimal Compute

Architecture discussions often ignore the budget constraint. But the compute budget is the most powerful forcing function for good engineering. When you have unlimited resources, you can afford sloppy scheduling, redundant processing, and unoptimized pipelines. When you have a strict ceiling, every design decision matters.

For example if platform runs on a Fabric F4 capacity — one of the smallest SKUs available — with a ceiling of 4 Capacity Units. The entire scheduling strategy should be designed around staying within that limit.

The budget constraint is not a limitation — it is a design principle. It forces incremental processing (no full table reprocessing), shared Spark sessions (no redundant cold starts), and staggered scheduling (no overlapping compute bursts). These same patterns reduce cost at any scale, not just on an F4.

06 · Investment Strategy: CAPEX vs. OPEX in Modern Data Platforms

The traditional approach to data infrastructure treats it as a capital expenditure: buy servers, license software, build a data center, amortize over 3–5 years. This model has a fundamental flaw for analytics platforms — you are locking in capacity before you know what the platform needs to do.

Cloud-native, consumption-based platforms flip this model. The entire compute cost becomes OPEX — adjustable month to month, tied to actual usage rather than projected capacity. This has three strategic implications:

1. Prove Value Before Scaling

A small-capacity SKU costs a fraction of what traditional on-premise infrastructure requires. The platform can run in production, serve real dashboards, and demonstrate measurable ROI — all before a single scaling decision needs to be made. Investment follows validated results, not projected ones.

2. Granular Scaling, Not Step-Function Upgrades

Scaling from F4 to F8, F16, or F64 is a configuration change, not a procurement cycle. More importantly, the architecture is designed so that scaling does not require re-engineering. Staggered scheduling works the same way at any capacity. The orchestrator pattern, incremental processing, and tail fetching all produce constant-scope workloads regardless of data growth. The platform scales by adding capacity, not by rewriting pipelines.

3. Reversible Decisions

If a processing domain turns out to deliver less value than expected, its pipelines can be decommissioned and the capacity reclaimed immediately. With CAPEX infrastructure, sunk costs create pressure to keep using systems that have outlived their usefulness. With OPEX, every component must continuously justify its compute consumption.

This does not mean CAPEX has no role. For extremely stable, high-volume workloads where utilization is predictable, reserved capacity or committed-use discounts can reduce per-unit costs significantly. The strategic approach is to start OPEX, prove value, then selectively convert proven workloads to reserved capacity as their patterns stabilize. This minimizes risk at every stage.

07 · Scalability Path and ROI

This architecture was designed with explicit scalability mechanisms — patterns that support growth without requiring re-architecture:

Staggered scheduling absorbs new pipelines. Adding new data streams does not require more compute — just more precise time-slot management. The same capacity budget can serve additional pipelines by adjusting offsets and frequency windows.

The orchestrator pattern reduces marginal cost. Adding a new analytical notebook to an existing processing domain requires minimal incremental compute, because secrets, file reads, and Spark sessions are already shared. The overhead per notebook decreases as the domain grows.

Incremental processing keeps costs constant. Tail fetching and Delta Lake MERGE ensure that each run processes only new and updated records. Whether the historical table holds 1 million or 50 million rows, the per-cycle cost stays the same.

The medallion pattern supports new consumers without new processing. Once data reaches Gold, any number of serving endpoints — new dashboards, new APIs, new reporting tools — can consume it without touching the processing pipelines. The marginal cost of a new consumer is nearly zero.

The ROI Equation

The return on a real-time analytics platform is not in the infrastructure — it is in the decisions the infrastructure enables. When KPIs are computed consistently and served in near real-time, the business can act on operational signals faster: identifying underperforming assets, reducing unnecessary maintenance cycles, optimizing resource allocation. In capital-intensive industries, even a small percentage improvement in asset utilization can pay for the entire platform many times over. The architecture’s job is to make those decisions trustworthy, fast, and affordable.

08 · Key Takeaways

Separate concerns cleanly. The medallion pattern is not a buzzword — it is a risk management architecture. Each layer absorbs a different failure mode. Bronze absorbs source instability, Silver enforces trust, Gold computes business logic once, and the serving layer delivers it everywhere.

Match ingestion patterns to data behavior. Orchestrated batch, hybrid event/scheduled, and incremental replication with tail fetching are three distinct patterns for three distinct data behaviors. Forcing one pattern onto all data sources creates fragility.

Embed governance in the pipeline, not beside it. Idempotent writes, validation at ingestion, and enforced quality rules are not polish added after the platform works — they are the reason the platform works. Without them, every new pipeline becomes a liability.

Use the budget as a design principle. Constraints force better engineering. Staggered scheduling, shared sessions, and incremental processing are not compromises — they are patterns that reduce cost at any scale.

Start OPEX, prove value, then optimize. Cloud-native consumption models eliminate the need to justify large upfront investments. Prove the platform delivers measurable results on minimal compute, then scale with confidence.

Invest in transferable patterns, not vendor lock-in. The orchestrator pattern, tail fetching, hybrid ingestion, and the medallion architecture work on Fabric, Databricks, Snowflake, or any modern lakehouse platform. The engineering discipline is the asset — the vendor is the vehicle.

Medallion Architecture · Data Governance · CAPEX/OPEX · Delta Lake · Microsoft Fabric Apache Spark · Cosmos DB · Incremental ETL · Platform Strategy · Budget Optimization

Compartir Sandro Gomez’s personal blog

Fifty-five microservices running across three environments are essential for critical operations, processing millions of events, messages, and records.

The challenge

The primary challenge is to unify the logging systems and standards to achieve comprehensive observability, enabling the detection of service deterioration, errors, and potential failures.

The Enabler

Claude Code

The result

The investment

The Impact

Final Thoughs

Be serious… AI is changing the game.

Are you interested in the real game of AI? Let’s talk…

Un “Agilista” es un término utilizado para describir a alguien que practica o aboga por metodologías ágiles en la gestión de proyectos y el desarrollo de software. Las metodologías ágiles y los diferentes marcos de trabajo, enfatizan la flexibilidad, la mejora continua y la capacidad de adaptarse a los requisitos en contextos cambiantes.

Algunas características clave que definen a un Agilista incluyen:

Colaboración

Los agilistas priorizan el trabajo en equipo y la colaboración entre los miembros del equipo y las partes interesadas. Facilitan la comunicación para garantizar que todos estén alineados y que los objetivos del proyecto se entiendan claramente.

Adaptabilidad y flexibilidad

Defienden la adaptabilidad y alientan a los equipos a responder a los cambios en lugar de seguir un plan preestablecido. Esto refleja el principio ágil central de responder al cambio en lugar de seguir un plan.

Desarrollo Iterativo

Los agilistas apoyan el desarrollo iterativo, donde los proyectos se dividen en segmentos manejables (iteraciones), lo que permite a los equipos reevaluar y ajustar con frecuencia su dirección.

Enfoque centrado en el cliente

Mantienen un fuerte enfoque en entregar valor al cliente, a menudo involucrándolo durante todo el proceso de desarrollo para garantizar que el producto final satisfaga sus necesidades y expectativas.

Mejora continua

Un agilista cree en la mejora constante de procesos, productos y prácticas a través de ciclos regulares de reflexión y adaptación.

Liderazgo y facilitación

En los diferentes roles que asumen, los Agilistas actúan como facilitadores en lugar de gerentes tradicionales, ayudando a los equipos a auto-organizarse y crear un entorno propicio para un trabajo eficaz y un aprendizaje continuo.

Comentarios

Compañías que abrazan los principios y valores de la agilidad, se destacan como organizaciones resilientes a los cambios y desafíos emergentes del mundo actual. Destaca la cultura centrada en las personas creando espacios seguros de experimentación e innovación. Con enfoque único en sus clientes y shareholders son hábiles en entregar valor de forma consistente y oportuna (What is agile? — Mckinsey).

Key Takeaways

Algunas recomendaciones

• Entender los principios ágiles y los gaps de tu organización.

• Documentate acerca de la agilidad, sus origenes, evolución e implementación (buenas y malas prácticas).

• Formaliza tus conocimientos buscando entrenamiento en el rol que mejor te defina.

• Lee mucho y busca ayuda en comunidades y foros.

Te comparto un par de libros que te ayudarán en el viaje.

1. The Age of Agile

2. Agile Transformation: Structures, Processes and Mindsets for the Digital Age

3. Lean Change Management

4. Agile Change Management

5. Succeding with Agile

6. Agile Estimating and Planning