Monolith into Microservices

The problem

This project started as a legacy PHP monolith where a single file.php handled the entire ETL workflow: fetching a thounsend of records from API, running multiple transformations, enriching the data with database queries, and saving the final result back into the database.

The main problem was maintainability. When one step failed, we did not have enough visibility to understand the root cause. There were no clear logs, no centralized error tracking, and the whole process had to be rerun from scratch.

Architecture & approach

Microservices were new to the team, so we started with a proof of concept. I defined how to split the ETL pipeline into small, reusable services, where each service had a single responsibility:

• Fetch data from the Social Media API
• Transform the raw data
• Enrich the data using database information
• Persist the final result

Each step was deployed independently on Kubernetes, which made the system easier to maintain, scale, and debug.

To avoid tight coupling between services, we introduced RabbitMQ as a message broker. Instead of having one long synchronous process, each service published and consumed messages from queues. This allowed the pipeline to continue step by step, retry failed jobs, and avoid rerunning the entire process from the beginning.

Expand

Observability & Error Handling

One of the biggest improvements was adding better observability to the pipeline.

We integrated Sentry to capture exceptions, track failed jobs, and identify exactly which service and step caused the error. This gave us more context when debugging issues, including the failed operation, error details, and the affected part of the ETL flow.

Combined with service-level logging, Sentry helped us move from a “black box” process to a traceable pipeline where failures were easier to investigate and recover from.

Outcome

We ended up with a fully traceable pipeline: when something fails, we know exactly where and why, and we resume from that point instead of rerunning from zero. Migrating complex processes became simpler because we reused existing services, and we stopped paying for large fixed servers — smaller on-demand ones took their place.