Cloud Governance
Platform Engineering

A mature cloud governance repository was being used continuously by multiple SRE teams supporting numerous development teams. It served as a central entry point for infrastructure requests, with changes going through automated validation, Terraform planning, ownership approvals and cost checks before they could be merged.

As adoption grew, so did the operational complexity.

There was limited visibility into repository activity, pipeline health and execution performance. Understanding failures, identifying bottlenecks and tracking the overall behaviour of the platform required a dedicated monitoring solution.

The merge process also introduced friction. Approvers frequently handled multiple infrastructure requests, and repeated rebasing and pipeline execution made the process slower than it needed to be.

I designed and developed a monitoring solution to provide operational visibility into the governance repository.

The application was built using Python and deployed on Google Kubernetes Engine (GKE). It consisted of three independent containers, each responsible for collecting a different category of GitLab information: jobs, pipelines and merge requests.

Each collector periodically retrieved information from the GitLab APIs, approximately every two minutes, and stored the collected data in a shared Redis instance.

A separate metrics exporter retrieved the consolidated information from Redis, converted it into metrics and exposed them through an HTTP endpoint. The existing VictoriaMetrics infrastructure automatically scraped these metrics, while Grafana provided the visualization layer.

One of the interesting design decisions was introducing Redis between data collection and metric generation. By consolidating the collected information in one place, the exporter could retrieve what it needed through a single lookup rather than coordinating multiple collectors or repeatedly querying the GitLab APIs.

The collected information was presented through three Grafana dashboards covering jobs, pipelines and merge requests.

The dashboards provided visibility into:

We also introduced alerting for abnormal failure levels. When failures exceeded expected thresholds, the team received notifications so potential problems could be investigated.

This gave us a much clearer understanding of how the repository was behaving. We could identify recurring failures, observe execution-time trends and use the collected information to improve pipeline performance and operational reliability.

Another important improvement was introducing GitLab Merge Trains.

The governance repository supported multiple SRE teams, each handling infrastructure requests for several development teams. As a result, individual approvers could be responsible for reviewing and merging numerous merge requests.

The existing process created unnecessary repetition.

Merge requests needed to be rebased against the latest target branch, and all required validation jobs had to pass before merging. If another merge request was merged in the meantime, the remaining requests needed to be rebased again, triggering another round of validation.

These pipelines included:

Since these validations ran on every commit, repeated rebasing resulted in additional pipeline executions and waiting time for both engineers and approvers.

Merge Trains helped address this by allowing merge requests to be queued for integration against the target branch. Instead of manually coordinating rebases and waiting for the branch to become available, approvers could add eligible merge requests to the train.

The train's validation process helped identify integration issues before changes reached the target branch, reducing the manual effort involved in coordinating multiple infrastructure changes.

I designed and implemented the monitoring application, including:

Beyond the monitoring solution, I contributed to improving the governance repository's CI/CD workflows and operational maturity.

This included helping introduce Merge Trains to reduce repeated rebasing and unnecessary pipeline executions, making the infrastructure request and approval process more efficient for the teams using the repository.

The governance repository gained a dedicated observability layer, providing visibility into its jobs, pipelines and merge requests through centralized dashboards and alerts.

The collected metrics helped the team understand execution patterns, investigate failures and identify opportunities to improve pipeline performance.

The introduction of Merge Trains also streamlined the merge process for teams handling multiple infrastructure requests, reducing the coordination and repeated validation effort associated with rebasing.

For me, the most rewarding part was building something that went beyond monitoring individual pipeline runs. It gave us the visibility needed to understand how an important engineering platform behaved and continuously make it better.