16 apps reconciled from Git

Git-reconciled · last verified September 2026

DevOps Infrastructure Engineer Malta 8+ years in IT infrastructure

I build reliable platforms that deploy and recover predictably.

Eight-plus years across IT infrastructure, networking and cloud automation. At Altenar I automate GCP with Terraform, Ansible and Python. On my own time I build and operate a four-node K3s cluster — four VMs on one bare-metal Proxmox host — and ship my own products on top of it.

In IT & infrastructure
8+ yrs

Employed · IT support → networking → DevOps

Cluster nodes
4

Personal platform · VMs on one host

Apps reconciled
16

Personal platform · ArgoCD from Git

Pay → playable
46s

WyrmHost · measured on one test purchase

Professional experience

IT support → networking → DevOps · 2016 to now

Employment Malta · 6 roles

Mar 2024 — Present

St Julian's, Malta

DevOps Infrastructure Engineer · Altenar

Automating and operating GCP infrastructure as code, and running the networking layer around it.

  • Modular Terraform configurations and Ansible playbooks that make GCP environments reproducible and version-controlled across multiple regions, instead of built by hand.
  • Scripts in Python and Bash written for specific operational use cases as they came up, taking work off the team that was otherwise done by hand.
  • A Prometheus and Grafana observability stack with automated alerting, giving SLI/SLO tracking and evidence for post-mortems.
  • Containerised workloads with Docker, plus Kubernetes cluster configuration, resource policies and rolling upgrades; Vault-based credential workflows and least-privilege IAM.
  • The networking layer: HAProxy load balancing, FortiGate rulesets, DNS zones and IPsec/SSL VPN gateways, every change version-controlled through Bash automation.
  • GCP
  • Terraform
  • Ansible
  • Python
  • Kubernetes
  • HAProxy
  • FortiGate
  • Vault

Oct 2022 — Mar 2024

Mosta, Malta

Network Administrator · Smart Technologies

Network design, configuration and hardening across client environments on Fortinet, Mikrotik, Cisco, Aruba and Ruckus — with quarterly security audits, gap analysis and prioritised remediation reporting supporting clients' ISO 27001 posture, and SSL/IPsec remote access built with redundant failover. Point of contact for networking support, and the escalation path for tier-1 whenever they couldn't resolve an issue.

  • Fortinet
  • Cisco
  • Mikrotik
  • IPsec / SSL VPN
  • ISO 27001
Earlier roles · 2016—2022

Aug 2021 — Oct 2022

Bugibba, Malta

IT Support Specialist · Pronet Gaming

Full IT infrastructure ownership for the Malta branch — endpoints, LAN, firewall and security perimeter — and the runbooks that turned recurring incidents into routine work.

Nov 2020 — 2021

Marsa, Malta

IT Specialist · Simply VC

Validator node infrastructure for proof-of-stake networks, run to a 24/7 availability target: Prometheus and Grafana monitoring with alerting, and Bash and Python deployment scripts that replaced manual node setup with a standard onboarding path.

Apr 2019 — Nov 2020

Birkirkara, Malta

IT Support Officer · Computime

Windows Server and Linux administration across managed client environments, PowerShell and Bash automation for diagnostics, patch verification and user provisioning, and Azure-hosted servers with Microsoft 365 Exchange tenants.

Apr 2016 — Sep 2018

Paola, Malta

Computer Engineering Assistant · Transport Malta

Early-career IT at a national agency: LAN maintenance, hardware support and web administration, plus end-user support of every kind — printers, user accounts and access, and day-to-day help for the business teams.

Certification: CCNA (200-301) · Cisco

Training: LPIC-1 (101/102) and MCSA coursework · ICE Malta

Education: MCAST Advanced Diploma in Computer Systems & Networking (EQF 4)

Full CV (PDF)

Provisioning path

My own platform · bare metal → running app

Fig. 1 — the pipeline Every layer declared in Git
  1. Proxmox One bare-metal host — the whole failure domain
  2. Terraform + Ansible 5 VMs provisioned, then configured
  3. K3s 4 VM nodes, 1 control plane
  4. ArgoCD 16 apps reconciled from Git
  5. Applications 20+ services on MetalLB + Longhorn

Observed by

Prometheus · metrics + custom exporters

Grafana · dashboards per workload

Alertmanager · routed to Discord

Selected work

Built and operated outside work · 3 featured · 4 smaller builds

kubernetes-platform Deployed

Infrastructure

Self-hosted Kubernetes platform

A four-node K3s cluster — four VMs on one Proxmox host — with every layer declared in Git.

View case study
Problem
Twenty-plus services on one box, every change made by hand. Nothing was reproducible, and a rebuild meant re-deriving undocumented steps.
What I built
Terraform provisions the VMs, Ansible brings up a four-node K3s cluster, ArgoCD reconciles every workload from Git. Plus MetalLB and Nginx Ingress, Longhorn with scheduled snapshots, a cert-manager internal CA for real TLS, and Prometheus/Grafana/Alertmanager with custom exporters alerting to Discord. Longhorn replicates volumes between the cluster VMs — since those VMs share one physical machine, that covers a node or a disk failing, not the host going down.
Result
16 applications reconciled from Git — 12 from this repository and 4 from separate application repos — with drift auto-corrected and promotion by pull request. Rebuilding from bare metal is a documented eight-step playbook rather than a weekend of remembering — and the four incidents below are the ones that shaped what it covers, and what it openly does not.
  • Terraform
  • Ansible
  • K3s
  • ArgoCD
  • Prometheus
Cluster topology

Cluster topology

Proxmox host — one physical machine

  • k3s-master control plane · VM
  • k3s-worker-1 worker · VM
  • k3s-worker-2 worker · VM
  • k3s-worker-3 worker · VM

wings-vm · Docker host, outside the cluster

  • MetalLB + Nginx Ingress · stable addressing
  • Longhorn · replicated storage, scheduled snapshots

Longhorn replicates volumes across the cluster VMs, which protects against a node or a disk failing. All five VMs sit on the same physical machine, so the Proxmox host is still a single failure domain — losing it loses everything at once.

Five VMs on one host — four form the cluster, one runs game servers.

Fig. 2 — how one application is wired

Fig. 2 — how one application is wired A resource graph: the homelab repository feeds the vaultwarden ArgoCD application, which owns a deployment, service, ingress, certificate and volume claim; those resolve to a pod, a TLS secret issued by the internal CA, and a Longhorn volume. GIT ARGOCD RESOURCES RUNTIME repo homelab application vaultwarden deployment vaultwarden service vaultwarden ingress · https vaultwarden certificate · cert-manager vaultwarden-tls pvc · longhorn vaultwarden-data pod vaultwarden secret · issued by the internal CA vaultwarden-tls volume · daily snapshot, retain 7 longhorn
One of the sixteen: the vaultwarden Application and everything ArgoCD reconciles for it — down to the cert-manager certificate that gives it real TLS and the Longhorn volume it stores data on.

Fig. 3 — the same five VMs, observed

A Grafana dashboard fed by a Proxmox exporter. A resource table lists five running QEMU guests — k3s-master with 2 vCPUs and 4 GiB at 85.7% memory, k3s-worker-1 to worker-3 with 2 vCPUs and 6 GiB each at 88.7%, 80.0% and 85.7%, and wings with 3 vCPUs and 10 GiB at 84.1%. Beside it, 24 hours of host CPU history sitting near one core of four, and memory history pinned flat near the top of the range. Gauges read 14.46% CPU and 94% memory, 29.4 GiB used of 31.3 GiB total. View the full-size dashboard (opens in a new tab)
The topology above as Prometheus sees it, through an exporter I run for Proxmox. The same five guests, and the reason a second host is first on the list below: 94% of the host’s 31.3 GiB is committed, and the memory line has been flat against that ceiling for a day. CPU is not the constraint — 14% of four cores. Captured September 2026.
Read the declarations
terraform/main.tf Every VM, from one map
resource "proxmox_virtual_environment_vm" "vm" {
  for_each = var.vms

  name      = each.key
  node_name = var.proxmox_node
  vm_id     = each.value.vmid

  cpu {
    cores = each.value.cores
    type  = each.value.cpu_type
  }
  memory {
    dedicated = each.value.memory
  }

  # The clone source is only used at creation time. The original
  # template VM may no longer exist, so ignore drift here to avoid
  # forcing VM replacement (which would destroy game-server data)
  # on routine changes like resizing.
  lifecycle {
    ignore_changes = [clone]
  }
}
Five VMs come out of one for_each over a variable map, so adding a node is a few lines of input rather than a new resource block. The lifecycle rule is there because a stale clone reference would otherwise replace a running VM — and take the game-server data with it.
kubernetes/argocd/apps.yml 1 of this repo’s 12
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vaultwarden
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/ToonKnight/homelab.git
    targetRevision: HEAD
    path: kubernetes/vaultwarden
  destination:
    server: https://kubernetes.default.svc
    namespace: vaultwarden
  syncPolicy:
    automated:
      prune: false      # deleting is never automatic
      selfHeal: true    # a hand-edit is reverted to Git
    syncOptions:
      - CreateNamespace=true
The Application behind Fig. 2. selfHeal is what makes “declared in Git” true rather than aspirational — edit the cluster by hand and it is put back. prune stays off on purpose: a removed manifest should need a decision, not silently delete a volume.
DISASTER-RECOVERY.md Bare metal → running cluster
StepPlaybookWhat it doesTime
100-provision-vmsCreates the VMs on Proxmox via Terraform~3 min
201-commonPackages and kernel settings on every node~3 min
302-k3s-masterInstalls the K3s control plane~2 min
403-k3s-workersJoins the three workers~2 min
504-k3s-addonsMetalLB, Nginx Ingress, Longhorn, Prometheus stack~10 min
605-kubernetes-appsApplies the application manifests~2 min
7inlineArgoCD install, ingress and this repo’s 12 Applications~3 min
806-wingsThe game-server VM: Docker and the Wings daemon~3 min
Documented total, before the manual steps the runbook lists~25–30 min
The same runbook is explicit about what it does not restore: Longhorn volumes, AdGuard’s config and stats, and Smokeping history are all listed as lost on a full destroy, next to what is safely in Git. Writing the gaps down beside the steps is what makes the document worth trusting at 2am.
wyrmhost Building

Platform automation

WyrmHost

Game-server hosting that provisions itself: pick a plan, pay, and the server boots.

View case study
Problem
Selling game-server hosting means provisioning a server per order. By hand it doesn't scale, and it can't run overnight.
What I built
Storefront, Stripe checkout, queue worker, and the panel API call that creates and boots the server. The storefront had no provider for the panel I run, so I ported one against its API. Each environment is its own GitOps repo with ArgoCD guardrails and manual promotion to production.
Result
Payment to a live, playable server in approximately 46 seconds with no manual steps — measured end to end on a test purchase, not an average over real orders. That includes automatic licence acceptance, which had previously crashed every new server on first boot.
  • Laravel
  • Stripe
  • Docker
  • ArgoCD
  • REST APIs
Provisioning sequence
  1. Checkout Customer picks a plan and pays
  2. Invoice paid Provisioning job queued
  3. Server created Allocation assigned, files installed
  4. Licence accepted Handled automatically, then start
  5. Online Joinable — ~46s total
Measured end-to-end on a live test purchase.
The WyrmHost storefront: a dark landing page headed “Summon a server. It breathes fire in seconds.”, with a Deploy Now button and a mock deploy console showing a server coming online. View the full-size storefront screenshot (opens in a new tab)
The storefront customers buy from — the front door to the sequence above. The provisioning run itself is a queue worker, so there is nothing to screenshot between checkout and the server coming online.
wyrmtable Live

Product · full-stack

Wyrmtable

A real-time tabletop toolkit that keeps a whole table on one shared game state.

View case study
Problem
Running games, I kept the map, the initiative order and everyone's character sheets on my laptop, and the table crowded round one screen. Players could not see their own sheet or roll their own dice without asking me to click something.
What I built
A real-time browser toolkit: shared table state over websockets, server-authoritative dice, combat tracker and character import. I built and run all of it — containerised, versioned migrations, nightly backups with a staleness alert, and its own Grafana dashboards.
Result
Live and public at wyrmtable.eu, behind a Cloudflare Tunnel rather than an open port, with the pod locked down by NetworkPolicy, running non-root and rate-limited on auth. Releases are pinned image tags, promoted by hand.
  • Next.js
  • TypeScript
  • Socket.IO
  • PostgreSQL
  • Kubernetes
The Wyrmtable site: a dark landing page headed “Run your table together, from any browser”, with Run a game and Join a game buttons. View the full-size Wyrmtable screenshot (opens in a new tab)
The public landing page at wyrmtable.eu. A table mid-session sits behind a login, so it is not shown here.
Smaller builds 4 tracked
touchline · fpl-ai · trading-harness · raid-companion
  • touchline

    Web app · coaching

    I coach under-5s to under-7s, so I built the session planner I wanted pitch-side. Say how many kids turned up and how confident they are, and 47 games rebuild to fit — extra areas, a scaled kit list. A run mode keeps time through a sleeping phone, and attendance exports to CSV for the head coach. No backend: data stays in the browser.

    • TypeScript
    • PWA
    • ArgoCD
    Live
  • fpl-ai

    Data · ML

    Ingests official Fantasy Premier League data, models expected points and recommends transfers — advice-only by design. Most of the work was data hygiene: unit mismatches and an unreliable upstream column would have quietly poisoned every recommendation.

    • Python
    • Pandas
    • Kubernetes
    Deployed
  • trading-harness

    Research · systems

    Walk-forward backtesting, realistic fee and slippage modelling, risk limits, and alerting that fires on silence as well as errors. The honest result: no edge. Win rate times payoff landed at 1.0 — the signature of random entries. Measuring that and stopping beat another round of tuning.

    • Python
    • Backtesting
    • GitHub Actions
    Research
  • raid-companion

    Desktop · tooling

    A desktop companion for raid nights, shipped as one self-contained executable — users double-click a file instead of reading an install guide.

    • Electron
    • TypeScript
    • Node.js
    Shipped

Incident history

Four incident write-ups

What broke, and what changed after Root-caused and written up
sev-1

Thin pool hit 100% and locked every VM

Failure
The storage pool filled completely. Writes froze and every VM on the host locked up at once.
Root cause
The hypervisor ran with discard=ignore, so TRIM inside the guests never reached the host. Deleted blocks were never reclaimed — the pool only grew, however much the VMs freed.
Fix
Enabled discard pass-through and reclaimed the orphaned extents with fstrim, bringing usage back under the ceiling.
Prevention
Pool utilisation is now alerted on, so the next approach to the ceiling arrives as a warning, not an outage. A setting that silently disables reclamation stays invisible until it's urgent — verify the whole path, not just the guest.
sev-1 An out-of-memory node took itself offline resolved
Failure
A cluster node stopped responding entirely instead of shedding one workload.
Root cause
A memory-hungry workload consumed the node's RAM. With no kubelet eviction thresholds set, the kubelet ran out of headroom before it could evict anything.
Fix
Set explicit eviction thresholds and system reservations so the kubelet always keeps enough memory to act, and right-sized the workload's limits against measured usage.
Prevention
A greedy pod is now a pod problem, not a node problem. Eviction config is part of cluster setup, not something added after the first incident.
sev-2 The NVR forgot everything on every restart resolved
Failure
Recordings accumulated on disk that the application no longer knew existed, and cleanup never ran.
Root cause
The video recorder's database sat on an emptyDir volume. Every pod restart wiped the index while the footage stayed on disk.
Fix
Moved the database onto a persistent volume and sized retention against measured write throughput rather than guesswork.
Prevention
If it survives a restart in your head, it needs a PVC in your manifest. Every stateful workload got the same audit.
sev-2 Intermittent API server timeouts under memory pressure resolved
Failure
Sporadic datastore and TLS timeouts from the API server, with no obvious trigger.
Root cause
The control-plane node was running with almost no free memory and no swap. Its datastore had grown large enough that queries slowed under cache pressure.
Fix
Traced the timeouts back to host memory rather than chasing the TLS errors, then relieved the pressure.
Prevention
Datastore size is now monitored, so growth is visible before it becomes latency. Symptoms surfaced several layers away from the cause — the metric that mattered wasn't the one throwing errors.
Long-form Written up in full
  • linkedin-pages-dev

    Write-up

    Why LinkedIn wouldn’t preview this site. The debugging tool read the page perfectly while every share path refused it — a control domain showed the hostname was the problem, and moving to a real domain hid a DNS trap of its own.

    Read

Next

Planned, not built

Planned, not built

Reviewed October 2026

  1. 01

    A second physical host

    Longhorn already replicates volumes between the cluster VMs, but all of them sit on one Proxmox box, so the host is still a single failure domain. A second host would let replicas land on separate machines and give back the memory headroom I keep rationing — Fig. 3 above is that squeeze, at 94% committed. It would not make the platform highly available on its own — that needs 02 as well, and a plan for what happens to the load balancer address.

  2. 02

    An HA control plane, and backups that leave the building

    One control-plane node is the other thing I would fix first: a multi-node control plane, and backups shipped off-cluster instead of living on the array they exist to protect.

  3. 03

    Inference on its own hardware, then AI-assisted operations

    Local model serving came off the cluster once it started competing for disk and RAM. I want it back on dedicated hardware — and then the thing it unblocks: I previously routed critical alerts into n8n, which called a local model and attempted remediation. I removed it in June; alerts have gone straight to Discord since. The version worth rebuilding keeps a person in the loop — n8n drafts the diagnosis, a human approves, every run leaves a trace.

  4. 04

    Wyrmtable past a single replica

    Live table state lives in one process's memory, so the app runs a single replica. A Redis adapter behind Socket.IO would let it scale out — with a fog-of-war battle map next on the feature list.

Stack

Used at work, on my own platform, or both

Infrastructure

  • GCP
  • Terraform
  • Ansible
  • Kubernetes / K3s
  • Helm
  • Docker
  • Proxmox
  • VMware / vCenter
  • Linux
  • Windows Server / AD
  • MetalLB
  • Longhorn
  • cert-manager

Network & security

  • HAProxy
  • FortiGate
  • Cisco
  • Mikrotik
  • IPsec / SSL VPN
  • TCP/IP · DNS
  • Vault
  • Least-privilege IAM
  • ISO 27001 audits

Delivery & observability

  • ArgoCD
  • GitOps
  • GitHub Actions
  • Prometheus
  • Grafana
  • Alertmanager
  • Exporters
  • n8n

Scripting

  • Python
  • Bash
  • PowerShell

Practice

  • Infrastructure as code
  • Incident root-cause analysis
  • Disaster recovery
  • Capacity planning
  • Observability design
  • Documentation

Hobbies

Outside work

Off the clock Football, tables and servers
  • Coaching

    Under-7s football

    I coach an under-7s team.

  • Sport

    Football

    I play as well as coach.

  • Sport

    Archery

  • Collecting

    Football shirts

  • Tabletop

    RPGs, as the GM

    I run the games — which is where Wyrmtable came from.

  • Games

    Game servers for friends

    Minecraft, Insurgency: Sandstorm and others, hosted on the same homelab.

  • Collecting

    Trading card games

    Collecting and trading.

Contact Usually replies within a day

Need someone who can build the platform and keep it running?

I do both — the provisioning pipeline and the 2am root-cause analysis. Based in Malta, currently a DevOps Infrastructure Engineer at Altenar. Happy to talk cloud automation, Kubernetes, networking, or anything that runs on a rack in a spare room.