Modernising a 20 Year Old Codebase with Observability

I recently visited Iceland and met with the Technical Director of Infrastructure at EVE Online, Nicholas Herring - a self-described "glorified plumber".

Migrating from Bare Metal to Containers in Kubernetes

EVE Online, with its two-decade-old codebase, faced the classic dilemma of maintaining and scaling a legacy infrastructure. The shift to containerization and Kubernetes was the strategic choice, offering scalability, agility, and operational efficiency.

The process involved:

1. Containerization: Transforming monolithic applications into containerized microservices allowed EVE Online to segment their codebase into smaller, more manageable pieces. This made it easier to update and scale individual components independently.

2. Kubernetes Orchestration: Kubernetes, the industry standard for container orchestration, enabled EVE Online to automate the deployment, scaling, and management of their containerized workloads. It provided the ability to dynamically allocate resources, ensuring optimal performance and resource efficiency.

3. Stateful Workloads: Given EVE Online's complex world state and database requirements, moving stateful workloads to Kubernetes StatefulSets and leveraging persistent storage solutions was crucial for data integrity.

Transitioning from bare metal to containers required meticulous planning, thorough testing, and a focus on maintaining a consistent player experience.

Launching the Quasar Technology Stack with Zero Impact on Players

Quasar, a game-changing technology stack designed to support EVE Online's future endeavors, was introduced seamlessly without any noticeable impact on players. This feat was achieved through careful planning and execution:

1. Testing and Staging Environments: Rigorous testing in controlled environments allowed EVE Online to identify and rectify potential issues before reaching production.

2. Phased Rollout: The deployment of Quasar was phased in incrementally, with a subset of players exposed to the new stack. Real-time monitoring and feedback loops enabled swift resolution of any unforeseen problems.

3. Continual Monitoring and Feedback: Sophisticated observability tools, including metrics, logs, and distributed tracing, provided EVE Online with invaluable insights into system performance. Player feedback played a vital role in identifying subtle issues that might otherwise have gone unnoticed.

Managing Cultural Changes Across Engineering Teams

Beyond the technical aspects, the transition to Quasar required significant cultural shifts within EVE Online' engineering teams:

1. Training and Resources: Comprehensive training programs and resources were provided to ensure that engineers were well-equipped to work with the new technology stack and observability tools.

2. Communication and Collaboration: An emphasis on open communication and collaboration fostered a culture of learning and adaptation. Engineers were encouraged to share insights and feedback, enabling continuous process improvement.

Watch a Honeycomb.io interview with Nicholas Herring & Liz Fong-Jones