Skip to content
All articles
Security27 June 2024·3 min read

From four to five 9s of uptime by migrating to Kubernetes

Almost every incident worth writing about starts with something that was working exactly as designed.

Paycux engineering

Almost every incident worth writing about starts with something that was working exactly as designed. The design was just describing a different world than the one the system ended up in.

This piece walks through how we think about it at Paycux, what we have changed our minds about, and where the sharp edges are.

User Management and Scale

Consider user management and scale. Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Blast radius is the metric that actually matters. A credential that can read one organization's data is an incident; the same credential scoped to every organization is a company-ending event, and the difference is a design decision made months earlier.

Our Own Kubernetes Flavor

Our Own Kubernetes Flavor is where this gets concrete. Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

  • Scope credentials to the smallest useful blast radius
  • Alert on the abuse you cannot prevent
  • Exercise the rotation path before you need it
  • Log the decision, not just the outcome

Secret Injection

Secret Injection is where this gets concrete. Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Automated Load Balancing, TLS and DNS

Consider automated load balancing, tls and dns. Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Rotate on a schedule, not on an incident. A rotation path that has been exercised is a routine change; one that has never been used is an outage waiting for the worst possible moment.

Rotate on a schedule, not on an incident.

Machine Provisioning

Machine Provisioning is where this gets concrete. Blast radius is the metric that actually matters. A credential that can read one organization's data is an incident; the same credential scoped to every organization is a company-ending event, and the difference is a design decision made months earlier.

Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Application Management

That brings us to application management. Blast radius is the metric that actually matters. A credential that can read one organization's data is an incident; the same credential scoped to every organization is a company-ending event, and the difference is a design decision made months earlier.

Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

The Path to Self Serving

That brings us to the path to self serving. Detection beats prevention when prevention is uncertain. If you cannot stop a class of abuse outright, make sure it produces a signal loud enough that someone sees it the same day.

Rotate on a schedule, not on an incident. A rotation path that has been exercised is a routine change; one that has never been used is an outage waiting for the worst possible moment.

Where this leaves us

The pattern repeats across every system we have looked at: the hard part is not the mechanism, it is keeping the mechanism honest as the surrounding assumptions change.

If you are working through the same problem and want to compare notes, the docs cover the mechanics and the console shows the behaviour on your own data.

Everything here, already built

Sign-in, enterprise SSO, directory provisioning, roles and an audit trail behind one API. Start with the quickstart and have a working sign-in this afternoon.

Start selling to enterprise customers

Create an account, point sign-in at Paycux, and get back to the part of the product that is actually yours.