To put that number in perspective: imagine every single person on Earth doing one long multiplication per second. To match an RTX 5090 you would need about 12,800 Earths full of people, all working together without a break.

Everybody now knows that GPUs are also the engine behind the AI boom, but few people can explain why a graphics card is so much better than a CPU at this kind of work, what a “tensor core” actually is, or why an RTX 5090 is technically a “defective” chip. In this article we take a look at all of it: the design philosophy behind a GPU, what’s physically inside a graphics card, how work is actually executed on thousands of cores, what tensor cores do, and how NVIDIA (and basically every chip maker) turns imperfect silicon into a full product lineup. We’ll use NVIDIA’s current Blackwell generation (RTX 50 series) as the example throughout.

A lot of the material here is inspired by Branch Education’s excellent video How do Graphics Cards Work? Exploring GPU Architecture, which explores a torn-down RTX 3090 with great 3D animations. The architecture has evolved since then, but the principles are exactly the same, and I strongly recommend it.

Part 1: Built for Throughput

Inside an RTX 5090, the GPU has over 21,000 cores. A high-end desktop CPU on the same motherboard might have 16 or 24. So the GPU is 1,000 times more powerful? Not really: the two are built for completely different jobs.

A useful analogy is to think of a CPU as a math professor and a GPU as a stadium full of students. The professor can solve any problem, even a long one full of steps and decisions, and solves it quickly. Each student can only do simple sums, and slowly. But hand out 10,000 simple sums, and the stadium finishes long before the professor is done with the first hundred. Likewise a GPU is good at a narrow set of simple operations (mostly arithmetic) applied to mountains of data, and it can’t run an operating system or talk directly to input devices or networks. In technical terms, a CPU is optimised for latency (finish one stream of instructions as fast as possible), while a GPU is optimised for throughput (finish millions of independent small tasks per second, even if each single task is slow).

That difference in goal leads to a completely different use of the silicon area:

              CPU                                       GPU
+-------------------------------+        +-------------------------------+
| +---------+  +---------+      |        | ##### ##### ##### ##### ##### |
| | Core    |  | Core    |      |        | ##### ##### ##### ##### ##### |
| | (big)   |  | (big)   |      |        | ##### ##### ##### ##### ##### |
| +---------+  +---------+      |        | ##### ##### ##### ##### ##### |
| +---------+  +---------+      |        | ##### ##### ##### ##### ##### |
| | Core    |  | Core    |      |        | ##### ##### ##### ##### ##### |
| +---------+  +---------+      |        | ##### ##### ##### ##### ##### |
|                               |        |-------------------------------|
|      Large L2 / L3 cache      |        |          L2 cache             |
+-------------------------------+        +-------------------------------+
  few powerful cores,                       thousands of simple ALUs,
  lots of control logic + cache             little control logic per ALU

Most of a CPU core’s area doesn’t do arithmetic at all: it goes into branch predictors, out-of-order execution and big caches, all there to keep a single thread from ever waiting. A GPU makes the opposite bet. It assumes the work is massively parallel: the same operation applied to millions of pixels, vertices or matrix elements. In that world, smart prediction is a waste of area; it’s far better to spend those transistors on more arithmetic units. Let’s open one up and see.

Part 2: Inside the Graphics Card

The board

Taking apart an RTX 5090, this is roughly what we find:

+-----------------------------------------------------------------------+
|  Heatsink + heat pipes / vapor chamber + fans                         |
|  (most of the weight of the card)                                     |
+-----------------------------------------------------------------------+
| PCB                                                                   |
|  [Display]   +------+ +------+          +---------+    [12V-2x6 power]|
|  [ ports ]   | VRAM | | VRAM |  ...     |   GPU   |    [  connector  ]|
|              +------+ +------+          |  GB202  |                   |
|              (16 GDDR7 chips total)     +---------+                   |
|              ..... VRM: dozens of inductors, capacitors, MOSFETs ..... |
+------------------------------||||||||||||||||-------------------------+
                               PCIe 5.0 x16 -> motherboard

• Display outputs (HDMI 2.1b, DisplayPort 2.1b) on one side.
• The 12 V power connector (the 16-pin 12V-2x6) on the other side.
• PCIe 5.0 x16 pins that plug into the motherboard, which carry data to and from the CPU and system memory.
• Most of the small components on the PCB form the Voltage Regulator Module (VRM). It takes the incoming 12 V and converts it to around 1 V for the GPU. An RTX 5090 is rated at 575 W, and at such a low voltage that means hundreds of amps flowing into the chip.
• All that power becomes heat, so most of the card’s weight is the cooler: heat pipes or a vapor chamber carry heat from the GPU and memory chips to the radiator fins, where the fans blow it away. (NVIDIA’s own 5090 Founders Edition even splits the electronics over separate boards so that air can flow straight through the card, and uses liquid metal between the GPU and the cooler.)
• 32 GB of graphics memory (GDDR7) spread across 16 chips around the GPU. We’ll get back to these.
• And in the center, the brain: the GPU itself.

The GPU chip: anatomy of GB202

The die inside the RTX 5090 is called GB202 (Blackwell architecture). It’s a chip of about 750 mm² made of 92 billion transistors, and the vast majority of its area is taken up by processing cores, organised hierarchically:

GB202 (full die)
 +-- 12 GPCs  (Graphics Processing Clusters)
       +-- 8 TPCs each  (Texture Processing Clusters)
             +-- 2 SMs each  (Streaming Multiprocessors)
                   +-- 1 RT core (ray tracing)
                   +-- 4 processing blocks, each with:
                         +-- 1 warp scheduler
                         +-- 32 CUDA cores
                         +-- 1 Tensor core

12 × 8 × 2   = 192 SMs
192 × 4 × 32 = 24,576 CUDA cores
192 × 4      = 768 Tensor cores
192 × 1      = 192 RT cores

(As we’ll see in the last part, the RTX 5090 doesn’t have all of these enabled.)

These three types of cores do all of the computation, each with a different job:

• CUDA cores (also called shading cores) are simple calculators with an “add” button, a “multiply” button and a few others. They do most of the work when running games.
• Tensor cores (5th generation on Blackwell) are matrix multiplication and addition calculators, used for geometric transformations and above all for neural networks and AI.
• Ray tracing cores (4th generation) are the largest but fewest, and they accelerate the ray tracing algorithms (finding which triangle a ray of light hits).

The SM is the real basic building block. Besides the cores, each SM has a big register file (256 KB), 128 KB of L1 cache / shared memory, and a few Special Function Units (SFUs).

Inside a CUDA core

Note that the word core is pure marketing: a CUDA core is a single arithmetic lane, not a core in the CPU sense. It has no instruction fetch, no branch predictor, nothing. A better comparison is that one SM is roughly a “core”, and the CUDA cores are its very wide vector unit.

What is left is a very simple calculator: it does basic math (add, multiply, compare and a few bit operations) on 32-bit numbers, either floating-point (FP32) or integer (INT32). On Blackwell, all 128 cores of an SM can do either type, while on the previous generations only half of them could handle integers.

Of these operations, one matters more than the others: the FMA (Fused Multiply-Add), a multiplication and an addition done in a single step:

FMA (Fused Multiply-Add):   result = A × B + C

It’s not the only thing a core does, but it’s the operation that almost all 3D graphics and neural network math boils down to, so it’s the one that defines the GPU’s performance numbers.

Each CUDA core completes one FMA per clock cycle, which counts as two operations (a multiply and an add). So for an RTX 5090:

21,760 cores × 2.41 GHz × 2 operations = ~105 trillion operations per second (or 105 TFLOPS)

There’s our 105 trillion. (For comparison: RTX 3090 ~36 trillion, RTX 4090 ~83 trillion.)

What about more complicated operations like division, square roots or trigonometric functions? These are handled by the Special Function Units, which are far fewer, only a handful per SM against 128 CUDA cores. GPUs are built for the assumption that the common case is multiply and add.

The rest of the chip

Around the cores we find:

• 16 memory controllers around the edge, each 32 bits wide, which talk to the GDDR7 chips (16 × 32 = 512-bit bus).
• The PCIe 5.0 interface to the CPU. (Older big chips like the RTX 3090’s GA102 also had NVLink to connect two cards together; consumer cards dropped it starting with the 40 series.)
• A big L2 cache shared by the whole chip: 128 MB on the full GB202, 96 MB enabled on the RTX 5090. For comparison, the RTX 3090 had only 6 MB.
• The GigaThread Engine, which manages all the GPCs and SMs and distributes work among them.
• Fixed-function blocks for video encoding/decoding (NVENC/NVDEC) and the display engine.

Graphics memory: feeding the beast

Even 96 MB of L2 can hold only a small fraction of a game scene. Everything else lives in the graphics memory (VRAM), and different chunks of the scene are continuously transferred between VRAM and the GPU. In fact, when you’re staring at a loading screen, most of that time is spent moving the 3D models and textures of the level from your SSD into the VRAM.

Because the cores are performing tens of trillions of calculations per second, GPUs are data hungry machines that need to be continuously fed. Back to our stadium: the students are only as fast as the people handing out the worksheets, so the memory system is built to hand out data on many lanes at the same time: the 16 memory chips on a 5090 together transfer 512 bits at a time (the bus width), each pin running at 28 Gbit/s:

512 bits × 28 Gbit/s / 8 = 1,792 GB/s  (~1.8 TB/s)

Compare that to the DRAM sticks supporting a CPU: each channel is only 64 bits wide, and even a dual-channel DDR5 desktop gets something around 80-100 GB/s. For many workloads, especially AI, memory bandwidth (not compute) is the real bottleneck.

More than ones and zeros

You may think computers only use binary 1s and 0s on their wires. To squeeze more data out of each wire, modern graphics memory uses multiple voltage levels:

• GDDR6X (RTX 30 and 40 series) used PAM-4: 4 voltage levels, so each symbol carries 2 bits.
• GDDR7 (RTX 50 series) switched to PAM-3: 3 voltage levels (-1, 0, +1), i.e. ternary digits. Bits are packed into trits with encoding schemes such as 3 bits → 2 trits (3² = 9 combinations, enough for 2³ = 8) and 11 bits → 7 trits (3⁷ = 2187, enough for 2¹¹ = 2048). Combined, they send 276 bits using only 176 ternary symbols, about 1.5 bits per symbol.

NRZ   (binary):   2 levels -> 1 bit per symbol
PAM-3 (GDDR7):    3 levels -> ~1.5 bits per symbol
PAM-4 (GDDR6X):   4 levels -> 2 bits per symbol

Why go from 4 levels down to 3? Because with fewer levels the voltage steps are bigger and easier to tell apart: better signal-to-noise ratio, simpler encoders, and better power efficiency, which allows much higher clock rates and in the end more bandwidth. That’s how the 5090 gets ~78% more bandwidth than the 4090.

HBM: memory for AI chips

Datacenter AI accelerators use a different kind of memory: HBM (High Bandwidth Memory). Instead of separate chips on the PCB, HBM is a stack of DRAM dies connected vertically by TSVs (Through-Silicon Vias), tiny vertical wires going through the silicon, forming a “cube” of memory placed right next to the GPU on the same package. A single HBM3E stack holds 24 to 36 GB, and a datacenter Blackwell chip like the B200 is surrounded by 8 of them, for 192 GB and about 8 TB/s of bandwidth.

Part 3: How the Work Actually Runs

Now that we know the hardware, how do we keep 20,000 cores busy at once?

Embarrassingly parallel problems

Video game rendering, scientific simulations and neural networks are what computer scientists call embarrassingly parallel problems: the work splits naturally into millions of small tasks that don’t depend on each other. GPUs run them with SIMD (Single Instruction, Multiple Data): one instruction, repeated across thousands of different numbers.

Take a cowboy hat on a table in a 3D game. The hat is made of about 14,000 vertices, each with X, Y and Z coordinates relative to the center of the hat. To place it in the game world, every vertex needs the same operation: “add the hat’s position in the world”.

world_x = model_x + hat_x
world_y = model_y + hat_y      <- same instruction, 14,000 vertices,
world_z = model_z + hat_z         one vertex per core

No vertex depends on any other, so they can all be computed at the same time. A full scene from the video has 8.3 million vertices, about 25 million additions, and that’s only one of the first steps of rendering a frame.

Threads, warps, blocks and grids

Each small task, like moving one vertex, is called a thread. The GPU doesn’t manage threads one by one: it groups them in teams of 32 called warps, and all 32 threads in a warp execute the same instruction at the same time, each on its own piece of data. Warps are then grouped into blocks, each assigned to one SM, and all the blocks of a job form a grid that covers the whole GPU:

Thread  (one small task)     ->  1 CUDA core
Warp    (32 threads)         ->  1 processing block
Block   (many warps)         ->  1 SM
Grid    (many blocks)        ->  the whole GPU

The downside of moving in teams of 32 is branches. If half the threads in a warp need to do one thing and the other half something else (an if/else), the warp has to run both paths one after the other, with half the threads waiting each time. This is called warp divergence, and it’s why GPUs are fast on uniform work and slow on code full of decisions.

Hiding latency instead of avoiding it

Reading data from memory is slow: it takes hundreds of cycles. A CPU fights this with big caches and clever prediction. A GPU takes a simpler approach: it keeps lots of warps ready at the same time, and whenever one is waiting for data, it immediately switches to another one that’s ready to go. Think of a chef with many dishes on the stove: while one pot is waiting to boil, they work on the next one instead of standing still.

Each SM can juggle up to 48 warps (1,536 threads) this way, so a 5090 can have over 260,000 threads in flight at once. That’s why the GPU never needs to be smart about any single thread: there’s always another one ready.

Part 4: Tensor Cores

Even with thousands of FP32 ALUs, NVIDIA realised around 2016 that deep learning was spending almost all of its time doing one single operation: matrix multiplication. Every fully connected layer, every convolution (after some rearranging), and every attention block in a transformer is, at its heart, a big A × B matrix multiply. Neural networks and generative AI need trillions to quadrillions of these operations, on very large matrices.

Doing that with regular CUDA cores means issuing one FMA instruction per element pair. That works, but a lot of energy goes into fetching instructions, reading registers and moving data around, not into the actual math.

So with the Volta architecture (V100, 2017), NVIDIA introduced the Tensor Core: a dedicated unit that takes three matrices, multiplies the first two, adds the third, and outputs the result:

D = A × B + C

  A (4x4)        B (4x4)        C (4x4)        D (4x4)
[ a a a a ]    [ b . . . ]    [ c . . . ]    [ d . . . ]
[ . . . . ]  x [ b . . . ]  + [ . . . . ]  = [ . . . . ]
[ . . . . ]    [ b . . . ]    [ . . . . ]    [ . . . . ]
[ . . . . ]    [ b . . . ]    [ . . . . ]    [ . . . . ]

d = a·b + a·b + a·b + a·b + c   (row of A × column of B, plus C)

Each value in the output is the sum of the products of a row of the first matrix with a column of the second, plus the corresponding value of the third. The key point: since all the values of the three input matrices are available at the same time, the tensor core computes all the output values concurrently, in one go. The first Volta tensor core performed 64 FMAs per clock (a 4×4×4 multiply); a regular CUDA core does one. Each generation since has made them bigger and faster. Larger matrices are simply broken into tiles and fed to the tensor cores tile by tile.

Why lower precision is the key

The second trick of tensor cores is reduced precision. First, a quick word on the names. FP stands for floating point, the standard way computers store numbers with decimals (a kind of scientific notation in binary), and the number after it is the total bits: FP32 uses 32 bits, FP16 uses 16, and so on. Each number is split into three parts:

FP32:  [sign 1 bit] [exponent 8 bits] [mantissa 23 bits]

sign      -> positive or negative
exponent  -> how big or small the number can be (the range)
mantissa  -> how many significant digits it keeps (the precision)

Fewer bits means less range, less precision, or both. You’ll also meet BF16 (brain floating point, invented at Google Brain) and TF32 (TensorFloat-32, NVIDIA’s own format): two formats that split their bits between exponent and mantissa differently, as we’ll see below.

Neural networks turn out to be remarkably tolerant to noise: you don’t need 32-bit floats to store a weight that will be nudged slightly in a random direction thousands of times anyway. Halving the number of bits means:

• half the memory and bandwidth per value,
• roughly a quarter of the silicon for a multiplier (multiplier area grows with the square of the mantissa width),
• so many more operations per clock in the same area and power.

Each generation of tensor cores has added support for lower-precision formats:

The new formats are added, not swapped in: a Blackwell tensor core still runs FP16, BF16, TF32, INT8 and FP8, and it simply runs FP4 too. (With few exceptions: INT4, for example, was dropped from datacenter chips starting with Hopper, since FP8 and FP4 do the job better.) There are no separate “FP16 cores” and “FP4 cores” either: the same tensor core handles all of them, and it gets faster as the numbers get smaller. Roughly, each time the precision is halved, the throughput doubles:

Same Blackwell tensor core, relative throughput:

TF32  (19 bits)   x0.5
FP16  (16 bits)   x1
FP8   (8 bits)    x2
FP4   (4 bits)    x4

The inputs can be tiny, but the results are usually accumulated in higher precision (FP16 or FP32), so the rounding errors don’t pile up over thousands of additions. The programmer, or more often the framework like PyTorch, picks the format for each job: higher precision for training, where accuracy matters, and the lowest precision the model can tolerate for running it, where speed matters.

A couple of these deserve a word:

• TF32 has the range of FP32 (8-bit exponent) but only a 10-bit mantissa. Its trick is that the tensor core accepts normal FP32 numbers and rounds them to TF32 internally, so code written for FP32 can use the tensor cores without being rewritten. Frameworks like PyTorch can turn this on with a single setting, for a big speedup at a precision loss most training doesn’t notice.
• BF16 keeps the 8-bit exponent of FP32 and only 7 mantissa bits. Same range as FP32, less precision, which is exactly the trade-off training wants (overflow is a much bigger problem than a little rounding).
• 2:4 structured sparsity: if in every group of 4 weights at least 2 are zero, the tensor core can skip them and double its throughput.
• FP4 (new on Blackwell) is just 4 bits per number: only 16 possible values! It works for running (not training) many models, with per-block scale factors to keep the accuracy acceptable, and it doubles throughput again compared to FP8.

On an RTX 5090 the plain FP32 CUDA cores deliver (as we calculated earlier) about 105 TFLOPS, that is 105 trillion floating-point operations per second. NVIDIA markets the same card at 3,352 “AI TOPS (Trillions of Operations Per Second)”, which is the tensor core throughput in FP4 with sparsity: about 30 times more. Even with the marketing tricks removed (no sparsity, higher precision), the tensor cores are many times faster than the CUDA cores. That’s why every serious deep learning framework goes out of its way to express work as large matrix multiplies in low precision: it’s the only path through the chip that runs at full speed.

Tensor cores are not idle silicon for gamers either: DLSS (NVIDIA’s AI upscaling) runs a neural network on the tensor cores to reconstruct a high-resolution frame from a lower-resolution one, and with DLSS 4 on the 50 series, Multi Frame Generation uses them to generate up to three extra frames for every frame actually rendered.

Tensor cores are an example of a general rule: the more specialised the hardware, the faster it is at its one job. Bitcoin mining shows the extreme case. GPUs took over mining in its early years, because hashing billions of candidate blocks is perfectly parallel, but they were soon replaced by ASICs, chips that can compute only the SHA-256 hash and nothing else, thousands of times faster than a graphics card. CPU → GPU → tensor core → ASIC is a scale from flexible to specialised.

Part 5: Binning, or How to Sell Broken Chips

Now to the most “business” part of chip design. Here’s a surprising fact: the RTX 5090 does not use a complete GB202 chip. Out of 192 SMs, only 170 are enabled (about 88%), along with 96 MB of the 128 MB of L2 cache. And the same GB202 die is also sold in the RTX PRO 6000 Blackwell, a professional card with 188 SMs and 96 GB of memory, for several times the price. Why?

Defects and yield

During manufacturing, patterning errors, dust particles and other small imperfections create defective areas on the chip. They land randomly across the wafer, so the bigger the chip, the more likely it is to catch at least one. With a typical defect rate for a mature process, the fraction of perfect dies (the yield) looks roughly like this:

So more than half of all GB202 dies would have at least one defect somewhere. And on a 300 mm wafer only around 70 of them fit. If you had to throw away every die with a single defect, high-end GPUs would cost a fortune (well, even more than they already do).

The trick: redundancy by design

The solution is the highly repetitive design we saw earlier. A GPU is made of dozens of identical SMs, so a small defect in one core only damages that particular SM and doesn’t affect the rest of the chip. Instead of throwing the chip away, engineers find the defective region and permanently isolate and deactivate the circuitry around it, by blowing tiny on-chip eFuses during testing, so the configuration can’t be changed afterwards. The same applies to L2 cache slices and memory controllers.

The chips are then tested and categorised, or binned, according to how many working units they have. Because the defect can land anywhere, two RTX 5090s don’t necessarily have the same SMs disabled, they just have the same number of working ones.

One die, many products

NVIDIA designs a handful of dies of different sizes and then cuts each one into several products, depending on how many units survived (and on market demand). Here’s the RTX 50 desktop lineup:

A few interesting things stand out:

• The RTX 5080 is a flawless GB203, all 84 SMs enabled, while GB203 dies with a few defects become 5070 Ti. Same chip, same memory bus, roughly a 20% difference in cores and a big difference in price.
• There is a huge gap between the 5090 (170 SMs) and the 5080 (84 SMs): GB202 is literally about twice the size of GB203.
• A common misconception is that a 5060 is just a “broken 5090”. It isn’t: they are completely different dies, and GB206 is less than a quarter of the size of GB202. Designing several dies is necessary because cutting a 750 mm² chip down to 30 SMs would mean selling a very expensive piece of silicon for 300 euros. But within each die family, binning is exactly what happens.

Speed binning

Defects are not the only thing that varies. Even among fully working dies, some can reach higher clocks at lower voltage than others, due to tiny variations in the manufacturing process. Testing sorts them by this too:

• the best dies can go into higher-clocked or more power-efficient products (top SKUs, laptop chips where every watt counts),
• the others are sold at lower clocks.

This is also why the “silicon lottery” exists among overclockers: two cards of the same model can overclock very differently.

Not only NVIDIA

Binning is universal in the semiconductor industry:

• AMD Ryzen CPUs are built from 8-core chiplets; a 6-core Ryzen 5 is very often an 8-core chiplet with two cores disabled.
• Intel “F” CPUs (like the i5-14400F) are chips where the integrated GPU is disabled, often because it was defective.
• The PlayStation 5 and Xbox Series X ship with some GPU compute units disabled on purpose, so that chips with a defect can still be used and console production is not limited by yield.

The uncomfortable part

As the process matures, defect density drops and more dies come out perfect. But the market still wants cheaper SKUs, so manufacturers sometimes take a perfectly good die and disable working units to sell it as a lower model. From a pure engineering point of view it sounds wasteful, but economically it’s the same chip, with the same manufacturing cost, sold at different prices to different customers. The price of a GPU has little to do with how much it costs to make that specific chip, and a lot to do with how many working SMs it has and who wants to buy it.

Conclusion

To recap:

• A GPU is the stadium, not the professor: it is built for throughput, with thousands of simple FMA calculators, fed by a very wide memory bus, hiding latency by switching between thousands of threads.
• The chip is a hierarchy of identical blocks: GPCs → TPCs → SMs → processing blocks → CUDA cores, and the SM is the real building block.
• Work is organised in threads, warps of 32, thread blocks and grids, and it shines on embarrassingly parallel problems like graphics and AI.
• Tensor cores compute small matrix multiply-accumulate operations all at once, at low precision, and they are the reason GPUs dominate AI.
• Binning turns imperfect silicon into a product lineup: the same die, with more or fewer SMs fused off, becomes a 5070 Ti or a 5080, a 5090 or an RTX PRO 6000.

So next time you see “21,760 CUDA cores” on a spec sheet, you’ll know it means 170 SMs that survived testing out of 192, each juggling dozens of warps of 32 threads, with 4 little matrix engines doing most of the heavy lifting for AI.

As always, I hope you found this article helpful! :)

Part 1: Probe Types

Kubernetes probes support four different protocols for monitoring pod health, and while httpGet is the most common, it isn’t always the ideal choice for every scenario. Let us review all available options to see how they work.

• httpGet

Kubernetes issues an HTTP GET against the container. Any 2xx or 3xx status is a success; everything else fails. For an HTTP application this is almost always the correct choice, because if the HTTP server has problems, the probe notices right away.

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080

Note that the probe is sent by the kubelet from the node, not from inside the container, so it bypasses anything you configured for in-cluster traffic. Avoid the tempting anti-pattern of wrapping curl in an exec probe:

# don't do this
readinessProbe:
  exec:
    command: ["curl", "-f", "localhost:8080/health"]  

That forks a process and loads a binary on every check to do something the kubelet does natively, for free.

• tcpSocket

The kubelet just opens a TCP connection to the port. If the handshake completes, the probe passes.

livenessProbe:
  tcpSocket:
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

This is a weak signal, and deliberately so. A listening socket says nothing about whether the application behind it works: the accept queue can be full, the worker pool exhausted, the event loop deadlocked, every request failing and yet the handshake still succeeds. Use it for things that genuinely have no health endpoint, such as a database or a plain TCP service, and don’t mistake it for a health check.

• exec

The kubelet runs a command inside the container; exit code 0 passes, anything else fails.

readinessProbe:
  exec:
    command: ["/app/bin/check-ready"]

Useful when the application ships a purpose-built health command, or for things with no network surface at all (batch workers, sidecars). The production caveat is cost: this spawns a process inside the container on every single check. At periodSeconds: 5 across 100 Pods that is 72,000 process creations an hour, so the command must be genuinely small and not a shell script that shells out six more times.

• grpc

For gRPC services, Kubernetes can speak the standard gRPC health checking protocol directly:

livenessProbe:
  grpc:
    port: 8080
  periodSeconds: 10

This replaces the old pattern of shipping a grpc_health_probe binary in the image and calling it with an exec probe.

Part 2: Timing Parameters

This is where most production problems actually come from, and the defaults deserve a look before anything else:

That timeoutSeconds: 1 default is the single most common cause of mysterious restarts. If your health endpoint occasionally takes 1.2 seconds under GC pressure or a cold connection pool, the default silently fails the probe and if it’s a liveness probe, it restarts a container that was working fine.

initialDelaySeconds

How long the kubelet waits after container start before probing at all.

initialDelaySeconds: 30

Try to avoid it since it is a fixed, unconditional waiting time: set it to 30 and every Pod in every rollout, forever, waits 30 seconds even when it was ready in two. Worse, setting it large to accommodate a slow start (initialDelaySeconds: 300 is a real thing people write) tells Kubernetes to ignore a genuinely dead container for five minutes.

A startup probe does the same job adaptively, it keeps checking, so a fast start is detected immediately and a slow one still gets its full budget. If you have a startup probe, you almost never need initialDelaySeconds.

periodSeconds

How often the check runs. Readiness generally wants a shorter period than liveness, because reacting quickly to “stop sending me traffic” is cheap and reversible, while “restart this container” is not cheap nor reversible:

Readiness: every 5 seconds
Liveness:  every 10 seconds

timeoutSeconds

How long the check may take before it counts as a failure. Set it above the realistic p99 of the endpoint. Setting it very high has the opposite problem: timeoutSeconds: 60 on a liveness probe means a hung container can sit there for minutes before anyone notices.

If your endpoint takes an entire minute to respond to a health check, it requires an upstream fix; a proper probe should reply in a fraction of a second and if cannot complete in well under a second is simply trying to do too much.

failureThreshold

How many consecutive failures are required before Kubernetes acts.

Probe 1 -> FAIL
Probe 2 -> FAIL
Probe 3 -> FAIL
             |
             v
       Threshold reached -> act

The counter resets on any success, which is the whole point: it distinguishes a persistent fault from the transient packet loss, GC pause, or CPU contention that every real cluster produces continuously.

## Part 3: Readiness During Rollouts and Termination

Readiness is not only a failure-handling mechanism it runs constantly, and it governs every rolling update.

A Pod failing readiness is simply removed from the Service’s endpoints while continuing to run:

Service
   |
   +---- Pod 1 [Ready]    <- receives traffic
   |
   +---- Pod 2 [Not Ready] <- running, no traffic
   |
   +---- Pod 3 [Ready]    <- receives traffic

Not Ready is not Not Alive, and during a deploy that distinction is doing real work: the Deployment waits for each new Pod to pass readiness before it proceeds to the next, so an honest readiness probe is what prevents a rollout from shifting traffic onto instances that aren’t warm yet. A readiness probe that returns 200 the instant the process starts makes a rolling update look successful while it drops requests.

Termination is where this gets complicated for critical systems where zero-downtime deployments matter, and it’s worth being precise because the common advice is wrong. When a Pod is deleted, two things happen in parallel:

1. Traffic routing stops: The cluster starts removing the pod’s IP address from all load balancers, ingress controllers, and proxies (EndpointSlices).
2. Shutdown begins: The pod receives a SIGTERM signal telling the application to shut down, and any preStop hook starts running.

Updating every router and proxy across the cluster takes a few seconds because it happens asynchronously. During that brief window, incoming traffic keeps arriving at the pod, even though the application has already been told to shut down. If the app stops immediately, those incoming requests fail with errors.

So there is a window where the application has already received SIGTERM but traffic is still arriving. Failing your readiness probe on shutdown does not close that window since deletion already triggered the service removal; the race is in the propagation. What actually closes it is a short preStop delay that keeps the Pod serving while the removal spreads:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

terminationGracePeriodSeconds: 30

The grace period is the total budget for preStop plus draining; overrun it and the container is SIGKILLed mid-request. Probes are one part of lifecycle management, not all of it.

Part 4: Configurations That Cause Outages

Probes that are too aggressive

periodSeconds: 1
timeoutSeconds: 1
failureThreshold: 1

This says: one slow response, and act. Production clusters produce transient slowness constantly GC pauses, CPU contention from a noisy neighbour, a node under memory pressure, ordinary packet loss and none of it means the application is broken. Aggressive thresholds on a liveness probe convert that background noise into restarts, which is how a mild latency incident becomes a crash loop.

Timeouts tuned too low

Covered above, but it’s the most common single misconfiguration: leaving timeoutSeconds at its default of 1 while the health endpoint’s p99 is well above that. The probe fails under load, which is precisely when you can least afford to lose replicas.

A startup budget smaller than the real startup time

periodSeconds × failureThreshold for the startup probe is a hard deadline. If it’s shorter than a cold start with migrations and an empty cache, the Pod is killed, restarted, and killed again presenting as CrashLoopBackOff with no error in the application logs.

The ones from part one, still true here

Three mistakes carry over from part one and are worth naming because they’re just as common at scale: checking external dependencies in liveness (restarting a Pod doesn’t fix a database), pointing all three probes at a single /health endpoint so they can’t express different answers, and making readiness too strict, so a degraded-but-useful Pod gets pulled from the Service over a dependency it could tolerate.

Part 5: Quick Checklist

Mechanism httpGet for HTTP, grpc for gRPC, tcpSocket only when there’s nothing better, exec only when it’s genuinely cheap.

Timing

• Is timeoutSeconds above the endpoint’s p99? (The default of 1 is rarely right.)
• Does periodSeconds × failureThreshold give acceptable detection time?
• Is the threshold tolerant of a transient network issue or a brief packet loss?
• Is the startup budget larger than the slowest realistic cold start?

Semantics

• Does liveness check only things a restart could fix?
• Does readiness answer “can I still serve traffic even if i am degradated state”?
• Are the three probes on separate endpoints?

Lifecycle

• Does readiness stay honest during a rolling update?
• Is there a preStop delay to cover endpoint propagation?
• Is terminationGracePeriodSeconds longer than preStop plus drain time?

Global

• If every replica fails this check simultaneously, what happens?

Kubernetes already provides all the tools we need to keep our applications healthy through probes; our only job is to configure them correctly.

As always, I hope you found both of these articles helpful! :).

The three probes answer three different questions, and each one triggers a different action:

That table is the whole article, really. The rest is about why mixing the three up is expensive: a badly designed liveness probe can turn a two-second network blip into a full outage.

Where Probes Are Defined

Probes are not part of the Service. They are defined on the container, inside the Pod template — so in practice in a Deployment, StatefulSet, or DaemonSet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          image: my-api:1.0

          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080

          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080

The Service never mentions probes at all. It simply routes traffic to whichever Pods are currently Ready:

                    Kubernetes Service
                           |
                           v
                Ready Pods receive traffic
                           |
              +------------+------------+
              |            |            |
           Pod #1       Pod #2       Pod #3
              |            |            |
          readiness    readiness    readiness
            probe        probe        probe

When a readiness probe fails, Kubernetes drops that Pod from the Service’s set of ready endpoints. Traffic stops, but the container keeps running. That separation between stop the traffic and restart the container is the entire reason there is more than one probe.

The Three Probes

Startup — “have I finished starting?”

Some applications simply take a long time to come up: loading configuration, warming caches, running migrations, loading large models. If liveness starts checking immediately, Kubernetes may kill the container before it ever had a chance to startup!

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  periodSeconds: 5
  failureThreshold: 30

While a startup probe is defined and still failing, the liveness and readiness probes are disabled entirely. Here the application gets up to 150 seconds (5 × 30) to initialize, and can finish much sooner if it is fast. Once the startup probe succeeds it is never run again, and the other two take over. This is far better than guessing a large initialDelaySeconds, which forces you to pay the worst case every single time.

Liveness — “should Kubernetes restart me?”

A liveness probe is for a container that is alive at the process level but no longer doing useful work: a deadlock, an exhausted thread pool, a wedged event loop. The process still exists, so without a probe Kubernetes would consider it perfectly healthy forever.

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

With these values the container is restarted only after roughly thirty seconds of consistent failure. That word matters: liveness should describe the application’s own ability to function, not the health of everything it happens to talk to.

Readiness — “should I receive traffic?”

Readiness answers a narrower question: this Pod is running, but should it be serving requests at this moment?

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

Readiness failure is cheap and fully reversible. The Pod stays alive, drops out of the Service endpoints, and returns as soon as the probe succeeds again — no restart, no lost in-memory state, no startup cost paid. This is why readiness, not liveness, is the right place for “I am temporarily unable to do my job”.

The Database Trap

This is the mistake worth internalizing, because it is both extremely common and actively harmful.

Imagine an API with three replicas, all talking to the same database, and a /health endpoint that checks database connectivity wired into the liveness probe. The database has a brief network problem, a few seconds of network issue, the kind of thing that happens routinely.

All three replicas share that dependency, so they all fail at the same time:

DB hiccup
   |
   +------> Pod 1 liveness fails
   |
   +------> Pod 2 liveness fails
   |
   +------> Pod 3 liveness fails
                 |
                 v
          Entire fleet restarts
                 |
                 v
       Cold caches, reconnect storm
                 |
                 v
     More pressure on the recovering DB

By the time the database is back, every Pod is in the middle of a cold start that can take few minutes depending from the application, and they all hit the recovering database at once. The miss-configured probes has taken a transient dependency failure and amplified it into a real outage.

The rule that prevents this is a single question:

Liveness asks whether restarting this container could plausibly fix the problem.

Restarting an API Pod does not repair a database. So database connectivity has no business being in a liveness probe. It belongs in readiness, where the consequence matches the problem:

Database unavailable
        |
        +----> Readiness fails
        |          |
        |          v
        |      Pod removed from Service
        |      (still running)
        |
        +----> Liveness still healthy
                   |
                   v
               No restart

The Pods stay up, stop advertising themselves as able to serve, and become Ready again on their own the moment the database recovers. Same failure, but back online in seconds with no restarts!

How Much Should Readiness Check?

The opposite mistake also exists. Readiness is not a dumping ground for every dependency either.

If your API genuinely cannot answer anything useful without the database, then checking it in readiness is correct. But suppose the application can still serve cached data, /metrics, and a handful of read paths when Redis is down. Making readiness fail on Redis pulls every Pod out of the Service — and since all replicas share that Redis, you have just taken the whole application offline over a dependency it could have tolerated.

The question to ask is:

Can this Pod still serve the traffic the Service is actually sending it?

If the answer is yes, it should stay Ready, even in a degraded state.

Designing Health Endpoints

The cleanest approach is one endpoint per question, rather than a single /health doing everything:

• /health/live — lightweight, and limited to the application itself. It should generally avoid touching databases, queues, or remote APIs unless there is a very specific reason.
• /health/ready — may check the dependencies that are genuinely required to serve requests, returning 503 when they are missing.
• /health/startup — reports only whether initialization has completed.

Whatever they check, these endpoints must be cheap. With 100 Pods and a five-second period, you are generating tens of thousands of health checks per hour, so they need to be fast, deterministic, side-effect free, and safe to call repeatedly. A readiness check that fans out to six backends is not a health check, it is a distributed tracing span you run twelve times a minute per Pod — and when one of those backends gets slow, the probe gets slow, Pods drop out of the Service, and the remaining Pods absorb more traffic. The probe itself becomes the outage.

Choosing the Right Probe

When you are unsure where a particular check belongs, don’t ask whether it indicates a problem. Ask what you want Kubernetes to do about it:

• “Restart the container, because restarting might fix it” → liveness
• “Stop sending traffic here until this resolves” → readiness
• “Give it more time to finish starting” → startup

This works because probes are not monitoring endpoints. Monitoring tells a human that something is wrong; a probe tells Kubernetes what action to take, and Kubernetes takes it immediately, on every replica, without asking. If you want to observe something, export a metric and alert on it. Put it in a probe only when you want the corresponding action.

What to remember

Choosing the right probe is extremely important, because that gives Kubernetes the right signal for the action you actually want. A temporary database outage should never be a reason to restart every API Pod.

So to conclude, liveness is a big hammer: it just restarts pods, so reserve it for problems a restart can genuinely fix. Readiness is the gentle, reversible one, and it is where most dependency conditions belong. Startup exists so your container has enough time to start, before the other two take over.

In the next article we will look at probe types, the timing parameters, and how to tune all of this for real production workloads.

Before we start, let’s get the standard legal disclaimer out of the way:

• This article is strictly for educational purposes. Do not use this knowledge to hurt poor consoles.
• I have only ever observed this exploit from a safe, theoretical distance and I have never ever ever actually performed it myself (I may recall a friend of a friend of a friend that did it)
• I hold a deep, unwavering reverence for the flawless, totally-not-sloppy security architectures of massive mega-corporations like Microsoft.
• «PLACEHOLDER FOR OTHER BS»

I hope that this suffices. Now lets start with an overview.

Some History

Launched in 2005, the Xbox 360 was built like a digital fortress, with an innovative secure-boot chain and hypervisor architecture designed to make homebrew impossible. For the first few years, the only available hacks were software exploits that Microsoft quickly patched through system updates.

Then in 2011, French researcher GliGli discovered an hardware vulnerability (called Reset Glitch Hack) baked into the CPU. This new vulnerability was purely hardware related, and completely unpatchable by Microsoft. For the next years, people started to exploit this vulnerability by using external modchips.

After several iterations like RGH1.2 and S-RGH, in late 2021, user 15432 presented the RGH3, a new exploit that skipped external chips entirely by hijacking the console’s internal standby processor. Because the RGH3 is the most complete hack, we will focus on that.

General Overview: The RGH Workflow

The Xbox 360 stores its operating system, bootloaders, and system-specific data (such as the console’s unique Keyvault) inside a physical flash memory chip on the motherboard, known as the NAND. The content of this memory is encrypted with encryption keys that are baked into the CPU and are unique for each console. This means that you cannot just clone nand data to a new console, because it would not be able to decrypt the nand content.

So for the RGH, the first thing to do is to read and dump the encrypted nand content. This is done by soldering an external SPI reader (or rpi-pico) to the motherboard to extract a exact image of this storage. It is extremely important that you make several backups of this file since, if something goes wrong, you can always restore the console with this (console-tailored) image.

Once you have the NAND dump extracted you can feed it into the J-Runner software. This software replaces the console’s factory second-stage bootloaders with XeLL (the Xenon Linux Loader), a bare-metal, open-source bootloader. At the same time, if selecting RGH3 tickbox, the software alters the firmware of the console’s SMC (System Management Controller) (a separate stand-by chip processor) to inject custom glitch-timing loops (more details in the section below). This modified image is then written back to the physical flash chip.

When the console is powered on, the modified SMC code activates. It synchronizes with the motherboard’s clock (The POST Wire) and drops the voltage to the CPU (The RST/PLL Wire) at a precise microsecond to bypass security verification, allowing unsigned XeLL binary to run. This binary is basically another OS similar to when you boot your PC from a usb stick.

Once Booted, the only thing that XeLL does is to sends direct instructions to the CPU to read its internal eFuses, displaying the unencrypted 128-bit CPU Key onto the screen.

With the CPU Key successfully recovered, they are entered to the J-Runner program. The software uses this key to fully decrypt the original NAND backup, extracting the system’s unique serial number and pairing data. It then takes a clean, official Microsoft Dashboard update, strips out its core security restrictions, re-encrypts the package using the console’s specific CPU Key, and generates a final custom image called a XeBuild Image. This new image can be written back to the console. The console is now cured, and you may proceed to install custom Dashboard such as Aurora.

Here is a quick overview of those steps:

+------------------+     +--------------------+     +---------------------+
| 1. Dump NAND     | --> | 2. Patch & Write   | --> | 3. Execute Glitch & |
|    (Raw Backup)  |     |    XeLL image      |     |    Boot XeLL        |
+------------------+     +--------------------+     +---------------------+
                                                               |
+------------------+     +--------------------+                v
| 6. Flash Custom  | <-- | 5. Build XeBuild   | <-- | 4. Read CPU Key     |
|    (Dashboard)   |     |    Custom Image    |     |    From eFuses      |
+------------------+     +--------------------+     +---------------------+

The Mechanics of the Glitch

To appreciate why RGH is so remarkable, one must look at how the physical exploit forces the CPU to run unauthorized software.

Voltage Fault Injection Attacks

The Xbox 360 utilizes a secure boot chain overseen by a hardware-level Boot ROM embedded in the CPU. When the console starts up, it runs a memory comparison loop (memcmp) to verify that the digital signature of the next bootloader matches Microsoft’s private RSA keys. If the signature doesn’t match, the CPU panics and halts immediately.

RGH breaks this mechanism via voltage fault injection:

1. Clock Slowdown: To widen the window of opportunity, the modded SMC code or an external modchip forces the motherboard’s clock generator to slow down the CPU’s operating frequency by a factor of 128.
2. The Timing Loop: The SMC counts individual clock cycles from a known hardware trigger (such as traffic on the POST or I2C buses) to track the exact nanosecond the signature check loop begins.
3. The Glitch Pulse: At the precise microsecond the CPU processes the branch instruction determining if the signature check passed or failed, the SMC drops the voltage on the CPU’s reset or PLL lines.
4. The Gate Flip: This intentional electrical instability starves the processor’s logic gates of power for a fraction of a clock cycle. Instead of evaluating the mismatched signature and executing a jump to the “shutdown” subroutine, a logic gate miscalculates and slips directly into the “success” pathway. The CPU is tricked into believing the unsigned code is completely authentic.

Because subtle factors like ambient temperature, wire routing, and silicon variances can shift the exact cycle the CPU processes this data, the glitch can occasionally miss. If it misses, the CPU panics, the SMC detects the failure, instantly triggers a hard reset of the system, and tries the exact sequence again on a brand-new boot loop.

Normal Boot Check:   [Verify Signature] ---> (Invalid) ---> [System Panic/Halt]
Glitched Boot Check: [Verify Signature] --(Voltage Drop)--> [Logic Error: Pass] ---> [Boot Code]

External Modchips vs. RGH3 (Internal Microcode Exploitation)

Historically, older RGH methods (RGH 1, 2, and S-RGH) relied on an external CPLD microchip (like the CoolRunner, Matrix or Ace) to perform a precise timing attack. This chip was listening to the motherboard’s signals and injecting the glitch, but because the timing had to be absolutely perfect, the chip often missed its tiny window of opportunity. Whenever a glitch failed, the CPU would panic and halt, forcing the chip to completely reboot the console to trigger a brand-new boot cycle and try again. This constant cycle of crashing and restarting meant users often had to wait minutes for the console to successfully boot. If you ever had the chance to observe one of those chip modded consoles, you would see a flashing LED on the modchip blinking at every single failed attempt.

RGH3 eliminated external silicon entirely. Instead of an external chip listening for signals, developers patched the firmware of the built-in System Management Controller (SMC) inside the NAND. Because the SMC lives natively on the motherboard architecture, it already has flawless synchronization with the master clock line. Only two basic connection wires are needed:

• The POST Wire: Connects the CPU’s boot status line (POST1) to the SMC’s monitoring point (SMC_POST1). This lets the SMC listen in and know exactly what stage of the boot sequence the CPU is in.

• The RST/PLL Wire: Connects a CPU control line (PLL_BYPASS on Slim models) directly to the SMC’s pulse point (SMC_PLL). This is the line SMC uses to drop the voltage and trigger the glitch loop.

The Patched Hypervisor

As we saw earlier, J-Runner uses the unique CPU key to decrypt, patch, and rebuild a completely custom version of the NAND image. Without this permanently modified system kernel, the console would immediately reject anything that lacks Microsoft’s official digital signature; you wouldn’t even be able to boot custom dashboards like Aurora, because the operating system would still be strictly blocking unauthorized file execution.

To fix this we need J-Runner to patch the nand. The primary target of the XeBuild image is the Hypervisor (HV), the highest privilege level of the Xbox 360 software environment. The Hypervisor acts as a strict ring-0 security layer that polices system memory. XeBuild patches the Hypervisor’s assembly code inside the system RAM to disable three fundamental defense mechanisms:

1. RSA Signature Elimination

The original Hypervisor checks the digital signature of every executable file (.xex) before letting it run. XeBuild modify the signature verification function to return “always pass”. This modification allows the console to load homebrew dashboards, game backups, and emulators flawlessly.

2. Overriding the W^X Memory Policy

To block modern buffer-overflow exploits, the official Hypervisor strictly enforces W^X (Write Xor Execute) memory isolation. A memory segment can be modified (written to), or it can execute instructions, but it can never do both simultaneously. XeBuild modifies the Hypervisor’s page table allocation rules, allowing specific memory spaces to be writable and executable at the same time. This modification is what enables real-time game modding engines, custom plugins, and trainers to run concurrently in the background.

3. Removal of Hardware and Regional Barriers

XeBuild patches the subroutines handling area identifiers (NTSC, PAL, NTSC-J) converting the console into region-free device.

Conclusions

I hope you enjoyed this deep technical article and found it useful for diving into the brilliant engineering behind the Xbox 360 mod.

At a high level, Kubernetes is split into two parts: the control plane that decides what should happen and the nodes that make it actually happen.

The control plane includes these components:

• Kube-apiserver that is the central entry point. Everything goes through this. Every component talks to it.
• Kube-scheduler it decides where a pod should run.
• Kube-controller-manager constantly tries to make reality match the desired state (replicas, nodes, jobs, etc.).
• etcd is the database storing the entire cluster state.

On the node side, things are more focused to make “things” run. There is container runtime (like containerd) that actually launches containers, Kubelet that makes sure containers actually run, the CNI plugin that wires up networking and kube-proxy that route traffic from Kubernetes Services to the pods.

In this article we will look more in detail the role of Kubelet and CNI plugins.

Kubelet — The primary node agent

The Kubelet is often described as a node agent, but that is a bit minimalistic respect to what it really does. It’s the core component that continuously reconciles what the control plane wants with what’s actually running on the machine. Notice that It doesn’t make decisions like the scheduler and it doesn’t store state like etcd. Instead, it’s constantly asking: “What pods am I supposed to run, and are they actually running?”. If the state is not the same, it will reconcile them. To achieve that, Kubelet operates as a loop:

• It watches the API server for pod specs assigned to its node.
• It compares that desired state with what’s running locally.
• If something is missing or broken, it fixes it.

For example actions that are normally performed by kubelet are:

• pulling images
• starting/stopping containers
• restarting crashed containers
• reporting status back to the control plane

However, is important to notice that kubelet does not get involved in any networking tasks. When a pod is about to start, Kubelet prepares everything, from filesystem to volumes to runtime configuration, but when it comes to networking, it hands off responsibility to CNI essentially saying: “Here’s a new container. Give it an IP and make sure it can talk to the cluster.”

This design is an architectural choice of Kubernetes. In fact, Kubernetes doesn’t include a native networking solution by default; instead, it delegates that entire responsibility to external CNI compliant plugins.

CNI — the network layer

CNI stands for Container Network Interface. It’s one of those things that sounds like a service, but it’s really just a contract. Indeed, strictly speaking is just a set of specifications that defines:

• how a runtime asks for networking
• what inputs/outputs look like
• how plugins should behave

Everything else is implementation detail. kubernetes only state that “Every pod gets its own IP, and all pods need to be able to talk to each other directly (no NAT in between).” This sounds simple, but it creates a big challenge:

• How do you make pod IPs routable across nodes?
• How do you scale that to thousands of pods?

That is where different CNI plugins take different approaches that will see in the next paragraph. So, when Kubelet asks the CNI plugin to “set up” a pod, the plugin executes a series of low-level Linux networking operations. This is where things get very real and very close to the OS. To wire up a pod, the plugin will:

• create a new network namespace for the pod (isolated networking stack)
• create a veth pair (two virtual interfaces connected like a cable)
• move one end into the pod namespace
• attach the other end to the host network (bridge, overlay, or routing layer)
• assign an IP address
• configure routes so packets know where to go

Once pods have IPs, how does traffic actually get routed the right pod behind a Service? This is where kube-proxy enters the picture. kube-proxy runs on every node and watches the API server for Service and Endpoint changes. Its job is to make sure that when you send traffic to a Service IP, it gets forwarded to one of the correct backend pods. In the traditional setup, kube-proxy implements this using iptables. It translates high-level Kubernetes concepts like:

• “Service X points to Pods A, B, and C” into low-level packet handling rules inside the Linux kernel.

A deeper look at CNI implementations -Cilium case

As we saw before, kubernetes only give specifications for CNI leave a lot of freedom to the actual implementation. Some plugins (like Flannel) use overlays which encapsulate traffic between nodes. Others (like Calico) rely more on routing and BGP. Newer ones like Cilium move the roting logic into the kernel using eBPF.

But regardless of implementation, they all sit behind the same CNI interface. To Kubelet, they look identical.

Ok, CNI plugins are responsible for wiring up networking, but how do they actually control traffic?

This is where the difference between traditional approaches (iptables) and modern ones (eBPF) really matters. For a long time, Kubernetes networking leaned heavily on iptables. iptables works by defining chains of rules that every packet is checked against. When a packet enters the system, it’s evaluated step by step until a matching rule is found. That rule might say:

• forward this packet
• rewrite its destination (NAT)
• drop it
• send it somewhere else

In Kubernetes, this mechanism gets used for several things at once:

• implementing Services (ClusterIP, NodePort)
• load balancing between pods
• handling outbound traffic (SNAT)
• enforcing some network policies

Over time, as the number of services and pods grows, the number of rules grows with it. And because iptables evaluates rules sequentially, performance becomes increasingly tied to how long those chains are and this is where eBPF changes the game. Instead of building long rule chains, eBPF allows you to attach small, purpose-built programs directly into the Linux kernel. These programs can intercept packets at various points and make decisions immediately, often using efficient data structures like maps and hash tables.

So instead of saying: “Check rule 1, then rule 2, then rule 3…”

with eBPF you’re saying:

“Run this program that already knows how to handle this packet.”

This reduces overhead and makes behavior far more performant at scale. Cilium is a CNI plugin that fully embrace this model. Rather than relying on iptables for service routing and policy enforcement, Cilium installs eBPF programs into the kernel that handle:

• service load balancing
• network policies (including L7 in some cases)
• packet forwarding decisions

In additions Cilium can replace kube-proxy entirely. Instead of using iptables to implement Services routing, it handles them directly via eBPF. From the outside, your Kubernetes cluster still behaves the same with Pods getting IPs, Services and Traffic flows working fine, but internally, the mechanics are very different. With eBPF (via Cilium), the system becomes more efficient with packets handled by logic in the kernel that can evolve without rewriting massive rule sets.

This leads to higher performance at scale, faster updates (no need to rebuild long rule chains) and deeper observability, since eBPF can inspect traffic more intelligently.

Key Takeaways

At this point, a short resume to make things clearer:

• The control plane (API server, scheduler, controllers) decides what should exist.
• Kubelet ensures pods actually run on a node.
• The CNI layer gives those pods a network identity.
• The underlying implementation (iptables or eBPF via Cilium) determines how traffic is handled.

If you zoom out far enough, Kubernetes networking is less about “one technology” and more about layers of delegation:

• Kubernetes delegates networking to CNI
• CNI plugins delegate packet handling to the kernel
• The kernel executes that logic via iptables or eBPF

I hope this article made Kubernetes appear now much less mysterious.

Skipping the Cloud and Gaining Independence

One of the best initial hacks is to force the Kobo into local-only account mode, which allows you to completely bypass the online registration and cloud login features. This is a must for those who prefer to sideload all their books and maintain privacy.

To enable local-only mode, you need to edit the Kobo’s configuration file. I did this on a freshly reset device, before you even turn it on the first time.

1. Access the Config File: Connect your Kobo to your PC and navigate to the .kobo/Kobo folder.
2. Locate Kobo Kobo eReader.conf: Open this file using a plain text editor (like Notepad++ or VS Code).

3. Under the [ApplicationPreferences] section, add the following line:

    SideloadedMode=true
   

4. Save and Eject: Save the file, safely eject the Kobo, and reboot.

Now, your Kobo will prompt you for local details instead of forcing a cloud registration, giving you complete sideloading freedom. I have also tested this for the Clara HD version.

The Hardware Fix: Repairing the Micro-USB Port

After few years of usage even a nice device like this can break up. For me the weak spot was the micro-USB connector. It eventually broke, preventing both charging and PC connectivity. So, I had to fix it. I considered upgrading the connector to USB-C but Since I already had similar Surface-Mount Technology (SMT) micro-USB connectors in my home lab, and I rarely need to charge the ebook reader, I decided to stick with the original port type instead of upgrading it.

The Soldering Challenge (Medium Difficulty)

The repair required careful soldering, a task of medium difficulty due to the small size of the contacts:

• Open the Case: the case is not glued but only clicked. so you can open it with help from a flat screwdriver
• Old Connector Removal: I first carefully desoldered the broken port and cleaned the remaining tin residue from the pads on the PCB.
• Alignment and Stabilization: The key was alignment. I placed the new connector, first soldering the two external anchor wings (the large structural tabs). This stabilized the connector perfectly. This worked fine because the new SMT micro usb port was basically identical to the old one!
• Data Pins: Finally, I soldered the four data/power pins to the board. Since the connector was already aligned and stable, this part was straightforward.

The repair was successful on the first attempt! The Kobo was charging and connecting to the PC again.

The Upgrade: 4GB to 32GB Memory Expansion

While the Kobo was open for the USB repair, I couldn’t resist to see what was inside. I immediately noticed an internal MicroSD card. Driven by curiosity, I unplugged the card and connected it to my PC. It turns out, this card is the device’s main storage, housing everything: the OS partition, a Recovery partition, and the Data partition for user books. My Kobo Glo HD came with a small 4GB card…clearly worth an upgrade.

The Expansion Process (From 4GB to 32GB)

The goal was to move everything to a new 32GB card and expand the user storage.

1. Full Image Backup: Using the dd command, I created a byte-for-byte clone of the original 4GB card. This is essential to transfer the OS and partitions exactly as they were.

    # Backup the original 4GB image to a file
    sudo dd if=/dev/mmcblk0 of=~/kobo_reader.img bs=4M status=progress
    # Restore the image onto the new 32GB card (inverse command)
    sudo dd if=~/kobo_reader.img of=/dev/mmcblk0 bs=4M status=progress

1. Resizing the Partition: After the image was restored to the 32GB card, I used disk to modify the partition table. The original image still defined the Data partition (partition 3) size as 4GB, so I extended the Data partition (/dev/mmcblk0p3) to utilize all the newly available space on the 32GB card. This step adjusted the boundary in the partition table.

2. The FAT32 Filesystem expansion: The Data partition was formatted as FAT32. Typically, after resizing the partition, you expand the filesystem to fill the new space. However, When I attempted to use fatresize (e.g., sudo fatresize -s max /dev/mmcblk0p3), the command failed to recognize the filesystem correctly and was unable to complete the resize operation.

The Workaround was basically: Backup, Delete, Recreate. Since direct filesystem resizing failed, I opted for a clean, reliable recreation:

1. I backed up all the original data files from the existing FAT32 filesystem.
2. I re-formatted the data partition as FAT32, ensuring it had the required volume label:

  sudo mkfs.vfat -F 32 -n "KOBOeReader" /dev/mmcblk0p3

1. Finally, I restored the original data files to the new, massive partition.

After inserting the 32GB card and reassembling the Kobo, it booted perfectly! The e-reader now fully recognizes and utilizes the 32GB of expanded memory, a massive increase that is even reflected in the device’s settings menu.

Final Thoughts

With a disabled cloud login, a fully repaired and reliable charging port, and a massive 32GB of internal storage, this Kobo Glo HD is now better than new. This project proves that the Kobo is not just an e-reader but a versatile piece of hardware that truly rewards the technically inclined people with easy to fix hardware and nice freedom… Good job Kobo!

As always I hope you enjoyed this article.

What Terraform Is

Terraform, created by HashiCorp, is an open-source tool for Infrastructure as Code (IaC). It allows you to define your entire infrastructure, from virtual machines to DNS records, in human-readable configuration files. Terraform then automates the provisioning and lifecycle management of these resources. With Terraform, you no longer need to click through dashboards or manually configure infrastructure: everything is defined in code. Creating a virtual machine, for example, is as simple as writing a 10-line block that declares its CPU, memory, and network. Want to change the CPU count? Just edit a single number. Want to remove it? Delete the block. It’s like magic, your entire infrastructure becomes reproducible, versioned, and self-documenting, with no need to remember how a particular network or database was configured. The key advantage is idempotency: Terraform knows what already exists and only applies the necessary changes to reach the desired state. It also provides a plan phase, where changes can be reviewed before being executed, ensuring predictability and control. For example, suppose you have a Terraform configuration that defines a Kubernetes namespace called monitoring with a Helm chart deploying Prometheus. If you later modify the Helm chart version or add a new configuration value, you can run:

terraform plan

Terraform will show a detailed execution plan, such as:

# helm_release.prometheus will be updated in-place
~ resource "helm_release" "prometheus" {
      version = "25.5.0" -> "25.6.1"
      values  = [
          ...
          + "alertmanager.enabled=true"
      ]
}
Plan: 0 to add, 1 to change, 0 to destroy.

This preview step lets you clearly see what Terraform intends to change, (in this case, upgrading Prometheus and enabling Alertmanager), before you apply the modifications. In a GitOps setup, this becomes even more powerful. The terraform plan command can be automatically triggered whenever a Pull Request is opened. The plan output is posted as a comment or status check in the PR, allowing the team to review the proposed infrastructure changes just like application code. Once the PR is approved and merged, a pipeline can then safely execute terraform apply, applying only the reviewed and approved changes. This ensures that every infrastructure modification is traceable, peer-reviewed, and auditable; aligned with gitops principles.

Terraform Beyond the Cloud

While Terraform is commonly used for cloud deployments, its provider ecosystem allows it to manage almost anything with an API, including on-premises infrastructure. This means you can use it to orchestrate:

• Kubernetes clusters running locally
• Bare-metal servers
• VMware or OpenStack virtual environments
• Network and storage configurations
• Helm charts and Kubernetes resources

In my case, the underlying infrastructure, Kubernetes clusters, load balancers, and base networking, is bootstrapped using Ansible. Ansible is great for low-level provisioning and configuration, especially when Terraform can’t directly manage the underlying hardware, making it ideal for setting up Kubernetes clusters or configuring operating systems.

Once that foundation is in place, Terraform takes over. Using its Kubernetes and Helm providers, Terraform manages the higher-level deployments: namespaces, ingress controllers, monitoring stacks, applications, and policies. This combination allows both flexibility and consistency: Ansible handles the initial provisioning, while Terraform governs the continuous state of the platform and applications.

Alignment with GitOps Principles

This hybrid approach fits perfectly within the GitOps philosophy. GitOps is about using Git as the single source of truth for your infrastructure and application configurations. All changes go through version-controlled pull requests, and automated pipelines apply those changes to the environment. The principles are simple but powerful:

• Declarative infrastructure — define your desired state in code.
• Versioned and auditable — all changes are tracked in Git.
• Automated reconciliation — systems automatically converge toward the declared state.
• Observability and feedback — visibility into what’s running versus what’s declared.

By using Terraform configurations stored in Git, combined with CI/CD or GitOps tools (like Argo CD or Flux), you can bring these same principles to your on-premises environments. Every infrastructure change becomes a pull request, every deployment is reproducible, and rollbacks are as simple as reverting a commit.

Benefits

Since adopting this approach, I’ve seen many benefits:

• Consistency across environments: development, staging, production, etc are defined the same way.
• Reproducibility: rebuilding an environment from scratch is straightforward. This is extremely important for disaster recovery!
• Transparency and collaboration: infrastructure changes are reviewed just like application code.
• Reduced drift: Terraform’s state tracking ensures the environment matches the desired configuration.
• Seamless integration with existing automation tools like Ansible, Jenkins, or Argo CD.

Final Thoughts

Terraform is not limited to the cloud. It’s a universal IaC tool, and when combined with Ansible and GitOps practices, it enables a modern, automated, and auditable on-premises infrastructure workflow.

This approach has been a success in my experience, bridging the gap between traditional datacenter automation and modern DevOps methodologies. It proves that the power of declarative infrastructure and GitOps principles is not bound to any specific platform: it’s a mindset that can transform how we manage infrastructure anywhere.

As always I hope this article was usefull.

But now, under the ownership of Broadcom, all that trust has been taken down. Instead of continuing their open source legacy, they’ve decided to cash in at the expense of the very community that made Bitnami what it was.

Broadcom has launched a new “Secure Images” subscription product and, in the process, effectively pulled the rug out from under thousands of developers and companies worldwide.

This wasn’t a simple “end of life” notice. That was an act that ensured maximum pain and minimal time for users to react.

The Deprecation That Destroys

Effective August 28th, 2025, Bitnami did not just stop updates to their free tier; they actively purged all versioned, stable image tags from the main docker.io/bitnami repository and moved to a new repository: docker.io/bitnamilegacy. In doing so, many tags were renamed or altered, breaking automated deployments and making migration intentionally more painful.

For teams that followed best practices by pinning versions to avoid surprises, this change turned stable systems into time bombs. Overnight, critical services that had been running fine for years were suddenly failing in the attempts to pull a versioned image from the old path (e.g., docker.io/bitnami/postgresql:13.7.0), getting ImagePullBackOff or ErrImagePull errors.

And if you wanted an “easy fix”? Well, Broadcom has a solution: pay for their new subscription service, that is reportedly priced in the tens of thousands of dollars annually. The message is basically “pay us or break”.

Quick fix

To stop the immediate, bleeding in your systems, you are forced to use the legacy images in your Helm chart configurations. For example in case of docker.io/bitnami/postgresql:13.7.0 you can use docker.io/bitnamilegacy/postgresql:13.7.0. Just make sure the tag is present in the new repo. The change is simple in syntax but do not rely on for long-term since those images are static and will have long term security implications.

The Path Forward: Migrate and Never Look Back

Bitnami and Broadcom have made their intentions clear: they are no longer interested in the community that built them up. They are only interested in extracting maximum profit from enterprises caught in the migration trap.

The real fix is simple: migrate away completely.

• Audit all your Bitnami Dependencies:

  kubectl get pods -A -o json | jq -r '..|.image? // empty' | sort -u | grep -i bitnami 
  

• Embrace Community Alternatives: Look to official upstream projects, the Kubernetes Community Helm Charts, or trusted vendors not owned by broadcom that value the open-source spirit.

Final Takeaway

This is a textbook example of vendor lock-in, where the vendor waits until you are fully reliant, then flips the switch to a massive paywall. The Bitnami deprecation is a business maneuver that provides a clear warning: do not build your production on a foundation controlled by a company that has proven it will break your systems for profit.

The real takeaway? Keep your SecOps and DevOps teams sharp (and well payed :P). Their skill is the only thing that will fix this mess without forcing you to surrender to the Bitnami pay-or-break blackmail.

Since that article, AWS has also introduced a new “EKS Cluster Auto-mode”, which provisions control plane and data plane resources more seamlessly. However, this comes with increased cost and reduced control. For many teams, especially where cost control is important, sticking with traditional EKS + Fargate still makes more sense.

However, the Terraform EKS module introduced breaking changes in version 21.0.0, notably removing native IRSA (IAM Roles for Service Accounts) support. This has significant implications, since by removing IRSA:

• Any workloads relying on it to assume IAM roles (e.g., Karpenter) will break.
• Fargate workloads, which must use IRSA for AWS permissions will break.

While EKS now supports Pod Identity as an alternative, Fargate cannot leverage that since is not supported. The core reason is that EKS Pod Identity relies on a dedicated agent (add-on) that runs as a DaemonSet on each worker node to provide credentials. Since AWS Fargate does not support DaemonSets, this agent cannot be deployed, making Pod Identity incompatible with Fargate. For pods on Fargate, you must continue to use IAM Roles for Service Accounts (IRSA).

It is particularly surprising that the EKS Terraform module developers, when deprecating native IRSA support in favor of Pod Identity, did not provide a documented solution for common use cases like Karpenter on Fargate, which cannot use the new approach..

So if you’re using Karpenter on Fargate, and you’re now stuck… you need to manually restore IRSA support. but before i show you how, lets clarify the difference between IRSA and Pod Identity.

IRSA vs Pod Identity

What is IRSA?

IAM Roles for Service Accounts (IRSA) is the original and widely adopted method of securely assigning AWS IAM permissions to Kubernetes pods. With IRSA, the flow works like this:

• You create an IAM Role with a trust policy that allows the EKS OIDC provider to assume it.
• The trust policy ensures only a specific service account in a specific namespace can use that role.
• The Kubernetes service account is annotated with the role ARN.
• When the pod starts, EKS injects a web identity token into the pod, which AWS STS uses to validate and issue temporary credentials.

Key advantages:

• Secure, least-privilege model.
• Easy to audit and manage.
• Works perfectly with EKS and Fargate.

What is Pod Identity?

EKS Pod Identity is a newer approach introduced by AWS in 2024, offering another way to associate IAM roles with pods.

Instead of relying on OIDC providers and complex trust policies, you create an association that directly links an IAM role to a Kubernetes service account within a specific namespace using the EKS API. This new approach relies on a daemonset running on worker nodes, and as stated above, is not supported by fargate.

Manually Restoring IRSA for Karpenter on Fargate

If you’re using the EKS Terraform module version 21.x or higher, the built-in IRSA configuration for Karpenter on Fargate is no longer available. To restore this functionality, you must manually define the necessary IAM role trust policy and pass it to the Karpenter module. This approach effectively recreates the IRSA trust relationship that was previously managed by the module, allowing your Karpenter pods on Fargate to assume their required IAM role. if you do not restore the policy, karpenter will fail to start with this error message

 failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 403, RequestID: 3d312d21-22d2-4d80-c222-227ae8c27cc1, api error AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity"

To restore the policy, here is the snippet of terraform code that implements these changes. It has been adapted from a GitHub discussion

data "aws_iam_policy_document" "karpenter_controller_assume_role_policy" {
    statement {
        effect = "Allow"
    
        principals {
          type        = "Federated"
          identifiers = [module.YOUR_EKS_CLUSTER.oidc_provider_arn]
        }
    
        condition {
          test     = "StringEquals"
          variable = "${module.YOUR_EKS_CLUSTER.oidc_provider}:sub"
          values   = ["system:serviceaccount:karpenter:karpenter"]
        }
    
        condition { # Required for OIDC trust
          test     = "StringEquals"
          variable = "${module.YOUR_EKS_CLUSTER.oidc_provider}:aud"
          values   = ["sts.amazonaws.com"]
        }
    
        actions = ["sts:AssumeRoleWithWebIdentity"]
    }
}

and then link this policy to karpenter:

module "karpenter" {
  ...
  iam_role_source_assume_policy_documents = [
    data.aws_iam_policy_document.karpenter_controller_assume_role_policy.json,
  ]
}

I’ve tested these changes and can confirm that Karpenter is successfully able to assume the role and provision new nodes.

As always, I hope you found this article helpful.

What is Watchtower?

Watchtower is a lightweight and powerful open-source Docker container that automates the process of updating your running Docker containers. It basically monitors your running containers and, when it detects a new version of their base image on Docker Hub (or other registry), it gracefully stops the old container, pulls the updated image, and restarts a new container using the same configuration parameters (environment variables, volumes, network settings, etc.) defined in your Docker Compose file.

Integrating Watchtower into your Docker Compose workflow offers many advantages:

• Automation: The most significant benefit is the automatic image updates. Once configured, Watchtower handles the entire update cycle without requiring manual intervention.
• Security: Regularly updating container images is an important security practice. Watchtower ensures you’re always running the latest versions, which often include critical security patches.
• Staying Up-to-Date with Features: New image versions often bring performance improvements and new functionalities.

Before we start to dig deeper into Watchtower usage, I want to clarify few things first. While Kubernetes and tools like Minikube offer powerful orchestration for complex deployments, for a small home environment, Docker Compose remains significantly more convenient and easier to manage. Its simplicity in defining and running multi-container applications with a single docker-compose.yml file is ideal for personal projects and home servers. Watchtower perfectly complements this setup by automating image updates, a task often manually handled in such environments.

It’s also important to note that Watchtower does not change the tag specified in your Docker Compose file; rather, it monitors the registry for updates to the image associated with the specified tag. This means that using :latest will trigger an update whenever a new version is pushed to the latest tag. However, if you specify a fixed tag like :1.2.1, Watchtower will only update if a new image is pushed to that exact tag and will not automatically pull versions tagged differently, such as :1.2.2.

Also, be aware that Watchtower automatic upgrade come with the risk of unexpected breaking of applications (eg latest tag moved across major version updates). Because these potential issues it should not be used anywhere else except your home/hobby projects, as specified in their official repo:

Watchtower is intended to be used in homelabs, media centers, local dev environments, and similar. We do not recommend using Watchtower in a commercial or production environment. If that is you, you should be looking into using Kubernetes. If that feels like too big a step for you, please look into solutions like MicroK8s and k3s that take away a lot of the toil of running a Kubernetes cluster.

How to Integrate Watchtower with Docker Compose

Imagine a typical Docker Compose setup where you have an application and a database running as services. In this common case, integrating Watchtower is very easy. You essentially add Watchtower as another service to your docker-compose.yaml file.

Here’s a general example of how to configure Watchtower in your docker-compose.yml:

version: '3.8'

services:
  your_app:
    image: your/app:latest
    # ... other configurations for your application ...
    restart: unless-stopped
    
  your_db:
    image: your/db:10.9
    # ... other configurations for your DB ...
    restart: unless-stopped

  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_SCHEDULE: "0 0 4 * * *"
    command: your_app

Let’s break down the watchtower configuration.

• image: containrrr/watchtower: This specifies the official Watchtower image from Docker Hub.

• restart: unless-stopped: This ensures that if the Watchtower container restarts for any reason, Docker will automatically bring it back up.

• volumes: - /var/run/docker.sock:/var/run/docker.sock: This is crucial. It mounts the Docker socket inside the Watchtower container, allowing it to communicate with the Docker daemon and manage other containers.

• environment: This section configures Watchtower using environment variables:
  • WATCHTOWER_CLEANUP: "true": This tells Watchtower to automatically remove the old containers after a successful update, keeping your system tidy.
  • WATCHTOWER_SCHEDULE: "0 0 4 * * *": This configures Watchtower to run on a cron schedule. The example 0 0 4 * * * specifies that Watchtower will check for updates every day at 4:00 AM.
• command: your_app: This instructs Watchtower to specifically monitor and update the container named your_app in the docker-compose.yml file. You can list multiple service names here, separated by spaces, if you want Watchtower to manage more than one service (e.g. command: your_app your_db) if you omit this command, watchtower will manage all the container used in the docker compose file. Alternatively, you can also add labels to your service and then configure watchtower to only manage container that have that label (leading to the same results).

One final note. if you tell Watchtower to monitor your database service (e.g., your_db) and you specified a minor version tag like :10.9, this will ensure it receives patch updates(eg a new image pushed with :10.9 tag) without the risk of automatic major version upgrades that could break the database.

Conclusion

Watchtower presents a powerful and convenient solution for automating container image updates within a home Docker Compose environment, simplifying its maintenance. Its ability to automatically fetch and deploy newer versions ensures you’re running with the latest security patches. However, leveraging this automation without caution can lead to problems. For critical services like databases, there is a serius risk of causing data corruption or service instability. In these cases I recommend to carefully select the tags manged by Watchtower and limiting the latest tag for less critical internet exposed services.