In the world of data engineering with Go, the OOMKilled (Exit Code 137) error is a common ghost in the machine. It often haunts developers who are moving large datasets between databases — like syncing tables from Oracle to a target destination.

The culprit isn’t usually a permanent memory leak, but a temporary memory spike. When you load a massive table into RAM, transform it, and then prepare to write it, you often end up holding two or even three copies of the same data in memory simultaneously.

To combat this, we use the “Clear-as-you-go” Pattern.

The Problem: The “Double-Copy” Trap

Imagine you are processing a table with 1,000,000 rows. Your workflow typically looks like this:

  1. Load: Read all 1M rows into an inputData slice.
  2. Transform: Iterate through inputData, create new outputRow objects, and store them in an outputData slice.
  3. Write: Send outputData to the target.

Between Step 2 and Step 3, your application’s memory usage peaks. You are holding the raw source data and the newly formatted target data. If each set is 800MB and your container limit is 1GB, the OS will trigger a SIGKILL, resulting in that dreaded 137 exit code.

The Solution: The “Clear-as-you-go” Pattern

The “Clear-as-you-go” pattern leverages the way Go’s Garbage Collector (GC) works. In Go, an object stays in memory as long as there is a pointer referencing it. By “nilling out” references to source data the very moment they are no longer needed, we allow the GC to reclaim that memory incrementally while the rest of the function is still running.

Implementation in a Transformation Loop

Instead of waiting for the function to return to free the input data, we destroy the input row as soon as the output row is generated:

go
func (s *Service) applyTransformations(inputData *model.TableData, outputData *model.TableData) {
    for rowIdx, inputRow := range inputData.Rows {
        // 1.Create the new data outputRow := s.transformSingleRow(inputRow) // 2.Store it in the
        output slice outputData.Rows[rowIdx] = outputRow // 3.THE PATTERN: Delete the reference to
        the source row // This makes the memory eligible for GC immediately inputData.Rows[rowIdx] =
        nil
    }
    // 4.Fully release the input slice header
    inputData.Rows = nil
}

Why This Works: Flattening the Memory Curve

Without this pattern, your memory usage looks like a steep mountain. With this pattern, your memory usage looks like a plateau. As your outputData slice grows, your inputData slice shrinks at exactly the same rate.

By the time you reach the end of the transformation loop, your RAM is only holding the outputData. You have effectively halved your peak memory requirement.

Pro-Tips for Implementation

  1. Mind the Pointers: This pattern only works if inputRow is a pointer or contains pointers (like a map or slice). If you are using a slice of primitive structs, nilling the index won’t have the same effect; you would need to clear the slice itself.
  2. Manual GC Triggers: If you are working in a highly constrained environment (like a small Kubernetes Pod), you can occasionally call runtime.GC() every few thousand rows to ensure the “cleared” memory is actually released back to the heap.
  3. Set GOMEMLIMIT: Always pair this pattern with the GOMEMLIMIT environment variable (introduced in Go 1.19). This tells the Go runtime exactly when it should become aggressive about garbage collection to avoid an OOM kill.

Conclusion

The “Clear-as-you-go” pattern is a simple yet powerful technique for Go developers handling large-scale data. It transforms a fragile, spike-prone application into a stable, memory-efficient pipeline. By being intentional about when you let go of data, you can process significantly larger datasets on the same hardware without ever seeing an exit code 137 again.