gRPC Streams Are Not Thread-Safe

Waken 2023-06-17

Hit a concurrency bug with gRPC streams. Turns out they’re not thread-safe for concurrent writes.

The Issue

Was trying to write to a gRPC stream from multiple goroutines. Got strange errors and corrupted messages.

The Rule

From the gRPC discussion:

Stream does not allow multiple concurrent writers. The general rule is that you should NOT assume it is safe to be called concurrently unless the comment explicitly states that.

What This Means

// DON'T do this
func sendMessages(stream pb.Service_StreamClient) {
    go func() {
        stream.Send(&pb.Message{Data: "from goroutine 1"})
    }()

    go func() {
        stream.Send(&pb.Message{Data: "from goroutine 2"})
    }()
}

Multiple concurrent Send() calls = undefined behavior.

The Fix

Serialize writes through a single goroutine:

func sendMessages(stream pb.Service_StreamClient) {
    msgChan := make(chan *pb.Message, 10)

    // Single writer goroutine
    go func() {
        for msg := range msgChan {
            stream.Send(msg)
        }
    }()

    // Other goroutines send to channel
    go func() {
        msgChan <- &pb.Message{Data: "from goroutine 1"}
    }()

    go func() {
        msgChan <- &pb.Message{Data: "from goroutine 2"}
    }()
}

Channel serializes access. Only one goroutine actually writes to the stream.

Lesson

gRPC streams work like most Go stdlib types: not concurrent by default. Check container/list, bufio.Writer, etc. - same pattern.

If docs don’t explicitly say “thread-safe”, assume it’s not.