Introducing Buf for Protobuf

Buf is an innovative tool that revolutionizes API development by promoting a schema-driven approach using Protocol Buffers (Protobuf). As the industry's most stable and widely adopted Interface Definition Language (IDL), Protobuf offers numerous advantages over traditional REST/JSON services. However, implementation difficulties have hindered its adoption. Buf aims to address these challenges by providing user-friendly tooling that makes Protobuf more accessible and efficient for service owners and clients.

Summary:

1. Buf’s primary goal is simplifying Protobuf usage while maintaining its technical superiority.

2. Key features of the Buf CLI include:
— A linter for enforcing good API design
— A breaking change detector for compatibility checks
— A code generator based on configurable templates
— A formatter for standardizing Protobuf files
— Integration with the Buf Schema Registry

3. Buf addresses common Protobuf adoption challenges such as:
— Inconsistent API designs
— Poor dependency management
— Compatibility issues
— Complicated stub distribution
— Limited tooling ecosystem

4. The tool offers streamlined processes for:
— Installing and configuring Buf
— Generating code from Protobuf schemas
— Linting APIs
— Managing dependencies

5. Buf supports various plugins and can be integrated with GitHub Actions for automated workflows.

6. The Buf Schema Registry (BSR) provides a centralized platform for exploring and sharing Protobuf schemas.

7. Buf is compatible with other modern gRPC frameworks like Connect, which offers additional features for building browser and gRPC-compatible HTTP APIs.

By addressing the complexities associated with Protobuf implementation, Buf is positioning itself as a comprehensive solution for organizations looking to leverage the power of schema-driven API development without the traditional overhead.

Buf’s goal is to shift API development toward a schema-driven paradigm and thus pave the way for a future in which APIs are defined in a way that service owners and clients can depend on.

Defining APIs using an IDL provides several benefits over simply exposing REST/JSON services, and today, Protobuf is the most stable, widely adopted IDL in the industry. But as things stand today, using Protobuf is much more difficult than using JSON as your data transfer format.

Buf is building tooling to make Protobuf reliable and user-friendly for service owners and clients while keeping it the obvious choice on the technical merits. Your organization should not have to reinvent the wheel to create, maintain, and consume Protobuf APIs efficiently and effectively. We’ll handle your Protobuf management strategy so you can focus on what matters.

The buf CLI is the best tool for working with Protocol Buffers. It provides:

A linter that enforces good API design choices and structure.
A breaking change detector that enforces compatibility at the source code or wire level.
A generator that invokes your plugins based on configurable templates.
A formatter that formats your Protobuf files following industry standards.
Integration with the Buf Schema Registry, including full dependency management.

Challenges in adopting Protobuf:

Inconsistent API designs: Lack of standards leads to inconsistency across organizations.
Poor dependency management: Manual file handling causes errors and lacks centralized tracking.
Compatibility issues: Maintaining backward-compatible APIs is difficult to enforce.
Complicated stub distribution: Organizations struggle with distributing Protobuf files and stubs efficiently.
Limited tooling ecosystem: Lack of user-friendly tools for common API tasks compared to REST/JSON APIs.

Installation

Install on Ubuntu

bash

go get github.com/bufbuild/buf/cmd/buf@v1.36.0
go install github.com/bufbuild/buf/cmd/buf@v1.36.0
buf
--version

It is possible to add it in bashrc

javascript

export PATH=$PATH:$(go env GOPATH)/binsource ~/.bashrcbuf --version

Try the Buf CLI

bash

buf config init

Add file module:

text

# For details on buf.yaml configuration, visit
https://buf.build/docs/configuration/v2/buf-yamlversion: v2modules:  - path: protolint:  use:    -
DEFAULTbreaking:  use:    - FILE

Verify that everything is set up properly and the module builds:

bash

buf build

The Buf CLI provides a user-friendly experience for generating code locally that’s compatible with any reasonable existing usage of protoc, so let's jump in and generate some code.

Full example of making proto with buf

Configure a buf.gen.yaml file

text

touch buf.gen.yaml‍‍‍‍‍‍

Add this content into buf.gen.yaml

yaml

version: v2managed:  enabled: true  override:    - file_option: go_package_prefix      value:
github.com/bufbuild/buf-tour/genplugins:  - remote: buf.build/protocolbuffers/go    out: gen    opt:
paths=source_relative  - remote: buf.build/connectrpc/go    out: gen    opt:
paths=source_relativeinputs:  - directory: proto

Given this config, the Buf CLI does two things:

It executes the protocolbuffers/go plugin to generate Go-specific code for your .proto files and places its output in the gen directory.
It executes the connectrpc/go plugin to generate client and server stubs for Connect-Go into the gen directory.

1. Managed mode:
— A setting that automatically configures file options for Protobuf languages
— Eliminates the need to set options in .proto files
— Reduces confusion and frustration for users

2. Remote plugins:
— Plugins hosted on the Buf Schema Registry
— Eliminates the need for local plugin management

3. Inputs for buf generate:
— Can use various input types (Buf modules, GitHub repos, archives)
— Example uses the ‘proto’ subdirectory in the workspace

Generate

bash

buf generate

Lint your API

Run this command to check all .proto files in the tour workspace for lint errors:

bash

buf lint

Version 1 Buf

As you may know, since the first introduction, the buf tool has served as a good linter for the gRPC ecosystem for a time. After all that time, they released a tool to help the work with gRPC become even easier. The buf generate tool has been announced for a long time, and I have also tried the beta version. It’s great, but it still lacks many features to convince me to adopt that.

Steps for generating

bash

buf config init
buf buildtouch buf.gen.yaml‍‍‍‍‍‍buf generate
buf lint

buf.gen.yaml

yaml

version: v2managed:  enabled: true  override:    - file_option: go_package_prefix      value:
connectrpc.com/connect/internal/genplugins:  - local: protoc-gen-go    out: gen    opt:
paths=source_relative

How to write buf.gen.yaml

Install the protoc-gen-go plugin, or have the corresponding protoc plugin for your output language of choice installed and in your $PATH. The code examples use the Go plugin.

properties

$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.31.0$ export
PATH="$PATH:$(go env GOPATH)/bin"

Sample of buf.gen.yaml for golanf

yaml

version: v2managed:  enabled: true  override:    - file_option: go_package_prefix      value:
github.com/your_addressplugins:  - local: protoc-gen-go    out: go    opt: paths=source_relative  -
remote: buf.build/connectrpc/go    out: go    opt: paths=source_relative

buf.yaml

text

# For details on buf.yaml configuration, visit
https://buf.build/docs/configuration/v2/buf-yamlversion: v2lint:  use:    - DEFAULTbreaking:  use:
- FILE

Protobuf file

protobuf

syntax = "proto3";
package models.v1;
option go_package = ".modelsv1" ;

message People {
  string user_uid = 1;

}

Github Action

Connectrpc

Connect is a slim library for building browser and gRPC-compatible HTTP APIs. You write a short Protocol Buffer schema and implement your application logic, and Connect generates code to handle marshaling, routing, compression, and content type negotiation. It also generates an idiomatic, type-safe client. Handlers and clients support three protocols: gRPC, gRPC-Web, and Connect’s own protocol.

Introducing Buf for Protobuf