{ "slug": "Introducing-Buf-for-Protobuf-06ebc98710c6", "title": "Introducing Buf for Protobuf", "subtitle": "Buf is an innovative tool that revolutionizes API development by promoting a schema-driven approach using Protocol Buffers (Protobuf). As…", "excerpt": "Buf is an innovative tool that revolutionizes API development by promoting a schema-driven approach using Protocol Buffers (Protobuf). As…", "date": "2024-08-12", "tags": [ "API" ], "readingTime": "5 min", "url": "https://medium.com/@mobinshaterian/buf-protobuf-06ebc98710c6", "hero": "https://cdn-images-1.medium.com/max/800/1*Z84e1iFw5VCdnpLV0azz9w.jpeg", "content": [ { "type": "heading", "level": 2, "text": "Introducing Buf for Protobuf" }, { "type": "paragraph", "html": "Buf is an innovative tool that revolutionizes API development by promoting a schema-driven approach using Protocol Buffers (Protobuf). As the industry's most stable and widely adopted Interface Definition Language (IDL), Protobuf offers numerous advantages over traditional REST/JSON services. However, implementation difficulties have hindered its adoption. Buf aims to address these challenges by providing user-friendly tooling that makes Protobuf more accessible and efficient for service owners and clients." }, { "type": "paragraph", "html": "Summary:" }, { "type": "paragraph", "html": "1. Buf’s primary goal is simplifying Protobuf usage while maintaining its technical superiority." }, { "type": "paragraph", "html": "2. Key features of the Buf CLI include:
 — A linter for enforcing good API design
 — A breaking change detector for compatibility checks
 — A code generator based on configurable templates
 — A formatter for standardizing Protobuf files
 — Integration with the Buf Schema Registry" }, { "type": "paragraph", "html": "3. Buf addresses common Protobuf adoption challenges such as:
 — Inconsistent API designs
 — Poor dependency management
 — Compatibility issues
 — Complicated stub distribution
 — Limited tooling ecosystem" }, { "type": "paragraph", "html": "4. The tool offers streamlined processes for:
 — Installing and configuring Buf
 — Generating code from Protobuf schemas
 — Linting APIs
 — Managing dependencies" }, { "type": "paragraph", "html": "5. Buf supports various plugins and can be integrated with GitHub Actions for automated workflows." }, { "type": "paragraph", "html": "6. The Buf Schema Registry (BSR) provides a centralized platform for exploring and sharing Protobuf schemas." }, { "type": "paragraph", "html": "7. Buf is compatible with other modern gRPC frameworks like Connect, which offers additional features for building browser and gRPC-compatible HTTP APIs." }, { "type": "paragraph", "html": "By addressing the complexities associated with Protobuf implementation, Buf is positioning itself as a comprehensive solution for organizations looking to leverage the power of schema-driven API development without the traditional overhead." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*Z84e1iFw5VCdnpLV0azz9w.jpeg", "alt": "Introducing Buf for Protobuf", "caption": "", "width": 1024, "height": 1024 }, { "type": "paragraph", "html": "Buf’s goal is to shift API development toward a schema-driven paradigm and thus pave the way for a future in which APIs are defined in a way that service owners and clients can depend on." }, { "type": "paragraph", "html": "Defining APIs using an IDL provides several benefits over simply exposing REST/JSON services, and today, Protobuf is the most stable, widely adopted IDL in the industry. But as things stand today, using Protobuf is much more difficult than using JSON as your data transfer format." }, { "type": "paragraph", "html": "Buf is building tooling to make Protobuf reliable and user-friendly for service owners and clients while keeping it the obvious choice on the technical merits. Your organization should not have to reinvent the wheel to create, maintain, and consume Protobuf APIs efficiently and effectively. We’ll handle your Protobuf management strategy so you can focus on what matters." }, { "type": "paragraph", "html": "The buf CLI is the best tool for working with Protocol Buffers. It provides:" }, { "type": "list", "ordered": false, "items": [ "A linter that enforces good API design choices and structure.", "A breaking change detector that enforces compatibility at the source code or wire level.", "A generator that invokes your plugins based on configurable templates.", "A formatter that formats your Protobuf files following industry standards.", "Integration with the Buf Schema Registry, including full dependency management." ] }, { "type": "paragraph", "html": "Challenges in adopting Protobuf:" }, { "type": "paragraph", "html": "Inconsistent API designs: Lack of standards leads to inconsistency across organizations.
Poor dependency management: Manual file handling causes errors and lacks centralized tracking.
Compatibility issues: Maintaining backward-compatible APIs is difficult to enforce.
Complicated stub distribution: Organizations struggle with distributing Protobuf files and stubs efficiently.
Limited tooling ecosystem: Lack of user-friendly tools for common API tasks compared to REST/JSON APIs." }, { "type": "heading", "level": 2, "text": "Installation" }, { "type": "paragraph", "html": "Install on Ubuntu" }, { "type": "code", "lang": "bash", "code": "go get github.com/bufbuild/buf/cmd/buf@v1.36.0\ngo install github.com/bufbuild/buf/cmd/buf@v1.36.0\nbuf\n--version" }, { "type": "paragraph", "html": "It is possible to add it in bashrc" }, { "type": "code", "lang": "javascript", "code": "export PATH=$PATH:$(go env GOPATH)/binsource ~/.bashrcbuf --version" }, { "type": "heading", "level": 2, "text": "Try the Buf CLI" }, { "type": "code", "lang": "bash", "code": "buf config init" }, { "type": "paragraph", "html": "Add file module:" }, { "type": "code", "lang": "text", "code": "# For details on buf.yaml configuration, visit\nhttps://buf.build/docs/configuration/v2/buf-yamlversion: v2modules: - path: protolint: use: -\nDEFAULTbreaking: use: - FILE" }, { "type": "heading", "level": 3, "text": "Verify that everything is set up properly and the module builds:" }, { "type": "code", "lang": "bash", "code": "buf build" }, { "type": "paragraph", "html": "The Buf CLI provides a user-friendly experience for generating code locally that’s compatible with any reasonable existing usage of protoc, so let's jump in and generate some code." }, { "type": "heading", "level": 2, "text": "Full example of making proto with buf" }, { "type": "heading", "level": 2, "text": "Configure a buf.gen.yaml file" }, { "type": "code", "lang": "text", "code": "touch buf.gen.yaml‍‍‍‍‍‍" }, { "type": "paragraph", "html": "Add this content into buf.gen.yaml" }, { "type": "code", "lang": "yaml", "code": "version: v2managed: enabled: true override: - file_option: go_package_prefix value:\ngithub.com/bufbuild/buf-tour/genplugins: - remote: buf.build/protocolbuffers/go out: gen opt:\npaths=source_relative - remote: buf.build/connectrpc/go out: gen opt:\npaths=source_relativeinputs: - directory: proto" }, { "type": "paragraph", "html": "Given this config, the Buf CLI does two things:" }, { "type": "list", "ordered": false, "items": [ "It executes the protocolbuffers/go plugin to generate Go-specific code for your .proto files and places its output in the gen directory.", "It executes the connectrpc/go plugin to generate client and server stubs for Connect-Go into the gen directory." ] }, { "type": "paragraph", "html": "1. Managed mode:
 — A setting that automatically configures file options for Protobuf languages
 — Eliminates the need to set options in .proto files
 — Reduces confusion and frustration for users" }, { "type": "paragraph", "html": "2. Remote plugins:
 — Plugins hosted on the Buf Schema Registry
 — Eliminates the need for local plugin management" }, { "type": "paragraph", "html": "3. Inputs for buf generate:
 — Can use various input types (Buf modules, GitHub repos, archives)
 — Example uses the ‘proto’ subdirectory in the workspace" }, { "type": "heading", "level": 2, "text": "Generate" }, { "type": "code", "lang": "bash", "code": "buf generate" }, { "type": "heading", "level": 2, "text": "Lint your API" }, { "type": "paragraph", "html": "Run this command to check all .proto files in the tour workspace for lint errors:" }, { "type": "code", "lang": "bash", "code": "buf lint" }, { "type": "heading", "level": 2, "text": "Version 1 Buf" }, { "type": "paragraph", "html": "As you may know, since the first introduction, the buf tool has served as a good linter for the gRPC ecosystem for a time. After all that time, they released a tool to help the work with gRPC become even easier. The buf generate tool has been announced for a long time, and I have also tried the beta version. It’s great, but it still lacks many features to convince me to adopt that." }, { "type": "heading", "level": 2, "text": "Steps for generating" }, { "type": "code", "lang": "bash", "code": "buf config init\nbuf buildtouch buf.gen.yaml‍‍‍‍‍‍buf generate\nbuf lint" }, { "type": "paragraph", "html": "buf.gen.yaml" }, { "type": "code", "lang": "yaml", "code": "version: v2managed: enabled: true override: - file_option: go_package_prefix value:\nconnectrpc.com/connect/internal/genplugins: - local: protoc-gen-go out: gen opt:\npaths=source_relative" }, { "type": "paragraph", "html": "How to write buf.gen.yaml" }, { "type": "paragraph", "html": "Install the protoc-gen-go plugin, or have the corresponding protoc plugin for your output language of choice installed and in your $PATH. The code examples use the Go plugin." }, { "type": "code", "lang": "properties", "code": "$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.31.0$ export\nPATH=\"$PATH:$(go env GOPATH)/bin\"" }, { "type": "heading", "level": 2, "text": "Sample of buf.gen.yaml for golanf" }, { "type": "code", "lang": "yaml", "code": "version: v2managed: enabled: true override: - file_option: go_package_prefix value:\ngithub.com/your_addressplugins: - local: protoc-gen-go out: go opt: paths=source_relative -\nremote: buf.build/connectrpc/go out: go opt: paths=source_relative" }, { "type": "paragraph", "html": "buf.yaml" }, { "type": "code", "lang": "text", "code": "# For details on buf.yaml configuration, visit\nhttps://buf.build/docs/configuration/v2/buf-yamlversion: v2lint: use: - DEFAULTbreaking: use:\n- FILE" }, { "type": "heading", "level": 3, "text": "Protobuf file" }, { "type": "code", "lang": "protobuf", "code": "syntax = \"proto3\";\npackage models.v1;\noption go_package = \".modelsv1\" ;\n\nmessage People {\n string user_uid = 1;\n\n}" }, { "type": "heading", "level": 2, "text": "Github Action" }, { "type": "heading", "level": 2, "text": "Connectrpc" }, { "type": "paragraph", "html": "Connect is a slim library for building browser and gRPC-compatible HTTP APIs. You write a short Protocol Buffer schema and implement your application logic, and Connect generates code to handle marshaling, routing, compression, and content type negotiation. It also generates an idiomatic, type-safe client. Handlers and clients support three protocols: gRPC, gRPC-Web, and Connect’s own protocol." }, { "type": "heading", "level": 2, "text": "Add another proto buf from a repository." }, { "type": "embed", "provider": "youtube", "url": "https://www.youtube.com/embed/OSdQlnoO0og?feature=oembed" }, { "type": "heading", "level": 2, "text": "Building Modern gRPC Service with Buf" } ] }