grpc

When building a microservice architecture, two of the most common approaches are a common messaging queue and HTTP APIs that are exposed and consumed by each element of the project. Despite those approaches being perceived as well-known and battle-tested, the community warmly welcomed another player to the game - gRPC. This is an introduction to this way of communication, and the story has, to begin with, a date with Protocol Buffers - something that sets gRPC apart.

What is gRPC?

In short, gRPC is a remote procedure call framework (ergo the name), that makes building scalable, distributed systems easy and fast. It has many advantages over an HTTP request-response approach because it requires both parties to implement the same data structures, therefore saves time on custom encoding and decoding of data. As gRPC is using Protocol Buffers, the sizes of the messages are also smaller, which allows your servers to process more requests.

What are Protocol Buffers?

Sending JSON values over the network seems to be a great idea, right? On each end of the communication stream, you can relatively easily implement some encoder and decoder, as most of the language provide such features out of the box. The thing is, that if you take a look at how long the process is, you may be shocked. It's not uncommon for even slightly more complicated systems to spend most of their time on JSON parsing (except for I/O calls, of course). That's not right! Protocol buffers allow you to send data in a binary form, saving a lot (a lot!) of time there. Let's get started then!

First of all, we need to build a .proto file with messages definitions. Each message needs to start with message directive and must contain a list of fields with their types and an ID. That ID represents an ordered place in the binary structure, and you should put most common values first. The reason is that fields with ID in (1,15) range take just one byte to encode, then the size increases, so if you have an attribute that appears in each message instance, it should be near the top to save space.

message Person {
    string first_name = 1;
    string last_name = 2;
    string date_of_birth = 3;
    bool cool = 4;
    int32 arguments_won = 5;
}

You can also use one message type as an attribute of another, just put its name as the type and you're good to go:

message Person {
    ...
    Hobby hobby = 6;
}

message Hobby {
    string name = 1;
    string description = 2;
}

If you want to have a list of some types, you can prefix an attribute with repeated keyword:

message Person {
    ...
    repeated Hobby hobbies = 6;
}

What you should know, as we'll be exploring gRPC using Golang, is that there are some important things that need to be defined in the file header. First is a version of the protocol, which needs to be set to proto3 as only this one is compatible with gRPC (we won't dig into previous versions so you're not missing anything). Then you should define a package, which is important if you ever want to use definitions from different files.

// file1.proto
syntax = 'proto3';

package mycodesmells.golangexamples.grpc.message;

Then it can me used somewhere else:

// file2.proto
syntax = 'proto3';

import "path/to/file1.proto"

message Another {
    mycodesmells.golangexamples.grpc.message.Person person = 1;
}

Last but not least, you need to define a package for Go code that later will be generated. It should contain a full path relative to your $GOPATH, just the way you'd import the file to another package:

option go_package = "github.com/mycodesmells/golang-examples/grpc/proto/message";

Once you do all this, you can perform what is probably the coolest things about gRPC and protobuf (short from Protocol Buffers) - generate code (you'll need to install protobuf before):

protoc --go_out=${GOPATH}/src proto/message/message.proto

And that's it! You'll have a cool autogenerated file called proto/message/message.pb.go that contains Person and Hobby structs.

// message.pb.go
...
package message
...
type Person struct {
    FirstName    string   `protobuf:"bytes,1,opt,name=first_name,json=firstName" json:"first_name,omitempty"`
    LastName     string   `protobuf:"bytes,2,opt,name=last_name,json=lastName" json:"last_name,omitempty"`
    DateOfBirth  string   `protobuf:"bytes,3,opt,name=date_of_birth,json=dateOfBirth" json:"date_of_birth,omitempty"`
    Cool         bool     `protobuf:"varint,4,opt,name=cool" json:"cool,omitempty"`
    ArgumentsWon int32    `protobuf:"varint,5,opt,name=arguments_won,json=argumentsWon" json:"arguments_won,omitempty"`
    Hobbies      []*Hobby `protobuf:"bytes,6,rep,name=hobbies" json:"hobbies,omitempty"`
}
...

Note: the IDs in the proto files should never change. Once the structure is out in public, you can only add new fields (with subsequent ID number), but never edit them. Protocol buffers provide backward compatibility, so that the old clients will ignore any extra files added to the structure, but will fail if eg. types of the field do not match. This is important to know and remember, to take full advantage of the protocol features. If you decide that a field is no longer necessary, you can delete it from the file but never reuse the ID. This can be enforced with reserved keyword:

// old file.proto
message Person {
    string name = 1;
    int32 age = 2;
    string favourite_team = 3;
}

// new file.proto    
message Person {
    reserved 2;
    string name = 1;
    string favourite_team = 3;
    string date_of_birth = 4;
}

Working example

You can probably imagine, that using the autogenerated code is a piece of cake, right? It's just code, see:

// main.go
...
func main() {
    p := message.Person{
        FirstName:    "John",
        LastName:     "Doe",
        DateOfBirth:  "1960-10-17T0:00:00Z",
        Cool:         true,
        ArgumentsWon: 7,
        Hobbies: []*message.Hobby{
            {
                Name:        "Running",
                Description: "Occasionally, about 10km a week",
            }, {
                Name:        "Computer games",
                Description: "Flappy bird, mostly",
            },
        },
    }

    log.Printf("Person created for .proto structure: %v\n", p)
}

Running it:

$ go run main.go 
Person created for .proto structure: {John Doe 1960-10-17T0:00:00Z true 7 [name:"Running" description:"Occasionally, about 10km a week"  name:"Computer games" description:"Flappy bird, mostly" ]}

What is cool about having this generated code is that you can extend its functionalities by implementing some custom functions on those created structs. All you need to do is create another file in the same folder where message.pb.go file is (don't edit *.pb.go file, they should always be generated!):

// message/message.go
package message
import "fmt"
func (p Person) FullName() string {
    return fmt.Sprintf("%s %s", p.FirstName, p.LastName)
}

And you can use that generated Person struct and your brand new shiny helper function:

func main() {
    p := message.Person{
        FirstName:    "John",
        LastName:     "Doe",
        ...
    }
    ...
    log.Printf("Full name (custom fn): %s\n", p.FullName())
}

Cool, right?