One of the things developers like about Go is that it has strong typing, which means that the compiler takes care of making sure that if you expect a string or a number, you will get one. Unfortunately, this approach has limitations, for example, it would be difficult to parse JSON structures since they come from the outside world. This blog post shows how you can use one specific package from the standard library so that you can handle situations like those.

Recently I faced a problem where I had to validate and parse many many string to string maps and I figured out, that it would be great to treat them like JSONs and automatically convert them into some defined structs. This is how the idea of slomek/mappy was born, which I created as a PoC, allowing me to understand more what reflect package is capable of.

reflect package

Since it is a part of the standard library, reflect is available to you out of the box. The most basic things you may do with it is checking the types and structures of any Go values, which can be used for many otherwise impossible actions. You can, for example, take any instance of any struct and print its value and type using:

...
type Person struct {
    FirstName string `json:"first_name,omitempty"`
    LastName  string `json:"last_name,omitempty"`
}
...
p := Person{FirstName: "Ben", LastName: "Wallace"}

fmt.Println("value", reflect.ValueOf(p)) // -> value {Ben Wallace}
fmt.Println("value", reflect.TypeOf(p))  // -> type main.Person

You can also do that with pointers to values:

type Person struct {
    FirstName string `json:"first_name,omitempty"`
    LastName  string `json:"last_name,omitempty"`
}
...
p2 := &Person{FirstName: "Ben", LastName: "Wallace"}

fmt.Println("value", reflect.ValueOf(p2)) // -> value &{Ben Wallace}
fmt.Println("type", reflect.TypeOf(p2))   // -> type *main.Person

Next, you can check how many fields and methods does the type of the value have (you can use both its value and type to check them):

...
func (p Person) FullName() string {
    return fmt.Sprintf("%s %s", p.FirstName, p.LastName)
}
func (p *Person) ChangeName(n string) {
    p.FirstName = n
}
...
p := Person{FirstName: "Ben", LastName: "Wallace"}
...
fmt.Println("value fields count", reflect.ValueOf(p).NumField())  // -> 2
fmt.Println("value method count", reflect.ValueOf(p).NumMethod()) // -> 1
fmt.Println("value fields count", reflect.TypeOf(p).NumField())   // -> 2
fmt.Println("value method count", reflect.TypeOf(p).NumMethod())  // -> 1

Note that since ChangeName func has a pointer receiver, the type main.Person has only one method, FullName!

The same check, however, will not work on the pointer, since it does not have any fields, but you still check the method count:

p2 := &Person{FirstName: "Ben", LastName: "Wallace"}
...
fmt.Println("value fields count", reflect.ValueOf(p2).NumField())  // -> panic
fmt.Println("value method count", reflect.ValueOf(p2).NumMethod()) // -> 2
fmt.Println("value fields count", reflect.TypeOf(p2).NumField())   // -> panic
fmt.Println("value method count", reflect.TypeOf(p2).NumMethod())  // -> 2

What is interesting here is that the pointer has access to both methods, even though only one has a pointer receiver.

This is not all since we would like to get some details of those fields and methods. For the simple value, we can iterate over fields and see their types, values and, yes, struct tags:

type Person struct {
    FirstName string `json:"first_name,omitempty"`
    LastName  string `json:"last_name,omitempty"`
}
...
p := Person{FirstName: "Ben", LastName: "Wallace"}
...
typeP := reflect.TypeOf(p)
for i := 0; i < typeP.NumField(); i++ {
    fmt.Printf("field #%d: %v -> %v (tags: %v)\n", i, typeP.Field(i).Name, valP.Field(i), typeP.Field(i).Tag)
}
// -> field #0: FirstName -> Ben (tags: json:"first_name,omitempty")
// -> field #1: LastName -> Wallace (tags: json:"last_name,omitempty")

We can do the same with methods, but accessing them via ValueOf(p) gives us the possibility to call it with some arguments (which we don't need now), so we'll stick to browsing the types:

for i := 0; i < valP.NumMethod(); i++ {
    fmt.Printf("method %d: %v -> %v\n", i, typeP.Method(i).Name, typeP.Method(i).Type)
}
// -> method 0: FullName -> func(main.Person) string

Again, we can to that with pointer as well:

p2 := &Person{FirstName: "Ben", LastName: "Wallace"}
...
typeP2 := reflect.TypeOf(p2)
for i := 0; i < typeP2.NumMethod(); i++ {
    fmt.Printf("method %d: %v -> %v\n", i, typeP2.Method(i).Name, typeP2.Method(i).Type)
}
// -> method 0: ChangeName -> func(*main.Person, string)
// -> method 1: FullName -> func(*main.Person) string

We can even go a step forward and check (access) what are the input and output types of each method:

p2 := &Person{FirstName: "Ben", LastName: "Wallace"}
...
typeP2 := reflect.TypeOf(p2)
for i := 0; i < typeP2.NumMethod(); i++ {
    method := typeP2.Method(i)
    fmt.Printf("method %d: %v -> %v\n", i, method.Name, method.Type)

    fmt.Printf(" INPUT: %d parameters\n", method.Type.NumIn())
    for j := 0; j < method.Type.NumIn(); j++ {
        fmt.Printf("  - %d -> %v\n", j, method.Type.In(j).Name())
    }

    fmt.Printf(" OUTPUT: %d parameters\n", method.Type.NumOut())
    for j := 0; j < method.Type.NumOut(); j++ {
        fmt.Printf("  - %d -> %v\n", j, method.Type.Out(j).Name())
    }
}
// -> method 0: ChangeName -> func(*main.Person, string)
// ->  INPUT: 2 parameters
// ->   - 0 ->
// ->   - 1 -> string
// ->  OUTPUT: 0 parameters
// -> method 1: FullName -> func(*main.Person) string
// ->  INPUT: 1 parameters
// ->   - 0 ->
// ->  OUTPUT: 1 parameters
// ->   - 0 -> string

Another interesting observation here is that even though both methods take the type on which they are called as their respective first parameter, we don't see their type names as the input parameter at index 0.

Last, but not least, is that using reflect we can modify the contents of the thing we are looking into. Actually, we cannot do that with the value itself (that's why we need pointer receivers on functions that modify the state):

valP := reflect.ValueOf(p)
typeP := reflect.TypeOf(p)
for i := 0; i < valP.NumField(); i++ {
    field := valP.Field(i)
    fieldType := typeP.Field(i)

    fmt.Printf("Field %s, can set? %t\n", fieldType.Name, field.CanSet())
    // -> Field FirstName, can set? false
    // -> Field LastName, can set? false

    // field.SetString(strings.ToUpper(field.String())) // -> panic
}

We can, however do that on the pointer to the value. But how do we do that, since we cannot browse through the fields of the pointer? We first need to access the element of the pointer, then proceed just like above (this time without panics):

p2 := &Person{FirstName: "Ben", LastName: "Wallace"}
...
el := reflect.ValueOf(p2).Elem()
typeEl := el.Type()
for i := 0; i < typeEl.NumField(); i++ {
    field := el.Field(i)
    fieldType := typeEl.Field(i)

    fmt.Printf("Field %s, can set? %t\n", fieldType.Name, field.CanSet())
    // -> Field FirstName, can set? true
    // -> Field LastName, can set? true

    field.SetString(strings.ToUpper(field.String())) // -> panic
}

fmt.Println(p2)
// -> &{BEN WALLACE}

Since on the level of reflect package we cannot use the strong typing that Go provides and we need to defer to simpler types, so that in order to set the value using some predefined functions such as SetInt, SetBool, or like in this case, SetString.

Marshaling map into struct

On top of all that was described above, we can access a struct tag by its name via fieldType.Tag.Get(NAME):

// github.com/slomek/mappy/marshal.go
func Marshal(data interface{}) (m map[string]string, err error) {
    ...
    m = make(map[string]string)
    rv := reflect.ValueOf(data)
    elT := reflect.TypeOf(data)
    for i := 0; i < rv.NumField(); i++ {
        field := rv.Field(i)
        ftype := elT.Field(i)

        tag := ftype.Tag.Get("map")
        if tag != "" {
            m[tag] = field.String()
        }
    }
    return m, nil
}

Unmarshaling struct into map

// github.com/slomek/mappy/unmarshal.go
func Unmarshal(m map[string]string, data interface{}) (err error) {
    ...
    rv := reflect.ValueOf(data)
    el := rv.Elem()
    elT := el.Type()
    for i := 0; i < el.NumField(); i++ {
        field := el.Field(i)
        ftype := elT.Field(i)

        tag := ftype.Tag.Get("map")
        val := m[tag]

        field.SetString(val)
    }

    return
}

Error handling trick

As you could have seen before, using reflect package can cause some unexpected panics (as they can come as a result of incorrect input parameters). In order to gain some control over those, we need to recover from those panics (easy part) and return some user-friendly error as a result (trickier part). Since anything we want to call in defer needs to be wrapped into a function, returning anything from there is not an option.

There is, however, a trick we can use here. We cannot return anything, but we can change values using closures, so all we need to do is use named returned parameters, and alter their values in the defer:

// Unmarshal transforms a string->string map into a custom struct.
func Unmarshal(m map[string]string, data interface{}) (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = errors.Wrapf(ErrMapUnmarshal, "%v", r)
        }
    }()
    ...
    return
}

Now our code returns our custom error, ErrMapUnmarshal instead of the terrible panics.

Source code

The source code of the library is available on Github: slomek/mappy.