goderive

Pros and Cons of Code Generation

Code generation definitely has its pros:

• Automated (less work for me)

• Safer (less chance for inconsistencies)

• Faster (compared to reflect)

Someone on the team always complains about adopting a new tool. In the case of code generation, I have heard them all:

• One more thing to learn

• Unreadable (hard to debug and reason about generated code)

• Too much code (slows down compilation)

• Slow to generate

• Not formatted

goderive has some design principles that try to address some of these concerns.

Design Principles

For goderive, I decided to have some code generation principles to address some of the negatives typically associated with code generation.

Generated code should be:

• Fast to generate

• Avoid duplication

• Readable

• Formatted

Goderive tries its best to generate readable code and be as close to the code you would have written yourself. It even makes sure that this generated code is already formatted. This seems unnecessary, but not needing to run gofmt over your generated code is a huge time saver. Formatting code can be one of the slowest things in your dev cycle outside of running tests. It is also important the code is generated quickly and that we try to reuse as much of the generated parts to keep compilation as fast as possible.

Demos

Enough ceremony, Let’s see it in action.

package main

import "fmt"

func main() {
  x := 19
  y := 39
  m := Min(x, y)
  fmt.Printf("%v\n", m)
}

// Code generated by goderive DO NOT EDIT.

package main

// Min returns the minimum of the two input values.
func Min(a, b int) int {
  if a < b {
    return a
  }
  return b
}

package main

import "fmt"

type Person struct {
  name string
  age  int
}

func main() {
  x := &Person{"Donna", 19}
  y := &Person{"Ron", 39}
  fmt.Printf("%v\n", Min(x, y))
}

// Code generated by goderive DO NOT EDIT.

package main

import "strings"

// Min returns the minimum of the two input values.
func Min(a, b *Person) *Person {
  if Compare(a, b) < 0 {
    return a
  }
  return b
}

// Compare returns:
//   * 0 if this and that are equal,
//   * -1 if this is smaller and
//   * +1 if this is bigger.
func Compare(this, that *Person) int {
  if this == nil {
    if that == nil {
      return 0
    }
    return -1
  }
  if that == nil {
    return 1
  }
  if c := strings.Compare(this.name, that.name); c != 0 {
    return c
  }
  ...
  return 0
}

type Person struct {
  name string
  age  int
}

func main() {
  x := Person{"Donna", 19}
  y := Person{"Ron", 39}
  people := []Person{y, x}
  fmt.Printf("%v\n", Sort(people))
}

// Code generated by goderive DO NOT EDIT.

package main

import (
  "sort"
  "strings"
)

// Sort sorts the slice inplace and also returns it.
func Sort(list []Person) []Person {
  sort.Slice(list, func(i, j int) bool { return Compare(list[i], list[j]) < 0 })
  return list
}
...

type Person struct {
  name string
  age  int
}

func main() {
  x := Person{"Donna", 19}
  y := Person{"Ron", 39}
  people := map[string]Person{x.name: x, y.name: y}
  for _, name := range Sort(Keys(people)) {
    fmt.Printf("%v\n", people[name])
  }
}

// Code generated by goderive DO NOT EDIT.

package main

import (
  "sort"
)

// Sort sorts the slice inplace and also returns it.
func Sort(list []string) []string {
  sort.Strings(list)
  return list
}

// Keys returns the keys of the input map as a slice.
func Keys(m map[string]Person) []string {
  keys := make([]string, 0, len(m))
  for key := range m {
    keys = append(keys, key)
  }
  return keys
}

type Person struct {
  name string
  age  int
}

func main() {
  x := Person{"Donna", 19}
  y := Person{"Ron", 39}
  people := map[string]Person{x.name: x, y.name: y}
  for _, name := range deriveSort(deriveKeys(people)) {
   fmt.Printf("%v\n", people[name])
  }
}

var re = deriveMem(func(r string) *regexp.Regexp {
  fmt.Printf("compiling regex <%s>\n", r)
  return regexp.MustCompile(r)
})

func main() {
  fmt.Printf("%v\n", re("ab.*").MatchString("abc"))
  // compiling regex <"ab.*">
  // true
  fmt.Printf("%v\n", re("cd.*").MatchString("cde"))
  // compiling regex <"cd.*">
  // true
  fmt.Printf("%v\n", re("ab.*").MatchString("cde"))
  // false
}

func deriveMem(f func(r string) *regexp.Regexp) func(r string) *regexp.Regexp {
  m := make(map[string]*regexp.Regexp)
  return func(param0 string) *regexp.Regexp {
    if v, ok := m[param0]; ok {
      return v
    }
    v := f(param0)
    m[param0] = v
    return v
  }
}

package main

var fib func(uint) uint

func init() {
  fib = deriveMem(func(i uint) uint {
    if i == 0 || i == 1 {
      return i
    }
    return fib(i-1) + fib(i-2)
  })
}

func main() {
  println(fib(1))
  println(fib(5))
  println(fib(64))
  // yes it really works.
  println(fib(1000))
}

// expensive function with two parameters.
func findLastName(firstname string, age *int) *string {
  // Mock: Search for a possible last name via an API
  ...
}

var getLastName = deriveMem(findLastName)

func main() {
  age := 2
  fmt.Printf("%v\n", getLastName("Donna", nil))
  fmt.Printf("%v\n", *getLastName("Ron", &age))
  fmt.Printf("%v\n", getLastName("Donna", nil))
}

// Code generated by goderive DO NOT EDIT.

package main

// deriveMem returns a memoized version of the input function.
func deriveMem(f func(firstname string, age *int) *string) func(firstname string, age *int) *string {
  type input struct {
    Param0 string
    Param1 *int
  }
  type mem struct {
    in  input
    out *string
  }
  m := make(map[uint64][]mem)
  return func(param0 string, param1 *int) *string {
    in := input{param0, param1}
    h := deriveHash(in)
    vs, ok := m[h]
    if ok {
      for _, v := range vs {
        if deriveEqual(v.in, in) {
          return v.out
        }
      }
    }
    res0 := f(param0, param1)
    m[h] = append(m[h], mem{in, res0})
    return res0
  }
}
...

Origins

Dean Learner from Garth Marenghi's Darkplace

I got this idea for goderive from Haskell, where you can generate an Equal, Compare, and String method by simply adding the statement deriving below the struct.

data Person = Person {
  name :: String,
  age :: Int
} deriving (Eq, Ord, Show)

I wanted it to be this easy to introduce these methods for code generation to Go, but I didn’t want to introduce extra syntax or use comments.

func (p *Person) Equal(q *Person) bool {
  return deriveEqual(p, q)
}

On a side note: What do you think this double colon is called that is used before every type field in Haskell? Since the double underscore in Python is called dunderscore, I think the double colon should be called dolon.

Anyway, I learned many other things from Haskell that I would have liked to use in Go.

Error Detention

For example, error handling!

In Haskell, I didn’t need to type if err != nil, but I still got the explicit error checks that I wanted. Let me show you an approximation in Go.

Here we can see a piece of code that translates a user type that is stored on another service. It calls the endpoint to get the user, unmarshals the user, upgrades the user, marshals the user again, and posts it to a new endpoint.

func upgradeUser(endpoint, username string) error {
    getEndpoint := fmt.Sprintf("%s/oldusers/%s", endpoint, username)
    postEndpoint := fmt.Sprintf("%s/newusers/%s", endpoint, username)

    resp, err := http.Get(genEndpoint)
    if err != nil {
        return err
    }
    data, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return err
    }
    olduser, err := user.NewFromJson(data)
    if err != nil {
        return err
    }
    newuser, err := user.NewUserFromUser(olduser),
    if err != nil {
        return err
    }
    buf, err := json.Marshal(newuser)
    if err != nil {
        return err
    }
    _, err = http.Post(
        postEndpoint,
        "application/json",
        bytes.NewBuffer(buf),
    )
    return err
}

We could also write the same code with goderive using a compose function.

func upgradeUser(endpoint, username string) error {
  getEndpoint := fmt.Sprintf("%s/oldusers/%s", endpoint, username)
  postEndpoint := fmt.Sprintf("%s/newusers/%s", endpoint, username)
  _, err := deriveCompose(
    http.Get,
    func(resp *http.Response) ([]byte, error) {
      return ioutil.ReadAll(resp.Body)
    },
    user.NewFromJson,
    user.NewUserFromUser,
    json.Marshal,
    func(buf []byte) (*http.Response, error) {
      return http.Post(
        postEndpoint,
        "application/json",
        bytes.NewBuffer(buf),
      )
    },
  )(getEndpoint)
  return err
}

The compose function takes two chain-able functions that return an error and chains them together into one callable function.

func compose(
  f func(*A) (*B, error),
  g func(*B) (*C, error),
) func(*A) (*C, error) {
  return func(a *A) (*C, error) {
    b, err := f(a)
    if err != nil {
      return nil, err
    }
    c, err := g(b)
    return c, err
  }
}

This is simple enough to now do with generics, but goderive is smarter and can compose as many functions as you would like:

func compose(
  a func(*A) (*B, error),
  b func(*B) (*C, error),
  ...
  y func(*Y) (*Z, error),
) func(*A) (*Z, error) {

Haskell has special syntax for this, so you don’t have to call compose, but this is the basic idea behind its error handling. Rust also gets around this problem using a question mark operator.

Do Concurrently

What if we wanted to execute two functions concurrently and return a possible error? This is exactly what the deriveDo function does:

func serviceCallOne() (string, error) {
  // some function that calls a service
}

func serviceCallTwo() (int, error) {
  // some function that calls a service
}

func serviceCalls() (string, int, error) {
 return deriveDo(serviceCallOne, serviceCallTwo)
}

The function starts up two Go routines and waits for their completion on an error channel. Once both are finished, it returns the values and the error. Again, deriveDo can take as many functions as input as you want to execute concurrently; it doesn’t just have to be two.

// deriveDo concurrently executes the input functions f0 and f1 and when all functions are finished the first error, if any, and results are returned.
func deriveDo(f0 func() (string, error), f1 func() (int, error)) (string, int, error) {
  errChan := make(chan error)
  var v0 string
  go func() {
    var v0err error
    v0, v0err = f0()
    errChan <- v0err
  }()
  var v1 int
  go func() {
    var v1err error
    v1, v1err = f1()
    errChan <- v1err
  }()
  var err error
  for i := 0; i < 2; i++ {
    errc := <-errChan
    if errc != nil {
      if err == nil {
        err = errc
      }
    }
  }
  return v0, v1, err
}

This function was also inspired by Haskell’s ApplicativeDo extension. I learned a lot more things about concurrency from Haskell, which are more complicated than this simple function, for example, proper cancellation of green threads. This is also something Erlang does really well, but we struggle with in Go.

Monads in Go

One of the hardest things to understand in Haskell is monads. The actual reason that motivated me to build this project was to explain monads to myself. Haskell has generics, but it also has a higher level of generics. When we think about generics, we only think about replacing the types inside our containers, for example, lists and channels with variables, like A or B. At a higher level, we can also replace the containers themselves with a variable, say M. For example, channels and slices are kind of the same, in that you can loop over both of them and could write generic code that could handle either. Each of the types M just need to implement some interface. Another example is the Kleisli arrow, which comes from Category Theory, but you can just think of it as the compose function.

type kleisliArrow = func(func(A) M<B>, func(B) M<C>) func(A) M<C>
type compose = func(func(A) (B, error), func(B) (C, error)) func(A) (C, error)
type pipeline = func(func(A) <-chan B , func(B) <-chan C) func(A) <-chan C
type listComprehension = func(func(A) []B, func(B) []C) func(A) []C

Category Theory studies how we compose things, which is quite important, since one of the few or maybe only ways we know how to solve problems is to break them up into smaller pieces and compose them back together again. In this case, the Kleisli arrow is an interface that can be implemented by errors, channels, and slices, to effectively abstract error handling, a concurrent pipeline, and a list comprehension. You can read more about this in another post: monads for Go programmers.

Monads for Go programmers or Gonads is what goderive allows you to generate.

So Much More

A gopher’s gonads

There is so much more I didn’t cover, but here it is in short.

There are various options for how to handle duplicate function names, which the command line flags help you to set:

--autoname: rename functions that are conflicting with other functions
--dedup: rename functions to functions that are duplicates
--pluginprefix: used to override function prefixes.  The input is a comma separated list of function and prefix pairs.  For example equal=deriveEqual,copyto=copyTo,fmap=fmap
--prefix: prefix of all functions (default "derive")

There are also a lot more functions to explore, such as the following:

Recursive Functions:

• Equal: deriveEqual(T, T) bool

• Compare: deriveCompare(T, T) int

• DeepCopy: deriveDeepCopy(dst *T, src *T)

• Clone: deriveClone(T) T

• GoString: deriveGoString(T) string

• Hash: deriveHash(T) uint64

Set Functions:

• Keys: deriveKeys(map[K]V) []K

• Sort: deriveSort([]T) []T

• Unique: deriveUnique([]T) []T

• Set: deriveSet([]T) map[T]struct{}

• Min: deriveMin(list []T, default T) (min T)

• Max: deriveMax(T, T) T

• Contains: deriveContains([]T, T) bool

• Intersect: deriveIntersect(a, b []T) []T

• Union: deriveUnion(a, b map[T]struct{}) map[T]struct{}

Fmap:

• deriveFmap(func(A) B, []A) []B

• deriveFmap(func(rune) B, string) []B

• deriveFmap(func(A) B, func() (A, error)) (B, error)

• deriveFmap(func(A) (B, error), func() (A, error)) (func() (B, error), error)

• deriveFmap(func(A), func() (A, error)) error

• deriveFmap(func(A) (B, c, d, ...), func() (A, error)) (func() (B, c, d, ...), error)

Join:

• deriveJoin([][]T) []T

• deriveJoin([]string) string

• deriveJoin(func() (T, error), error) func() (T, error)

• deriveJoin(func() (T, ..., error), error) func() (T, ..., error)

More functional functions:

• Filter: deriveFilter(pred func(T) bool, []T) []T

• All: deriveAll(pred func(T) bool, []T) bool

• Any: deriveAny(pred func(T) bool, []T) bool

• TakeWhile: deriveTakeWhile(pred func(T) bool, []T) []T

• Flip: deriveFlip(f func(A, B, ...) T) func(B, A, ...) T

• Curry: deriveCurry(f func(A, B, ...) T) func(A) func(B, ...) T

• Uncurry: deriveUncurry(f func(A) func(B, ...) T) func(A, B, ...) T

• Tuple: deriveTuple(A, B, ...) func() (A, B, ...)

• Compose

• Mem: deriveMem(func(A...) (B...)) func(A...) (B...)

• Traverse: deriveTraverse(func(A) (B, error), []A) ([]B, error)

• ToError: deriveToError(error, func(A...) (B..., bool)) func(A...) (B..., error)

• Apply: deriveApply(f func(...A, B) C, B) func(...A) C

Concurrency functions:

• Fmap: deriveFmap(func(A) B, <-chan A) <-chan B

• Join: deriveJoin(chan T, chan T, ...) <-chan T

• Pipeline: derivePipeline(func(A) <-chan B, func(B) <-chan C) func(A) <-chan C

• Do: deriveDo(func() (A, error), func() (B, error)) (A, B, error)

• Dup: deriveDup(c <-chan T) (c1, c2 <-chan T)

The tool is pretty extendable if you would like to add your own functions or even remove some you wouldn’t like to see being applied in your code base.

Thank you

• Marius van Zyl for having the idea and pushing me to do talk on goderive.

• Neil Garb for organising, recording and supporting the talk at the Go Cape Town meetup.