Guava-like EventBus for Go

I needed a simple solution to notify components about changes in some global settings.
In my Java days, I have been using the Guava Eventbus to solve similar problems, so I decided to cook something that does the basics.

From Guava: “The EventBus allows publish-subscribe-style communication between components requiring the components to explicitly register with one another (and thus be aware of each other).”

The Java implementation uses annotations to register components and subscribe its methods to event types. In my Go version, with lack of such annotations, I decided to keep it simpler; just subscribe a function to an event by name. The singleton EventBus value will be an exposed package variable.

Example

	bus := NewEventBus()
	bus.Subscribe("test", func(e Event) {
		fmt.Printf("%#v\n", e)
	})
	bus.Post("test")
	bus.Post("test", EventData{"pi": 3.14159})

The code

package lang

// EventData represents the context of an event, a simple map.
type EventData map[string]interface{}

// Event is send on the bus to all subscribed listeners
type Event struct {
	Name string
	Data EventData
}

// EventListener is the signature of functions that can handle an Event.
type EventListener func(Event)

// The EventBus allows publish-subscribe-style communication between components
// without requiring the components to explicitly register with one another (and thus be aware of each other)
// Inspired by Guava EventBus ; this is a more lightweight implementation.
type EventBus struct {
	listeners map[string][]EventListener
}

func NewEventBus() EventBus {
	return EventBus{map[string][]EventListener{}}
}

// Subscribe adds an EventListener to be called when an event is posted.
func (e EventBus) Subscribe(name string, listener EventListener) {
	list, ok := e.listeners[name]
	if !ok {
		list = []EventListener{}
	}
	list = append(list, listener)
	e.listeners[name] = list
}

// Post sends an event to all subscribed listeners.
// Parameter data is optional ; Post can only have one map parameter.
func (e EventBus) Post(name string, data ...map[string]interface{}) {
	list, ok := e.listeners[name]
	if !ok {
		return
	}
	event := Event{Name: name}
	if len(data) == 1 {
		event.Data = EventData(data[0])
	}
	for _, each := range list[:] { // iterate over unmodifyable copy
		each(event)
	}
}

Smalltalk collect on Go slice of int

package main

import "fmt"

type intSlice []int

func (i intSlice) collect(block func(i int) int) intSlice {
	r := make(intSlice, len(i))
	for j, v := range i {
		r[j] = block(v)
	}
	return r
}

func main() {
	numbers := intSlice{1, 2, 3}
	squared := func(i int) int { return i * i }

	fmt.Printf("%v", numbers.collect(squared))
}

Try it.

Implementations of select,inject,detect are left as an excercise for the reader.

Javascript CLI in Go using Otto

Today, I was playing with Otto and hacked together a small command-line-interface program in Go.

package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"

	"github.com/robertkrimen/otto"
)

var Otto = otto.New()

func main() {
	fmt.Println("otto\n")
	loop()
}

func dispatch(entry string) string {
	if len(entry) == 0 {
		return entry
	}
	value, err := Otto.Run(entry)
	if err != nil {
		return err.Error()
	} else {
		return fmt.Sprintf("%v", value)
	}
}

func loop() {
	for {
		fmt.Print("> ")
		in := bufio.NewReader(os.Stdin)
		entered, err := in.ReadString('\n')
		if err != nil {
			fmt.Println(err)
			break
		}
		entry := strings.TrimLeft(entered[:len(entered)-1], "\t ") // without tabs,spaces and newline
		output := dispatch(entry)
		if len(output) > 0 {
			fmt.Println(output)
		}
	}
}

One cool feature of Otto is the ability to expose your own Go functions in the Javascript namespace.
This example below adds the function log to print its arguments using the standard log package.

	Otto.Set("log", func(call otto.FunctionCall) otto.Value {
		log.Printf("%v\n",call.ArgumentList())
		return toValue("")
	})

With this setup, I can call this function on the command line like this

> log("hi")
> hi

From multiple to single-value context in Go

These are the results of exploring one aspect of the Go language; functions can return multiple values.
Consider the following simple function:

func ab() (a,b int) {
	a,b = 1,2
	return
}

Clearly, this function cannot be used in other function calls that expect a single value.

fmt.Printf("%v\n",ab())

The compiler will help you: “multiple-value ab() in single-value context”.

Now suppose you want to take only the first value without introducing an intermediate variable.
One possible solution might be to create this helper function:

func first(args ...interface{})interface{} {
	return args[0]
}

Such that you can write:

fmt.Printf("%v\n",first(ab()))

But what if you want to generalize this such that you won’t end up with functions like second(),third(),…
To achieve this, I tried the following:

func pick(index int, args ...interface{})interface{} {
	return args[index]
}
fmt.Printf("%v\n",pick(1,ab()))

However, again the compiler prompts with the same error message: “multiple-value ab() in single-value context”.
It is my interpretation that the compiler (in this case) cannot implicitly convert the multiple returns into a slice because of the extra function parameter.

So instead I created this slice converter function:

func slice(args ...interface{}) []interface{} {
	return args
}

And with that, you can write:

fmt.Printf("%v\n",slice(ab())[1])

A case of sizing and draining buffered Go channels

Update: Based on several comments posted on the G+ Go community, I rewrote the solution presented here. See past the Conclusion section.

Recently, I was asked to write a small program for collecting product promotion information for a large collection of products. Given an input file of product identifiers, the following tasks had to be done:

1. read all identifiers from an input file
2. for each identifier, fetch product offer information in XML
3. parse the offer XML into in a DOM document
4. extract all promotions from the offer document
5. for each promotion, export its data into a CSV record

A straightforward implementation is to write a function for each task and call them in a nested loop. For this program, I created functions such that the basic flow is quite simple.

xml := fetchSellingOfferXml(id)
doc := parseStringXml(xml)
promos := extractPromotions(doc)
exportPromotion(promo)

Next, I created the loops (reading identifiers and iterating the promotions) and the program was completed.

However, as predicted, some of these tasks took relative more time to execute. For getting product offer information, a Http request was needed to access a remote WebService.
The next task, parsing the response in XML representing an offer with promotion information, is also taking some time. Finally, given the requirement of processing 50.000 identifiers, I decided to use Go concurrency here.

Concurreny to the rescue

Again, a straightforward implementation of concurreny in Go is to use goroutines and channels. Channels will be used to communicate input and output data between functions which are called inside goroutines.

chaining functions with channels

Because one input (an identifier) can result in zero (no offer) or more promotions and the observation that some functions take more time then others, I decided to use buffered channels. To balance the time between waiting for fetching an offer XML and processing it, I expected to use multiple goroutines for fetching.

The following channels are initialized:

id_channel := make(chan string, CHAN_CAP)
xml_channel := make(chan string, CHAN_CAP)
offer_channel := make(chan *dom.Document, CHAN_CAP)
promotion_channel := make(chan *promotion, CHAN_CAP)

Next, I created new functions to wrap each task function into a loop. For example, the fetchXmlLoop will block read on id-channel. After receiving an identifier, it will call fetchSellingOfferXml. Unless the result is empty (no offer available) the result is send on the xml-channel. Now, my program can setup the chain of channels by spawning goroutines for each loop.

go fetchXmlLoop(id_channel, xml_channel)
go parseXmlLoop(xml_channel, offer_channel)
go extractPromotionsLoop(offer_channel, promotion_channel)
go writePromotionsLoop(promotion_channel, quit_channel)

Real challenges

Up to this point the story is pretty straightforward and easy to understand without getting into details. Now I will discuss the real challenges for this program because it is not complete.

Draining

At one point in time, the program has to terminate gracefully. All identifiers have to be processed and as a result all available promotions have to be exported. Because the program uses multiple buffered channels, the program can only terminate if all channels are drained ; no data can be left unconsumed.

The simple method of counting cannot be used here; for a given identifier there might not be offer information and if such offer information is available then there can be zero or more promotions available.
Another method would be polling the length of each channel. Again this method cannot be used here as detecting an empty xml-channel does not imply there is nothing more to receive in the future.

Fortunately, I found a solution which involves the use of a termination value for each channel type. This termination value is sent through the chain of channels “notifying” each goroutine loop to discontinue. After sending a termination value on the next channel it can stop.

For this program, after reading all product identifiers from the file, the terminating identifier (“eof”) is sent on the id-channel. The fetchXmlLoop will receive this value and sends the termination Xml string (“eof”) on the xml-channel. For each channel datatype, a termination value is defined.

no_more_ids = "eof"
no_more_xml = "eof"
no_more_offers = new(dom.Document)
no_more_promotions = new(promotion)

The fetchXmlLoop tests each received identifier against the termination value and acts accordingly.

	func fetchXmlLoop(id_channel chan string, xml_channel chan string) {
		for {
			id := <-id_channel
			if no_more_ids == id {
				xml_channel <- no_more_xml
				break
			} else {
				xml := fetchSellingOfferXml(id)
				if xml != "" { // happens if no offer is found
					xml_channel <- xml
				}
			}
		}
	}

This pattern has been applied to all loop functions as listed above. There is one caveat though. The thread (or goroutine) that runs the main function cannot exit until the termination value has been received by the last channel in the chain. To synchronize this, I introduced the non-buffered quit-channel. After reading all identifiers from the input file, a boolean value is blocked sent on the quit-channel. When the termination value of the promotion-channel (in the writePromotionsLoop) is received, it takes the boolean value from the quit-channel. Now, the program is terminated after all work is done.

	// start reading and feeding the chain
	scanner := bufio.NewScanner(fileIn)
	for scanner.Scan() {
		id_channel <- scanner.Text()
		observer.Observe()
	}
	// send terminating identifier
	id_channel <- no_more_ids

	// and wait for the last channel to be drained
	quit_channel <- true

Sizing

The concurrent program now uses multiple buffered channels and can create multiple goroutines each executing a loop. In general, each channel must be given a capacity and for each loop I need to decide how many goroutines could run in parallel. Even for this simple program, there are potentially 4 + 4 variables to size.

A practical way to find these values, such that the program runs as fast as possible, is to measure the use of each channel. Setting the channel capacity too low, then the sending goroutines will have to wait. Creating too many goroutines for one loop function, then the channel may not provide information fast enough to process.

For this purpose, I created a simple ChannelsObserver object. It runs a goroutine that periodically inspects the length of all registered channels and logs basic statistical (min,man,average,..) information. To compute the statistics, I am using the go-metrics package. Because in Go it is not possible to store different typed channels as values in a single map, I defined the ChannelLengthReporter type.

type ChannelLengthReporter func() int

Now, each channel can be registered on the ChannelsObserver using a closure.

observer := ChannelsObserver{}
observer.Register("id", func() int { return len(id_channel) })
observer.Register("xml", func() int { return len(xml_channel) })
observer.Register("offer", func() int { return len(offer_channel) })
observer.Register("promotion", func() int { return len(promotion_channel) })
timeBetweenLogEntry, _ := time.ParseDuration("5s")
observer.Run(timeBetweenLogEntry)

In the loop for reading identifiers from the input file, I added the observer.Observe() call to measure the current status of all channels.
The obsever will log the metrics of all its channels every 5 seconds.

Channel Length Observer

	type ChannelsObserver struct {
		reporters  map[string]ChannelLengthReporter
		histograms map[string]metrics.Histogram
	}

	func (c *ChannelsObserver) Register(name string, reporter ChannelLengthReporter) {
		if c.reporters == nil {
			// lazy init
			c.reporters = make(map[string]ChannelLengthReporter)
			c.histograms = make(map[string]metrics.Histogram)
		}
		c.reporters[name] = reporter
		sample := metrics.NewUniformSample(100)
		c.histograms[name] = metrics.NewHistogram(sample)
		metrics.Register(name, c.histograms[name])
	}

	func (c ChannelsObserver) Observe() {
		for name, reporter := range c.reporters {
			c.histograms[name].Update(int64(reporter()))
		}
	}

	func (c ChannelsObserver) Run(logEvery time.Duration) {
		go metrics.Log(metrics.DefaultRegistry, logEvery, log.New(os.Stderr, "metrics: ", log.Lmicroseconds))
	}

Conclusion

Using goroutines and channels, two of the Go concurrency features, I was able to apply concurrency to this program. Because the program uses buffered channels to decouple from time-varying processing tasks, correctly draining the channels was solved by passing “termination” data. Finally, to size the capacity of each channel and to decide how many goroutines per processing tasks is needed, measurement of channel usage is a valid method.

Update from feedback

First, given the solution described earlier and its measurement results, I observed that only the fetching process is taking significantly more time that any other processing. This is simply due to the network latency involved. Consequently, all but one channel remained empty. The id_channel was always saturated. Using buffering for the channels is not a right solution here.

Second, studying the comments added by several Gophers on the Go Community post, I decided to re-implemented the solution using a pattern suggested and prototyped by Bryan Mills. Basically his advice is “You don’t need buffers for any of these: the goroutines themselves form all the buffer you need”. Using the standard WaitGroup synchronization types, this solution spawns goroutines for the different stages (fetch,parse,export) and make them terminate and close the channels gracefully. I put in extra logging to view the actual flow ; see below the code for an example output.

As can be seen from the code, no buffered channels are used. Each stage is implemented using the same pattern for which a WaitGroup is created, goroutines are created and the cleanup is done using a deferred function call. All channels are consumed using the range operator.

	// spawn 1 goroutine to export promotions
	var promoWG sync.WaitGroup
	defer func() {
		glog.V(1).Info("waiting for the promo group")
		promoWG.Wait()
	}()
	promos := make(chan *promotion)
	promoWG.Add(1)
	go func() {
		defer func() {
			glog.V(1).Infof("promo writer done")
			promoWG.Done()
		}()
		for promo := range promos {
			writePromotion(promo)
		}
	}()

	// waitgroup to synchronize finished goroutines for parsing
	var parseWG sync.WaitGroup
	defer func() {
		glog.V(1).Info("waiting for the parse group")
		parseWG.Wait()
		close(promos)
	}()

	// spawn NumCPU goroutines to process the XML documents
	xml := make(chan string)
	for i := 0; i < runtime.NumCPU(); i++ {
		parseWG.Add(1)
		go func(parser int) {
			defer func(parser int) {
				glog.V(1).Infof("parser %d done", parser)
				parseWG.Done()
			}(parser)
			for each := range xml {
				doc, err := dom.ParseStringXml(each)
				if err == nil {
					extractPromotions(doc, promos)
				} else {
					// log it
				}
			}
		}(i)
	}

	// waitgroup to synchronize finished goroutines for fetching
	var fetchWG sync.WaitGroup
	defer func() {
		glog.V(1).Info("waiting for the fetch group")
		fetchWG.Wait()
		close(xml)
	}()

	// spawn FETCHER_CAP goroutines to process each read identifier
	nFetchers := 0
	scanner := bufio.NewScanner(fileIn)
	id_channel := make(chan string)
	for scanner.Scan() {
		if nFetchers < FETCHER_CAP {
			nFetchers++
			fetchWG.Add(1)
			go func() {
				defer func(fetcher int) {
					glog.V(1).Infof("fetcher %d done", fetcher)
					fetchWG.Done()
				}(nFetchers)
				for id := range id_channel {
					xml <- fetchSellingOfferXml(id)
				}
			}()
		}
		id_channel <- scanner.Text()
	}
	close(id_channel)
	glog.V(1).Info("done scanning")
I1015 11:09:38.632012 06942 getpromos_waitgroup.go:190] fetchSellingOffer: http://offer.services.xxxxx/slo/products/9200000019409664/offer
I1015 11:09:38.632309 06942 getpromos_waitgroup.go:190] fetchSellingOffer: http://offer.services.xxxxx/slo/products/9000000011756074/offer
I1015 11:09:38.632505 06942 getpromos_waitgroup.go:190] fetchSellingOffer: http://offer.services.xxxxx/slo/products/9200000011859583/offer
I1015 11:09:38.632543 06942 getpromos_waitgroup.go:123] done scanning
I1015 11:09:38.632551 06942 getpromos_waitgroup.go:97] waiting for the fetch group
I1015 11:09:38.694256 06942 getpromos_waitgroup.go:112] fetcher 1 done
I1015 11:09:38.694920 06942 getpromos_waitgroup.go:158] extractPromotions
I1015 11:09:38.694950 06942 getpromos_waitgroup.go:140] writePromotion
I1015 11:09:38.700120 06942 getpromos_waitgroup.go:112] fetcher 2 done
I1015 11:09:38.701317 06942 getpromos_waitgroup.go:158] extractPromotions
I1015 11:09:38.701367 06942 getpromos_waitgroup.go:140] writePromotion
I1015 11:09:38.701429 06942 getpromos_waitgroup.go:140] writePromotion
I1015 11:09:38.701535 06942 getpromos_waitgroup.go:112] fetcher 3 done
I1015 11:09:38.702002 06942 getpromos_waitgroup.go:158] extractPromotions
I1015 11:09:38.702019 06942 getpromos_waitgroup.go:69] waiting for the parse group
I1015 11:09:38.702029 06942 getpromos_waitgroup.go:140] writePromotion
I1015 11:09:38.702092 06942 getpromos_waitgroup.go:80] parser 3 done
I1015 11:09:38.702098 06942 getpromos_waitgroup.go:80] parser 4 done
I1015 11:09:38.702102 06942 getpromos_waitgroup.go:80] parser 5 done
I1015 11:09:38.702105 06942 getpromos_waitgroup.go:80] parser 6 done
I1015 11:09:38.702109 06942 getpromos_waitgroup.go:80] parser 7 done
I1015 11:09:38.702113 06942 getpromos_waitgroup.go:80] parser 0 done
I1015 11:09:38.702116 06942 getpromos_waitgroup.go:80] parser 1 done
I1015 11:09:38.702120 06942 getpromos_waitgroup.go:80] parser 2 done
I1015 11:09:38.702124 06942 getpromos_waitgroup.go:51] waiting for the promo group
I1015 11:09:38.702128 06942 getpromos_waitgroup.go:58] promo writer done

Hopwatch – a debugging tool for Go

Hopwatch is an experimental tool that can help debugging Go programs. Unlike most debuggers, hopwatch requires you to insert function calls at points of interest in your program. At these program locations, you can tell Hopwatch to display variable values and suspend the program (or goroutine). Hopwatch uses Websockets to exchange commands between your program and the debugger running in a HTML5 page.

Using Hopwatch

Basically, there are two functions Display and Break. Calling Display will print the variable,value pairs in the debugger page. Calling Break will suspend the program until you tell the debugger to Resume. The Display function takes one or more pairs of (variable name,value). The Break function takes one or more boolean values to be conditionally. Below is a test example.

package main

import "github.com/emicklei/hopwatch"

func main() {
	for i := 0; i < 6; i++ {
		hopwatch.Display("i",i)
		j := i * i
		hopwatch.Display("i",i, "j", j ).Break(j > 10)
		hopwatch.Break()
	}
}

Starting your Go program that includes a Hopwatch function call waits for a connection to the debugger page.
You can disable Hopwatch by passing the command line parameters -hopwatch false.

2012/12/14 17:24:47 [hopwatch] open http://localhost:23456/hopwatch.html ...
2012/12/14 17:24:47 [hopwatch] no browser connection, wait for it ...

Once the debugger is connected, it can receive Display and Break command requests from your Go program. You can look at the stack trace, resume a suspended program or disconnect from your program.

Hopwatch debugger page

I made a short movie (1.5Mb .mov) that demonstrates using hopwatch in a multiple goroutines program. This example is included in the tests folder of the hopwatch package.

What’s next?

Creating this tool has been fun and educative. Hopwatch uses two channels to communicate between the sendLoop and receiveLoop goroutines and uses the go.net Websocket package to talk to Javascript in a browser. As a tool Hopwatch may be useful (perhaps for remote debugging?) but ofcourse it lacks the important features of a real debugger (GDB) such as setting breakpoints at runtime (and therefore without source modification).
If you have suggestions then please leave a comment on the announcement in the go-nuts discussion group.

The project is located at github ; documentation can also be found on godoc.org

Methods as objects in Go

In an earlier post, I discussed an example of using plain functions as objects. Today, I investigated solutions for using Methods as objects. As I understand it, Go methods are functions that are “bound” to a named type that is not a pointer or an interface.

package main

import "fmt"

type T struct {
	S string
}

func (t T) GetS1() string {
	return t.S
}

func (t *T) GetS2() string {
	return t.S
}

func main() {
	t := T{"hello"}
	f := (*T).GetS1
	g := (T).GetS1
	h := (*T).GetS2
	// i := (T).GetS2   // invalid method expression T.GetS2 (needs pointer receiver: (*T).GetS2)
	fmt.Printf("%v, %v %v, who is there?", f(&t) ,g(t), h(&t))
}

Run it yourself.

In the example, I defined a type T and the value method GetS1 and pointer method GetS2 both returning the value of field S. In the main() function, I create a T and three variables (f,g and h) to hold references to the methods GetS1 and GetS2. The statement for the variable i is commented ; this is an illegal construction. Finally, in the printing statement, I put the expressions to call the functions using the t parameter ; either using its address, or as a value.

go-restful first working example

In a previous post, I discussed the design of go-restful which is a package for building REST-style WebServices using the Google Go programming language.

Today, I completed the implementation of that package which provides the basics:

  • Creating a WebService with Routes (mapping between Http Request and a Go function)
  • Each Route requires information about the Http method (GET,POST,…), URL Path (/users..), Mime-types and the function it binds to
  • Functions get passed in a Request and a Response
  • The Request is used to access Path and Query parameters , Headers and request content (XML, JSON,…)
  • The Response is used to set Status, Headers, and response content
  • Both Request and Response can Unmarshal and Marshal objects to and from XML,JSON using the standard packages

All of this can best be illustrated with a small example ; a Webservice for CRUD operations on User objects.
We start by creating the file userservice.go inside its own folder userservice.

package userservice

import (
	"github.com/emicklei/go-restful"
	"log"
)

type User struct {
	Id, Name string
}

Type User represents our domain object.

Next section in this file is the API specification of the WebService. The API is a collection of Route objects which specify how an incoming Http Request is mapped to a function.

func New() *restful.WebService {
	service := new(restful.WebService)
	service.
		Path("/users").
		Consumes(restful.MIME_XML, restful.MIME_JSON).
		Produces(restful.MIME_XML, restful.MIME_JSON)

	service.Route(service.GET("/{user-id}").To(FindUser))
	service.Route(service.POST("").To(UpdateUser))
	service.Route(service.PUT("/{user-id}").To(CreateUser))
	service.Route(service.DELETE("/{user-id}").To(RemoveUser))

	return service
}

First, the service is initialized with a root URL path for all routes, the MIME-types each route can receive (Consumes) and the MIME-types it can respond (Produces). It is possible to specify this for each individual Route. Next, the service specifies what Routes it can provide. The function call GET(“/{user-id}”) is just a shortcut for Method(“GET”).Path(“/{user-id}”) and creates a RouteBuilder object. Using the RouteBuilder we specify that the route with method GET will be handled by the function FindUser.

Next section in this file will be the function definitions for each Route.

func FindUser(request *restful.Request, response *restful.Response) {
	id := request.PathParameter("user-id")
	// here you would fetch user from some persistence system
	usr := &User{Id: id, Name: "John Doe"}
	response.WriteEntity(usr)
}

The signature for functions that are used in Routes are all the same and contain a restful Request,Response pair. Request is a wrapper for a http Request that provides convenience methods. Response is a wrapper on the actual http ResponseWriter. This design gives you as a developer complete access to the underlying http structures and access to common restful functions such as WriteEntity. The WriteEntity function will inspect the Accept header of the request to determine the Content-Type of the response and how to Marshal the object (in this case the User).

The remainder of the userservice.go file will contain the other function definitions.

func UpdateUser(request *restful.Request, response *restful.Response) {
	usr := new(User)
	err := request.ReadEntity(&usr)
	// here you would update the user with some persistence system
	if err == nil {
		response.WriteEntity(usr)
	} else {
		response.WriteError(http.StatusInternalServerError,err)
	}
}

func CreateUser(request *restful.Request, response *restful.Response) {
	usr := User{Id: request.PathParameter("user-id")}
	err := request.ReadEntity(&usr)
	// here you would create the user with some persistence system
	if err == nil {
		response.WriteEntity(usr)
	} else {
		response.WriteError(http.StatusInternalServerError,err)
	}
}

func RemoveUser(request *restful.Request, response *restful.Response) {
	// here you would delete the user from some persistence system
}

Now,we have completed the specification and implementation of the UserService. The next snippet shows an example how to use this service in an application.

package main

import (
	"github.com/emicklei/go-restful"
	"log"
	"net/http"
	"userservice"
)

func main() {
	restful.Add(userservice.New())
	log.Fatal(http.ListenAndServe(":8080", nil))
}

The go-restful project source can be found on github and is documented on go.pkgdoc.org

First class functions in Go

Today, I played with functions as objects in the Go programming language. If functions are first class citizens in Go then it must be possible to store them in fields of a struct, pass them as arguments to other functions and use them as return values of other functions.

So I visited play.golang.org for putting together a simple program that demonstrates this.

package main

import "fmt"

func CallWith(f func(string), who string) {
	f(who)
}

type FunctionHolder struct {
	Function func(string)
}

func main() {
	holder := FunctionHolder{ func(who string) { fmt.Println("Hello,", who) }}
	CallWith(holder.Function,"ernest")
}
  • The CallWith function takes a one string parameter function f and a string parameter who. The body of the CallWith function evaluates the function parameter with the string parameter.
  • The FunctionHolder is a struct type with a field called Function which must be a function with a string parameter.
  • In the main function, a FunctionHolder is created initializing the Function field with a function that prints an Hello message.
  • Next in the main function, the CallWith function is called with the Function value of the holder and a string value.

On play.golang.org can run this program yourself

go-restful api design

I have been using JAX-RS for many REST-based service implementations in Java.
As part of my journey into the Google Go programming language, I am exploring designs for such REST support using the standard net/http package in Go.

JAX-RS provides a set of Annotation classes which can be used to add meta-data to classes, methods and method arguments. In JAX-RS these annotation are used to specify the mapping between a Http Request to a Java method and its arguments.

With the exception of tags (string) for struct fields, there is no such concept in Go (AFAIK). So, I tried several approaches to specify this mapping information when defining a REST Resource. For that I wrote a simple example that defines a UserResource which has the REST versions of the CRUD operations on a domain type “User”.

Iteration 1: ResourceMethod and ResourceMethodContainer

The first part below shows the implementation of a UserResource with functions for each GET,PUT,POST and DELETE. The second part shows the Go code to specify the mappings for each function.

type UserResource struct {
	restful.ResourceMethodContainer
}
func (self UserResource) GetUser(userid restful.PathParam) restful.Response {
	response := restful.Response{Status: http.StatusOK}
	// someUser := fetch by userid
	// response.SetEntity(someUser)
	response.AddHeader(restful.HeaderLastModified, time.Now().Add(time.Duration(1000)))
	return response
}
func (self UserResource) UpdateUser(userid restful.PathParam, who User) restful.Response {
	// update User with id = userid
	return restful.Response{Status: http.StatusOK}
}
func (self UserResource) CreateUser(userid restful.PathParam, who User) restful.Response {
	// new User with id = userid
	return restful.Response{Status: http.StatusCreated}
}
func (self UserResource) DeleteUser(userid restful.PathParam) restful.Response {
	// delete User where id = userid
	return restful.Response{Status: http.StatusOK}
}

The functions in UserResource all return a restful.Response which is used to specifiy the status, the response body (bytes) and optional Http headers.

A mapping is specified with a ResourceMethod struct ; its Function field must match to a function defined in the UserResource package. That is the reason why the UserResource has an anonymous field ResourceMethodContainer.

// Create a new UserResource and register all mappings between HTTP requests and UserResource functions
func New() UserResource {
	res := UserResource{restful.ResourceMethodContainer{Root: "/"}}
	userPath := "/users/{user-id}"
	res.Register(restful.ResourceMethod{
        Method:   restful.GET,
        Path:     userPath,
        Produces: restful.XML,
        Function: "GetUser"})

    res.Register(restful.ResourceMethod{
        Method:   restful.POST,
        Path:     userPath,
        Consumes: restful.XML,
        Function: "UpdateUser"})

    res.Register(restful.ResourceMethod{
        Method:   restful.PUT,
        Path:     userPath,
        Consumes: restful.XML,
        Function: "CreateUser"})
    res.Register(restful.ResourceMethod{
        Method:   restful.DELETE,
        Path:     userPath,
        Function: "DeleteUser"})
	return res
}

Iteration 2: Apply Builder pattern to reduce clutter

The idea here is to use a ResourceMethodBuilder to build ResourceMethod structs using a fluent api. The builder is initialized with defaults and has methods to change its state in order to build a ResourceMethod. For example, the call GET(“GetUser”) tells the builder to change the current Http method to “GET” and that it is mapped to the function “GetUser”.

func New() *UserResource {
	res := new(UserResource)
	res.Path("/users")

	builder := restful.ResourceMethodBuilder {
	   Produces: restful.XML,
	   Consumes: restful.XML }
	builder.Path("/{user-id}")

	res.Add(builder.GET("GetUser").Build())
	res.Add(builder.POST("UpdateUser").Build())
	res.Add(builder.PUT("CreateUser").Build())
	res.Add(builder.DELETE("DeleteUser").Build())

	res.Add(builder.GET("FindUsersByName").Path("/").QueryParam("name").Build())

	return res
}

Below, the restful.Response is replaced by a function parameter of type restful.ResponseBuilder. It has convenience build functions to compose the response. Its implementation will use the http.ResponseWriter for actually writing the response content including the marshalled entity.

func (self UserResource) FindUsersByName(response restful.ResponseBuilder, name restful.QueryParam) {
    users := []User{} // select from User where name  = ?
    response.StatusOK()
    response.AddHeader(restful.HeaderLastModified, time.Now().Add(time.Duration(1000)))
    response.Entity(users)
}

Iteration 3: Introduce defaults on the Resource and override per method if needed

func New() *UserResource {
    resource := new(UserResource)
    // set the root path and defaults for accept and contentType
    resource.Path("/users").Accept(restful.XML).ContentType(restful.XML)

    // create a builder for Resource Methods
    builder := resource.NewMethodBuilder().Path("/{user-id}")

    // add methods using current path, contentType and accept
    resource.Add(builder.GET("GetUser"))
    resource.Add(builder.POST("UpdateUser"))
    resource.Add(builder.PUT("CreateUser"))
    resource.Add(builder.DELETE("DeleteUser"))

    // change current contentType and path for function FindUsersByName
    builder.ContentType(restful.JSON).Path("/")
    resource.Add(builder.GET("FindUsersByName").QueryParam("name"))

    return resource
}

Iteration 4:  Binding Routes to Functions

In this iteration, I decided to change the name of ResourceMethodContainer to WebService. The combination “restful.WebService” is clear about what its functionality will provide.

The next (and biggest) change is to use Functions as first class citizens. Instead of storing the function name in the Resource and resolving this to a function at runtime, I introduced Route structs that are initialized with a reference to the function. In the example below, a Route is created that binds a GET on “/users/{user-id}” to the function GetUser.

Finally, I simplified the signature for functions that are used for mapping (Routes) from Http requests. A restful.Request is used to access Path,Query,URI and Header information. A restful.Response is used to write back the response composed of Status,Header and Body (bytes). In previous api designs, I tried to use an explicit argument mapping strategy i.e. the parameter value “{user-id}” could have been mapped to the “userid PathParam”. This complicates the implementation of go-restful but also causes challenges for functions that need more information from the Request or need to add more information to the Response.

type UserService struct {
	restful.WebService
}
func New() *UserService {
	service := new(UserService)
	service.Path("/users").Accept("application/xml").ContentType("application/xml")

	service.Route(service.Method("GET").Path("/{user-id}").To(GetUser))
	return service
}

func GetUser(request *restful.Request, response *restful.Response) {
	id := request.PathParameter("user-id")
	wantsDetails := request.QueryParameter("details")
	x := request.Header.Get("X-Something")
	// ... fetching the user
	response.StatusOK()
	response.AddHeader("Last-Modified", time.Now().Add(time.Duration(1000)).String())
	response.Entity(user)
}

Iteration 5 (2012-12-04): Document the API

To allow for a more detailed auto-generated API document, the RouteBuilder (and therefore Route) is extended to include comment Doc(string), the request payload type Reads(…) and the response payload type Write(…).

	service.Route(service.GET("?from={from}&to={to}&type={type}&center={center}").
		Doc(`Get all (filtered) connections for all applications and the given scope`).
		To(getFilteredConnections).
		Writes(model.Connection{}))

	service.Route(service.PUT("/from/{from}/to/{to}/type/{type}?allowCreate={true|false}").
		Doc(`Create a new connection using the from,to,type values`).
		PathParam("from", "comma separated list of application ids").
		PathParam("to", "comma separated list of application ids").
		PathParam("type", "comma separated list of application ids").
		QueryParam("allowCreate", "if true then create any missing applications").
		To(putConnection).
		Reads(model.Connection{}))

Update 1: Others designs (and implementations) are rest2go and gorest

Update 2: go-restful is now work-in-progress