distchan
Package distchan enables Go channels to be used for distributed computation.
NOTE: This library is very young, and as such its API is very much subject
to change. Until this notice is removed, it should be assumed that the API is in
an alpha state and subject to breakage. That said, the changes shouldn't be too
drastic, and feedback is very much encouraged, so please give it a shot!
Also, check it out on
Chisel!
Why?
While Go's concurrency story around single-process data pipelines is
great, its ability to distribute workloads
across multiple machines is relatively lacking. There are several options here
(notably glow and
gleam), but they lack the type-safety and
ease-of-use that Go's built-in channels provide.
How?
In a nutshell: standard net primitives, Gob encoding, and reflection.
Architecturally, the package assumes a client-server model of workload
distribution. One server can have as many clients connected to it as you want,
and work will be distributed across them using a simple round-robin algorithm.
Example
As a simple example, let's say that capitalizing letters in a string is very
computationally expensive, and you want to distribute that work across a number
of nodes. First, you'll need to create a server in charge of defining the work
to be done:
Server
package main
import (
"log"
"net"
"github.com/dradtke/distchan"
)
func main() {
ln, err := net.Listen("tcp", "localhost:5678")
if err != nil {
log.Fatal(err)
}
var (
out = make(chan string)
in = make(chan string)
server, _ = distchan.NewServer(ln, out, in)
)
server.Start()
server.WaitUntilReady() // wait until we have at least one worker available
go producer(out)
for s := range in {
println(s)
}
}
func producer(out chan<- string) {
// send strings to be capitalized to out
out <- "hello world"
// don't forget to close the channel! this is how all connected
// clients know that there's no more work coming.
close(out)
}
Then you'll need to create a client, or worker. It's similarly easy to get wired
up, so you can focus on the hard part: capitalizing strings:
Client
package main
import (
"log"
"net"
"strings"
"github.com/dradtke/distchan"
)
func main() {
conn, err := net.Dial("tcp", "localhost:5678") // must be able to connect to the server
if err != nil {
log.Fatal(err)
}
var (
out = make(chan string)
in = make(chan string)
client, _ = distchan.NewClient(conn, out, in)
)
client.Start()
// Loop over all input from the server...
for input := range in {
capitalized := strings.ToUpper(input)
// ...and send the results back.
out <- capitalized
}
close(out)
<-client.Done()
}
Check out the example folder for more examples.