Consider the case where there are multiple gophers at work alone the pipeline at each place:
- Multiple are loading books from a pile into a cart.
- Multiple are moving the filled cart to the incinerator (and later returning the empty cart).
- Multiple are moving the books from the cart into the incinerator.
The data is sending through the channels in sequences (pipelines).
What is the proper way to close the channels/pipelines in proper sequences?
To help people that need to understand with code, I've prepared one @ https://go.dev/play/p/Q8ej8-fgnDL:
package main
import (
"fmt"
"time"
)
// Here's the worker, of which we'll run several
// concurrent instances. These workers will receive
// work on the `jobs` channel and send the corresponding
// results on `results`. We'll sleep a second per job to
// simulate an expensive task.
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs {
fmt.Println("worker", id, "started job", j)
time.Sleep(time.Second)
fmt.Println("worker", id, "finished job", j)
results <- j * 2
}
}
func main() {
// In order to use our pool of workers we need to send
// them work and collect their results. We make 2
// channels for this.
jobs := make(chan int, 100)
results := make(chan int, 100)
// This starts up 3 workers, initially blocked
// because there are no jobs yet.
for w := 1; w <= 3; w++ {
go worker(w, jobs, results)
}
// Here we send 5 `jobs` and then `close` that
// channel to indicate that's all the work we have.
for j := 1; j <= 5; j++ {
jobs <- j
}
close(jobs)
// Finally we collect all the results of the work.
for a := 1; a <= 5; a++ {
<-results
//for _ := range results {
}
}
The question in turn becomes, what if I comment out line 44~45, and uncomment line 46, how to make it still work?
UPDATE:
Found the best theoretical explanation of the problem -- the Fan-out, fan-in concurrency pattern explained and illustrated at https://go.dev/blog/pipelines