#WebRTC#Go#WebSockets#Distributed System#Heavy Workload

Building Ethio-Peer

May 10, 2025

Why, and why a platform

The official answer to "how do CS students actually learn this stuff" at our college was: lecture, assignment, exam. But the thing that actually worked for me and for basically everyone I studied with, was the small group. Three or four of us, one shared screen, one person explaining something they'd just figured out, the rest asking the questions no lecture ever answered. That was the real class.

We had no infrastructure for it, though. No easy way to spin up a live session and make it public in a centralized manner, no way to share materials across a study group, no record of what students were actually stuck on so an instructor could step in. So we built one. Ethio-Peer is a real-time collaborative workspace and streaming thing for peer-learning groups in Ethiopia, scoped first to HILCOE School of Computer Science, my college, but built so it could be pointed somewhere else later.

This is the journal of what got built, what I figured out, and the parts that still surprise me when I open the repo.

What it looks like

Seven services, one shared docker-compose.yml, one VPS doing absolutely everything:

gateway/             # C# + YARP, the single front door
auth-service/        # Go
peer-service/        # Go
streaming-service/   # Go + LiveKit
resource-service/    # C# + MinIO, the upload pipeline
bridge-service/      # Go, glues things together
mailing-service/      # Go

The split was on purpose. Auth is auth, peer groups are peer groups, live sessions are live sessions, files are files, and the gateway is the only thing the browser talks to. Each service owns its data, calls its neighbors over gRPC when it needs a synchronous answer, and drops events on RabbitMQ for everything async. Redis sits in front of the hot reads.

The gateway is C# because YARP is honestly the nicest reverse-proxy story I've used in any language, and the resource-service is C# because ASP.NET's multipart upload plus the MinIO SDK is a couple of lines, whereas Go's upload story is, in 2025, still a small fight you don't need to have.

Everything else is Go.

The infrastructure that really shouldn't have worked

The whole thing runs on one VPS: 50GB of storage, 1.9GB of RAM. LiveKit self-hosted on it. RabbitMQ self-hosted on it. Redis self-hosted on it. All seven services self-hosted on it. I handled deploying every piece of it.

For context, LiveKit's documented minimum for a small production deployment is more memory than I had for the entire stack. RabbitMQ alone defaults to eating more RAM than I could spare. Redis, the upload service, the gateway. Every default config in every container assumed I had a real machine. I didn't.

So nothing ran on defaults. docker-compose.yml ended up with explicit deploy.resources.limits on the heavy hitters. RabbitMQ capped at 1 CPU and 1GB, Redis at half a CPU and 512MB, everything else fighting over whatever was left. The trick wasn't cleverness, it was just restraint. I turned off stuff I didn't need (RabbitMQ's management UI, Redis's disk persistence, LiveKit's recording), grabbed Alpine images where I could, and tuned the Go services to keep idle memory low.

It runs. It's been running. The thing I keep relearning: when your hardware is the constraint, you stop reading about what's possible and start reading about what you can turn off.

The Go services: vertical slice architecture

All the Go services follow the same internal layout:

internal/
├── broker/      # RabbitMQ producers/consumers
├── db/          # database connection + migrations
   ├── features/    # the actual slices: login, signup, peer, profile, ...
├── genproto/    # generated gRPC code
├── models/      # domain types
├── protobuf/    # .proto sources
└── server/      # the gRPC server wiring

VSA, vertical slice architecture. Instead of the usual handlers/, services/, repositories/ split (which sorts code by what kind of thing it is), each feature is a self-contained folder that owns its handler, its validation, its persistence calls, and its events. auth-service/internal/features/login/ is everything login needs; nothing outside that folder has to know it exists.

It feels like overkill on day one. On day thirty, when you need to add a permission check to signup and the whole change lives inside the signup folder, you stop questioning it. The only things that have to know about anything else are the cross-cutting bits: broker, db, models.

A slice looks roughly like this:

// features/signup/signup.go
func Signup(ctx context.Context, req *pb.SignupRequest) (*pb.SignupResponse, error) {
    // validate
    // persist
    // emit signup.created
    // return
}

The request comes in through the gRPC server, hits the slice, goes out through the broker if there's downstream work. No Service struct, no IRepository interface, no DI container. Just a function that does the thing.

The fan-out / fan-in pattern

Some requests genuinely need to do N things at once and gather the results. Like when the streaming service starts a new session, it has to provision a LiveKit room, tell the peer group, queue an analysis job for the admin dashboard, and write an audit event, all before responding.

The pattern I used is the standard Go one: a sync.WaitGroup plus per-result channels, with a small dispatcher that closes them. Each fan-out worker writes to its own channel, the fan-in goroutine ranges over them with a timeout, and the function returns the merged result. Shape of it:

results := make(chan result, len(tasks))
var wg sync.WaitGroup

for _, t := range tasks {
    wg.Add(1)
    go func(t Task) {
        defer wg.Done()
        results <- do(t)
    }(t)
}

wg.Wait()
close(results)

It's not exotic. It's also not something I'd have written a year ago without thinking hard about who closes what and when. The discipline of "every goroutine that writes to a channel is the one that signals done via WaitGroup" looks obvious in hindsight and feels like defusing a bomb the first time.

Where I used it in the project:

Session creation in the streaming service: provision room + notify group + queue analysis + audit, in parallel.
Material upload completion in the resource-service: confirm MinIO write + index for search + notify subscribers, in parallel.
Admin analytics rollup in the bridge-service: pull per-session stats from multiple services and aggregate.

gRPC everywhere

Every service-to-service call is gRPC. The browser never talks to a service directly. It goes through the gateway, which terminates HTTP, checks the auth claim, and routes to the right gRPC backend via YARP. The internal .proto files live in each service's protobuf/ folder and the generated Go code is in genproto/.

The win: one IDL, typed clients, generated stubs, streaming when you need it. The cost: you generate code, you commit it, you regenerate when the proto changes, and the first time you forget to run make gen and ship a build talking to a slightly older service version, you lose a Saturday to a mismatch.

The thing that saved me: every .proto change ships in the same commit as the regenerated code, and the generated files are checked in so reviewers can see what actually changed. CI runs buf breaking against main to catch backward-incompatible changes before they land.

The resource-service: where files live

The resource-service is the C# one that owns the material pipeline. It exposes REST endpoints for upload and download, talks to MinIO for the bytes, talks to RabbitMQ to announce a new resource is available, and exposes a gRPC interface for other services to ask "what's in course X, topic Y?"

The upload flow is the part I'm happiest with:

Client asks the gateway for a presigned upload URL.
Gateway calls resource-service over gRPC, which asks MinIO for a presigned PUT URL with a 15-minute expiry and a content-length cap.
Client uploads straight to MinIO. The bytes never touch the gateway.
MinIO fires a webhook to resource-service when it's done.
Resource-service verifies the object, computes the checksum, indexes the metadata, and publishes a resource.uploaded event to RabbitMQ.
The peer-service and streaming-service, which had subscribed, update their views.

The gateway is completely out of the byte-handling business. A 200MB lecture video doesn't go through ASP.NET. It goes browser → MinIO → webhook. The gateway is for control plane, not data plane.

The admin view: AI on the discussion stream

The piece I'm quietly most proud of is the admin analytics surface.

While a peer session is live, the streaming service emits periodic events. "Alice joined", "Bob spoke", "screen-share pointed at slide 7", "session ended", onto a dedicated RabbitMQ stream. A worker in the streaming service (internal/worker/score.go) tails that stream, batches the events per session, and at session-end fires them at a small analysis pipeline that:

summarizes what topics came up,
flags recurring questions or confusion patterns,
notes the language register and engagement level,
rolls it into a per-session report.

That report is what the college admin sees. Not a transcript, a summary of what students were actually struggling with during the live session, and what it felt like from the inside. The instructor can then decide to follow up.

This is the part where "for peer learning" stops being a tagline and becomes a feature. A platform that records what students understand during a study session, not just that they showed up, is a different tool from a video-call app. The summary isn't perfect, the model's not always right, and a bad summary is worse than no summary. So we keep the human in the loop: the admin sees the report, the admin decides.

Deployment, or "I own all of it"

I deployed every service on this stack. The deploy.sh at the repo root is a small loop that tags and pushes the locally-built images to Docker Hub under my namespace; the VPS pulls and restarts. One docker-compose.yml knows about every service and every dependency.

No Kubernetes. No Helm chart. No service mesh. One machine, one compose file, a docker compose up -d to roll out a new build. For a single-tenant college pilot, that's the right answer. I'm not too proud to admit I considered fancier setups and rejected them. Not because they wouldn't work, but because running them on 1.9GB of RAM would've eaten the whole budget.

The lesson, again: the right architecture is the one you can actually run on the hardware you've got.

What I learned

Microservices on a tiny machine is a forcing function. You stop reaching for defaults, you actually read the resource limits on every image, and you find which features in every dependency you were never using. Tight constraints are clarifying.
VSA scales to real services. The features/<slice>/<slice>.go layout, easy to dismiss as a toy, is the reason seven services stay navigable as they grow. The cross-cutting folders stay small and boring, which is exactly what you want from them.
gRPC + RabbitMQ is a good pair. gRPC for "I need an answer now," RabbitMQ for "I need to know eventually." Once that split is clear, the rest of the architecture kind of follows.
Self-hosting LiveKit is doable but only barely. If you go down this road, budget more memory than you think, and more time than you think, and more time to debug the timeouts.
The admin analytics is the actual product. Everything else is plumbing. The thing that justifies the platform existing is the report the instructor reads Monday morning.

What it isn't, yet

It's a pilot. It runs. It serves one college. The peer-group creation, the material sharing, the live sessions, the admin summaries. They all work. They don't all work well under load, and I haven't stress-tested any of it the way a production system would need. The summaries are sometimes wrong in ways that are subtle rather than obvious, and we're still figuring out how to surface that uncertainty to the admin.

I started it in April 2025. It's not done. I'm still keeping it.

References

Code: github.com/barnabasSol/ethio-peer, Go + C# microservices, LiveKit + MinIO + RabbitMQ + Redis, single-VPS deployment via docker-compose.yml.
Pattern references: VSA / feature-folder layout for the Go services; fan-out/fan-in via sync.WaitGroup + per-task channels; gRPC for sync service calls, RabbitMQ for async events; YARP as the gateway reverse proxy.