It's not just about receiving. You have to send, too.
When people learn about building services, they focus on the ingress side. How do I accept connections? How do I parse requests? How do I handle load? Good questions. But only half the picture.
A service that can only receive is a dead end. Real systems are conversations. Your service receives a request, processes it, and then talks to other things — databases, queues, other services, hardware. The send side is just as important as the receive side, and the patterns you use for it change everything about reliability, latency, and debuggability.
Channels, control planes, and back channels
Here's a pattern that shows up everywhere, and once you see it, you can't unsee it: active and passive communication.
FTP: The original two-channel protocol
FTP uses two connections: a control channel (port 21) where commands are sent, and a data channel (a separate port) where actual file data flows. In active mode, the server connects back to the client on a port the client specified. In passive mode, the client connects to a port the server specified.
This separation of control and data is everywhere. VDI protocols (Citrix ICA, VMware PCoIP) use a control channel for session management and separate channels for display, audio, USB, and clipboard — each optimized differently. Kubernetes uses the API server as a control plane and kubelet connections as the data plane.
The back channel: reverse communication
Sometimes the most powerful pattern is the one that goes backwards. Instead of the client always initiating, the server pushes. Instead of polling, you subscribe. Instead of the installer pulling packages from the internet, the ISO carries everything with it.
An installer is nothing more than a process where hardware and services
communicate to accomplish a feat. And the smartest installers use back channels —
the target system reporting its state back to the installer, the bootloader telling the
init system what to mount, the kernel passing parameters to userspace through /proc/cmdline.
Protocol brokers: the right tool for the job
Ansible is great at push. "Go to these ten machines and run this playbook." But it's stateless — it doesn't know what happened between runs. Salt is a protocol broker. It maintains persistent connections (ZeroMQ under the hood), receives events in real-time, and can react. It does what Ansible does, plus everything Ansible can't.
The trick isn't choosing one. It's understanding that push and pull, active and passive, request-reply and pub-sub — these are all tools. A hammer and a screwdriver aren't competing. You use the one that fits. The person who only has a hammer thinks everything is a nail. The person who understands the toolbox builds things that work.
The recipe metaphor
Building systems is like cooking. You don't need to be a chef to feed yourself, but knowing what heat does to protein is the difference between a steak and a shoe. You don't need to memorize every protocol RFC, but understanding that TCP guarantees delivery (at a cost) and UDP doesn't (but is faster) is the difference between a system that works and one that "works on my machine."
It's not rocket science. It's recipes. And we're writing them down so you don't have to reverse-engineer them from Stack Overflow posts written in 2014.