Shared memory and IPC

Why IPC is a spectrum

Two extremes:

• Tightly coupled, same machine, high throughput: shared memory. Zero copy. Microsecond latency. You handle synchronization.
• Loosely coupled, possibly across machines: sockets. Two copies. API works locally and over network. Kernel handles delivery.

Everything in between (pipes, FIFOs, unix sockets, message queues) is a different point on the trade-off curve.

POSIX shm vs SysV shm

POSIX (recommended):

int fd = shm_open("/name", O_CREAT|O_RDWR, 0660);
ftruncate(fd, size);
void *p = mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
close(fd);  // mmap holds the reference
// ... use *p ...
munmap(p, size);
shm_unlink("/name");  // when done forever

shm objects appear in /dev/shm/name (a tmpfs). You can ls /dev/shm to see them. They persist until shm_unlink or reboot.

SysV (older):

int shmid = shmget(IPC_PRIVATE, size, 0660|IPC_CREAT);
void *p = shmat(shmid, NULL, 0);
// ...
shmdt(p);
shmctl(shmid, IPC_RMID, NULL);

SysV uses integer IDs and system-wide limits set via sysctl. Older code uses it; new code should prefer POSIX shm.

mmap variations worth knowing

• MAP_SHARED + file fd: classic memory-mapped file. Writes go back to the file. Used by databases, file servers.
• MAP_SHARED + MAP_ANONYMOUS: shared memory without a backing file. Only useful between related processes (via fork).
• MAP_PRIVATE + file fd: load file into memory but writes don't go back. COW for changes.
• MAP_PRIVATE + MAP_ANONYMOUS: this is what malloc for large allocations uses under the hood.
• MAP_HUGETLB: use 2MB pages instead of 4KB. Reduces TLB pressure.
• MAP_LOCKED: pin in RAM, never swap. Requires CAP_IPC_LOCK or a high RLIMIT_MEMLOCK.
• MAP_POPULATE: pre-fault all pages. Slower mmap, faster first access.

A mmap MAP_SHARED of a file is how databases like LMDB and Boltdb operate: the file IS the database, the OS handles the cache via page cache, transactions use COW techniques.

Synchronization options

Atomic operations

For counters, flags, single-word coordination:

#include <stdatomic.h>
struct shared { atomic_int counter; };
atomic_fetch_add(&s->counter, 1);

C11 atomics, C++11 std::atomic, Rust AtomicUsize. These compile to single CPU instructions (LOCK XADD on x86) and are safe across processes if the atomic lives in shared memory.

Process-shared mutexes

pthread_mutexattr_t attr;
pthread_mutexattr_init(&attr);
pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
pthread_mutex_init(&shared->mutex, &attr);

The mutex itself must live in shared memory. With PTHREAD_PROCESS_SHARED, it works across processes. Underneath, it uses a futex (fast userspace mutex): uncontended lock/unlock is a single atomic op in userspace; contention falls into the futex syscall.

Beware of robust mutexes (PTHREAD_MUTEX_ROBUST): if the holding process dies, the next lock attempt gets EOWNERDEAD and must call pthread_mutex_consistent to reclaim. Otherwise a process crash with a lock held = permanent deadlock.

Lock-free ring buffers

For high-throughput producer/consumer, locks are too slow. Use a single-producer single-consumer (SPSC) ring buffer:

[ head atomic ]  [ tail atomic ]  [ buffer N slots ]
producer: load tail, store data at tail, store tail+1
consumer: load head, load data at head, store head+1

With proper memory ordering (release on producer write, acquire on consumer read), this is wait-free. Used by LMAX Disruptor, DPDK, kernel ring buffers, perf_events.

For multi-producer or multi-consumer, the algorithms get harder (MPSC, MPMC). Crossbeam in Rust, moodycamel::ConcurrentQueue in C++, Disruptor in Java.

Futexes: the primitive behind mutexes

A futex is a 32-bit integer in shared memory plus a syscall (futex). Userspace can implement a mutex as: "try to atomically swap 0 to 1; if you got 0, you have the lock. If you got 1, call futex(WAIT) to sleep until someone wakes you."

The unlock is "atomic swap 1 to 0; if there were waiters, call futex(WAKE) to wake one."

The win: uncontended lock/unlock never calls into the kernel. Only contention does. This is why glibc's pthread_mutex is essentially free when uncontested.

Process-shared futexes work because the futex's address is in shared memory; the kernel hashes the physical address (not virtual) to find the wait queue, so both processes contend correctly.

Eventfd, signalfd, timerfd

Modern Linux gives you "events as file descriptors," so you can integrate them into an epoll loop.

• eventfd: a counter you can read/write; combined with epoll, lets one thread wake another or signal across threads.
• signalfd: receive signals via read() instead of via async handlers. Easier to reason about in event loops.
• timerfd: timers that fire as readable events on an fd.

These solve the "I'm in an epoll loop and I want to be woken by X (a signal, a timer, another thread)" problem cleanly.

Pointers in shared memory: don't

When process A maps shared memory at virtual address 0x7f0000000000 and process B maps it at 0x7f1000000000, a pointer stored in A is meaningless in B. Both pointers point to the same physical memory, but they're different virtual addresses.

Two strategies:

1. Map at the same address in all processes. Use mmap(addr, ..., MAP_FIXED, ...). Fragile because ASLR fights you. Boost.Interprocess does this.
2. Use offsets, not pointers. Store relative offsets from the start of the shared region. Reconstruct pointers at access time. This is what most production systems do (LMDB, Apache Arrow IPC, capnproto with file mode).

Languages with safety features (Rust) make this explicit. C/C++ lets you shoot yourself in the foot.

Why sockets are sometimes still right

• Topology change-friendly: moving from "same machine via unix socket" to "different machines via TCP" is a config change, not a code rewrite.
• Multiple consumers: broadcast/fan-out is awkward in raw shared memory; trivial with sockets or pubsub.
• Backpressure: sockets have built-in buffering and flow control; you have to design these yourself in shared memory.
• Security boundaries: unix sockets with SO_PEERCRED let you authenticate the peer. Shared memory has no built-in identity.
• Language interop: every language has socket support out of the box; shared memory APIs are less ubiquitous.

For most application IPC, unix sockets are the right default. Use shared memory when you have measured a copy/syscall bottleneck.

Message passing primitives

• POSIX message queues (mq_open, mq_send, mq_receive): kernel-managed queues with priorities. Bounded. Useful for distinct messages with structure.
• SysV message queues: older, similar idea, less commonly used.
• Datagrams over unix socket: flexible, language-friendly, supports SOCK_SEQPACKET for ordered messages with boundaries.

For higher-level patterns (pub/sub, request/reply, work distribution), use ZeroMQ, NATS, or gRPC. These hide IPC choice and give you patterns.

Cache-line considerations

Shared memory between threads or processes on the same machine still goes through CPU caches. Two cores writing to the same cache line cause false sharing: every write invalidates the other's cache, ping-ponging the line over the interconnect.

Pad your shared structures to cache-line size (64 bytes on x86_64). For producer/consumer rings, put head and tail in separate cache lines so producer and consumer don't bounce the line.

C++17 has std::hardware_destructive_interference_size for this. Rust has #[repr(align(64))]. C does it via alignas(64).

Real-world examples

• Postgres: uses shared memory for the buffer pool. Backends mmap a shared region at startup, all access pages through it.
• Chrome: shared memory between renderer process and GPU process for compositing.
• Redis: shared memory between parent and forked child during BGSAVE (COW after fork).
• DPDK: shared memory ring buffers between NIC poll thread and application.
• systemd: unix sockets everywhere for IPC between systemd and services.
• gRPC: uses unix sockets or TCP; same code path. Shared memory transport is rare but exists for IPC-heavy uses.

Common pitfalls

Mental model

Shared memory is like roommates sharing a kitchen: super fast access, but you need a rule for who gets the stove when (locks/atomics). Pipes are passing notes under the door: simple, slow, sequential. Sockets are mail: works across the street or across the world, but every letter is copied and stamped. Pick by whether your problem is "share a lot fast with someone next door" or "talk reliably to anyone."