What you will learn
- Why
send()andrecv()can move fewer bytes than requested. - Reusable
send_all()andrecv_all()loops. - Three ways to mark message boundaries on a byte stream, and why length prefixes are the usual choice.
- How to assemble complete messages from arbitrary chunks in a non-blocking server.
Short counts are normal
TCP is a byte stream (lesson 1). The kernel moves bytes between your buffers and its own send and receive buffers in whatever amounts are available:
send()returns as soon as it has copied some bytes into the send buffer. On a blocking socket that’s usually everything, but not always: a signal, a full buffer on a non-blocking socket, or a very large write can all produce a short count.recv()returns whatever has arrived so far, from 1 byte up to your buffer size. A 10 KB message might arrive as 1448 + 1448 + … bytes, matching network segment sizes, or all at once on loopback.
That last point explains a common surprise: “on my machine I get all the data in one recv(), but over the network I get it 1460 bytes at a time”. Loopback has a huge MTU and no real network delay; real links deliver data in segments. Your code must work for any split.
send_all() and recv_all()
These two loops belong in every TCP program you write:
/* Keep calling send() until every byte is written. 0 on success, -1 on error. */
int send_all(int fd, const void *data, size_t len);
/* Read exactly len bytes. 1 = done, 0 = peer closed early, -1 = error. */
int recv_all(int fd, void *data, size_t len);
Their full implementations are at the top of the program below. recv_all() is only appropriate when you know how many bytes to expect, which brings us to framing.
Putting message boundaries back
The receiver needs a rule to know where one message ends. Three common choices:
| Method | Example | Pros | Cons |
|---|---|---|---|
| Delimiter | Lines ending in \n (SMTP, Redis inline, chat) | Human-readable, easy with nc | Payload can’t contain the delimiter without escaping; must scan every byte |
| Fixed size | Every message is 64 bytes | Trivial to parse | Wastes space or limits content |
| Length prefix | 4-byte length, then payload (many binary protocols) | Any payload; receiver knows exactly how much to read | Not human-readable |
Length-prefixed frames
Each frame is a 4-byte unsigned length in network byte order followed by that many payload bytes. Here’s a complete implementation, tested by sending three frames through a socketpair() (a connected pair of local sockets, perfect for tests):
/* framing.c - send_all/recv_all and length-prefixed messages over a stream.
* Self-test: runs over a socketpair, feeding bytes in awkward chunks. */
#include <errno.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/types.h>
/* Keep calling send() until every byte is written. 0 on success, -1 on error. */
int send_all(int fd, const void *data, size_t len)
{
const char *p = data;
while (len > 0) {
ssize_t n = send(fd, p, len, MSG_NOSIGNAL);
if (n == -1) {
if (errno == EINTR)
continue;
return -1;
}
p += n;
len -= (size_t)n;
}
return 0;
}
/* Read exactly len bytes. 1 = done, 0 = peer closed early, -1 = error. */
int recv_all(int fd, void *data, size_t len)
{
char *p = data;
while (len > 0) {
ssize_t n = recv(fd, p, len, 0);
if (n == 0)
return 0;
if (n == -1) {
if (errno == EINTR)
continue;
return -1;
}
p += n;
len -= (size_t)n;
}
return 1;
}
#define MAX_FRAME 65536
/* Frame = 4-byte big-endian length, then that many payload bytes. */
int send_frame(int fd, const void *payload, uint32_t len)
{
uint32_t header = htonl(len);
if (send_all(fd, &header, sizeof header) == -1)
return -1;
return send_all(fd, payload, len);
}
/* Returns payload length, or -1 on error/close. buf must hold MAX_FRAME bytes. */
long recv_frame(int fd, void *buf)
{
uint32_t header;
if (recv_all(fd, &header, sizeof header) != 1)
return -1;
uint32_t len = ntohl(header);
if (len > MAX_FRAME) /* never trust a length from the network */
return -1;
if (len > 0 && recv_all(fd, buf, len) != 1)
return -1;
return (long)len;
}
int main(void)
{
int sv[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == -1) {
perror("socketpair");
return 1;
}
const char *msgs[] = { "hi", "", "a slightly longer message" };
for (int i = 0; i < 3; i++)
send_frame(sv[0], msgs[i], (uint32_t)strlen(msgs[i]));
static char buf[MAX_FRAME];
for (int i = 0; i < 3; i++) {
long n = recv_frame(sv[1], buf);
printf("frame %d: %ld bytes \"%.*s\"\n", i, n, (int)n, buf);
if (n != (long)strlen(msgs[i]) || memcmp(buf, msgs[i], (size_t)n) != 0)
return 2;
}
close(sv[0]);
close(sv[1]);
return 0;
}
frame 0: 2 bytes "hi"
frame 1: 0 bytes ""
frame 2: 25 bytes "a slightly longer message"
The most important line is the size check in recv_frame(). A length field comes from the network, so it’s untrusted input. Without MAX_FRAME, a malicious or buggy peer could send 0xFFFFFFFF and make you try to allocate or read 4 GB.
Framing in a non-blocking server
recv_all() blocks until the whole frame arrives, which a poll()-based server can’t afford. Instead, give each connection its own buffer. Append whatever each recv() returns, then extract as many complete frames as the buffer holds, keeping any leftover partial frame for next time:
/* frame_buffer.c - collect bytes from a non-blocking socket until whole frames arrive. */
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>
#define BUF_CAP (4 + 1024) /* room for one maximum-size frame */
struct conn_buffer {
unsigned char data[BUF_CAP];
size_t used;
};
/* Append freshly received bytes. Returns -1 if they don't fit. */
int buffer_append(struct conn_buffer *b, const void *bytes, size_t n)
{
if (n > BUF_CAP - b->used)
return -1;
memcpy(b->data + b->used, bytes, n);
b->used += n;
return 0;
}
/* If a complete frame is buffered, copy its payload out and return its length.
* Returns -1 if more bytes are needed, -2 if the frame is too big. */
long buffer_take_frame(struct conn_buffer *b, void *out, size_t out_cap)
{
if (b->used < 4)
return -1;
uint32_t len;
memcpy(&len, b->data, 4);
len = ntohl(len);
if (len > out_cap || len > BUF_CAP - 4)
return -2;
if (b->used < 4 + (size_t)len)
return -1; /* header here, payload not yet */
memcpy(out, b->data + 4, len);
size_t rest = b->used - 4 - len;
memmove(b->data, b->data + 4 + len, rest); /* keep leftover bytes */
b->used = rest;
return (long)len;
}
int main(void)
{
/* Two frames, "PING" and "hello", arriving split in odd places. */
unsigned char wire[] = { 0,0,0,4, 'P','I','N','G', 0,0,0,5, 'h','e','l','l','o' };
size_t cuts[] = { 3, 6, 9, sizeof wire };
struct conn_buffer b = { .used = 0 };
size_t from = 0;
char payload[1024];
for (int i = 0; i < 4; i++) {
buffer_append(&b, wire + from, cuts[i] - from); /* pretend recv() gave us this */
from = cuts[i];
long n;
while ((n = buffer_take_frame(&b, payload, sizeof payload)) >= 0)
printf("after chunk %d: frame \"%.*s\"\n", i, (int)n, payload);
}
return b.used == 0 ? 0 : 1;
}
after chunk 2: frame "PING"
after chunk 3: frame "hello"
The test feeds two frames split at awkward places (inside the header, inside the payload, across the boundary). Nothing is produced until a frame is complete, and the leftover bytes of the second frame are kept with memmove(). This pattern, a per-connection input buffer plus a “take next complete message” function, is the heart of every event-driven protocol parser.
Designing a small protocol
Framing gives you messages; the payload still needs structure. A typical binary message looks like:
+---------+--------+-------------+------------------------+
| length | type | request id | body ... |
| 4 bytes | 1 byte | 4 bytes | (length - 5) bytes |
+---------+--------+-------------+------------------------+
- A type byte says how to interpret the body (login, chat message, ping…).
- A request id lets replies be matched to requests when several are in flight.
- A version field in the first message lets you evolve the protocol later.
How to encode the numbers and strings in the body is the topic of the next lesson.
Key takeaways
- Partial sends and receives are normal; wrap them in
send_all()/recv_all()loops. - TCP has no message boundaries; add them with delimiters, fixed sizes or (usually) length prefixes.
- Always cap length fields received from the network.
- Event-driven servers keep a per-connection buffer and extract complete frames as they arrive.
Exercises
- Add
send_all()to the poll chat server’s relay and switch it to newline-delimited messages using a per-client buffer. - Write
recv_line()that reads up to\nusing an internal buffer (not one byte perrecv()call). - Extend
frame_buffer.cwith a 1-byte message type after the length, and a dispatcher that printsPINGandTEXTmessages differently.
Quiz
- A sender calls
send()once with 5000 bytes over the Internet. How manyrecv()calls might the receiver need? - Why must
recv_frame()check the length against a maximum? - In an event-driven server, what happens to bytes that arrive after the end of a complete frame?
Show answers
- Any number from 1 up to 5000; commonly a few, sized by network segments (around 1448–1460 bytes each on Ethernet).
- The length is untrusted input. A huge value could exhaust memory or make the server wait forever for data that never comes.
- They stay in that connection’s buffer (moved to the front) as the beginning of the next frame.