Lesson 15 of 18 Data on the Wire

Partial send() and Message Framing

Fix partial send() and recv() with send_all and recv_all loops, then add message boundaries to TCP with length-prefix framing and a per-connection buffer.

50-60 min intermediate 0/18 lessons done
Course contents Lesson 15 of 18 · 0 done
On this page (10 sections)
  1. What you will learn
  2. Short counts are normal
  3. send_all() and recv_all()
  4. Putting message boundaries back
  5. Length-prefixed frames
  6. Framing in a non-blocking server
  7. Designing a small protocol
  8. Key takeaways
  9. Exercises
  10. Quiz

What you will learn

  • Why send() and recv() can move fewer bytes than requested.
  • Reusable send_all() and recv_all() loops.
  • Three ways to mark message boundaries on a byte stream, and why length prefixes are the usual choice.
  • How to assemble complete messages from arbitrary chunks in a non-blocking server.

Short counts are normal

TCP is a byte stream (lesson 1). The kernel moves bytes between your buffers and its own send and receive buffers in whatever amounts are available:

  • send() returns as soon as it has copied some bytes into the send buffer. On a blocking socket that’s usually everything, but not always: a signal, a full buffer on a non-blocking socket, or a very large write can all produce a short count.
  • recv() returns whatever has arrived so far, from 1 byte up to your buffer size. A 10 KB message might arrive as 1448 + 1448 + … bytes, matching network segment sizes, or all at once on loopback.

That last point explains a common surprise: “on my machine I get all the data in one recv(), but over the network I get it 1460 bytes at a time”. Loopback has a huge MTU and no real network delay; real links deliver data in segments. Your code must work for any split.

send_all() and recv_all()

These two loops belong in every TCP program you write:

/* Keep calling send() until every byte is written. 0 on success, -1 on error. */
int send_all(int fd, const void *data, size_t len);

/* Read exactly len bytes. 1 = done, 0 = peer closed early, -1 = error. */
int recv_all(int fd, void *data, size_t len);

Their full implementations are at the top of the program below. recv_all() is only appropriate when you know how many bytes to expect, which brings us to framing.

Putting message boundaries back

The receiver needs a rule to know where one message ends. Three common choices:

MethodExampleProsCons
DelimiterLines ending in \n (SMTP, Redis inline, chat)Human-readable, easy with ncPayload can’t contain the delimiter without escaping; must scan every byte
Fixed sizeEvery message is 64 bytesTrivial to parseWastes space or limits content
Length prefix4-byte length, then payload (many binary protocols)Any payload; receiver knows exactly how much to readNot human-readable

Length-prefixed frames

Each frame is a 4-byte unsigned length in network byte order followed by that many payload bytes. Here’s a complete implementation, tested by sending three frames through a socketpair() (a connected pair of local sockets, perfect for tests):

/* framing.c - send_all/recv_all and length-prefixed messages over a stream.
 * Self-test: runs over a socketpair, feeding bytes in awkward chunks. */
#include <errno.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/types.h>

/* Keep calling send() until every byte is written. 0 on success, -1 on error. */
int send_all(int fd, const void *data, size_t len)
{
    const char *p = data;
    while (len > 0) {
        ssize_t n = send(fd, p, len, MSG_NOSIGNAL);
        if (n == -1) {
            if (errno == EINTR)
                continue;
            return -1;
        }
        p += n;
        len -= (size_t)n;
    }
    return 0;
}

/* Read exactly len bytes. 1 = done, 0 = peer closed early, -1 = error. */
int recv_all(int fd, void *data, size_t len)
{
    char *p = data;
    while (len > 0) {
        ssize_t n = recv(fd, p, len, 0);
        if (n == 0)
            return 0;
        if (n == -1) {
            if (errno == EINTR)
                continue;
            return -1;
        }
        p += n;
        len -= (size_t)n;
    }
    return 1;
}

#define MAX_FRAME 65536

/* Frame = 4-byte big-endian length, then that many payload bytes. */
int send_frame(int fd, const void *payload, uint32_t len)
{
    uint32_t header = htonl(len);
    if (send_all(fd, &header, sizeof header) == -1)
        return -1;
    return send_all(fd, payload, len);
}

/* Returns payload length, or -1 on error/close. buf must hold MAX_FRAME bytes. */
long recv_frame(int fd, void *buf)
{
    uint32_t header;
    if (recv_all(fd, &header, sizeof header) != 1)
        return -1;
    uint32_t len = ntohl(header);
    if (len > MAX_FRAME)               /* never trust a length from the network */
        return -1;
    if (len > 0 && recv_all(fd, buf, len) != 1)
        return -1;
    return (long)len;
}

int main(void)
{
    int sv[2];
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == -1) {
        perror("socketpair");
        return 1;
    }

    const char *msgs[] = { "hi", "", "a slightly longer message" };
    for (int i = 0; i < 3; i++)
        send_frame(sv[0], msgs[i], (uint32_t)strlen(msgs[i]));

    static char buf[MAX_FRAME];
    for (int i = 0; i < 3; i++) {
        long n = recv_frame(sv[1], buf);
        printf("frame %d: %ld bytes \"%.*s\"\n", i, n, (int)n, buf);
        if (n != (long)strlen(msgs[i]) || memcmp(buf, msgs[i], (size_t)n) != 0)
            return 2;
    }
    close(sv[0]);
    close(sv[1]);
    return 0;
}
frame 0: 2 bytes "hi"
frame 1: 0 bytes ""
frame 2: 25 bytes "a slightly longer message"

The most important line is the size check in recv_frame(). A length field comes from the network, so it’s untrusted input. Without MAX_FRAME, a malicious or buggy peer could send 0xFFFFFFFF and make you try to allocate or read 4 GB.

Framing in a non-blocking server

recv_all() blocks until the whole frame arrives, which a poll()-based server can’t afford. Instead, give each connection its own buffer. Append whatever each recv() returns, then extract as many complete frames as the buffer holds, keeping any leftover partial frame for next time:

/* frame_buffer.c - collect bytes from a non-blocking socket until whole frames arrive. */
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>

#define BUF_CAP (4 + 1024)             /* room for one maximum-size frame */

struct conn_buffer {
    unsigned char data[BUF_CAP];
    size_t used;
};

/* Append freshly received bytes. Returns -1 if they don't fit. */
int buffer_append(struct conn_buffer *b, const void *bytes, size_t n)
{
    if (n > BUF_CAP - b->used)
        return -1;
    memcpy(b->data + b->used, bytes, n);
    b->used += n;
    return 0;
}

/* If a complete frame is buffered, copy its payload out and return its length.
 * Returns -1 if more bytes are needed, -2 if the frame is too big. */
long buffer_take_frame(struct conn_buffer *b, void *out, size_t out_cap)
{
    if (b->used < 4)
        return -1;
    uint32_t len;
    memcpy(&len, b->data, 4);
    len = ntohl(len);
    if (len > out_cap || len > BUF_CAP - 4)
        return -2;
    if (b->used < 4 + (size_t)len)
        return -1;                     /* header here, payload not yet */

    memcpy(out, b->data + 4, len);
    size_t rest = b->used - 4 - len;
    memmove(b->data, b->data + 4 + len, rest);  /* keep leftover bytes */
    b->used = rest;
    return (long)len;
}

int main(void)
{
    /* Two frames, "PING" and "hello", arriving split in odd places. */
    unsigned char wire[] = { 0,0,0,4, 'P','I','N','G', 0,0,0,5, 'h','e','l','l','o' };
    size_t cuts[] = { 3, 6, 9, sizeof wire };

    struct conn_buffer b = { .used = 0 };
    size_t from = 0;
    char payload[1024];
    for (int i = 0; i < 4; i++) {
        buffer_append(&b, wire + from, cuts[i] - from);   /* pretend recv() gave us this */
        from = cuts[i];
        long n;
        while ((n = buffer_take_frame(&b, payload, sizeof payload)) >= 0)
            printf("after chunk %d: frame \"%.*s\"\n", i, (int)n, payload);
    }
    return b.used == 0 ? 0 : 1;
}
after chunk 2: frame "PING"
after chunk 3: frame "hello"

The test feeds two frames split at awkward places (inside the header, inside the payload, across the boundary). Nothing is produced until a frame is complete, and the leftover bytes of the second frame are kept with memmove(). This pattern, a per-connection input buffer plus a “take next complete message” function, is the heart of every event-driven protocol parser.

Designing a small protocol

Framing gives you messages; the payload still needs structure. A typical binary message looks like:

+---------+--------+-------------+------------------------+
| length  | type   | request id  | body ...               |
| 4 bytes | 1 byte | 4 bytes     | (length - 5) bytes     |
+---------+--------+-------------+------------------------+
  • A type byte says how to interpret the body (login, chat message, ping…).
  • A request id lets replies be matched to requests when several are in flight.
  • A version field in the first message lets you evolve the protocol later.

How to encode the numbers and strings in the body is the topic of the next lesson.

Key takeaways

  • Partial sends and receives are normal; wrap them in send_all() / recv_all() loops.
  • TCP has no message boundaries; add them with delimiters, fixed sizes or (usually) length prefixes.
  • Always cap length fields received from the network.
  • Event-driven servers keep a per-connection buffer and extract complete frames as they arrive.

Exercises

  1. Add send_all() to the poll chat server’s relay and switch it to newline-delimited messages using a per-client buffer.
  2. Write recv_line() that reads up to \n using an internal buffer (not one byte per recv() call).
  3. Extend frame_buffer.c with a 1-byte message type after the length, and a dispatcher that prints PING and TEXT messages differently.

Quiz

  1. A sender calls send() once with 5000 bytes over the Internet. How many recv() calls might the receiver need?
  2. Why must recv_frame() check the length against a maximum?
  3. In an event-driven server, what happens to bytes that arrive after the end of a complete frame?
Show answers
  1. Any number from 1 up to 5000; commonly a few, sized by network segments (around 1448–1460 bytes each on Ethernet).
  2. The length is untrusted input. A huge value could exhaust memory or make the server wait forever for data that never comes.
  3. They stay in that connection’s buffer (moved to the front) as the beginning of the next frame.

Finished this lesson?

Mark it complete to track your progress (saved in this browser).