Lesson 16 of 18 Data on the Wire

Serialization: Packing Binary Data

Send integers, floats and strings safely between machines: text vs binary, why raw structs break, portable big-endian packing in C, and IEEE 754 floats.

45-55 min advanced 0/18 lessons done
Course contents Lesson 16 of 18 · 0 done
On this page (11 sections)
  1. What you will learn
  2. Three approaches
  3. Encoding integers
  4. Encoding floats
  5. Strings
  6. The complete program
  7. Hardening the decoder
  8. Or use an existing format
  9. Key takeaways
  10. Exercises
  11. Quiz

What you will learn

  • Three ways to send structured data and their trade-offs.
  • Why send(fd, &my_struct, sizeof my_struct, 0) is a bug waiting to happen.
  • Portable encoders and decoders for unsigned and signed integers, floats and strings.
  • When to reach for an existing format instead of rolling your own.

Three approaches

Say you need to send a reading: sensor id 4021, offset -17, temperature 21.375, room “server-room”.

1. Text. Format it: "4021 -17 21.375 server-room\n". Easy to debug with nc, works across every language. Costs: more bytes, parsing work on both ends, and floating-point values must be printed with enough digits to survive the round trip.

2. Dump the raw struct. Tempting, fast, and fragile:

struct reading r = { 4021, -17, 21.375f, "server-room" };
send(fd, &r, sizeof r, 0);   /* don't */

It breaks when the two ends differ in byte order, type sizes (long is 4 bytes on Windows, 8 on 64-bit Linux), or padding (compilers insert invisible gaps to align fields, and different compilers or flags insert different gaps). It also leaks whatever garbage sits in padding bytes.

3. Explicit binary encoding. Write each field into a byte buffer in a fixed, documented layout. Compact and fast like option 2, portable like option 1. That’s what we build below.

Encoding integers

Shift and mask each byte into place, most significant first (big-endian, matching network order). This never depends on the host’s byte order, alignment or padding:

static unsigned char *put_u32(unsigned char *p, uint32_t v)
{
    p[0] = (unsigned char)(v >> 24);
    p[1] = (unsigned char)(v >> 16);
    p[2] = (unsigned char)(v >> 8);
    p[3] = (unsigned char)v;
    return p + 4;
}

Returning the advanced pointer lets you chain calls: p = put_u32(p, a); p = put_u16(p, b);.

Signed integers. Every machine you’ll meet uses two’s complement, so send the bit pattern as unsigned. Decoding back needs care, because converting an out-of-range unsigned value to a signed type is implementation-defined in older C standards; the helper below does it arithmetically.

64-bit integers follow the same pattern with eight shifts.

Encoding floats

Two practical options:

  • Copy the IEEE 754 bits. memcpy the float into a uint32_t and send that integer. Every mainstream CPU uses IEEE 754 for float and double, so this is portable in practice and exact. (If you ever target a platform without IEEE 754, you’d convert with frexp()/ldexp() into sign, exponent and mantissa yourself.)
  • Fixed point. If you know the precision you need, send an integer: 21.375 °C becomes 2138 hundredths. Simple and language-neutral, but you choose the precision up front.

Never cast the pointer (*(uint32_t *)&f): it violates C’s aliasing rules. memcpy is the correct, and equally fast, way.

Strings

Send a length followed by the bytes, without the C terminator. Our encoder uses a 16-bit length, limiting strings to 65,535 bytes. On decode, clamp to the destination size: the length is untrusted input.

The complete program

/* pack.c - portable binary encoding of integers, floats and strings. */
#include <math.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

/* ---- unsigned integers, big-endian (network order) ---- */
static unsigned char *put_u16(unsigned char *p, uint16_t v)
{
    p[0] = (unsigned char)(v >> 8);
    p[1] = (unsigned char)v;
    return p + 2;
}
static unsigned char *put_u32(unsigned char *p, uint32_t v)
{
    p[0] = (unsigned char)(v >> 24);
    p[1] = (unsigned char)(v >> 16);
    p[2] = (unsigned char)(v >> 8);
    p[3] = (unsigned char)v;
    return p + 4;
}
static const unsigned char *get_u16(const unsigned char *p, uint16_t *v)
{
    *v = (uint16_t)((p[0] << 8) | p[1]);
    return p + 2;
}
static const unsigned char *get_u32(const unsigned char *p, uint32_t *v)
{
    *v = ((uint32_t)p[0] << 24) | ((uint32_t)p[1] << 16) |
         ((uint32_t)p[2] << 8) | (uint32_t)p[3];
    return p + 4;
}

/* ---- signed integers: send the two's-complement bit pattern ---- */
static unsigned char *put_i32(unsigned char *p, int32_t v)
{
    return put_u32(p, (uint32_t)v);
}
static const unsigned char *get_i32(const unsigned char *p, int32_t *v)
{
    uint32_t u;
    p = get_u32(p, &u);
    /* Convert without relying on implementation-defined casts. */
    *v = u <= INT32_MAX ? (int32_t)u : -(int32_t)(UINT32_MAX - u) - 1;
    return p;
}

/* ---- floats ----
 * Option A: copy the IEEE 754 bits (fine when both ends use IEEE 754 floats,
 * which is true of every mainstream CPU today). */
static unsigned char *put_f32(unsigned char *p, float f)
{
    uint32_t bits;
    memcpy(&bits, &f, sizeof bits);
    return put_u32(p, bits);
}
static const unsigned char *get_f32(const unsigned char *p, float *f)
{
    uint32_t bits;
    p = get_u32(p, &bits);
    memcpy(f, &bits, sizeof *f);
    return p;
}

/* Option B: fixed point. Send temperature 21.37 as the integer 2137. */
static int32_t to_centi(double x) { return (int32_t)lround(x * 100.0); }

/* ---- strings: 16-bit length, then the bytes (no terminating zero) ---- */
static unsigned char *put_str(unsigned char *p, const char *s)
{
    size_t len = strlen(s);
    p = put_u16(p, (uint16_t)len);
    memcpy(p, s, len);
    return p + len;
}
static const unsigned char *get_str(const unsigned char *p, char *out, size_t cap)
{
    uint16_t len;
    p = get_u16(p, &len);
    size_t n = len < cap - 1 ? len : cap - 1;
    memcpy(out, p, n);
    out[n] = '\0';
    return p + len;
}

struct reading {
    uint32_t sensor_id;
    int32_t  offset;
    float    celsius;
    char     room[32];
};

static size_t encode_reading(unsigned char *buf, const struct reading *r)
{
    unsigned char *p = buf;
    p = put_u32(p, r->sensor_id);
    p = put_i32(p, r->offset);
    p = put_f32(p, r->celsius);
    p = put_str(p, r->room);
    return (size_t)(p - buf);
}

static void decode_reading(const unsigned char *buf, struct reading *r)
{
    const unsigned char *p = buf;
    p = get_u32(p, &r->sensor_id);
    p = get_i32(p, &r->offset);
    p = get_f32(p, &r->celsius);
    get_str(p, r->room, sizeof r->room);
}

int main(void)
{
    struct reading in = { 4021, -17, 21.375f, "server-room" }, out;
    unsigned char wire[128];
    size_t n = encode_reading(wire, &in);

    printf("encoded %zu bytes:", n);
    for (size_t i = 0; i < n; i++)
        printf(" %02x", wire[i]);
    printf("\n");

    decode_reading(wire, &out);
    printf("decoded: id=%u offset=%d celsius=%.3f room=%s centi=%d\n",
           out.sensor_id, out.offset, out.celsius, out.room, to_centi(out.celsius));
    return !(out.sensor_id == in.sensor_id && out.offset == in.offset &&
             out.celsius == in.celsius && strcmp(out.room, in.room) == 0);
}
encoded 25 bytes: 00 00 0f b5 ff ff ff ef 41 ab 00 00 00 0b 73 65 72 76 65 72 2d 72 6f 6f 6d
decoded: id=4021 offset=-17 celsius=21.375 room=server-room centi=2138

Reading the bytes: 00 00 0f b5 is 4021; ff ff ff ef is -17 in two’s complement; 41 ab 00 00 is the IEEE 754 encoding of 21.375; 00 0b is the string length 11, followed by the ASCII of “server-room”. Compile with gcc pack.c -o pack -lm (lround lives in the math library).

Hardening the decoder

The decoder above trusts that the buffer is long enough. A real one, reading data from the network, must also:

  • track how many bytes remain and fail cleanly instead of reading past the end;
  • reject string lengths larger than what remains;
  • validate enum-like fields (an unknown message type is an error, not a crash).

Every decoding bug is a potential security hole, because the attacker controls the input.

Or use an existing format

For anything beyond a small protocol, consider a well-tested format: Protocol Buffers, FlatBuffers, Cap’n Proto, MessagePack, CBOR, or plain JSON when bandwidth doesn’t matter. They handle versioning, optional fields and cross-language support. Writing your own encoder, as here, is still the best way to understand what they do.

Key takeaways

  • Never send raw structs: byte order, type sizes and padding differ between machines.
  • Encode integers big-endian with shifts; send signed values as their two’s-complement bits.
  • Send floats as IEEE 754 bits via memcpy, or as fixed-point integers.
  • Prefix strings with a length and validate every length you decode.

Exercises

  1. Add put_u64() / get_u64() and a double encoder, and extend the reading with a 64-bit timestamp.
  2. Make decode_reading() take a buffer length and return -1 if the data is truncated anywhere. Test it with every prefix length from 0 to 24.
  3. Combine this lesson with lesson 15: send three encoded readings as length-prefixed frames over a socketpair() and decode them on the other end.

Quiz

  1. Name three reasons sending a raw C struct over a socket is unsafe.
  2. Why use memcpy rather than a pointer cast to get a float’s bits?
  3. A peer sends a string length of 60,000 but your buffer holds 32 bytes. What should the decoder do?
Show answers
  1. Different byte order, different sizes of types such as long, and different (and uninitialised) padding between fields.
  2. Accessing a float through a uint32_t * breaks C’s strict aliasing rules and can be miscompiled; memcpy is well defined and optimised to the same instruction.
  3. Treat it as untrusted: copy at most what fits (or reject the message), and never read past the end of the received data.

Finished this lesson?

Mark it complete to track your progress (saved in this browser).