Data Serialization Options

Basics

But, why?

Why? (contd.)

Thoughts going into this:

save on bandwidth
save on cpu usage

Does Gzip solve all these problems?

Not quite.

Gzip from the client to server is not recommended. https://stackoverflow.com/a/48572297

Examples of Scale

Link

Formats

JSON

https://www.json.org/

// Example:
{
  "name": "Something",
  "weight": 42,
  "contents": [
    "widgets",
    "cogs"
  ]
}

XML

https://w3.org/XML/

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<root>
  <name>Something</name>
  <weight>42</weight>
  <contents>widgets</contents>
  <contents>cogs</contents>
</root>

Warning

There be ~~dragons~~ binary

The following string representations are only for visualization purposes.

Avro

https://avro.apache.org/

Something(Bwidgetscogs

Avro Schema

const example1K = avro.Type.forSchema({
  type: 'record',
  fields: [
    { name: 'id', type: 'string' },
    { name: 'index', type: 'int' },
    { name: 'guid', type: 'string' },
    { name: 'isActive', type: 'boolean' },
    { name: 'balance', type: 'string' },
    { name: 'picture', type: 'string' },
    { name: 'age', type: 'int' },
    { name: 'eyeColor', type: 'string' },
    { name: 'name', type: 'string' },
    { name: 'gender', type: 'string' },
    { name: 'company', type: 'string' },
    { name: 'email', type: 'string' },
    { name: 'phone', type: 'string' },
    { name: 'address', type: 'string' },
    { name: 'about', type: 'string' },
    { name: 'registered', type: 'string' },
    { name: 'latitude', type: 'float' },
    { name: 'longitude', type: 'float' },
    { name: 'greeting', type: 'string' },
    { name: 'favoriteFruit', type: 'string' },
    { name: 'tags', type: {type: 'array', items: 'string'} },
    {
      name: 'friends',
      type: {
        type: 'array',
        items: {
          type: 'record',
          fields: [
            { name: 'id', type: 'int'},
            { name: 'name', type: 'string'},
          ]
        }
      }
    },
  ]
});

Avro Schema (contd.)

const exampleArraySchema = avro.Type.forSchema({
  type: 'record',
  fields: [
    {
      name: 'data',
      type: {
        type: 'array',
        items: exampleSchema
      }
    }
  ],
});

Bencode

https://en.wikipedia.org/wiki/Bencode

Pronounced “B-encode”
Used by the Bittorrent protocol

d8:contentsl7:widgets4:cogse4:name9:Something6:weighti42ee

BSON

http://bsonspec.org/

Used by MongoDB
Pronounced [bee · sahn]
Stands for “Binary JSON”

Oname
Somethingweight*contents 0widgets1cogs

MessagePack

https://msgpack.org/index.html

��name�Something�weight*�contents��widgets�cogs

Protobuf

https://developers.google.com/protocol-buffers/

By Google

Something(Bwidgetscogs

Protobuf Schema (proto)

syntax = "proto3";

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

Protobuff Schema (json)

const root = protobuf.Root.fromJSON({
  nested: {
    Friend: {
      fields: {
        id: {
          type: 'int32',
          id: 1,
        },
        name: {
          type: 'string',
          id: 2,
        },
      }
    },
    Example: {
      fields: {
        // Primary
        id: {
          type: 'string',
          id: 1,
        },
        index: {
          type: 'int32',
          id: 2,
        },
        guid: {
          type: 'string',
          id: 3,
        },
        // Non-Primary
        isActive: {
          type: 'bool',
          id: 20,
        },
        balance: {
          type: 'string',
          id: 21,
        },
        picture: {
          type: 'string',
          id: 22,
        },
        age: {
          type: 'int32',
          id: 23,
        },
        eyeColor: {
          type: 'string',
          id: 24,
        },
        name: {
          type: 'string',
          id: 25,
        },
        gender: {
          type: 'string',
          id: 26,
        },
        company: {
          type: 'string',
          id: 27,
        },
        email: {
          type: 'string',
          id: 28,
        },
        phone: {
          type: 'string',
          id: 29,
        },
        address: {
          type: 'string',
          id: 30,
        },
        about: {
          type: 'string',
          id: 31,
        },
        registered: {
          type: 'string',
          id: 32,
        },
        latitude: {
          type: 'float',
          id: 33,
        },
        longitude: {
          type: 'float',
          id: 34,
        },
        tags: {
          type: 'string',
          id: 35,
          rule: 'repeated'
        },
        friends: {
          id: 36,
          rule: 'repeated',
          type: 'Friend'
        },
        greeting: {
          type: 'string',
          id: 37,
        },
        favoriteFruit: {
          type: 'string',
          id: 38,
        },
      }
    },
    Examples: {
      fields: {
        data: {
          type: 'Example',
          id: 1,
          rule: 'repeated'
        }
      }
    }
  }
});

Others (unimplemented here)

Flatbuffers
Thrift
YAML
SOAP

Testing Methodology

Different Object Sizes

medium (~10kb)
large (~100kb)

Metrics

Content-Length (Uncompressed)

Content-Length (Gzipped)

Library overhead

Browser Runtimes

Browser: Encoding 10K

Browser: Encoding 100K

Browser: Decoding 10K

Browser: Decoding 100K

Node CPU Runtime

Node: Encoding 10K

Node: Encoding 100K

Node: Decoding 10K

Node: Decoding 100K

Other Languages

Go

https://github.com/alecthomas/go_serialization_benchmarks

Java

https://dzone.com/articles/is-protobuf-5x-faster-than-json

Summary

Browser Speed

Nothing beats JSON

Desktop Browser Inbound Data

You can save 8% with Avro (after gzip)

Desktop Browser Outbound Data

You can save up to ~30% with Avro
but that data is free anyway

Mobile Data

Data is not free for the user (in most cases)

mobile games
realtime stock or cryptocurrency data
bandwidth limited IoT

Server to Server

Speed* depends on your server languages
AWS still charges for outbound “Inter-region” data transfer
Gzip can almost always help (when the source is trusted)