Buffer Overview in Node.js

Everyone who worked in Node.js, should have came across the term Buffer. Few may think that Buffer is only for library developers and its not in the scope of application developer. Here this blog is for you to provide an overview about the Buffer and its usage.

In Node.js, Buffer objects are used to represent the binary data, which is used for data transmission in Node.js. Usage of Buffer is supported in Streams, File system and other Node.js internal API's. To interact with operating system (OS) using Node.js File API, the interactions were happens as a binary data which OS generally understand.

Common usage Buffer Object

Listing out some the common usage of Buffer object

Streams API
File System API
Crypto API
Http/Https API
Character Encoding (base64, utf8, utf16 and other supported encoding)
Image Processing

Can I convert binary data (Buffer) as a String ?

To convert Buffer as a String, Buffer uses character encoding for the conversion. Default character encoding is utf8, which also supports options for utf16le, latin1 and others.

In simple words, Once the buffer is processed to String it is encoded to the human readable string format.

Hands on with Buffer

To create a new buffer instance require the buffer global object and create instance using

Buffer.from()

to decode the buffer use

Buffer.toString()

. Following snippet illustrates the examples for creating Buffer instance with different character encoding and with decoding.

const buffer = require('buffer').Buffer;

// Encoding : Creates a Buffer containing default utf8 string
let encBuffDefaultUtf8 = buffer.from("hello world");
console.log(encBuffDefaultUtf8); // Output: <Buffer 68 65 6c 6c 6f 20 77 6f 72 6c 64>

// Decoding : Covert a Buffer to utf8 string.
let decBuffDefaultUtf8 = encBuffDefaultUtf8.toString()
console.log(decBuffDefaultUtf8); // Output: hello world

// Encoding : Creates a Buffer containing the Latin-1 bytes [0x74, 0xe9, 0x73, 0x74].
const encBuffLatin = Buffer.from('tést', 'latin1');
console.log(encBuffLatin) // <Buffer 74 e9 73 74>

// Decoding : Covert a Buffer to Latin-1 string.
const decBuffLatin = encBuffLatin.toString('latin1');
console.log(decBuffLatin) // tést

The usage of Buffer in https/https API returns the Buffer object chunk (partial data) on each event and its stored into an Array. Once the stream ends Buffer object is concatenated using

Buffer.concat()

and then parsed back to JSON

const https = require('https');
https.get("https://jsonplaceholder.typicode.com/users", 
    { rejectUnauthorized: false }, (res) => {
        let data = []; 
        // Response stream
        res.on('data', (chunk) => {
            // Check the type of chunk is Buffer, if not create as a buffer object
            const isBuffer = Buffer.isBuffer(chunk);
            console.log(`Is Buffer : ${isBuffer} - Chunk :`, chunk);
            let bufferChunk = isBuffer ? chunk :  Buffer.from(chunk)
            data.push(bufferChunk);
        })
        // On end of the response, concatnate the Buffer and parse it to JSON object result
        res.on('end', () => {
            let result = Buffer.concat(data)
            result = JSON.parse(result);
            // console.log("Result :", result);
        })
    }
)

Buffer Performance Benchmark

Consider the scenario, which you want to convert an Array as a String, users prefers to use

Array.toString()

but using

Buffer.toString()

is more faster.

const Benchmark = require('benchmark');
var suite = new Benchmark.Suite;

let input = [28929,53068,96487,73074,8615,13068,86607,60890,44297,57022];

suite.add('Buffer.toString', function() {
  Buffer.from(input).toString()
})
.add('Array#toString', function() {
    input.toString()
  })
// add listeners
.on('cycle', function(event) {
  console.log(String(event.target));
})
.on('complete', function() {
  console.log('Fastest is ' + this.filter('fastest').map('name'));
})
// run async
.run({ 'async': true });

Benchmark Result

The usage of Buffer is wrapped into most of the Node.js core module and most of the popular libraries. In order to get hands on in it try implementing it whenever you found a right usecase

For better understanding check out this blog Buffer in Node.js