When node.js was first developed there where no typed arrays such as Uint8Array to help work with binary data. As such Buffer was introduced to help work with binary data in a node.js environment. Buffers are something that I run into when working with streams, ether file io streams, or from http requests. In any case Buffers are helpful when doing anything that involves working with raw binary data. So lets take a look at some examples of buffers in node.js.
THis is a post on using buffers in nodejs to work with binary data in a node.js environment. There is a lot to know about when working with buffers, such as how to create them in the first place, and how it has changed compared to older versions of node.js. Anyway I hope that you have at least some background with node.js, and javaSciprt in general, as I will not be coving the basics of those topics here.
In this post I was using nodejs 8.x LTS which at the time of this writing is still supported, but it coming to the end of its support cycle. If you are using an older or newer version of nodejs what is written here may not be applicable.
So lets start out with some very basic examples of buffers that have to do with just creating a new buffer. There is more than one way to make them, and in both safe, and unsafe ways as well. There is also the old way of making them that you might only bother with if you are still using a real old version of node.js for some reason. So lets get started with some buffer hello world examples of making a new buffer instance in nodejs.
So the old way of making a buffer was to use the Nodejs Buffer global like that of any old jaavScript constructor method. In other words like that of the Date constructor that is used with the new keyword to create a new instance of Date. I call Buffer with the new keyword, and pass whatever it is that I want placed in a buffer.
This should not be used any more unless for some reason you are using a real old version of node.js that does not support the newer ways to make a buffer, in which case I guess you have to. This way of creating a new buffer still shows up a lot with code examples on the open web, but it does open up some security concerns, and it has been deprecated when it comes to more modern versions of nodejs. I will not be getting into this in detail in this post as I have done so in another post on creating a new buffer with the new keyword, you can check that out if you want to.
As of late (node 8.x+) it is advised to not use the Buffer constructor directly, in fact doing so is depreciated, instead when dealing with buffers the various methods of Buffer are what should be used in order to work with buffers. For example the buffer from method works great for quickly creating a new buffer with an initial value that is derived from a string.
So then Buffer.from is one such method and in most cases however maybe it is desired to create a blank buffer that is filled with zeros, or maybe you do not event want to do that. That being said there are of course other options such as Buffer.alloc, and Buffer.allocUnsafe.
The Buffer.alloc method is great for just setting up a buffer of a fixed byte size in a safe way that will assure that noting old in memory will end up being part of the buffer.
If I want to I can also give a string to pre fill the buffer with, and give a character encoding to observe when parsing the string to a buffer.
So it might be fun to play around with unsafe buffers to see if I can make sense of old data in memory, and find out if they really are unsafe. I can understand why in theory at least, and it is not to hard to reproduce why. Just try out on yourself, and then look at what you get, a whole bunch of old data in memory. Compare that to what happens when you just use Buffer.alloc, and you get nothing but zeros.
Concatenating buffers is just a matter of using Buffer.concat. This method works bu just calling it, and passing it an array of two or more buffers. The method will figure out the length of the new buffer that will be returned, this will take a little overhead as you might expect, so if you do know what the length of the result will be before hand it would make sense to pass that as the second argument when calling the method.
It is possible to set the encoding that should be used when making a buffer. This can be important when say, creating a buffer from a string that contains characters that are above the ascii range. The number of bytes that a character takes up is different depending on the character encoding. The ascii character encoding is easy to understand, because it is just a single byte per character, however that is not how Unicode works. With Unicode that number of bytes can range from one upwards depending on what the character is.
So one of the big things about buffers is that they comes up a lot when working with files. When using the file system module, or any kind of readable, writable, or duplex stream involving data from the file system buffers are the node.js standard way of working with binary data stored in on then hard drive. So lets take a look at a few examples that involve files\
So when it comes to reading a file using the fs.createReadStream method in the file system module, this will result in a readable stream. This has some events that can be used to define a callback that will fire each time some more chunks of data come along in the readable stream. The chunks are of course instances of buffer, and now that we know a thing or two about buffers we should have no problem working with them when it comes to doing something with a readable stream.
There is of course writable streams as well, for example when using the fs.createWriteStream method, a buffer can be given as an argument to the write method. When creating, or appending a file.
With this example I can use this script from the command line like this if it is called write.js, and the above example on read streams is called read.js.
I hope you found this post on buffers a little helpful, if you did not know much about them before hand. If you enjoyed reading this post you might want to also check out my post on using request to work with streams when scripting http. Buffers do come up a lot when it comes to working with file io, streams, and just about anything that has to do with binary data, so this is one aspect of node.js development that developers should get solid.