When making a command line interface program in node.js that needs to walk a file system recursively there are many options. If you do want to work within the core set of node.js modules without installing any additional from npm there is of course the fs.readdir method in the file system module that may be of interest. However in this post I will be writing about an npm package option that I seem to like a little for this known as klaw, that can also be used with another popular project known as through2. I will be giving file system walking examples mainly using this, but will also touch base on some alternatives as well.
In this post I am using klaw 2.1.1 which as of july 2018 is still the latest version, and there has not been much activity in the repo for the past few months. That is not a deal breaker for me as long as it seems that the project is still fairly solid, and so far I can not say I have not run into any problems with klaw.
I put together some examples of klaw that have to do with walking a file system folder. Like many of my posts on an npm package I make a test folder, and install the package along with any additional packages I might need to make some demos. For the examples in this section the only additional package I am using is throught2.
For a basic.js file example I made a quick script that just walks the root name space of a path that I give as the first argument in the command line, else it defaults to the current working directory.
I am so far only using a single option when calling klaw that can be used to set the depth level of recursion when it comes to walking through a file system, a key feature that just about any walker should have.
One of the most important aspects of klaw is the data event, as this is what will be called for each file that is not filtered out before hand, more on that later in this post. A single object is given for the method that is provided as the callback for this event that contains the absolute path of the item, as well as the stats for that item.
The stats object is an instance of what is returned if I where to call an fs.stat for each file, there is no need to do that as it is done foe each file by default. The path given is always an absolute path, as it should be for something like this.
By default klaw uses the fs module replacement package known as graceful-fs, If I want to have klaw use something else such as fs-extra for example, then I just need to pass it as the fs option when using klaw.
So then I just need to install it.
And then pass it as the file system module via the fs option.
So as you can see fs-extra makes each of the file system methods return a promise, if I comment one the fs option this will of course result in an error. If interested I have written a post on fs-extra a while back if you want to read more about that fs module drop in replacement.
In this post I am also using through2 as a way to make streams to use with the pipe method. If for some reason I do not want to use through2 one alternative is to just directly use stream.Transform in object mode.
The projects readme file recommends that I use through2 to to quickly help with more advanced filtering operations that, may involve preforming one or more steps, some of which may require additional file io operations such as reading the actual contents of a file rather than just the meta data. Turns out that through2 is a fairly popular project, so I thought I would give it a try.
So if I want to filter by extension I can pass a method to the through2 object method that will return
In many cases I might want to filter files based on the content of a file as well. For example I with html files I might only want files that have a certain meta tag, and I also might want to append some properties to the item object as well.
For this I can use another npm project that is useful for working with html content, without having to have a working browser environment in node.js, such as cases like this called cheerio.
This works great at getting all the files that meat that criteria. I could expand this into some kind of actual useful tool that will give me information about a public html folder such as total site wide word count for all blog posts, or something to that effect, but you get the idea.
So now that I have covered how to use klaw as a node.js file system walker solution, it might be a good call to briefly take a look at some alternatives.
One of the first file systems walkers I have come across is nodedir, and I have written a post on this on a while back. As of this writing it would look as though the project is no longer supported, as there has not been a single commit at least for over a year now. Still the main method of interest does seem to work okay for what it was designed to do if you want to give it a try anyway.
This solution works okay, but I can not say I would prefer it over klaw these days. I have written a post on node-dir a while back if you are interested in reading more about this one anyway for whatever the reason.