I have been making a few threejs videos lately in which I am testing out how to go about syncing video with audio. Thus far though I am just working out the timing in terms of time stamps and duration and thus I am not taking into account any kind of actual audio sample data to adjust things that are going on when rendering my frames. However I have found that I can export audio sample data in an HTML file format when using Audacity to work out the audio tracks that I want in this video. So I can adjust the sample rate so that I have just one sample per frame, and then I can export the sample data in an HTML file in which I have at least one sample point for each frame of the video. I can then also do this on a track by track basis, so that I have an HTML file of sample data for say drums, then another for bass, and yet another of samples, and so forth.
I then just need to make another threejs examples project that will use something like the THREE.FileLoader to load these HTML files of audio sample data. The just do a little additional work to create a kind of standard object of this data that I extract from the HTML files. I can then use an array of audio sample numbers that have been adjusted to be in the range of 0 and 1 as alpha values for anything and everything that will call for such a value to adjust things like scale, position, rotation, and any additional effects for any module that I am using and so forth. In other words the goal here is to create music visualization using threejs, and audio sample data exported from Audacity.
The source code that I am writing about here as well as the audio sample data files can be found in my test threejs repository on Github. This is also where I am parking the sample data export files that I am loading in the demos for this post as well. This repo is also where I am parking the source code, notes, assets, and much more that I am using for all my other blog posts on threejs as well.
When I first wrote this post I was using r146 of threejs, and things where working fine on my end with that revision.
In this section I will be going over the source code of the first revision of the module as well as a few demos of the module as well. The main goal with this first version is to just simply load one or more html files that contain sample data, and create a result object. This result object will then have processed data for each file at the ready to work with by making use of additional public methods, or just directly working with the arrays of sample data when making a video project.
I am all ready running into a lot of things that I would like to do when it comes to future revisions, but for now I will just be exploring what these things are to some extent when it comes to the demos that make use of this first revision.
The main method of interest for getting started with this is the load public method of the module. When I call this I give a base URL where I store the sample data files that I exported from Audacity. On top of giving a base URL I then give a list of HTML files that I would like to load from that base url. The file names for these files do very much matter a lot as the string values of the file names are what i will be using for key values of the resulting object that will contain the final sample objects.
I am using the THREE.Fileloader to load the HTML files of sample data, and all seems to go well with that, but then I had to find a way to pare the HTML into a workable DOM Object. When it comes to this there must be some kind of browser built in way to do this sort of thing now, and yes there is, what I want is the DOMParser. So I pass the HTML string to the DOMParser and the returned result is then a DOM object that I can use the variousDOM methods on such as query selector all to help me extract the sample data that I want into an array.
When it comes to parsing the sample data into an array I am all ready sure that I will want more than one array for a sample object actually. I do not need to get all this nailed down for this first revision just yet as I am sure much of this I can leave for future revisions. So for now in terms of the core of the module at least I will have a raw, and abs array. As the name suggests the raw array is just the raw data extracted from what is in the html file, while the abs array is short for absolute value in which I am creating an array in which all the numbers are positive. When it comes to anything and everything else I can work methods that create other arrays from these arrays when making my demos for this module. What i work out and refine with them might then be baked into future revisions.
For this demo I wanted to test out using more than one audio sample data file so I am calling the load method of the module and passing an array of two files one for a base line and another for drums. I am then using the final audio data to effect the scale position and rotation of a mesh of a cube.
The idea came to me that in some cases I might want to have an array where each element is the sum of the last element plus the current element. I can then get the max value of this array of values and use tht as a way to get an alpha value. This can then be used in situations in which I want to use sample data to effect something over a long period of time. For example say I want to move a box form one side of a scene to the other, over the length of the full duration of a video, but do so in sync with an audio track of a drum beat.
I have found that often the reactions are a little jumpy, that is that the scale for example will jump all the way up and then back down often over the course of just one or two frames. I am thinking that a better visual effect is the the scale will jump up, but then slowly fall bak down over at least a little time. So I thought that maybe one way to go about something things out would be to have a method where I am going by not just a current sample point, but also a mean of the current sample as well as a few samples back. This seems to work well when it comes to a desire to smoothing things out a little.
The module is working okay as I have it all ready, but I can all ready see that I am going to want to make at least one revision in which I bake some of the things that I have worked out in my demos into the module itself. There is also not just loading and processing sample data, but also what it is that I can do with just data. I would like to have better methods for things like certain paths that are the result of sample data and things to that effect, however maybe all of that is something that should all be part of another project.