Visualizing google analytics data with three.js
I have been writing posts on express.js, and am now at a point in which I am just making some projects based that include express.js. I have a post on a basic express todo app, a middleware that responds to requests with json, and now the project that I am going to write about in this post that has to do with using three.js to visualizing my google analytics data that I am just calling express_visual_analytics. I think one of the best ways to learn something a little complicated, is to just start building something interesting with it, and learn as I go. That has been the case with this project, and as such it only makes sense that I write about it.
1 - what to know before continuing
1.1 - This is a work in progress
When I am writing this I am working on version 0.1.x of this project, and it may never be a done deal. There are many additional ideas for features that I wanted to add, and I am also relatively new to full stack development.
1.2 - Not meant for deployment at this time.
This project is just one of several projects that I want to mention in my main post on express.js as examples of full stack applications using express. The aim with this project was to just simply make some three.sj models that reflect changes in google analytics data that is imported from CSV files. I do not intent to deploy this project, but I do intent to make it into some kind of offline tool that can be used to make some interesting three.js models that I can then make images, and maybe animations from that can then potential be shard somewhere.
1.3 - Relationship with express_flyjson
This project borrows from a tool, and middle ware that I have development as a separate independent project called express_flyjson. Mainly The tool at /lib/csv_ga_import, and the middleware at /mw/json_fly_va are copyes of what I am also using in that project. Changes and improvements that I make with them here may be adopted into that, and vise versa.
2 - Setup
Because this project is a little complex I will not be getting into how to go about making the whole thing from the ground up. If you are interested in following alone locally it would be best to just clone down the project.
As of this writing I am working on 1.x, if doing this gives you a later version what I am writing about here may be out of date.
3 - importing CSV
To import csv download a csv from the audience overview section in google analytics. At present the library that I have made only imports csv data that is exported for daily user counts there. At some point if I do continue to suport this there may me more options.
Once I have a csv file download I then need to place it in a csv folder at the root of the project folder, if it is not there just created it.
Then I just need to call the main build script at root, and the lib at /lib/csv_ga_import should build a json database at /db/db.json that will be used by the project as the date to be visualized.
3.1 - The main build.js file
The script that I call from the command line to build the json database is located in the root name space and is simply called build.js.
As you can see I use a lib called csv_ga_import which is something I put together in a separate project called express_flyjson.
3.1 - csv importing thanks to csvtojson
The procress of importing csv was not as time consuming as it would have been if I had not used csvtojson, which is a great tool for doing as the name implies. If interested in learning more about that I have a post on csvtojson.
4 - json_fly_va middle ware
The json_fly_va middleware is custom hacked over version of a similar middleware that I developed in my express_flyjson project. ALthoigh I will not get into full detail about the full sorce code of this I will of course cover some of the features of it that I use to make my three.js models.
4.1 - The main index.js of json_fly_va
The main index.js of this middleware creates an instance of the express.js app object by calling the express top level function, and this of course is what is exported by the middleware, and is therefor what is used in the main app.js file via app.use.
4.2 - json_fly_va/response_send_tab for sending tabulated totals
This is a part of the midddleware that allows for be to make queries where I can give a start date, a number of days, and a count of number of days to go back from the start date to give tabulated totals of users. So in other words if I want to go back 28 days from a start date, and break those 28 days down into 7 day chunks, and have a total for each 7 day chunk of that 28 day total this will come in handy.
So then this kind of query…
… will give me this json.
5 - The Public folder
6 - The themes folder
The themes folder as the name suggests is where I place my ejs templates when it comes to rendering a theme. As of this writing there is only one theme that is in use that is based on jQuery and bootstrap rather than the usual angular powered theme common in mean stack applications. At a later point I might make an angular powered theme, but for now I am happy with anything that just works, as this is not the bottom line of this project.
7 - The main app.js file
The main app.js file in this project makes use of static paths, sets some settings for the port, and ejs rendering that is used in some paths. Here is also where the /flyjson path is defined that is used by my client to get the data that is used in this project as well.
8 - Current lists of Models, or Works
Here is a current list of all the works that I have so far. So far there are four works that I made for 0.x, and am currently working on 1.x where I have a few more as well.
8.1 - The first Model that I just call ‘First’
For the first project I made a model that just shows bars for each day of a standard start day to end day query.
When looking at this model it shows in great detail how much traffic to this site has grown from when I first started in early 2017.
8.2 - The ‘over_time’ project
This one is just like the first project, but I used a different material that changes over time based on the date property of the day objects that are given in a query.
8.3 - ‘spiraled’
In this project I have the day objects broken down into parts for each month, and have them spaced out in a spiraled pattern from the center outward, where the center most groups are the oldest in the query.
8.4 - ‘spiraled_canvas’
This is the same as spiraled only I experimented with using canvas elements to generate a simple texture that I thne use as a material in the model.
9 - Conclusion
I did not cover everything there is to write about this project as well as the additional projects that I used to help speed things along. If you are interested in reading more about express.js my main post on express would be a good starting point, I also have many posts on three.js as well.