↓ Skip to Content Start of content

How it's Made

At the moment these reports are generating using two main sources:

The process of building new reports is a little long-winded, but thankfully it’s almost entirely automated so it only takes a few commands to build.

Importing Data

The first step in the process is to put all the data into a more usable format, in this case an SQLite database. It adds a few extra steps, but makes answering questions like “How many hours were spent using browsers in March 2012” much easier to answer.

RescueTime provides an API, so getting data from them isn’t too difficult, and it’s stored in the database in pretty much the same format. Org data takes a few more steps to process.

Org-Mode supports plenty of export options, but none of them really gave me the details I needed. Emacs is about as extendable as you could ever want, but my lisp is a little rusty so I used Perl with Org::Parser to extract tasks from my org directory. Tasks (and their children) are stored in a JSON file, which is then read by a PHP script and stored in the database. It’s not a very elegant system, but it does the trick.

In case you’re interested, the final layout looks a little like this:

CREATE TABLE `activity` (
    id INTEGER PRIMARY KEY AUTO_INCREMENT,
    `timestamp` INTEGER,
    `duration` INTEGER,
    `activity` TEXT,
    `category` TEXT,
    `productivity` INTEGER
);

CREATE TABLE `task` (
    `id` INTEGER PRIMARY KEY AUTO_INCREMENT,
    `file` TEXT,
    `title` TEXT,
    `state` TEXT,
    `is_archived` INTEGER,
    `deadline` INTEGER,
    `scheduled` INTEGER,
    `added` INTEGER,
    `closed` INTEGER
);

There are also tables to store properties, tags and tracked time.

Processing

The main processing script is written in PHP, and uses RedBean for the database layer and Twig for templates. Templates were originally just PHP code, but after a while they became unwieldy as more processing code was shoved inside them. This is a Bad Thing™, so converting templates to Twig forced the querying code into better defined objects.

Once all the various files and fragments have been created, the site HTML is built using Jekyll and then deployed to the site. The whole process takes less than a minute, and most of that is spent parsing org. If no new data needs loading, it takes under 10 seconds to build & deploy.

All of this is wrapped in a Phing build script, so the whole process takes a few keystrokes to execute.

$ phing fetch:stats build:stats build:site deploy:site

Software & Libraries

The following libraries and tools are used during generation: