How it's Made
At the moment these reports are generating using two main sources:
- emacs org files
- RescueTime data
The process of building new reports is a little long-winded, but thankfully it’s almost entirely automated so it only takes a few commands to build.
Importing Data
The first step in the process is to put all the data into a more usable format, in this case an SQLite database. It adds a few extra steps, but makes answering questions like “How many hours were spent using browsers in March 2012” much easier to answer.
RescueTime provides an API, so getting data from them isn’t too difficult, and it’s stored in the database in pretty much the same format. Org data takes a few more steps to process.
Org-Mode supports plenty of export options, but none of them really gave me the
details I needed. Emacs is about as extendable as you could ever want, but my
lisp is a little rusty so I used Perl with Org::Parser
to extract tasks from
my org directory. Tasks (and their children) are stored in a JSON file, which is
then read by a PHP script and stored in the database. It’s not a very elegant
system, but it does the trick.
In case you’re interested, the final layout looks a little like this:
CREATE TABLE `activity` (
id INTEGER PRIMARY KEY AUTO_INCREMENT,
`timestamp` INTEGER,
`duration` INTEGER,
`activity` TEXT,
`category` TEXT,
`productivity` INTEGER
);
CREATE TABLE `task` (
`id` INTEGER PRIMARY KEY AUTO_INCREMENT,
`file` TEXT,
`title` TEXT,
`state` TEXT,
`is_archived` INTEGER,
`deadline` INTEGER,
`scheduled` INTEGER,
`added` INTEGER,
`closed` INTEGER
);
There are also tables to store properties, tags and tracked time.
Processing
The main processing script is written in PHP, and uses RedBean for the database layer and Twig for templates. Templates were originally just PHP code, but after a while they became unwieldy as more processing code was shoved inside them. This is a Bad Thing™, so converting templates to Twig forced the querying code into better defined objects.
Once all the various files and fragments have been created, the site HTML is built using Jekyll and then deployed to the site. The whole process takes less than a minute, and most of that is spent parsing org. If no new data needs loading, it takes under 10 seconds to build & deploy.
All of this is wrapped in a Phing build script, so the whole process takes a few keystrokes to execute.
$ phing fetch:stats build:stats build:site deploy:site
Software & Libraries
The following libraries and tools are used during generation:
- Perl::OrgParse – Perl library for parsing org-mode files.
- RedBean – Lightweight PHP database library.
- Twig – Template engine for PHP.
- Phing – PHP build tool.
- Jekyll – Static site generator.