Tracking my diet with Emacs - part II

In part one, I showed how I keep track of my diet using Emacs. In this part I'll explain how I extract that data into something that can be used outside of Emacs.

The full source code is available at the end of the post.

What it does

I wrote a small ruby script that converts all of my daily entries into a json file. It turns something like this:

** CAL-OUT Diet for day <2019-10-05 Sat>
  :PROPERTIES:
  :Weight:   167.3
  :END:

| Timestamp              | Food                   | Calories | Quantity |   Total |
|------------------------+------------------------+----------+----------+---------|
| [2019-10-05 Sat 08:42] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 11:34] | Chocolate Chip Pancake |      2.7 |      202 |   545.4 |
| [2019-10-05 Sat 14:46] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 14:46] | Vanilla yogurt raisins |     4.33 |       30 |   129.9 |
| [2019-10-05 Sat 17:45] | Beef meatballs         |     2.02 |      370 |   747.4 |
| [2019-10-05 Sat 17:45] | Fresh Pasta            |     1.31 |      306 |  400.86 |
| [2019-10-05 Sat 17:45] | Garlic bread           |     2.75 |       94 |   258.5 |
| [2019-10-05 Sat 19:58] | Cup of tea             |       54 |        1 |      54 |
|------------------------+------------------------+----------+----------+---------|
| Total                  |                        |          |          | 2244.06 |
#+TBLFM: $5=$3*$4::@>$5=vsum(@2$5..@-I$5)

into this:

[
  {
    "keyword": "CAL-OUT",
    "date": "2019-10-05 Sat",
    "properties": {
      "Weight": "167.3"
    },
    "entries": [
      [
	"Timestamp",
	"Food",
	"Calories",
	"Quantity",
	"Total"
      ],
      [
	"[2019-10-05 Sat 08:42]",
	"Cup of tea",
	"54",
	"1",
	"54"
      ],
      [
	"[2019-10-05 Sat 11:34]",
	"Chocolate Chip Pancake",
	"2.7",
	"202",
	"545.4"
      ],
      [
	"[2019-10-05 Sat 14:46]",
	"Cup of tea",
	"54",
	"1",
	"54"
      ],
      [
	"[2019-10-05 Sat 14:46]",
	"Vanilla yogurt raisins",
	"4.33",
	"30",
	"129.9"
      ],
      [
	"[2019-10-05 Sat 17:45]",
	"Beef meatballs",
	"2.02",
	"370",
	"747.4"
      ],
      [
	"[2019-10-05 Sat 17:45]",
	"Fresh Pasta [Ronzoni Homestyle]",
	"1.31",
	"306",
	"400.86"
      ],
      [
	"[2019-10-05 Sat 17:45]",
	"Garlic bread",
	"2.75",
	"94",
	"258.5"
      ],
      [
	"[2019-10-05 Sat 19:58]",
	"Cup of tea",
	"54",
	"1",
	"54"
      ]
    ]
  }
]

I went with json as there are far more libraries available that parse json compared to org-mode, which makes it easier to use the data elsewhere.

Other potential solutions

There were a couple of other ideas I considered.

1. Write an elisp org-mode exporter

org-mode has a very flexible exporting system; it's what turns my .org blog posts into html so that Jekyll can publish them. There are ten exporters included with org-mode, including exporters for html, latex, and OpenDocument.

This was my first idea, but my elisp knowledge in 2012 was limited at best. There's also the issue that Emacs has to start up and run to extract data this way. It's fine if I'm exporting data whilst using Emacs, but not quite as quick if I'm running it from the command line.

It might be something I try again in the future now that I have more lisp experience.

2. Write a script in a different language

I went with ruby as it has an easy-to-use org-mode library that is fairly up-to-date. There are other org-mode parsers available, but not all of them are as robust as ruby's.

I'd like to try the common lisp parse, cl-org-mode, at some point.

3. Export the content to SQLite instead

Json is useful for passing data around, but a dedicated database with SQL is a better tool when it comes to querying. I've worked with SQLite for smaller projects and it's always quick to get started with.

Ruby has some nice libraries for interacting with SQLlite, so it could be integrated fairly easily into my existing script. Emacs also has SQL mode which can query databases and return the results into a buffer.

The full script

The script requires the following gems to be installed:

It wasn't designed to be used as a standalone script as it's part of a larger library that I use to build this site. It's also not as ruby-fied as I would like, but it does the job for now.

Usage

Add the following to the bottom of the full script, replacing f.input and f.output with the full input and output paths of the diet file.

extractor = OrgDietExtractor.new do |f|
  f.from     = '2020-01-01'
  f.to       = '2020-12-31'
  f.keywords = %w[CAL-OUT CAL-CANCEL]
  f.headline = 'Diet for day'
  f.input    = 'diet.org'
  f.output   = 'diet.json'
end

extractor.generate

Available configuration variables are:

from
Any entries made before this date will be excluded from the export file.
to
Like from, except it excludes entries made after this date.
keywords
An array of headline keywords to include in the export. Any headlines with a different keyword (such as CAL-IN) will be excluded.
headline
The starting words of headlines to include.
input
The full path of the diet file to parse.
output
The full path where the json data should go.

The script

# frozen_string_literal: true

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

require 'json'
require 'org-ruby'

class OrgDietExtractor
  attr_accessor :format
  attr_accessor :input
  attr_accessor :output
  attr_accessor :from
  attr_accessor :to
  attr_accessor :headline
  attr_accessor :keywords
  attr_accessor :current_parent
  attr_accessor :current_level

  def initialize
    yield self
  end

  def headline_contents(headline)
    text_lines = headline.body_lines
    text_lines.shift

    # Strip properties stuff.
    text_lines = text_lines.map do |line|
      task_line = line.output_text.strip
      next if task_line.start_with?(':')

      task_line
    end

    text_lines = text_lines.reject { |c| c.nil? || c.empty? }
    text_lines.join("\n")
  end

  def belongs_to?(headline)
    @current_parent == headline
  end

  def entry_in_range?(headline)
    return true unless @from || @to

    # Extract the dates
    date       = Date.parse(diet_date(headline))
    start_date = Date.parse(@from) if @from
    end_date   = Date.parse(@to) if @to

    return false if start_date && date < start_date
    return false if end_date && date > end_date

    true
  end

  def extract_table_headers(content)
    header_line = content.split("\n").first
    return if header_line.nil?

    header_cols = header_line.split('|').map(&:strip)

    header_cols.reject { |col| col.empty? }
  end

  def extract_table_rows(content)
    table_lines = content.split("\n")
    return if table_lines.nil?

    # Remove first + last line
    table_lines = table_lines.drop(2)
    table_lines.pop(3)

    table_lines.map do |line|
      line.split('|').map(&:strip).reject(&:empty?)
    end
  end

  def diet_node?(headline)
    @keywords.include?(headline.keyword) && headline.headline_text.start_with?(@headline)
  end

  def diet_date(headline)
    headline.headline_text.scan(/\<(.*?)\>/).last.first
  end

  def diet_properties(headline)
    headline.property_drawer
  end

  def diet_entries(contents)
    headers = extract_table_headers(contents)
    return nil unless headers

    table = []
    table << headers
    table += extract_table_rows(contents)

    table
  end

  def remap_property(task, from, to)
    return unless task[:properties].key?(from)

    task[to] = task[:properties][from]
    task[:properties].delete(from)
  end

  def generate()
    # Get full paths.
    input_file  = File.absolute_path(@input)
    output_file = File.absolute_path(@output)

    # Load the org file and parse it.
    file_in = IO.read(input_file)
    doc     = Orgmode::Parser.new(file_in)

    # Parse all headlines.
    diet_days = []

    doc.headlines.each do |headline|
      # Want top-level headlines in date range only.
      next unless diet_node?(headline)
      next unless entry_in_range?(headline)

      # Create diet info container.
      diet_entry = {
	keyword:    headline.keyword,
	date:       diet_date(headline),
	properties: diet_properties(headline),
	entries:    diet_entries(headline_contents(headline))
      }

      diet_days << diet_entry
    end

    # Generate the file.
    File.open(output_file, 'w') do |file_out|
      file_out.write(diet_days.to_json)
    end
  end
end

Tracking my diet with Emacs

I've been using Emacs with org-mode to track my diet since 2012. I've had some breaks along the way, the overall setup has stayed the same.

I use this system to track how much I weigh, as well as how many calories I'm consuming in a single day. There are plenty of apps and online services that provide this functionality, but I prefer to own my data in an open format I can use elsewhere.

Seeing as I work from home and always have an Emacs session open, it made sense for me to try to utilize Emacs in some way.

My set up uses the following Emacs and org-mode functionality:

  • Individual org-mode headlines for each day
  • org-mode properties for storing my weight
  • org-mode tables and spreadsheet formulas
  • org-capture for weighing in
  • Some elisp functions for adding individual food entries

Let's take a closer look at how all of this fits together.

Diet file setup

My diet file looks like this (with headings collapsed):

#+TITLE: Diet Tracker
#+SEQ_TODO: CAL-IN | CAL-OUT CAL-CANCEL

* Daily Logs
** CAL-OUT Diet for day <2020-07-03 Fri>...
** CAL-OUT Diet for day <2020-07-02 Thu>...
** CAL-OUT Diet for day <2020-07-01 Wed>...
** CAL-OUT Diet for day <2020-06-30 Tue>...
** CAL-OUT Diet for day <2020-06-29 Mon>...
... And more

I tried to keep the file format as simple as possible. Each day has its own entry underneath the main "Daily Logs" header. The main "Daily Logs" header is used by org-capture to find where to place new items.

The top-level SEQ_TODO property is used to set the "done" state for each headline. I use three different types of headline:

CAL-IN
For days where I haven't entered all of my diet information. There's usually only one of these open at a time.
CAL-OUT
For finished days where all data is final.
CAL-CANCEL
For days where I'm tracking weight but not calorie information. I use this for Sundays, days where I go out to eat, or when I have social events planned.

Daily entries

A daily entry looks something like this:

** CAL-OUT Diet for day <2019-10-05 Sat>
  :PROPERTIES:
  :Weight:   167.3
  :END:

| Timestamp              | Food                   | Calories | Quantity |   Total |
|------------------------+------------------------+----------+----------+---------|
| [2019-10-05 Sat 08:42] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 11:34] | Chocolate Chip Pancake |      2.7 |      202 |   545.4 |
| [2019-10-05 Sat 14:46] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 14:46] | Vanilla yogurt raisins |     4.33 |       30 |   129.9 |
| [2019-10-05 Sat 17:45] | Beef meatballs         |     2.02 |      370 |   747.4 |
| [2019-10-05 Sat 17:45] | Fresh Pasta            |     1.31 |      306 |  400.86 |
| [2019-10-05 Sat 17:45] | Garlic bread           |     2.75 |       94 |   258.5 |
| [2019-10-05 Sat 19:58] | Cup of tea             |       54 |        1 |      54 |
|------------------------+------------------------+----------+----------+---------|
| Total                  |                        |          |          | 2244.06 |
#+TBLFM: $5=$3*$4::@>$5=vsum(@2$5..@-I$5)

It's a fairly simple table that uses some org-mode magic for calculating totals. The "Calories" column is usually "calories per gram", but for some items it's "calories per item". Likewise, the quantity column either refers to the weight in grams or the number of items consumed.

The #+TBFLM: part underneath the table is an org-mode spreadsheet formula. It uses two formulae:

$5=$3*$4
Sets column 5 (the "Totals" column) to Calories x Quantity. org-mode columns indexes start from 1 rather than 0.
@>$5=vsum(@2$5..@-I$5)

Calculates the total amount of calories consumed during the day. It uses relative references so that it works no matter how many lines

I previously used $LR5 instead of @>$5 to reference the footer row, but this no longer worked after upgrading to org-mode 9.4.

The Spreadsheet section of the org-mode manual goes into detail on formulas. It took me a while to get the hang of, but it's a really powerful system.

Weigh-ins

I use an org-capture-template for my weigh-ins. I weigh myself every morning, depending on my schedule.

org-capture is bound to C-c o r, and then my weigh-in template is bound to w. So every day I run C-c o r w, enter my weight, then use C-c C-c to save it to my diet file. And that's it.

My capture template is below:

(add-to-list
 org-capture-templates
 '("w" "Weigh-in" entry
   (file+headline "~/self/diet.org" "Daily Logs")
   "* CAL-IN Diet for day %t
%^{Weight}p
| Timestamp | Food | Calories | Quantity | Total |
|--------------+---+----------+----------+-------|
|--------------+---+----------+----------+-------|
| Total        |   |          |          |       |
#+TBLFM: $5=$3*$4::@>$5=vsum(@2$5..@-I$5)"
   :prepend t))

Adding new food entries

I have a couple of elisp functions I use for adding new data. The primary function is org-diet-copy, which is bound to C-c C-C.

org-diet-copy is used on a row of another table. It copies the food name, calorie amount, and quantity to the top table and replaces the timestamp with the current date and time.

The process of adding a new entry typically goes like this:

  • Hit C-s to search for the food I want to add. If I wanted to add a new entry for "french fries" I would probably do something like "C-s fren" to find the first "french fries" entry.
  • Hit C-c C-C to copy the entry to my active day.
  • Replace the quantity with whatever amount I ate. I try to keep the same portion size for breakfast and snacks, so this isn't always necessary.
  • Run M-x org-table-recalculate to update the table.

It's a simple system, but it works well enough. Eventually I may add function that prompts for a food and quantity, and then automatically fills in the calorie amount.

All of the functions I use are below:

(defun org-diet-move-today ()
  "Move to today's entry in the org-diet file."
  (interactive)
  (beginning-of-buffer)

  ;; Move to first heading (Daily Logs) and expand.
  (outline-next-visible-heading 1)
  (show-children)
  (outline-next-visible-heading 1))

(defun org-diet-move-last-entry ()
  "Move to the last entry in the current diet table."
  (interactive)

  (search-forward "#+TBLFM")

  ;; Move to start of final line.
  (previous-line 1)
  (previous-line 1)
  (move-beginning-of-line 1))

(defun org-diet-copy ()
  "Copy the current table line to today.

Copies the current table line and moves it to the bottom of
today's diet table.  Changes the timestamp to the current time
and day."
  (interactive)

  (let ((diet-line (thing-at-point 'line t)))
    ;; Jump to today & last line.
    (org-diet-move-today)
    (org-diet-move-last-entry)

    ;; Insert the copied line.
    (insert diet-line)

    ;; Move back to last line.
    (org-diet-move-last-entry)
    (forward-line -1)

    ;; Replace the date.
    (beginning-of-line)
    (search-forward "| ")
    (zap-to-char 1 93)
    (insert (format-time-string "[%Y-%m-%d %a %H:%M]" (current-time)))

    ;; Update the table.
    (forward-line 2)
    (org-table-recalculate)

    ;; Move to inserted line.
    (beginning-of-line)
    (forward-line -2)))

;; Bind copying to `C-c C-C`.
(global-set-key "\C-cC" #'org-diet-copy)

In part two I'll cover the system I use to extract data from the org-mode file. It's not pretty.


Groundhog Day Resolutions - July 2020

June ended up being quite a busy month for work and went far faster than I had anticipated.

June's Primary Goals

1. Release version 0.2 of The Book

I released two new chapters and added a little polish to the existing ones. I would have liked to have written more, but I'm still fairly happy with how things turned out.

2. Keep running

Motivation for running has been at an all time low. It's hot and humid outside so I need to run before the sun comes up. I've managed it a few times, but my legs were stiff and the whole thing was thoroughly unpleasant.

3. Release version 1.2 of beeminder.el

I didn't even touch this during the month. It probably only needs a couple of hours to finish off, but I didn't make the time.

4. Complete one of my "other goals"

I finally settled on zetteldeft to manage my notes. I've been trying out a couple of other services and systems, but zetteldeft fits my setup nicely. I'd like to write more about this in the future, especially the tweaks I've made and how I take notes now.

Primary Goals for July

1. Release version 0.3 of The Book

I really want to get the section on syntax checkers completed by August. If time (and energy) permits, I'd also like to get the section on using lsp-mode published. lsp-mode is something I've only been using the last 6 months or so, but it's made a huge different to my productivity and I'd like others to benefit from it.

2. Run some more

My weekly miles are going to start increasing now that July is here, so I need to keep on top of things before August arrives. October isn't that far away when it comes to improving my running.

3. Release version 1.2 of beeminder.el

If at first you don't succeed…

Secondary goals for July

1. Complete another "other goal"

Maybe I'll try the art thing again.

2. Create a short screencast

I have a couple of topics in mind that I'd like to record a screencast for. It will probably be Emacs or productivity related, so don't get too excited.

Recording a screencast is something I've thought about doing for a while, but never really felt motivated enough to do. Last week I gave an absolutely disastrous presentation which made me realize my public speaking skills have completely atrophied. I'd like to fix that.


My 2020 development setup

I was recently looking through some old projects and found a screenshot of my development environment from 2006:

Development screen from 2006

That's a lot of green.

The language is BlitzPlus, and the editor is Protean IDE (a custom Blitz IDE that is no longer available).

I think it's interesting to look over old screenshots like this to see how my work environment has changed (and how it's stayed the same).

Here's a shot I took during some work on beeminder.el version 1.2 this week.

Developing the beeminder.el package

One thing that stood out to me is how much visual clutter was in the old screenshot. I've removed as many toolbars and menus as possible, and try to stick to keyboard shortcuts where possible.

Current setup

I'm currently using Emacs 26.3 with the doom-nord theme. The status bar at the bottom uses doom-modeline with some slight tweaks. Font is Ubuntu Mono.

For work projects I use php-mode and web-mode. Recently I've started using lsp-mode with intelephense to add better auto-completion and documentation tooltips (which is something I'll be writing about in my book, "Developing PHP with Emacs").

I use magit for working with git repositories which has been a huge boost for efficiency. For navigating projects I use projectile with projectile-helm; they make jumping from one project to another extremely quick, which is helpful when I'm switching between client projects during the day. For project planning I use org-mode.

I'd say 80% of my professional work these days is done from within Emacs. I still use external applications like browsers and database tools, but any source code editing is done from within Emacs.

Eventually I'd like to get more comfortable using some other Emacs features like tramp (for editing files on remote servers) and EmacSQL for working with databases.

I still develop hobby projects in Blitz (using BlitzMax), although I've changed my setup a couple of times. I ended up writing blitzmax-mode so that I could write code with Emacs, and I've built a bunch of other Blitz tools over the years.


Groundhog Day Resolutions - June 2020

May went okay.

May's Primary Goals

1. Read another book

I read "Mastering Software Technique: Conscious Practice for Writing Software". I've been writing software for a long time, but this book covered some ideas I'd never really thought about before.

Creating small test projects to throw away is definitely something I want to put into practice.

2. Make some art

I drew a few pencil sketches to get warmed up, but that's as far as I got.

Drawing was not as relaxing as I'd hoped.

3. Release the initial version of The book

The initial version of Writing PHP with Emacs is now available. I'd been holding back on publishing in order to get everything "just right", but that attitude defeats the purpose of leanpub. So I cut out everything I wasn't 100% happy with and then hit "publish".

Now the real work begins.

4. Run more

I ran a "stay away" 5K for charity at the end of the month. It wasn't my fastest, but it was top 3 for me which has given me a bit of a confidence boost.

I've also started training for a half marathon in October. Running in the summer heat is no fun.

Primary Goals for June

1. Release version 0.2 of The Book

I've published some minor updates, but by July 7th I want to complete the next major section. I've linked leanpub updates to a Beeminder goal to help keep me on track.

2. Keep running

My training plan for June is pretty light, but this time I'm adding in some pace runs which I've never done before.

3. Release version 1.2 of beeminder.el

It's been over six years since I released the first version of beeminder.el. For the past few weeks I've been working on adding a goal dashboard and details page. It needs some polish, but is otherwise finished, so I'd like to get it cleaned up and released this month.

4. Complete one of my "other goals"

I'm not sure which one yet, but we're nearly 50% through 2020 and I've only checked off a single goal.