How I use org-mode

A few days ago, someone in /r/emacs asked the following question:

Hi! I am getting started with org and wanted to ask you why do you use it and how did org changed your life/workflow.

Answering it made me realize just how much of my daily workflow uses org-mode. Even more interesting was how much I use it for non-task related work. This is an expanded version of my original answer.

Tracking my diet, weight, and workout routines

This is something I wrote about in July. I've used org-mode to track my diet for years; the combination of properties and inline tables makes it extremely flexible.

I also use org-mode for tracking my exercise routines. This is integrated with beeminder to keep me on track.

Keeping on top of recurring tasks

I use org-habits for tracking recurring tasks and habits that I want to create. It integrates with org-agenda and shows a progress bar of days I have completed it.

All of my habits are also tied to beeminder goals (via beeminder.el) to give me some additional incentive.

A single habit entry looks like this:

* TODO Weekly review
  SCHEDULED: <2020-08-28 Fri .+7d>
  :PROPERTIES:
  :beeminder: weekly-review
  :STYLE:     habit
  :beeminder-skip-deadlines: true
  :END:

Marking the task as completed sends a datapoint to beeminder, and because the task is a habit the TODO remains open for the next date. I use the :beeminder-skip-deadlines: property to prevent habit deadlines from clogging up my agenda.

Keeping work and personal notes

I've been using zetteldeft to create and organize my notes for a couple of months. I was using deft before, so there wasn't much of a learning curve. I really like how the search narrows things as I type, and linking notes together makes it easier to keep things organized.

I'm not quite as organized as some other systems I've seen, but for the amount of notes I have (~150) it works quite well.

The full setup deserves an entire post of its own.

My full setup is detailed here: "Keeping notes with zetteldeft".

Keeping track of important dates

I use org-anniversary and a single file called dates.org to store birthdays and anniversaries.

An entry in my dates.org looks like this:

* Anniversaries
  :PROPERTIES:
  :CATEGORY: Important Dates
  :END:

%%(org-anniversary 2020  6 16) Gingersnap's %d adoptaversary

Unlike a scheduled TODO item, these dates show up in the agenda with the correct anniversary or birthday number. So in 2021 this entry would show as Gingersnap's 1 adoptaversary.

Planning out projects and organizing my day

The combination of plaintext for tasks and the dynamic agenda are extremely powerful for organizing projects. The agenda shows everything I have scheduled for the current week, and I'll usually narrow it down to the current day to get a better view.

This is another thing that deserves its own post.

Writing my personal blog

I use org-mode's html exporter to convert posts to html. I wrote about this in "From WordPress to Jekyll", but that process is a little outdated.

My current system involves storing posts in git, and then building the html + site on the server whenever content is pushed to it. I should probably write a post about this one too.

~~~
If you found this helpful and would like to read more content like it, follow me on Twitter.


Groundhog Day Resolutions - August 2020

July was was busy. Really busy. I'm also convinced it was the shortest month of the year.

July's Primary Goals

1. Release version 0.3 of The Book

I wrote a lot of notes but didn't get them in a publishable state.

2. Run some more

I ran more miles in July than any month since March. I can't say I enjoyed running them, but the longer runs were not as painful as last time I trained for a half.

3. Release version 1.2 of beeminder.el

I finished a couple more features and straightened out some bugs, but there is one last part I need to get finished and tested before release.

July's Secondary Goals

1. Complete another "other goal"

Success, I planted a tree! Or rather, I donated money to an organization that plants trees. I would have preferred to do the planting myself, but given the various restrictions in place due to COVID-19 that wasn't looking likely.

2. Create a short screencast

I underestimated how difficult this was going to be. Not just from a technical point of view, the also the amount of writing and editing required.

This is still something I'd like to try, but I don't think it's something to attempt without more planning and research.

Primary Goals for August

1. Release version 0.3 of The Book

Let's try this one again. I really want to have the first complete edition published by the end of the year. There are still 4 months left, but that time goes by pretty darn quick.

2. Bodyweight workouts, 3 times a week

My October half marathon has been canceled, so there is no need for me to run as many miles as I had planned. I'll still be running, but only 3 times a week instead of 5 days like before. I'm okay with that.

To replace the running I'd like to get back into bodyweight workouts. I already have most of the equipment I need, which is handy given that everything is sold out.

I expect there will be some Beeminding involved with this goal.

3. Release version 1.2 of beeminder.el

Third time's a charm?

Secondary Goals for August

1. Publish 8 posts on this site

There are a couple of topics I've been meaning to write about but haven't made the time for. August seems like as good a time as any.

2. Maintain a frog stand for 30 seconds

Learning three bodyweight skills is one of my major goals for 2020, but I've been quite lax lazy about practicing them.

By the end of August I want to be able to maintain a frog stand for 30 seconds without falling over.

July absolutely flew by. Work was especially busy, but I still should have spent more time on my goals. I have a bad habit of setting GHD goals and then forgetting about them until it's too late.

For August I'll be putting them all into my GTD system (like I should have been doing). We'll see if that makes a difference.


Tracking my diet with Emacs - part II

In part one, I showed how I keep track of my diet using Emacs. In this part I'll explain how I extract that data into something that can be used outside of Emacs.

The full source code is available at the end of the post.

What it does

I wrote a small ruby script that converts all of my daily entries into a json file. It turns something like this:

** CAL-OUT Diet for day <2019-10-05 Sat>
  :PROPERTIES:
  :Weight:   167.3
  :END:

| Timestamp              | Food                   | Calories | Quantity |   Total |
|------------------------+------------------------+----------+----------+---------|
| [2019-10-05 Sat 08:42] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 11:34] | Chocolate Chip Pancake |      2.7 |      202 |   545.4 |
| [2019-10-05 Sat 14:46] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 14:46] | Vanilla yogurt raisins |     4.33 |       30 |   129.9 |
| [2019-10-05 Sat 17:45] | Beef meatballs         |     2.02 |      370 |   747.4 |
| [2019-10-05 Sat 17:45] | Fresh Pasta            |     1.31 |      306 |  400.86 |
| [2019-10-05 Sat 17:45] | Garlic bread           |     2.75 |       94 |   258.5 |
| [2019-10-05 Sat 19:58] | Cup of tea             |       54 |        1 |      54 |
|------------------------+------------------------+----------+----------+---------|
| Total                  |                        |          |          | 2244.06 |
#+TBLFM: $5=$3*$4::@>$5=vsum(@2$5..@-I$5)

into this:

[
  {
    "keyword": "CAL-OUT",
    "date": "2019-10-05 Sat",
    "properties": {
      "Weight": "167.3"
    },
    "entries": [
      [
	"Timestamp",
	"Food",
	"Calories",
	"Quantity",
	"Total"
      ],
      [
	"[2019-10-05 Sat 08:42]",
	"Cup of tea",
	"54",
	"1",
	"54"
      ],
      [
	"[2019-10-05 Sat 11:34]",
	"Chocolate Chip Pancake",
	"2.7",
	"202",
	"545.4"
      ],
      [
	"[2019-10-05 Sat 14:46]",
	"Cup of tea",
	"54",
	"1",
	"54"
      ],
      [
	"[2019-10-05 Sat 14:46]",
	"Vanilla yogurt raisins",
	"4.33",
	"30",
	"129.9"
      ],
      [
	"[2019-10-05 Sat 17:45]",
	"Beef meatballs",
	"2.02",
	"370",
	"747.4"
      ],
      [
	"[2019-10-05 Sat 17:45]",
	"Fresh Pasta [Ronzoni Homestyle]",
	"1.31",
	"306",
	"400.86"
      ],
      [
	"[2019-10-05 Sat 17:45]",
	"Garlic bread",
	"2.75",
	"94",
	"258.5"
      ],
      [
	"[2019-10-05 Sat 19:58]",
	"Cup of tea",
	"54",
	"1",
	"54"
      ]
    ]
  }
]

I went with json as there are far more libraries available that parse json compared to org-mode, which makes it easier to use the data elsewhere.

Other potential solutions

There were a couple of other ideas I considered.

1. Write an elisp org-mode exporter

org-mode has a very flexible exporting system; it's what turns my .org blog posts into html so that Jekyll can publish them. There are ten exporters included with org-mode, including exporters for html, latex, and OpenDocument.

This was my first idea, but my elisp knowledge in 2012 was limited at best. There's also the issue that Emacs has to start up and run to extract data this way. It's fine if I'm exporting data whilst using Emacs, but not quite as quick if I'm running it from the command line.

It might be something I try again in the future now that I have more lisp experience.

2. Write a script in a different language

I went with ruby as it has an easy-to-use org-mode library that is fairly up-to-date. There are other org-mode parsers available, but not all of them are as robust as ruby's.

I'd like to try the common lisp parse, cl-org-mode, at some point.

3. Export the content to SQLite instead

Json is useful for passing data around, but a dedicated database with SQL is a better tool when it comes to querying. I've worked with SQLite for smaller projects and it's always quick to get started with.

Ruby has some nice libraries for interacting with SQLlite, so it could be integrated fairly easily into my existing script. Emacs also has SQL mode which can query databases and return the results into a buffer.

The full script

The script requires the following gems to be installed:

It wasn't designed to be used as a standalone script as it's part of a larger library that I use to build this site. It's also not as ruby-fied as I would like, but it does the job for now.

Usage

Add the following to the bottom of the full script, replacing f.input and f.output with the full input and output paths of the diet file.

extractor = OrgDietExtractor.new do |f|
  f.from     = '2020-01-01'
  f.to       = '2020-12-31'
  f.keywords = %w[CAL-OUT CAL-CANCEL]
  f.headline = 'Diet for day'
  f.input    = 'diet.org'
  f.output   = 'diet.json'
end

extractor.generate

Available configuration variables are:

from
Any entries made before this date will be excluded from the export file.
to
Like from, except it excludes entries made after this date.
keywords
An array of headline keywords to include in the export. Any headlines with a different keyword (such as CAL-IN) will be excluded.
headline
The starting words of headlines to include.
input
The full path of the diet file to parse.
output
The full path where the json data should go.

The script

# frozen_string_literal: true

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

require 'json'
require 'org-ruby'

class OrgDietExtractor
  attr_accessor :format
  attr_accessor :input
  attr_accessor :output
  attr_accessor :from
  attr_accessor :to
  attr_accessor :headline
  attr_accessor :keywords
  attr_accessor :current_parent
  attr_accessor :current_level

  def initialize
    yield self
  end

  def headline_contents(headline)
    text_lines = headline.body_lines
    text_lines.shift

    # Strip properties stuff.
    text_lines = text_lines.map do |line|
      task_line = line.output_text.strip
      next if task_line.start_with?(':')

      task_line
    end

    text_lines = text_lines.reject { |c| c.nil? || c.empty? }
    text_lines.join("\n")
  end

  def belongs_to?(headline)
    @current_parent == headline
  end

  def entry_in_range?(headline)
    return true unless @from || @to

    # Extract the dates
    date       = Date.parse(diet_date(headline))
    start_date = Date.parse(@from) if @from
    end_date   = Date.parse(@to) if @to

    return false if start_date && date < start_date
    return false if end_date && date > end_date

    true
  end

  def extract_table_headers(content)
    header_line = content.split("\n").first
    return if header_line.nil?

    header_cols = header_line.split('|').map(&:strip)

    header_cols.reject { |col| col.empty? }
  end

  def extract_table_rows(content)
    table_lines = content.split("\n")
    return if table_lines.nil?

    # Remove first + last line
    table_lines = table_lines.drop(2)
    table_lines.pop(3)

    table_lines.map do |line|
      line.split('|').map(&:strip).reject(&:empty?)
    end
  end

  def diet_node?(headline)
    @keywords.include?(headline.keyword) && headline.headline_text.start_with?(@headline)
  end

  def diet_date(headline)
    headline.headline_text.scan(/\<(.*?)\>/).last.first
  end

  def diet_properties(headline)
    headline.property_drawer
  end

  def diet_entries(contents)
    headers = extract_table_headers(contents)
    return nil unless headers

    table = []
    table << headers
    table += extract_table_rows(contents)

    table
  end

  def remap_property(task, from, to)
    return unless task[:properties].key?(from)

    task[to] = task[:properties][from]
    task[:properties].delete(from)
  end

  def generate()
    # Get full paths.
    input_file  = File.absolute_path(@input)
    output_file = File.absolute_path(@output)

    # Load the org file and parse it.
    file_in = IO.read(input_file)
    doc     = Orgmode::Parser.new(file_in)

    # Parse all headlines.
    diet_days = []

    doc.headlines.each do |headline|
      # Want top-level headlines in date range only.
      next unless diet_node?(headline)
      next unless entry_in_range?(headline)

      # Create diet info container.
      diet_entry = {
	keyword:    headline.keyword,
	date:       diet_date(headline),
	properties: diet_properties(headline),
	entries:    diet_entries(headline_contents(headline))
      }

      diet_days << diet_entry
    end

    # Generate the file.
    File.open(output_file, 'w') do |file_out|
      file_out.write(diet_days.to_json)
    end
  end
end

Tracking my diet with Emacs

I've been using Emacs with org-mode to track my diet since 2012. I've had some breaks along the way, the overall setup has stayed the same.

I use this system to track how much I weigh, as well as how many calories I'm consuming in a single day. There are plenty of apps and online services that provide this functionality, but I prefer to own my data in an open format I can use elsewhere.

Seeing as I work from home and always have an Emacs session open, it made sense for me to try to utilize Emacs in some way.

My set up uses the following Emacs and org-mode functionality:

  • Individual org-mode headlines for each day
  • org-mode properties for storing my weight
  • org-mode tables and spreadsheet formulas
  • org-capture for weighing in
  • Some elisp functions for adding individual food entries

Let's take a closer look at how all of this fits together.

Diet file setup

My diet file looks like this (with headings collapsed):

#+TITLE: Diet Tracker
#+SEQ_TODO: CAL-IN | CAL-OUT CAL-CANCEL

* Daily Logs
** CAL-OUT Diet for day <2020-07-03 Fri>...
** CAL-OUT Diet for day <2020-07-02 Thu>...
** CAL-OUT Diet for day <2020-07-01 Wed>...
** CAL-OUT Diet for day <2020-06-30 Tue>...
** CAL-OUT Diet for day <2020-06-29 Mon>...
... And more

I tried to keep the file format as simple as possible. Each day has its own entry underneath the main "Daily Logs" header. The main "Daily Logs" header is used by org-capture to find where to place new items.

The top-level SEQ_TODO property is used to set the "done" state for each headline. I use three different types of headline:

CAL-IN
For days where I haven't entered all of my diet information. There's usually only one of these open at a time.
CAL-OUT
For finished days where all data is final.
CAL-CANCEL
For days where I'm tracking weight but not calorie information. I use this for Sundays, days where I go out to eat, or when I have social events planned.

Daily entries

A daily entry looks something like this:

** CAL-OUT Diet for day <2019-10-05 Sat>
  :PROPERTIES:
  :Weight:   167.3
  :END:

| Timestamp              | Food                   | Calories | Quantity |   Total |
|------------------------+------------------------+----------+----------+---------|
| [2019-10-05 Sat 08:42] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 11:34] | Chocolate Chip Pancake |      2.7 |      202 |   545.4 |
| [2019-10-05 Sat 14:46] | Cup of tea             |       54 |        1 |      54 |
| [2019-10-05 Sat 14:46] | Vanilla yogurt raisins |     4.33 |       30 |   129.9 |
| [2019-10-05 Sat 17:45] | Beef meatballs         |     2.02 |      370 |   747.4 |
| [2019-10-05 Sat 17:45] | Fresh Pasta            |     1.31 |      306 |  400.86 |
| [2019-10-05 Sat 17:45] | Garlic bread           |     2.75 |       94 |   258.5 |
| [2019-10-05 Sat 19:58] | Cup of tea             |       54 |        1 |      54 |
|------------------------+------------------------+----------+----------+---------|
| Total                  |                        |          |          | 2244.06 |
#+TBLFM: $5=$3*$4::@>$5=vsum(@2$5..@-I$5)

It's a fairly simple table that uses some org-mode magic for calculating totals. The "Calories" column is usually "calories per gram", but for some items it's "calories per item". Likewise, the quantity column either refers to the weight in grams or the number of items consumed.

The #+TBFLM: part underneath the table is an org-mode spreadsheet formula. It uses two formulae:

$5=$3*$4
Sets column 5 (the "Totals" column) to Calories x Quantity. org-mode columns indexes start from 1 rather than 0.
@>$5=vsum(@2$5..@-I$5)

Calculates the total amount of calories consumed during the day. It uses relative references so that it works no matter how many lines

I previously used $LR5 instead of @>$5 to reference the footer row, but this no longer worked after upgrading to org-mode 9.4.

The Spreadsheet section of the org-mode manual goes into detail on formulas. It took me a while to get the hang of, but it's a really powerful system.

Weigh-ins

I use an org-capture-template for my weigh-ins. I weigh myself every morning, depending on my schedule.

org-capture is bound to C-c o r, and then my weigh-in template is bound to w. So every day I run C-c o r w, enter my weight, then use C-c C-c to save it to my diet file. And that's it.

My capture template is below:

(add-to-list
 org-capture-templates
 '("w" "Weigh-in" entry
   (file+headline "~/self/diet.org" "Daily Logs")
   "* CAL-IN Diet for day %t
%^{Weight}p
| Timestamp | Food | Calories | Quantity | Total |
|--------------+---+----------+----------+-------|
|--------------+---+----------+----------+-------|
| Total        |   |          |          |       |
#+TBLFM: $5=$3*$4::@>$5=vsum(@2$5..@-I$5)"
   :prepend t))

Adding new food entries

I have a couple of elisp functions I use for adding new data. The primary function is org-diet-copy, which is bound to C-c C-C.

org-diet-copy is used on a row of another table. It copies the food name, calorie amount, and quantity to the top table and replaces the timestamp with the current date and time.

The process of adding a new entry typically goes like this:

  • Hit C-s to search for the food I want to add. If I wanted to add a new entry for "french fries" I would probably do something like "C-s fren" to find the first "french fries" entry.
  • Hit C-c C-C to copy the entry to my active day.
  • Replace the quantity with whatever amount I ate. I try to keep the same portion size for breakfast and snacks, so this isn't always necessary.
  • Run M-x org-table-recalculate to update the table.

It's a simple system, but it works well enough. Eventually I may add function that prompts for a food and quantity, and then automatically fills in the calorie amount.

All of the functions I use are below:

(defun org-diet-move-today ()
  "Move to today's entry in the org-diet file."
  (interactive)
  (beginning-of-buffer)

  ;; Move to first heading (Daily Logs) and expand.
  (outline-next-visible-heading 1)
  (show-children)
  (outline-next-visible-heading 1))

(defun org-diet-move-last-entry ()
  "Move to the last entry in the current diet table."
  (interactive)

  (search-forward "#+TBLFM")

  ;; Move to start of final line.
  (previous-line 1)
  (previous-line 1)
  (move-beginning-of-line 1))

(defun org-diet-copy ()
  "Copy the current table line to today.

Copies the current table line and moves it to the bottom of
today's diet table.  Changes the timestamp to the current time
and day."
  (interactive)

  (let ((diet-line (thing-at-point 'line t)))
    ;; Jump to today & last line.
    (org-diet-move-today)
    (org-diet-move-last-entry)

    ;; Insert the copied line.
    (insert diet-line)

    ;; Move back to last line.
    (org-diet-move-last-entry)
    (forward-line -1)

    ;; Replace the date.
    (beginning-of-line)
    (search-forward "| ")
    (zap-to-char 1 93)
    (insert (format-time-string "[%Y-%m-%d %a %H:%M]" (current-time)))

    ;; Update the table.
    (forward-line 2)
    (org-table-recalculate)

    ;; Move to inserted line.
    (beginning-of-line)
    (forward-line -2)))

;; Bind copying to `C-c C-C`.
(global-set-key "\C-cC" #'org-diet-copy)

In part two I'll cover the system I use to extract data from the org-mode file. It's not pretty.


Groundhog Day Resolutions - July 2020

June ended up being quite a busy month for work and went far faster than I had anticipated.

June's Primary Goals

1. Release version 0.2 of The Book

I released two new chapters and added a little polish to the existing ones. I would have liked to have written more, but I'm still fairly happy with how things turned out.

2. Keep running

Motivation for running has been at an all time low. It's hot and humid outside so I need to run before the sun comes up. I've managed it a few times, but my legs were stiff and the whole thing was thoroughly unpleasant.

3. Release version 1.2 of beeminder.el

I didn't even touch this during the month. It probably only needs a couple of hours to finish off, but I didn't make the time.

4. Complete one of my "other goals"

I finally settled on zetteldeft to manage my notes. I've been trying out a couple of other services and systems, but zetteldeft fits my setup nicely. I'd like to write more about this in the future, especially the tweaks I've made and how I take notes now.

Primary Goals for July

1. Release version 0.3 of The Book

I really want to get the section on syntax checkers completed by August. If time (and energy) permits, I'd also like to get the section on using lsp-mode published. lsp-mode is something I've only been using the last 6 months or so, but it's made a huge different to my productivity and I'd like others to benefit from it.

2. Run some more

My weekly miles are going to start increasing now that July is here, so I need to keep on top of things before August arrives. October isn't that far away when it comes to improving my running.

3. Release version 1.2 of beeminder.el

If at first you don't succeed…

Secondary goals for July

1. Complete another "other goal"

Maybe I'll try the art thing again.

2. Create a short screencast

I have a couple of topics in mind that I'd like to record a screencast for. It will probably be Emacs or productivity related, so don't get too excited.

Recording a screencast is something I've thought about doing for a while, but never really felt motivated enough to do. Last week I gave an absolutely disastrous presentation which made me realize my public speaking skills have completely atrophied. I'd like to fix that.