Limiting goaccess reports to a date range

A few days ago I wrote about filtering referer spam from my server logs before they're processed by goaccess. goaccess doesn't support log filtering out of the box, but tools like grep or sed can be used to filter the log before it's processed by goaccess.

There are two main reports I use for this blog: today's log, and the last 30 days of activity.

Viewing today's log

This is nice and simple. It searches the log for entries that have today's date string, and the results are then piped to goaccess.

grep "$(date '+%d\/%b\/%Y' -d 'today')" path/to/access_log \
    | goaccess -a

This can also be adapted to show log entries for a specific month (or year). I use this when generating monthly reports on how "Writing PHP with Emacs" is performing:

grep 'Nov/2020' path/to/access_log \
    | grep "books/php-with-emacs" \
    | goaccess -a

Note: This could also be done in a single pass using grep's -e syntax, but I find this version more readable.

Viewing this month's log

This one is a little trickier as we need to filter out everything before a specific date. Thankfully sed can do this:

sed -n '/'$(date '+%d\/%b\/%Y' -d '1 month ago')'/,$ p' path/to/access_log \
    | goaccess -a

There are a couple of parts here:

-n
This flag tells sed to hide its output. Normally it will display every line that is processed, which in this case would output the entire log file.
$(date '+%d\/%b\/%Y' -d '1 month ago')
This is evaluated and replaced by a timestamp for 1 month ago.
'/'$(date '+%d\/%b\/%Y' -d '1 month ago')'/,$ p'
This is the expression that sed will evaluate, which searches all entries that match a specific range. The , symbol is the range delimiter, and $ is the end of the range (which in this case is any date after the start date).
p
This final modifier tells sed to output lines that matched.

All together this filters log entries to the last 30 days before goaccess processes them. The date command is pretty flexible so this approach can filter entries to the current week, quarter, or any other range.


Lessons from 11 years of freelancing

I've been working as a freelance software engineer for just over 11 years now. My first contract came through Upwork (which was called oDesk back in 2009) and I charged $10 an hour. All of my early work was for individual bloggers or small businesses, and usually involved building or tweaking WordPress themes and plugins.

Since then I've been able to help dozens of different organizations and work on some really interesting projects. It's not quite the career I had in mind, but I enjoy the variety of work that comes my way.

Here's what I've learned.

It takes time

It took me about two years for my income to stabilize. The first year I could often go a week or two without any money coming in. This situation improved once I found a few steady clients, most of which came through referrals. After three years I was able to pay all of my bills (rent/food/etc) entirely through freelance work.

Use what you know

There is a time and a place to experiment with new technology. It's not when you're doing billed work for a client.

Charge more

This is a pretty common refrain in freelance circles, but it's still good advice. Raising prices does several things:

  • Signals that you're serious about what you do.
  • Weeds out clients that you may not want.
  • Allows you to be more selective about what you do.

It took me a long time to build the experience - and courage - to increase my rates. I could have been self-sustainable much earlier if I'd been brave enough to charge more.

Sometimes people won't pay

This has only happened a few times, but it's extremely frustrating when it does. These days I won't do any work without signing a contract first and being very clear about what is expected from both parties.

Not all clients are a good fit

This doesn't mean that they're bad people, just that their work styles don't mesh with mine. It's usually better to cut things early rather than trying to make things work.

Don't rely on a single client

After my first year I stopped looking for new projects and took all of my work from the same client. When they decided to switch to another company my income dried up overnight. There's no severance pay when you're a freelancer.

Communicate. Communicate more

Early on I developed a very bad habit where I wouldn't show what I'd done until it looked perfect. This sounds like a logical approach, but it causes all kinds of problems:

  1. I know that I've been working on something, but the client doesn't know that unless I tell them. They are paying for my services, and they want to see what they're getting.
  2. Often what they originally asked for isn't quite what they want. Showing things early gives time for feedback.
  3. It's not my place to decide what's perfect for the client.

I solved all of these by communicating clearly. I don't particularly like writing emails, but they're an important part of my work.

You need to learn a lot of non-work skills

Keeping accurate records, sending invoices, paying estimated taxes (and finding out who to pay), and filing taxes are all things I've had to learn since I started.

I keep a single spreadsheet for all of my financial information, and I store any printed receipts for the year in a manila folder. Once taxes are done, finances get printed and put in the filing cabinet. It's not glamorous, but it means I have an actual record if I need it.

Hint: scribbling "paid $xxx" on a post-it note is not good bookkeeping.

It can be expensive

My experience is based on living in the US, but I've found freelancing to have a lot of hidden costs. I have to pay health insurance, pay into social security (including the employer contribution), pay into a retirement account, and I don't get paid holidays.

Non-billable tasks take a lot of time

Hours of time can quickly get eaten by emails and administrative tasks. There are also non-work things like eating, drinking, and bathroom breaks that take time. Take these into account.

You're on your own

Unless you hire outside help, everything that needs doing rests on the shoulders of one person: you. This includes planning projects, scheduling work days, writing proposals, sending invoices (and following up on them), setting up meetings, and then actually doing the work you're being paid for on top of all that.

It's a lot.

Take all of this advice with a grain of salt; there's a hefty chunk of survivorship bias thrown in.


Post or Pay

Post or Pay

Aleix contacted me about their new service, "Post or Pay":

Post or Pay is a website that allows bloggers to put a price to not blogging. The user sets a deadline for her blog post and a price. If she doesn't publish it on time, she pays.

20% of the payment goes to charity (the World Literacy Foundation).

I use Beeminder for tracking my blog posting. As long as I post frequently enough, I'll stay on target and not get billed. It's working well for my 30 Days of Blogging trial, although it wasn't totally painless to set up - I had to hook up my RSS feed to Zapier in order to automate data submission.

Post or Pay works a little differently; you commit to writing a single post by a specific date, and then send proof once it's published. It's a much more focused approach.

If you're struggling to blog more, or you're looking to dip your toes into the world of commitment services, Post or Pay might be a good place to start.


Time confetti

A couple of days ago I read an article on the concept of "Time Confetti", about how our leisure time is slowly being fragmented into smaller pieces throughout the day:

I (and other researchers) call this phenomenon time confetti (a term coined by Brigid Schulte), which amounts to little bits of seconds and minutes lost to unproductive multitasking. Each bit alone seems not very bad. Collectively, though, all that confetti adds up to something more pernicious than you might expect.

Source: Time Confetti and the Broken Promise of Leisure

I'm certainly guilty of letting myself get distracted like this. I'll often take a short break after completing a work-related task; they don't give me enough time to really relax, but they add up over the course of the day. By the end of the day I'll have spent a couple of hours "relaxing", but I'll feel no better for it.

One solution to this is using an unschedule from "The NOW Habit". The idea is to create a schedule for the day, but to add fun activities first (rather than work tasks). I've tried it in the past with some success, but over time my schedule started skewing more heavily towards work. Eventually it ended up being entirely work-focused with tiny breaks scattered in ad-hoc.

I think now might be a good time to try it again.


The end of fun

I've been messing around with C++ in preparation for some of my 2021 goals. I haven't actually written anything in C++ since I was in university, and given the quality of what I wrote I'm surprised I graduated at all.

But what I've been trying for the last few days has been a lot of fun. Maybe it's because I'm writing what I want instead of code to pass an assignment, or maybe it's because there's a familiarity to it so I feel like I'm learning quicker.

I really enjoy trying to code up an idea and then seeing it come to life. Even after 25+ years of writing code, I still get enjoyment from seeing the ideas in my head becoming a reality.

But at some point I get a little carried away and try to create something outside of my abilities. And then the fun stops.

In my recent C++ adventure I managed to get a little pixel sprite moving around, all working on the Atari ST. That's something I never thought I'd be able to do.

But already my mind is thinking 200 steps ahead. Because the tool allows me to create something good, surely I must use it to create something good. I think this is similar to the problem I have with art supplies; part of my brain thinks better tools will improve my skills, rather than the other way around.

Even if I focus on something small, eventually I get to a point where working on it just feels bad. I'm not sure if that means I only enjoy the first 20% of a project, or if it's something deeper.

My hunch is that I enjoy the first part because it's all about discovery. I get to learn new things, and there's usually a quick turnaround between trying something and seeing it work. When I'm learning something new, I'm only concerned about the result, not how it's achieved. I allow myself to fail multiple times until it works.

Once I move past learning new things, everything changes. I start bumping up against areas where my skillset isn't so good. I have to make design decisions. Things start breaking and my motivation tanks. So I put it aside and learn something else, but all that gives me is a hard-drive full of half-baked prototypes. That's not what I want.

So where does that leave my little pixel sprite? My normal approach is to make a small prototype and then slowly grow it into the (un)finished article. I think this time I'll take a more practice-oriented approach, testing out new ideas and techniques separately, before building on that experience to create something new.