In this post, I talk about how I keep track of my personal finance by piecing
together some command-line-based open-source software (
reckon) and utilizing the open standard OFX
for downloading transactions from banks.
I believe there is value in using software to keep track of one’s personal finance. Some people might choose to do so because they want to be financially responsible adults. I mostly enjoy the process of collecting data about myself—I should at least possess the same data that various credit card companies and banks already have about me. This data could answer questions about my finances and provide insight into my lifestyle should the need arise, but I’m perfectly happy collecting and organizing it for its own sake—as a hobby.
I’d been using Mint for over three years. I’d log in every couple of days, diligently fix the inaccuracies of the automatically imported transactions, and put in those rare cash expenses when needed. It worked fine for the most part, but I never really liked this solution because of the following:
Security: Giving a third-party service my bank passwords is never the correct security solution, no matter how much I “trust” that party. For some reason, banks won’t offer oauth.
Flexibility: There are use cases I need that Mint can’t handle in any non-convoluted way, e.g. keeping track of other people’s money that’s physically in my accounts but not mine to spend, and recording stocks, frequent flyer miles, credit card reward points, and other liquid monetary assets that aren’t US dollars. I find myself creating separate spreadsheets to handle these, which seems silly.
Philosophy: While I appreciate that I can truly own my data by downloading all my transactions from Mint in csv, I’m also needlessly giving my data to a company that does questionable things.
Given my affinity for GNU/Linux, my quest to find an alternative personal
finance software landed me on GnuCash. While I’m sure it is a very capable
piece of software, my initial impression is that its GUI is complex,
unintuitive, and unappealing. But by trying it out, I was introduced to some
basic accounting concepts, including the double-entry accounting system,
which led me to
ledger, a command-line-based program that implements such a
system. I also found out that GnuCash has an extension that allows you to
download transactions from banks, which led me to its underlying protocol, OFX.
You can learn about
ledger from its homepage or a number of well-written
tutorials online, so I won’t rehash them here. Instead I want to talk about why
I like it from the following angles.
The most distinguishing feature of
ledger is its plain text approach—the
user supplies a text file containing all the transactions in a particular
ledger helps the user analyze this data by generating reports
according to the command.
ledger is a program that does one thing well, and because of its plain text
interface, it’s easy to combine it with other programs that do other things
well. For example:
It’s trivial to write a script to convert my previous transactions in csv into
ledger’s format, whereas it’s unclear how I would go about, for example, importing the csv into GnuCash precisely.
It doesn’t care how you record new transactions. You could manually write them with a text editor, or you could create a sophisticated system that uses a neural network that detects when you’ve made a purchase from the signals captured by your phone and automagically records the transaction into the text file.
You could use
gitor any version control system on the text file, without having to come up with a custom versioning and syncing solution, and without worrying about
ledgermessing things up—since it only reads the file.
You could pipe
ledger’s output to
gnuplotto create graphs and charts if that’s your cup of tea, or you could read the output using python and do some more serious data crunching.
ledger’s features page:
It figures out from looking at your data what you mean by it and how you want it reported back to you. Accounts are created as they appear; currencies are created as they’re referenced.
ledger makes no assumptions about your use case, so it’s possible to use it
for anything. For example:
ledgeris currency-agnostic, I could create my own units for frequent flyer miles, and they would be treated by
ledgerno differently than well-known units like “$” and “GOOG”. This means that if I were an obsessed member of the frequent flyer community, I could easily keep track of the cent-per-mile (cpm) of my mileage redemptions.
Similarly, I could craft a command that generates a report for a credit card, telling me how much I’m spending on its annual fee versus how much value I’m getting out of redeeming its reward points. Since I design my own account structure in
ledger, I could generate such a report for each individual card or across all cards, serving as valuable data in helping me decide whether the credit card game is still worth playing.
If the length of its documentation is any indication,
definitely not a “simple” program. But its complexity never made me feel
overwhelmed. In order to start using it, you could just follow a tutorial,
write a journal file using the most basic syntax, and run a few simple reports
to check your account balance or net worth. This can all be done within
minutes. When the need arises, you could dig into the documentation and for
example create a virtual account, set up some automated
transactions, or craft a complex expression to query for exactly
what you need. These can all be slowly built up while your text file is safe
under version control. There is no cluttered UI with dozens of undecipherable
menu options, where you worry that doing the wrong thing might corrupt your
data. There is just a text file and the command line; they are only as complex
as you need them to be.
Having the right accounting system is not enough—I would not be willing to use such a system if I had to manually put in every transaction. Fortunately, some companies behind some well-known commercial personal finance products created an open standard for exchanging financial data and pushed the banks to adopt it. It’s an ambitious protocol that basically supports everything you could do on a bank’s website—checking statements, transferring funds, paying bills, etc.
To talk to your bank using OFX, you make an HTTP POST request to a special URL
set up by the bank, attaching an SGML (or XML in the latest version) -formatted
payload containing your request, and the bank replies back in the same format.
There are a number of open source clients that take care of the heavy lifting
by constructing the requests and parsing the responses for you. (I am using the
Sounds lovely? Unfortunately, the real world execution leaves a lot to be desired. Each financial institution implements the protocol a little differently, if at all. PNC Virtual Wallet accounts and Capital One credit cards don’t support OFX. Chase decides to ignore the multi-factor authentication features of the protocol but instead send you a “secure message” in your account center containing a link that you have to manually log in and click in order to authorize your client. Best of all, Discover throws the HTTP specification out the window by passive-aggressively denying you unless you use a particular set of HTTP headers in a particular order—wat.
While these idiosyncrasies are never documented by the banks, there is a small but active community at ofxhome that created a crowd-sourced database of protocol parameters and a forum where people share their discoveries about each bank’s quirkiness. While I have my ideals for a protocol that’s better designed and more consistently implemented, this is what we currently have and it mostly meets my immediate needs, so I’ll take it.
Putting It Together
I now have a way of downloading transactions from banks, and I know I want
to organize them into
ledger’s text file. The missing piece is
a tool that converts csv into
ledger’s format, while taking care of
deduplication, and automatically categorizing transactions based on previous
So I end up with a python script that pieces all these components
together. I would run the script every couple of days to download and
categorize new transactions, keep the text file safe under
git, and use
ledger’s immense reporting power to answer any questions I might have
about my finances.
Note that my script also integrates with
pass, the command-line
password manager I use. Refer to this post for an overview of my security
While I would not describe this system as “easy to use”, there is a certain aesthetic behind piecing together open source software to create a uniquely customized solution that fits my needs. I have been using this system for 3 months and it indeed fixed the shortcomings of my previous solution outlined in the first section, but the process of discovery and experimentation gave me as much satisfaction as the end result. Through this exercise I’ve come to better appreciate the Unix philosophy, especially how plain text is the universal interface that allows easy composition of different programs that each does one thing well. While I would hesitate to eschew all modern software with a GUI, integrating more command-line software into my workflow has allowed me to see a special kind of elegance in this computing paradigm that I’m now unwilling to abandon.