Personal Finance with Ledger and OFX

In this post, I talk about how I keep track of my personal finance by piecing together some command-line-based open-source software (ledger, ofxclient, and reckon) and utilizing the open standard OFX for downloading transactions from banks.

Background

I believe there is value in using software to keep track of one’s personal finance. Some people might choose to do so because they want to be financially responsible adults. I mostly enjoy the process of collecting data about myself—I should at least possess the same data that various credit card companies and banks already have about me. This data could answer questions about my finances and provide insight into my lifestyle should the need arise, but I’m perfectly happy collecting and organizing it for its own sake—as a hobby.

I’d been using Mint for over three years. I’d log in every couple of days, diligently fix the inaccuracies of the automatically imported transactions, and put in those rare cash expenses when needed. It worked fine for the most part, but I never really liked this solution because of the following:

• Security: Giving a third-party service my bank passwords is never the correct security solution, no matter how much I “trust” that party. For some reason, banks won’t offer oauth.

• Flexibility: There are use cases I need that Mint can’t handle in any non-convoluted way, e.g. keeping track of other people’s money that’s physically in my accounts but not mine to spend, and recording stocks, frequent flyer miles, credit card reward points, or other liquid monetary assets that aren’t US dollars. I find myself creating separate spreadsheets to handle these, which seems silly.

• Philosophy: While I appreciate that I can truly own my data by downloading all my transactions from Mint in csv, I’m also needlessly giving my data to a company that does questionable things.

• Reliability: I’ve personally run into various bugs over the years, the last of which had no workaround and prompted me to seriously research an alternative to Mint.

Given my affinity for GNU/Linux, my quest to find an alternative personal finance software landed me on GnuCash. While I’m sure it is a very capable piece of software, my initial impression is that its GUI is complex, unintuitive, and unappealing. But by trying it out, I was introduced to some basic accounting concepts, including the double-entry accounting system, which led me to ledger, a command-line-based program that implements such a system. I also found out that GnuCash has an extension that allows you to download transactions from banks, which led me to its underlying protocol, OFX.

Ledger

You can learn about ledger from its homepage or a number of well-written tutorials online, so I won’t rehash them here. Instead I want to talk about why I like it from the following angles.

Composability

The most distinguishing feature of ledger is its plain text approach—the user supplies a text file containing all the transactions in a particular format, and ledger helps the user analyze this data by generating reports according to the command.

ledger is a program that does one thing well, and because of its plain text interface, it’s easy to combine it with other programs that do other things well. For example:

• It’s trivial to write a script to convert my previous transactions in csv into ledger’s format, whereas it’s unclear how I would go about, for example, importing the csv into GnuCash precisely.

• It doesn’t care how you record new transactions. You could manually write them with a text editor, or you could create a sophisticated system that uses a neural network that detects when you’ve made a purchase from the signals captured by your phone and automagically records the transaction into the text file.

• You could use git or any version control system on the text file, without having to come up with a custom versioning and syncing solution, and without worrying about ledger messing things up—since it only reads the file.

• You could pipe ledger’s output to gnuplot to create graphs and charts if that’s your cup of tea, or you could read the output using python and do some more serious data crunching.

Flexibility

From ledger’s features page:

It figures out from looking at your data what you mean by it and how you want it reported back to you. Accounts are created as they appear; currencies are created as they’re referenced.

ledger makes no assumptions about your use case, so it’s possible to use it for anything. For example:

• Since ledger is currency-agnostic, I could create my own units for frequent flyer miles, and they would be treated by ledger no differently than well-known units like “$” and “GOOG”. This means that if I were an obsessed member of the frequent flyer community, I could easily keep track of the cent-per-mile (cpm) of my mileage redemptions.

• Similarly, I could craft a command that generates a report for a credit card, telling me how much I’m spending on its annual fee versus how much value I’m getting out of redeeming its reward points. Since I design my own account structure in ledger, I could generate such a report for each individual card or across all cards, serving as valuable data in helping me decide whether the credit card game is still worth playing.

Well-Managed Complexity

If the length of its documentation is any indication, ledger is definitely not a “simple” program. But its complexity never made me feel overwhelmed. In order to start using it, you could just follow a tutorial, write a journal file using the most basic syntax, and run a few simple reports to check your account balance or net worth. This can all be done within minutes. When the need arises, you could dig into the documentation and for example create a virtual account, set up some automated transactions, or craft a complex expression to query for exactly what you need. These can all be slowly built up while your text file is safe under version control. There is no cluttered UI with dozens of undecipherable menu options, where you worry that doing the wrong thing might corrupt your data. There is just a text file and the command line; they are only as complex as you need them to be.

OFX

Having the right accounting system is not enough—I would not be willing to use such a system if I had to manually put in every transaction. Fortunately, some companies behind some well-known commercial personal finance products created an open standard for exchanging financial data and pushed the banks to adopt it. It’s an ambitious protocol that basically supports everything you could do on a bank’s website—checking statements, transferring funds, paying bills, etc.

To talk to your bank using OFX, you make an HTTP POST request to a special URL set up by the bank, attaching an SGML (or XML in the latest version) -formatted payload containing your request, and the bank replies back in the same format. There are a number of open source clients that take care of the heavy lifting by constructing the requests and parsing the responses for you. (I am using the python-based ofxclient.)

Sounds lovely? Unfortunately, the real world execution leaves a lot to be desired. Each financial institution implements the protocol a little differently, if at all. PNC Virtual Wallet accounts and Capital One credit cards don’t support OFX. Chase decides to ignore the multi-factor authentication features of the protocol but instead send you a “secure message” in your account center containing a link that you have to manually log in and click on in order to authorize your client. Best of all, Discover throws the HTTP specification out the window by passive-aggressively denying you unless you use a particular set of HTTP headers in a particular order—wat.

While these idiosyncrasies are never documented by the banks, there is a small but active community at ofxhome that created a crowd-sourced database of protocol parameters and a forum where people share their discoveries about each bank’s quirkiness. While I have my ideals for a protocol that’s better designed and more consistently implemented, this is what we currently have and it mostly meets my immediate needs, so I’ll take it.

Putting It Together

I now have a way of downloading transactions from banks, and I know I want to organize them into ledger’s text file. The missing piece is reckon, a tool that converts csv into ledger’s format, while taking care of deduplication, and automatically categorizing transactions based on previous ones.

So I end up with a python script that pieces all these components together. I would run the script every couple of days to download and categorize new transactions, keep the text file safe under git, and use ledger’s immense reporting power to answer any questions I might have about my finances.

Note that my script also integrates with pass, the command-line password manager I use. Refer to this post for an overview of my security solution.

Thoughts

While I would not describe this system as “easy to use”, there is a certain aesthetic behind piecing together open source software to create a uniquely customized solution that fits my needs. I have been using this system for 3 months and it indeed fixed the shortcomings of my previous solution outlined in the first section, but the process of discovery and experimentation gave me as much satisfaction as the end result. Through this exercise I’ve come to better appreciate the Unix philosophy, especially how plain text is the universal interface that allows easy composition of different programs that each does one thing well. While I would hesitate to eschew all modern software with a GUI, integrating more command-line software into my workflow has allowed me to see a special kind of elegance in this computing paradigm that I’m now unwilling to abandon.