My Own Private Audit Trail...

23 Dec 2012 - Nic Ho Chee

When I was a child I felt that I had a mind like a steel trap, any fact that made it through my conscious barrier would be there forever. I took no photos since my memory would suffice, and taking photos was too invasive anyway... As I grew older I realised that rather like someone suggesting that 640 KB would be enough memory for anyone my younger self had failed to comprehend the sheer amount of data that would pass through my grey matter during my working life. As a computer professional I can honestly say that I have not stopped learning since I was even younger than the young self above, with new code, facts, problems and processes presenting themselves quite literally every day.

Over time I've realised that my grey matter is rather less like a trap and more like a Primary/Secondary/Tertiary cache stack. Issues and languages I've used in the past year are in my brain's primary cache and I've got pretty much instantaneous recall of the data there, anything in the last five years takes a few seconds or more to recall, anything longer than that and I'm reaching into my tertiary cache and there is a good chance that the data might have been subtly or entirely corrupted along the way.

This might not seem like much of a problem, however given the instinctive way in which a human reacts to problems that slight corruption could lead you down a blind coding alley. Daniel Kahneman has been looking into this with his work on cognitive bias covered in the book Thinking, Fast and Slow. As humans we appear to have a couple of modes of thought, one which is fast/automatic/stereotypic and another which is slow/effortful/conscious. As a programmer with almost a couple of decades of coding under your belt, some of your reactions to problems will start to become automatic, which in most cases will be correct, but in some will be woefully inadequate. I have worked with a very small number of people in which their responses to problems had become so automatic that they ceased to learn anything new and stagnated as programmers, remaining stuck using techniques and languages which were twenty years out of date.

This is not a place you want to be as a knowledge worker.

In a bid to reduce the probability of falling into this rut I realised I had to take a few actions very early on in my career:

I would need to create a process to capture my decisions, learning and work items so that I could cogitate over them at a later date.
I would need to try and ensure that my approach to work was a conscious decision, that I applied the maximal mental surface to problems I encountered to reduce errors and traversal of blind alleys.
As much as possible I would need to expose myself to new processes, techniques and languages in a bid to stave off Programming Calcification.

Item three seemed a matter of keeping abreast of literature and blogs, so I spent some time on item one. The decision capture started as a Work Diary in a series of text files. As the data there began to grow I noticed that my behaviour started to change, I became much more focused on the issues and work items that needed to be fixed rather than constantly context switching as items were thrown over the wall. At the time we did crazily long hours, had a number of people/teams attempting to get us to prioritise their work above others and with no real ticketing system for work or bugs. Now they were logged in my work diary, and I could quickly place them in order of priority, forget about them and continue with what I was doing. I'd reduced the cost of context switching with very little effort. As I completed an item I could go to my work diary, pick the next item off the stack and get it done.

Keeping a TO DO list is a relatively obvious step, but by choosing a decent item format and judiciously capturing the right granularity of data over time you can make some surprising discoveries. Over time I started to realise I could mine my own data for interesting emergent behaviours in a similar way as Dr Wolfram outlines here. You can for example:

Start looking at your past efforts to see which tools actually saved you time rather than were a perceived time saver.
Understand the scope of previous undertakings, what worked and what didn't.
Keep a more accurate time log to show which processes should be targeted with automation through scripting, and what kind of time you can spend doing this before you waste more time scripting than you would save.
Discover un-expected long term trends.

After a short time I realised that text files or spreadsheets wouldn't cut it so I started looking around for other means of data capture. I settled on MediaWiki to capture my work items/learning/thoughts. I've written a couple of blog posts about setting this up on both Windows and Apple Mac. If you're computer literate and can spare a weekend I can highly recommend setting up your own wiki and starting to record how much time you spend on various tasks over the day, and what you're doing. You'll begin to process the data and find some un-expected behaviours emerge.

Work as an ever-present conscious decision should start to become second nature as you have a large amount of verifiable data to sift through. You can even create testable hypotheses to the patterns that you start to see appearing in your private and working life that might have not been discoverable previously.

Creating my own audit trail has made my work and general life better, give it a try and see what it can do for you.