Being Organized is No Fun

The number one professional skill to have is the ability to be organized about your work.  If you have a good organizational system and mindset, then no matter what your profession, you will almost certainly succeed. Given the importance of being organized, it surprises me how lacking many people (even highly paid ones) are in this area.  

I believe we have even created a bit of a social stigma against being organized.  Being accused of being "OCD" or "anal" simply because one wants to do something in an organized or systemic way is one example of this negative bias.  "Too much process" is the another phase used in work environments to ensure things continue to slip through the cracks and bad communication patterns persist.  

There is this saying: 

"Let's organize this thing and take all the fun out of it."

So maybe the collective population just wants to have fun and I am just a curmudgeon.  

Though there is a lot of truth to that quote, I think it is best applied to your individual activities and hobbies, not things where a lack of thoughtfulness is going to negatively impact other people, your career or your company. 

Assuming you are an organized person, or want to become one, you will need to adopt or come up with some sort of system and find tools (software) to help you.  Everyone is a bit different and not every system/tool is going to work for everybody. There's far too many tools out there claiming to help you be organized. Most of them are lacking and many of them catering to people that think using some organization tool will magically transform them into an organized person.  Kind of like fad diets, gym memberships or exercise equipment: most people buy them with an ambition, but then never realize the benefits because there is no silver bullet: it has to come from within, not from a subscription payment.

My Organizational System

That is all the lead up to my describing the organizational system I have developed over the years.  This is what I use for work-related organization where there are tasks, projects, meetings, people, etc. This is the result of many iterations and refinements over decades. The tool I use is org-mode inside the Emacs text editor, but my system can work with any tool that supports some fundamental requirements.  I'll point out what I believe these requirements are that makes org-mode one of the best tools out there for being organized.

My more recent experience have been in management, so my system has developed with a bias toward those needs. If you are someone whose work tasks are defined by others, you may not need something like this because some other tool is likely where your tasks and priorities will be managed. When I was a software engineer, I also used this system, but my roles usually had broader responsibility with a lot of self-directed work and project management. 

The Solution

My Real Notes File (top levels)

I use a single text file. It is always open and I edit it within Emacs using the Org Mode package. That package just adds some features that allow it to interpret the text with a markdown type syntax and syntax highlighting.  The files names I have used for this are "ToDo" or "Notes", neither of which is a good name for what it contains, but consistency matters more than good name and I am the only one that needs to know.

Is a single file scalable? Yes.  In three years of keeping copious notes, though many projects, tons of  responsibilities and with it including the entire history of all my work, the file from my last job sits at just over 57K lines and is just over 2MB in size. With the computers of today, this size is trivial and splitting this up into multiple files or some fancy database would just be overkill, adding unnecessary complexity and overhead.  I think I have had more varied responsibilities and was involved in more initiatives than the average person, so this file size is probably a lot larger than many would have after three years.

This file does not contain all the detailed assets I need for projects, but serves as a to-do list, a jumping off point and a "diary" about those action items, meeting notes, reminders, handy reference material, etc. How I organize the different categories of information in this file is discussed in more detail below.

Hierarchical Note Taking

Notes File with Hierarchy Expanded

Any note-taking app that cannot organize information in a nested, hierarchical way should not be considered an organizational tool.  Hierarchical structures are the most natural way to organize information: it is how our brains work. We operate on multiple levels of abstraction all the time, in work and in life, so there is no hope in capturing information in a natural way without this feature.

It must also be trivial to manipulate the hierarchy: moving items, promoting or demoting items, or shifting around sections of the hierarchy.  The more friction there is to re-organizing the structure, the more likely you will avoid doing it and wind up with something out of sync from your mental model, reality and/or needs.  Any time the information in your system does not match your mental model, you are adding friction that diminished the value of the tool.

Status and Timestamps

All items, and especially "ToDo" items need some indicator about their state.  We do not want to actually delete anything, since there is value in history, so we need to be able to differentiate those things that still need to be done and those that have already been done.  Labeling items with its state and easily being able to change its state is crucial.  In Org Mode, a simple keystroke toggles the state.

Org Mode also allows me to customize the states and the ones I have found useful are:

• TODO - Needs action on my part.  The order they appear in the file defines the relative priority of my action items.  If there is a deadline, I annotate this with a square bracket comment. e.g.,
• TODO [Due May 5] Create Board Presentation
• WAIT - I am blocked waiting on someone or something.  I usually put a note in square brackets to remind me what I am waiting for. e.g.,
• WAIT [Email reply from Tom] Communicate new security policy
• DONE - Things I have accomplished that have no more work for me. Org Mode also automatically adds a timestamp when something is moved to the "DONE" state and this is handy for referencing in the future.
• DEFERRED - A variation of "DONE" to indicate that it was not completed, but that I consciously decided to deferred this (usually indefinitely).  Anything that is TODO that I decided to do later would still remain "TODO", but just being moved down the list in priority.
• DELEGATED - A variation of "DONE" where the task was done by someone else. I'll usually annotate who I delegated it to for future reference.

Searching

Seamless and fast searching is an essential feature.  With a single text file, in Emacs I can search everything quickly with a few keystrokes.  Anything I need to reference from the current or past is accessible nearly instantly, even in a 2MB file.

Linking

One of the features of Org Mode is that it will render hyperlinks in color and make them clickable.  My notes file is the "meta" view of my activities and thoughts, but they often relate to the details of projects or references. An organization tool need to allow you to trivially allow to navigate from your tasks and notes to the detailed, external documents.

Privacy

This file is meant to be private, for my eyes only.  I treat this like the most sensitive document that is liable to contain my inner thoughts that I may not want anyone else on the planet to read.  I do this because I do not want to have to think twice about what I add there nor have the extra cognitive friction of trying to word things differently than they initially come out in my mind.  Capturing notes and idea as quick as possible is the the important part, not word smithing. Thus, I am not interested in access for anyone but myself and a tool with a "sharing" feature actually represents a risk.

I also only use this file in the context of work, on my work laptop, so I do not have much of a need for it to be accessible from multiple devices (i.e., cloud based).  It might be a little convenient for me at times, but I'd rather contain my working hours to my working hours, so ubiquitous access is more of a burden than a featue.  Given my distrust of for-profit corporations, storing this file in a 3rd party database also represents a risk.

I do want a backup though. I do this with a simple copy script that puts it on my home desktop machine. I work from home, so my work laptop and person computer are on the same private network.  I copy it with a date in the file name to preserve history in case there is any disastrous event or mistaken edits I want to revert.  A revision control system is a possibility, but then I have to treat that repository in a similarly secure manner.  If I had multiple files, I would definitely use revision control (via a local, private git repo), but since I am only managing a single text file, a simple shell script to copy works fine.

Workflows

Here is a view of the way information flows in my system:

• The information flows into the notes file from any and all sources: deep thinking, conversations, emails, etc.
• This information is usually in the form of something I need to do (TODO) or something I would like to discuss with someone (PEOPLE) and flows into one of those sections.
• At the beginning of the week, and sometimes also at the beginning of the day, I will look at the order of the items in the TODO section and reprioritize as needed. I find it a good way to start the day to take a broad view of the upcoming tasks. (Daily I usually just prioritize for the day, but weekly, I will look at the list more deeply and try to set overall priorities.)
• Interrupting someone every time an discussion idea pops into my head is not good for productivity, so I accumulate discussion items in the PEOPLE section, in the sub-node associated with their name.
• Some discussions items may need immediate action, but I try to consciously not interrupt people unless I really need an immediate answer.
• The accumulated items in the PEOPLE section for a person defines the agenda for our next 1-on-1 meeting.
• I also include recurring 1-on-1 meeting agenda items in the PEOPLE section (if applicable).
• During meetings, I take notes, but any action item I sign up for in the meeting I will indicate with an exclamation mark at the start of the line in the meeting notes.
• At the day of the day (or next day), I review all my meeting notes and also look for any of the action items I have, which I will add to the TODO section. 
• I do not add TODO items during meetings because I like to let ideas simmer a bit and do not want to be worrying about where in the priority order they should go in the TODO section during the meeting.
• Once I have taken care of the action items for a meeting, I set the state of that node in the Meeting Notes section as DONE to reflect that I have reviewed it. When I get busy, sometimes a few days may pass before I review my notes, usually when they contain lower priority action items.
• Sometimes the meeting action item is trivial, and I just do it rather than creating a TODO item.
• At the end of the week (or so), I'll take all the TODO items in the DONE state and move them the the HISTORY section.
• I deliberately do not move completed TODO items immediately to the history section because it is often useful to reference recently completed items and often times when you first think you are done, something comes up and it turns back into a TODO. 

Content

Here is the full list of the top level categories in the notes file and details about how I use them.

• TODO - All the tasks I need to do, in priority order, often containing nested TODO items for subtasks, especially for bigger scoped projects or multi-steps processes. Accumulates thoughts, notes, references, etc. about the task. For things requiring documents or communications, I usually put a subsection for the draft of the outline or content. I cut and paste the draft into some other system when I need to share and add some styling.  I always prefer to focus on the content first and doing it in a text file ensures I am not distracted with making it pretty while the ideas are still forming.
• PEOPLE - First divided by categories of people, then a node for each individual people.  Besides discussion items and 1-on-1 agenda items, I often have a "Background" section where I add any personal information I find out about them (e.g., their birthday, or work anniversary date). For my last job, I had the following divisions for people:
• members of the management team;
• direct reports;
• vendors/reps;
• members of the engineering team;
• "others" for everyone else; and
• "defunct" for people that I no longer interact with (e.g., they left the company).
• MEETING NOTES - The first level division here is by date and then  there is a division by each meeting I have that day.  This could be project-related meetings, vendor rep or individual 1-on-1 meetings.  I create the dated section name at the beginning of the day, or with the first meeting of the day.  I create the subsection for the meeting at the start of the meeting, or sometimes before if I want to put some notes ahead of time.  Even if I do not wind up with any notes for the meeting, I have a node for that meeting with the person or topic I met with as I have found it handy to know when meetings happened.  Action items coming out of the meeting will be annotated with an exclamation mark as the first character of the line that describes what I need to do.  Action items from meetings will get turned into TODO items later.
• RECRUITING - Particular to my role of being involved in recruiting people.  I create a subsection for each candidate to hold notes and agenda items. I also notate what position they were interviewing for and what type of interview it was (phone screen, tech screen, etc.)  My official candidate assessment goes into a different system, but the notes form the meeting help jog my memory when creating that assessment report.
• HISTORY - Where all the TODO items go after they have been marked DONE and are in no danger of being re-opened. Either once a week I move things, or if the accumulated DONE items starts to grow large and is cluttering up the TODO section.  I often will create a separate history section for every 6 months to aid in finding things later.
• REFERENCE - What's our corporate address, tax ID number? How do I get a shell in a Docker container? There's a collection of useful info I run across and need infrequently, so I'll gather it here in categories to save me time looking it up elsewhere.

Meeting Notes Expanded View

Auxiliary Tools

Although I rely heavily on this text file to organize my work, this does not cover everything and I need a few other tools to make a more comprehensive organizational system.

• Calendar reminders (for recurring or scheduled tasks).  
• These calendar tasks wind up as emails in my inbox and since action items from emails flow into the TODO items, this makes sure I capture these when action is needed.
• Meeting reminders.  
• I review my week's upcoming meetings at the start of the week, and the day's meetings at the start of the day. 
• If prep time is needed for the meeting, I block off time on my calendar for that work. 
• The event and meeting notification reminders, by default, fire at 5 and 1 minute before the meeting: both are necessary:
• 5 minutes - tells me to wrap up any current meeting I may be in, take a bio-break if I need one before the meeting, or generally makes sure I don't get too involved in some task.
• 1 minute - What seems like something I can do in 5 minutes risks leading to a longer distraction and resulting my being late to the meeting.  This happened often and resulted in my being late often enough that I added this one minute reminder to my system.

Conclusion

I've evolved this process to this point over decades and it seems a fairly simple organizational system to me.  However, this was the first time I ever tried to write this down, and after seeing how much I needed to explain, I now question the premise of it being easy for anyone to adopt.  Still, I hope some of the ideas in here may have some value in adopting or refining whatever organizational system you may work with.  

Digital Transformation, Really?

Judging by the solicitations I get from various vendors, "Digital Transformation" seems to be a big buzz word in the business circles.  I find this strange.  When the Internet and Web first took off, 20 to 25 years ago, I could see companies needing a "digital transformation".  But now?  Maybe there's some fringe businesses that still rely on paper, or very large legacy use cases but for any company that is less than 20 years old, what do they mean by trying to sell me digital transformation services? And what larger company has not been working on doing things "digitally" for the last 20+ years? Why is this term still being used? Is it just the consulting industry's ploy to sell us services?

The Startup Equation

This is what I call the startup equation for staffing.  If time is "T" and work is "W", then:

 

At no point in time at any of the startups I worked at did we have enough people to do all the work we needed to be doing.  At first I complained, then I understood and now I accept it.  If you are at a startup and feel you need more people, understand that it is no one's fault and there is nothing to be done about it: it is just the way the math works.

Albers Projection for Maps

 I wrote this Python code a couple months ago when I needed a simple stand-alone way to plot cities on an SVG representation of a map.  There were no good, standalone Python references I could find.  Even when I found something in other languages, the readability of it was somewhat lacking.  

It seems the more math you know, the fewer characters you put in your variable names. Sigh.  Are they using meters or miles? Are they using radians or degrees? You have no idea from looking at the code, so you *have* to understand the math to be able to decipher it.

Developing this code required some investment of time in not just the math, but the geographic uses of the Albers projection. I figured it might be generally useful, so have made the code available in GitHub here:

https://github.com/cassandra/geo_maps

It includes an example that deals with some tricky issues that can come up if you want to plot points against a map of the U.S.  

When you see those U.S. maps where they have relocated Alaska and Hawaii to be in the lower left of the map, it is obvious they are not in the right location and are of different scale than the contiguous 48 states. However, did you know that they are using completely different projection parameters *and* have rotated them?  I learned this the hard way.

Starting Over

I read a thought provoking article:

How to hire senior developers: Give them more autonomy

It contains the provocatively titled section "Your Code is Worthless", and it makes a good case for this being true.  The context here is not that the running code is worthless to the business, but that the code itself is of no value to anyone else. i.e., Worrying about a competitor stealing your code base is pointless.

A basis of the argument is that the software engineering process is not about creating an artifact (the code base), but about building a "theory".  Another way to express this idea that I have seen is: the software development process is a group exercise in the organization of information.  The code is nothing more that the way to capture the outcome of that collaborative process.

To reach the conclusion that the resulting code is worthless, the article borrows from work by Peter Naur (of BNF fame) which says it is impossible for software, or its documentation, to encode all the information that was collectively organized and which is necessary to efficiently maintain it.  

There are a lot of good supporting details in that article, so it is definitely worth reading. Many of the points are aligned with my experiences.  The one conclusion that got me thinking is the premise that it is cheaper and faster to start from scratch than to try to adopt an unfamiliar code base. I do agree that this is true for a competitor getting a hold of other company's code. But could this apply to the company that owns the code?

What if a company had 100% turn-over of their developers, is starting over the best choice for the company?

Engineers usually tend to think a rebuild is the right answer in this case, but the business side never does. Is the business perspective wrong?

I have been in a situations where there was 100% turnover in a complicated code base.  I pushed to keep some of the previous developers on contract because I knew how crucial it was to leverage the information in their brains. Even if a rewrite was the right answer, this was not going to happen instantaneously and the business needed to keep operating. 

So maybe it is this continuity aspect that changes the equation.  If you have no choice but to figure out the code base, then a rewrite is only an added cost, not a replacement cost.

Can We Please Keep it Simple?

There is a famous quote about software that I have heard attributed to Brian Kernighan:

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

I do not think you would find a software developer that disagrees with that statement, yet the field is overrun by a culture that seems to value succinctness more than readability.

When I began to write a lot of Perl, I wrote it like I wrote C programs and caught the ridicule of the Perl wizards who told me it was "better" to rewrite it as something that would wind up looking like this:

@P=split//,".URRUU\c8R";@d=split//,"\nrekcah xinU / lreP&&
close$_}%p;wait until$?;map{/^r/&&<$_>}%p;$_=$d[$q];

I continued to write the code simply with the fewest "Perl-isms" I could. Our hiring budget would only allowed us to hire mere mortals to help maintain it.

Ten year later, the pattern repeats itself. When I write Python, people point out all the extra syntax I do not need, even though it is my deliberate attempt to make things more explicit and readable. Optimizing character and line counts seems to be a reflex for many programmers.

I have grown to abhor this sentence:

"Look what I can do in just one line of code."

This is usually pitched to me in the context of someone trying to convince me of the wonders of the latest, greatest programming language.

Of all the languages I have learned, there have been only a handful of language advances I have encountered where truly "better" syntax was introduced in terms of readability.  Some of them include:

• try-catch exception handling
• "finally" blocks
• "else" for for loops
• list comprehensions (but only if used judiciously)
• string interpolations

In general, if the syntactic feature is specific to a language, I will avoid using it.  I shift around writing code in many languages and the more I can do to minimize the context shift, the better.

With software, readability is 99% of the problem to be solved. As Martin Fowler has said:

"Any fool can write code that a computer can understand.  Good programmers write code that humans can understand."

I was inspired to share these thoughts after reading this article that also makes the case for keeping it simple:

 Simplistic programming is underrated

I have also recently read this related article:

Developers spend most of their time figuring the system out

This is definitely in line with my experiences and helps to emphasize how important code readability truly is.

Collective Fictions of Software Methodologies

Software methodologies are a collective fiction. Necessary, but a fiction nonetheless.  The author of this article is where I first saw this concept:

My 20-Year Experience of Software Development Methodologies

My experiences and conclusions have been similar.  A software team needs something to organize them, but exactly what is used does not matter that much.  Really, all you need is:

1. A system to organize the work
2. A system to communicate.

All the software methodologies provide these elements. The bottom line is that any methodology can work with good people and every methodology will fail with bad people.  

As one of the cartoons in that article alludes to: the goal of a methodology is predominantly about avoiding chaos.