Research is a process focused on answering technical questions which, at least initially, have unknown or uncertain solutions. This is in contrast to development, which is usually focused on implementing (mostly) known solutions. While there is no clear separation between the two, it can be useful to place tasks somewhere between the two ends of the research-development continuum.

While many words have been spent on principles of optimal development, fewer have been spent on principles of optimal research. Research comes in many forms - from long-term, fundamental, academic research to short-term, applied, corporate work. It may not be be possible to apply the same principles in all cases, especially where the approach has to work with the people, rather than the people with the approach. And the principles may not manifest themselves in the same way, each team and application is different. But principles are still be a useful reference point across the different forms, teams and applications.

Traditionally, research has been viewed as an individual, isolated task with intermittent formal collaboration - even in a corporate setting. Researchers spend time studying, then write lengthy documents (papers, specifications, etc.) and present them only when finished. However, this isn’t the only approach, and it’s often far from ideal. A more productive method can be called ‘Agile Research’, which abandons this individual approach and places collaboration and rapid iteration at its core. Drawing parallels to ‘Agile Software Development’, this concept can be further extended to ‘Extreme Research’, where each principal is taken to its extreme.

The principles presented here are far from new, but are rarely presented in this way.

Principles of Agile Research

1. Know your customer (and have your customer know you)

Know them, speak with them, interact with them.

Well defined scope. Know when to stop.

Communicate risk.

2. Collaborate constantly (but allow people space to focus)

Of course, all else being equal then two heads are better than one. But in practice, the worry is that two people working on one problem are less productive than two people working on two problems (or that two people collaborating on two problems are less productive than two people laser-focused only on their two separate problems).

But collaboration, when done well, should boost productivity rather than decrease it, even for focused work like research. Some models for collaboration, in order of increasing intensity, are:

a. Staying with a system of primary researchers per project, but with frequent feedback opportunities. This feedback can be in-person and synchronous - meetings, presentations, coffees etc - but should also fully exploit asynchronous opportunities - shared documents, code reviews, informal messages etc.

b. Breaking down projects into parallel steps, and attacking different parts of the problem simultaneously. These parallel steps should intersect at more than one point during the project, leading to feedback and shared learning.

c. Two researchers working together through the day, working on the same problem. Analogous to pair-programming, this can be termed pair-researching. Similar to pair-programming, this may not suit everyone all the time, but can be useful exercise. It may be tempting to use it for an imbalanced pair - an experienced researcher mentoring a new or junior team member - but can drive productivity more when in a balanced pair.

An additional benefit of constant collaboration is that it often requires some of the other principles listed here. It is difficult to collaborate without defining the project’s scope and outlining the expected steps involved, and frequent feedback can help with knowing when to pivot and when to ignore non-essential branches of work.

A downside to constant collaboration can be the lack of space to focus: being bombarded with requests for feedback, floods of messages begging for attention, meetings scattered through the day. Discipline over meetings is needed, and true asynchronous communication can help: requests and messages that do not require immediate response, but that can be dealt with at some point during the day. As with any endeavour that coordinates a group of people, it is necessary to allow time for taking stock and evolving the system.

3. Automate everything (that you can)

Productive research relies on good infrastructure. Continuously increasing the productivity relies on continuously investing in infrastrcture.

Build infrastructure to make this and future projects more productive. Automation also provides reliability and tracability.

Less extreme: proportional investment

4. Outline rapidly, fill in (and update) later

Break the task down into smaller steps (knowing that the steps may be wrong).

Produce an outline of documentation, fill in and edit later

5. Implement while you research (or ‘Do while you think’)

Don’t wait

Research isn’t a linear process.

Prototyping as experimentation

6. (Know when to) Take the tangent

A blessing and a curse of research is it’s open-ended, uncertain nature. It can lead to unanticipated solutions and places, even paradigm-changing conclusions that would not have been seen as in-scope at the beginning of the project. This should be seen as a blessing, and pursued and exploited where possible.

However, when unchecked, this can lead to endless rounds of investigation that take the company nowhere. In an academic context there is more freedom to pursue tangents because the risks of incomplete projects are usually lower, by design. In a corporate context, the risks are higher. Hence, as with all the principles, this principle needs to go hand-in-hand with the other principles. Specifically, ‘Know your customer’ brings knowledge of the scope and risks of the project, ‘Constant collaboration’ can bring feedback on scope-creep, and ‘Outline rapidly’ bring the ability to reevaluate priorities given new information.

Other References