1993

(This essay is from the introduction to On Lisp [blocked].)

It's a long-standing principle of programming style that the functional elements of a program should not be too large.

If some component of a program grows beyond the stage where it's readily comprehensible, it becomes a mass of complexity which conceals errors as easily as a big city conceals fugitives.

Such software will be hard to read, hard to test, and hard to debug.

In accordance with this principle, a large program must be divided into pieces, and the larger the program, the more it must be divided.

How do you divide a program?

The traditional approach is called top-down design: you say "the purpose of the program is to do these seven things, so I divide it into seven major subroutines.

The first subroutine has to do these four things, so it in turn will have four of its own subroutines," and so on.

This process continues until the whole program has the right level of granularity-- each part large enough to do something substantial, but small enough to be understood as a single unit.

A long-standing principle: the functional elements of a program shouldn't be too large. A component that grows past being comprehensible becomes a mass of complexity which conceals errors as easily as a big city conceals fugitives.

So large programs get divided, the larger the more so. The traditional approach is top-down design: the program does seven things, so you split it into seven subroutines; each gets its own subroutines, on down until every part is small enough to grasp as a unit.

It's a long-standing principle that the functional elements of a program shouldn't be too large. So large programs get divided — and the traditional way to divide them is top-down design.

Experienced Lisp programmers divide up their programs differently.

As well as top-down design, they follow a principle which could be called bottom-up design-- changing the language to suit the problem.

In Lisp, you don't just write your program down toward the language, you also build the language up toward your program.

As you're writing a program you may think "I wish Lisp had such-and-such an operator."

So you go and write it.

Afterward you realize that using the new operator would simplify the design of another part of the program, and so on.

Language and program evolve together.

Like the border between two warring states, the boundary between language and program is drawn and redrawn, until eventually it comes to rest along the mountains and rivers, the natural frontiers of your problem.

In the end your program will look as if the language had been designed for it.

And when language and program fit one another well, you end up with code which is clear, small, and efficient.

Experienced Lisp programmers also work bottom-up— changing the language to suit the problem. You don't just write your program down toward the language; you build the language up toward it. You think "I wish Lisp had such-and-such an operator," and write it.

Language and program evolve together — like the border between two warring states, drawn and redrawn until it rests along the natural frontiers of your problem. In the end your program looks as if the language had been designed for it.

Experienced Lisp programmers also work bottom-up: they change the language to suit the problem, writing the operators they wish they had. Language and program evolve together until the boundary settles along the natural frontiers of the problem.

It's worth emphasizing that bottom-up design doesn't mean just writing the same program in a different order.

When you work bottom-up, you usually end up with a different program.

Instead of a single, monolithic program, you will get a larger language with more abstract operators, and a smaller program written in it.

Instead of a lintel, you'll get an arch.

In typical code, once you abstract out the parts which are merely bookkeeping, what's left is much shorter; the higher you build up the language, the less distance you will have to travel from the top down to it.

This brings several advantages:

Bottom-up design isn't the same program reordered — you get a different one: not a monolith but a larger language of abstract operators, with a smaller program written in it. Instead of a lintel, you'll get an arch.

Bottom-up design isn't writing the same program in a different order — you end up with a different program: a larger language and a smaller program written in it. The higher you build the language, the shorter the distance down to it.

By making the language do more of the work, bottom-up design yields programs which are smaller and more agile.

A shorter program doesn't have to be divided into so many components, and fewer components means programs which are easier to read or modify.

Fewer components also means fewer connections between components, and thus less chance for errors there.

As industrial designers strive to reduce the number of moving parts in a machine, experienced Lisp programmers use bottom-up design to reduce the size and complexity of their programs. 2.

Bottom-up design promotes code re-use.

When you write two or more programs, many of the utilities you wrote for the first program will also be useful in the succeeding ones.

Once you've acquired a large substrate of utilities, writing a new program can take only a fraction of the effort it would require if you had to start with raw Lisp.

Bottom-up design makes programs easier to read.

An instance of this type of abstraction asks the reader to understand a general-purpose operator; an instance of functional abstraction asks the reader to understand a special-purpose subroutine. [1] 4.

Because it causes you always to be on the lookout for patterns in your code, working bottom-up helps to clarify your ideas about the design of your program.

If two distant components of a program are similar in form, you'll be led to notice the similarity and perhaps to redesign the program in a simpler way.

First, making the language do more yields smaller, more agile programs — fewer components, fewer connections between them, less chance for errors. Like designers cutting a machine's moving parts, Lisp programmers shrink their programs' complexity.

Second, it promotes code re-use: utilities written for one program serve the next, so a new one takes a fraction of the effort.

Third, it makes programs easier to read: this abstraction asks the reader to understand a general-purpose operator, not a special-purpose subroutine.

Fourth, it keeps you on the lookout for patterns: spot two distant components alike in form, and you may redesign more simply.

Bottom-up design yields programs that are smaller and more agile, promotes code re-use, makes programs easier to read, and — by keeping you on the lookout for patterns — clarifies your ideas about the program's design.

Bottom-up design is possible to a certain degree in languages other than Lisp.

Whenever you see library functions, bottom-up design is happening.

However, Lisp gives you much broader powers in this department, and augmenting the language plays a proportionately larger role in Lisp style-- so much so that Lisp is not just a different language, but a whole different way of programming.

Bottom-up design happens elsewhere too — wherever you see library functions. But Lisp gives far broader powers, so much so that it's not just a different language but a whole different way of programming.

Bottom-up design happens in other languages too — wherever you see library functions. But Lisp gives you far broader powers here, so much so that it's not just a different language but a whole different way of programming.

It's true that this style of development is better suited to programs which can be written by small groups.

However, at the same time, it extends the limits of what can be done by a small group.

In The Mythical Man-Month, Frederick Brooks proposed that the productivity of a group of programmers does not grow linearly with its size.

As the size of the group increases, the productivity of individual programmers goes down.

The experience of Lisp programming suggests a more cheerful way to phrase this law: as the size of the group decreases, the productivity of individual programmers goes up.

A small group wins, relatively speaking, simply because it's smaller.

When a small group also takes advantage of the techniques that Lisp makes possible, it can win outright [blocked].

New: Download On Lisp for Free [blocked].

This style suits small groups — and extends their limits. In The Mythical Man-Month, Brooks proposed that productivity doesn't grow linearly with group size: as a group grows, individual productivity goes down.

Lisp suggests a cheerier phrasing: as the group shrinks, individual productivity goes up. A small group wins simply because it's smaller — and armed with Lisp, it can win outright [blocked].

This style suits small groups but also extends their limits. Brooks said productivity per programmer falls as a group grows; the cheerful inversion is that it rises as the group shrinks. A small Lisp-armed group can win outright.

[1] "But no one can read the program without understanding all your new utilities." To see why such statements are usually mistaken, see Section 4.8.

A footnote answers the objection that no one can read the program without understanding all your new utilities — and shows why that's usually mistaken.

A footnote rebutting the worry that no one can read your program without first understanding all your new utilities.