Clojure Tradeoffs (design implications and why you should care)
EDIT: HN thread: https://news.ycombinator.com/item?id=5943982
Clojure as a language and community is very sensitive to the definition and design of tradeoffs. This post is an attempt to elucidate the tradeoffs chosen by the language, what they mean to interested parties, and an attempt to predict the future based on these choices.
Rich Hickey’s said a few things about design and the role of tradeoffs, in a recent talk he described design as consciously making choices about tradeoffs. He has another important design tenet: design by decoupling concepts from each other. So, clojure is in the interesting position of being an extra layer of abstractions that claims to actually simplify the task of programming in the long run. It does this by pulling apart concepts programmers take for granted in order to assemble them more effectively. Below are some tradeoffs I noticed through working with clojure for the past year and a half. Some of them, I had never thought about in my previous languages, but I can see that by accepting a language, I also accepted a set of tradeoffs that guided how I work. Because design tradeoffs (manifested as abstractions) determine what is easy and what is difficult, I think it’s valuable to see what tradeoffs are made. It’s valuable to know what they encourage, what they discourage, and how they interact, so that we can have more control over our tools and environments.
Clojure is a lisp. That fact alone means it builds on 50 years of infrastructure and thought, some of which has been absorbed into mainstream languages, but it also presents a foreign and scary interface to users that are used to other syntaxes. You can perform ‘syntactic abstraction’ at the expense of visual clarity, but it also becomes easier to express ideas and give them names. The programmer that works on a team is forced to become more judicious about these design choices, but the greater ease of expression means you’re never bound by what your language provides. It is the ultimate tool to identify and remove repetition.
Clojure has a concurrency focus. This tradeoff interacts with the decoupling desire in order to diminish the numbers and effects of problematic sites in programs. With regards to state change over time, this is similar to Python’s mantra, ‘better explicit than implicit.’ Even though Clojure has all the nifty concurrency features you could ever want, thread-pools, async methods, etc., the most important simplifying feature to consider is in fact immutability by default, which is a decoupling of state-change and value that most languages freely intermix. This, as a concept, is difficult to make compelling without an already captive audience, but there are great treatments of it from multiple sources. The pitch usually starts off with emphasizing the pain of building shared-state concurrent systems, but it is applicable and useful in other contexts.
The Value of Values: http://www.infoq.com/presentations/Value-Values
In practice, it means you don’t lose any capabilities of java, but since you accept that what’s easy, idiomatic, and low-friction in clojure is the right thing to do, you will by default write safe and moderately performant code. Reliable concurrency semantics fall out from the intentional convenience of a suite of core functions built around this ‘feature’. It’s a similar conversation to telling a C programmer that you’re taking away their malloc, but the data structures needed to express it are written in normal java. The brilliance of clojure’s core library is that it makes using these safe/fast immutable data structures (relatively) more convenient/effective than any other option, without artificially making java-like things less (absolutely) convenient/effective.
Shared-memory over other computing paradigms, ie message-passing.
Like C, C++, Java, Ruby, Python, etc.. clojure maps closely to the actual semantics of the machines that run those languages, which in our case is the Von-Neumann shared state bit-banging model. We generally don’t even think about this tradeoff, but there are examples of systems that hold some other construct as fundamental, such as Erlang’s actor model. In practice, this means there’s no wall of abstraction preventing you from using lower-level constructs that map efficiently to the hardware. You can write low-level code just as well as or better than java, while writing high-level code very easily. It’s simple to switch modes of thought and mix and match levels of abstraction due to the ‘Composition Tradeoff’. The resulting abstraction soup is a little unnerving at first, but you come to appreciate it after a few months of use. In my opinion, dealing with it is a worthwhile meta-skill :-).
Dynamic over Static
It’s the same in every dynamic vs static debate. Static languages have the advantage of stronger compiler support of domain-level assertions encoded directly into a type system. Dynamic languages trade that for increased flexibility, which is helpful when existing code is repurposed or used unpredictably. It becomes harder to reason about contracts of programmatic interfaces when the compiler and IDE isn’t helping you. Increased documentation is more necessary as a result. Automated tests can fill the role of compile-time assertions at run-time. As an added benefit, 90% of my time is spent in interactive development, building things in a live, running environment. Usually, static languages have a speed advantage, but in Clojure this is not the case…
Speed over Convenience
Clojurists love their neat dynamic tools, however they are also speed junkies. There is no pervasive run-time system in clojure to slow everything down, however idiomatic use promotes heavy use of the immutable data structures, which are optimized to perform competitively to other choices. More relevantly, Clojure lets you apply the 80/20 rule. Given the ease of interop with java, it’s possible to pick and choose your own performance tradeoffs to implement components, without losing the benefits of dynamic languages. For instance, Clojure Records provide fast java field-access for known-ahead-of-time fields, but they are also backed by a standard immutable hashmap for additional properties. They can be treated with the conveniences of standard hash-maps, but they are also efficient. Clojure’s deftype is equivalent to raw java if you need to go a step further. At the core, care is taken to provide fast implementations for common operations. Nothing prevents a user from using their own abstractions and data structures, and extending clojure’s abstractions over them.
Composition over Inversion-of-control
The emphasis on concurrency via shared, immutable data promotes standard, dynamic methods for libraries and functions to interact. Since a user can trust that data is immutable and reliable, there is no need for things like defensive copying as is standard practice (or should be) in multi-threaded object systems like java. It’s simply not easy or expedient to go out of your way to destroy someone else’s data. This trust in data integrity coupled with the syntactic abstraction afforded via macros and higher-order functions means you can write concise code that composes in intuitive ways, without the need to hook yourself into someone else’s sandbox (eg Spring or Rails).
The most troublesome thing about such frameworks is the use of polymorphism and inversion of control as a sledgehammer to get around the inherent problems of OO. Namely, OO couples state-change to objects (binds the effects of time to a specific bucket of memory), and functions to classes. Clojure, on the other hand, feels like nothing you write is actually ‘doing’ anything at all. Functions generally simply transform data, and occasionally you might fire off a side-effect or perform some coordinated state change. You can trust that there is usually a simple relationship of inputs to outputs. When you want polymorphism, you can get it in spades, but you’ll end up sprinkling it in occasionally instead of being bound to a particular style throughout the construction of your application.
When was the last time you tried to switch some Spring beans or Rails controllers over to another framework, or use two such frameworks in one application? This is problematic primarily due to inversion of control binding all your code to the framework’s assumptions. In clojure, you compose functions yourself, making more choices along the way, but the benefits of doing so coupled with the ease of dealing in data overshadows the need to trust in someone else’s choices. Code becomes actually reusable, and it usually even reads more like a tree-expansion than a graph traversal. The language features themselves are mostly orthogonal, and are similarly composable.
Community over Individualism
Lisp has a history of promoting an individualist spirit. There’s such a thing as the ‘Lisp Curse’ http://www.winestockwebdesign.com/Essays/Lisp_Curse.html . I personally believe Clojure is positioned to beat the curse, due to this generation’s emphasis on open-source, social media, friendly and productive chat rooms and newsgroups, and clojure’s intentional design decisions to promote interop between libraries. One example is Clojure’s standardized Lisp Reader, which is more restricted than Common Lisp’s, but enables source code to be shared more easily. There are excellent conferences with highly interesting talks, and the bootstrapping by java’s pre-existing momentum meant clojure was uniquely positioned to be useful at an early state. At this point, I feel there is enough momentum to keep clojure moving forward for the foreseeable future.
Long-term benefits over short-term approachability
Clojure optimizes for long-term use and long-term simplicity over familiarity and initial ease. However, at each decision point, there is compelling rationale driving the design decisions. Things are made very easy when not at the expense of primary design concerns.
Tradeoffs for individuals
Tradeoffs for companies
Companies have to worry about a number of things with regard to technology choices, namely there is a question of the ability to hire good developers to work in a language. Clojure is still not yet mainstream, and the developers are few and far between. However, if you manage to find one, you are guaranteed that they will be someone who cares about optimizing their workflow, productivity, and relevance. Interest in clojure is a good indicator of respect for the above tradeoffs and good design sensibility. The language and toolchain is increasing in popularity. The community is very much engaged and invested in its success, and it continues to grow. General hardware trends and competitive trends are going to push more interest in clojure’s direction, and the language itself will keep pace with innovation. The bottom line is that it takes a bit of effort to learn, but it stays out of the way and presents safety and simplicity as the convenient things to do. It integrates well with any JVM solution, and many companies such as Twitter leverage a JVM polyglot infrastructure that includes clojure. I think the most relevant analysis was the recent ThoughtWorks Radar: http://www.thoughtworks.com/radar , which both placed clojure in the ‘adopt’ category and promoted small composable libraries, a hallmark of clojure’s approach.
Tradeoffs for me
Personally, through my experiences at work, multiple conferences, IRC and newsgroups, I’m convinced that the Clojure community is a melting pot of innovation from many walks of developers. I have confidence that there won’t be a more relevant language for me for at least five years. Given trends in hardware, concurrency will become more of a driving force in language decisions, and I want to be on the cusp. For larger numbers of cores and distributed systems in the future, clojure is making its way into message-passing, and will certainly have a solid offering. I can stop worrying about languages for a while, and I can instead focus on the JVM platform itself and general systems problems until we hit a point where the tradeoffs are no longer a match for the systems that need to be built.
What they say about lisp is true, I really feel like I’m learning the truths of computing without getting bogged down by the act of expression. After I got over the initial hump, I now spend 99% my time thinking about the problems I’m trying to solve instead of fussing with the tools. When I have to learn something new about the language by doing a deep-dive, I always feel it’s a worthwhile exercise due to the readability of concise, idiomatic, well-composed code.
The right tool for the job
The ‘right tool for the job’ might be a more relevant thing to say in the material world, where you have to go somewhere to pay money for tools, and repurposing them would be too costly. In open-source tech, we are able to trade our time and speculation to improve our own tools. Taking advantage and contributing back to someone else’s tools is freely encouraged. A 50-year hammer has an opportunity to influence the design of a future toolmaker’s swiss army knife in combination with other tools built by experts from other disciplines. I hope I’ve been persuasive that it’s more interesting to talk about actual design tradeoffs and their implications.
In conclusion, Clojure’s a more right tool for more jobs than one might think. By making opinionated yet cautious choices, Clojure allows the user enough breathing room to compose and extend constructs in whichever way is appropriate, while maintaining a set of standards that promote safe, performant, and beautiful code.