Clojure Tradeoffs (design implications and why you should care)

June 26, 2013 3 comments

EDIT: HN thread: https://news.ycombinator.com/item?id=5943982

Clojure as a language and community is very sensitive to the definition and design of tradeoffs. This post is an attempt to elucidate the tradeoffs chosen by the language, what they mean to interested parties, and an attempt to predict the future based on these choices.

Motivation

Rich Hickey’s said a few things about design and the role of tradeoffs, in a recent talk he described design as consciously making choices about tradeoffs.  He has another important design tenet: design by decoupling concepts from each other.  So, clojure is in the interesting position of being an extra layer of abstractions that claims to actually simplify the task of programming in the long run.  It does this by pulling apart concepts programmers take for granted in order to assemble them more effectively.  Below are some tradeoffs I noticed through working with clojure for the past year and a half.  Some of them, I had never thought about in my previous languages, but I can see that by accepting a language, I also accepted a set of tradeoffs that guided how I work.  Because design tradeoffs (manifested as abstractions) determine what is easy and what is difficult, I think it’s valuable to see what tradeoffs are made.  It’s valuable to know what they encourage, what they discourage, and how they interact, so that we can have more control over our tools and environments.

LISPiness

Clojure is a lisp.  That fact alone means it builds on 50 years of infrastructure and thought, some of which has been absorbed into mainstream languages, but it also presents a foreign and scary interface to users that are used to other syntaxes.  You can perform ‘syntactic abstraction’ at the expense of visual clarity, but it also becomes easier to express ideas and give them names.  The programmer that works on a team is forced to become more judicious about these design choices, but the greater ease of expression means you’re never bound by what your language provides.  It is the ultimate tool to identify and remove repetition.

Concurrency

Clojure has a concurrency focus.  This tradeoff interacts with the decoupling desire in order to diminish the numbers and effects of problematic sites in programs.  With regards to state change over time, this is similar to Python’s mantra, ‘better explicit than implicit.’  Even though Clojure has all the nifty concurrency features you could ever want, thread-pools, async methods, etc.,  the most important simplifying feature to consider is in fact immutability by default, which is a decoupling of state-change and value that most languages freely intermix.  This, as a concept, is difficult to make compelling without an already captive audience, but there are great treatments of it from multiple sources.  The pitch usually starts off with emphasizing the pain of building shared-state concurrent systems, but it is applicable and useful in other contexts.

The Value of Values: http://www.infoq.com/presentations/Value-Values

In practice, it means you don’t lose any capabilities of java, but since you accept that what’s easy, idiomatic, and low-friction in clojure is the right thing to do, you will by default write safe and moderately performant code.  Reliable concurrency semantics fall out from the intentional convenience of a suite of core functions built around this ‘feature’.  It’s a similar conversation to telling a C programmer that you’re taking away their malloc, but the data structures needed to express it are written in normal java.  The brilliance of clojure’s core library is that it makes using these safe/fast immutable data structures (relatively) more convenient/effective than any other option, without artificially making java-like things less (absolutely) convenient/effective.

Shared-memory over other computing paradigms, ie message-passing.

Like C, C++, Java, Ruby, Python, etc.. clojure maps closely to the actual semantics of the machines that run those languages, which in our case is the Von-Neumann shared state bit-banging model.  We generally don’t even think about this tradeoff, but there are examples of systems that hold some other construct as fundamental, such as Erlang’s actor model.  In practice, this means there’s no wall of abstraction preventing you from using lower-level constructs that map efficiently to the hardware.  You can write low-level code just as well as or better than java, while writing high-level code very easily.  It’s simple to switch modes of thought and mix and match levels of abstraction due to the ‘Composition Tradeoff’.  The resulting abstraction soup is a little unnerving at first, but you come to appreciate it after a few months of use.  In my opinion, dealing with it is a worthwhile meta-skill :-).

Dynamic over Static

It’s the same in every dynamic vs static debate. Static languages have the advantage of stronger compiler support of domain-level assertions encoded directly into a type system.  Dynamic languages trade that for increased flexibility, which is helpful when existing code is repurposed or used unpredictably.  It becomes harder to reason about contracts of programmatic interfaces when the compiler and IDE isn’t helping you.  Increased documentation is more necessary as a result.  Automated tests can fill the role of compile-time assertions at run-time.  As an added benefit, 90% of my time is spent in interactive development, building things in a live, running environment.  Usually, static languages have a speed advantage, but in Clojure this is not the case…

Speed over Convenience

Clojurists love their neat dynamic tools, however they are also speed junkies.  There is no pervasive run-time system in clojure to slow everything down, however idiomatic use promotes heavy use of the immutable data structures, which are optimized to perform competitively to other choices.  More relevantly, Clojure lets you apply the 80/20 rule.  Given the ease of interop with java, it’s possible to pick and choose your own performance tradeoffs to implement components, without losing the benefits of dynamic languages.  For instance, Clojure Records provide fast java field-access for known-ahead-of-time fields, but they are also backed by a standard immutable hashmap for additional properties.  They can be treated with the conveniences of standard hash-maps, but they are also efficient.  Clojure’s deftype is equivalent to raw java if you need to go a step further.  At the core, care is taken to provide fast implementations for common operations.  Nothing prevents a user from using their own abstractions and data structures, and extending clojure’s abstractions over them.

Composition over Inversion-of-control

The emphasis on concurrency via shared, immutable data promotes standard, dynamic methods for libraries and functions to interact.  Since a user can trust that data is immutable and reliable, there is no need for things like defensive copying as is standard practice (or should be) in multi-threaded object systems like java.  It’s simply not easy or expedient to go out of your way to destroy someone else’s data.  This trust in data integrity coupled with the syntactic abstraction afforded via macros and higher-order functions means you can write concise code that composes in intuitive ways, without the need to hook yourself into someone else’s sandbox (eg Spring or Rails).

The most troublesome thing about such frameworks is the use of polymorphism and inversion of control as a sledgehammer to get around the inherent problems of OO.   Namely, OO couples state-change to objects (binds the effects of time to a specific bucket of memory), and functions to classes.  Clojure, on the other hand, feels like nothing you write is actually ‘doing’ anything at all.  Functions generally simply transform data, and occasionally you might fire off a side-effect or perform some coordinated state change.  You can trust that there is usually a simple relationship of inputs to outputs.  When you want polymorphism, you can get it in spades, but you’ll end up sprinkling it in occasionally instead of being bound to a particular style throughout the construction of your application.

When was the last time you tried to switch some Spring beans or Rails controllers over to another framework, or use two such frameworks in one application?  This is problematic primarily due to inversion of control binding all your code to the framework’s assumptions.  In clojure, you compose functions yourself, making more choices along the way, but the benefits of doing so coupled with the ease of dealing in data overshadows the need to trust in someone else’s choices.  Code becomes actually reusable, and it usually even reads more like a tree-expansion than a graph traversal.  The language features themselves are mostly orthogonal, and are similarly composable.

Community over Individualism

Lisp has a history of promoting an individualist spirit.  There’s such a thing as the ‘Lisp Curse’ http://www.winestockwebdesign.com/Essays/Lisp_Curse.html .  I personally believe Clojure is positioned to beat the curse, due to this generation’s emphasis on open-source, social media, friendly and productive chat rooms and newsgroups, and clojure’s intentional design decisions to promote interop between libraries.  One example is Clojure’s standardized Lisp Reader, which is more restricted than Common Lisp’s, but enables source code to be shared more easily.  There are excellent conferences with highly interesting talks, and the bootstrapping by java’s pre-existing momentum meant clojure was uniquely positioned to be useful at an early state.  At this point, I feel there is enough momentum to keep clojure moving forward for the foreseeable future.

Long-term benefits over short-term approachability

Clojure optimizes for long-term use and long-term simplicity over familiarity and initial ease.  However, at each decision point, there is compelling rationale driving the design decisions.  Things are made very easy when not at the expense of primary design concerns.

Tradeoffs for individuals

Certainly, by pursuing clojure, you are not pursuing other things, but the design elements are excellent to study, and the language itself is small.  Additionally, you are not leaving the JVM, which will be a relevant platform for many years to come.  Clojurescript has recently also come into existence, offering similar design tradeoffs targeting the Javascript VM, which will also stay relevant for the foreseeable future.  An investment in clojure will position the user to take advantage of many platforms in perhaps more convenient ways.  It’s not a wall of abstraction, and it doesn’t protect you from learning the host environment.  For these reasons, it’s less of an isolating ‘language’ commitment than other languages might be, while conferring substantial benefits.  Sure, languages come and go, but working in a lisp makes it easy to not get distracted by syntax, and to instead focus on semantics.  Time spent dealing with abstractions this way makes it easy to disregard superficial differences in other languages, and the experience makes it easy to learn them quickly should the need arise.

Tradeoffs for companies

Companies have to worry about a number of things with regard to technology choices, namely there is a question of the ability to hire good developers to work in a language.  Clojure is still not yet mainstream, and the developers are few and far between. However, if you manage to find one, you are guaranteed that they will be someone who cares about optimizing their workflow, productivity, and relevance.  Interest in clojure is a good indicator of respect for the above tradeoffs and good design sensibility.  The language and toolchain is increasing in popularity.  The community is very much engaged and invested in its success, and it continues to grow.  General hardware trends and competitive trends are going to push more interest in clojure’s direction, and the language itself will keep pace with innovation.  The bottom line is that it takes a bit of effort to learn, but it stays out of the way and presents safety and simplicity as the convenient things to do.  It integrates well with any JVM solution, and many companies such as Twitter leverage a JVM polyglot infrastructure that includes clojure.  I think the most relevant analysis was the recent ThoughtWorks Radar: http://www.thoughtworks.com/radar , which both placed clojure in the ‘adopt’ category and promoted small composable libraries, a hallmark of clojure’s approach.

Tradeoffs for me

Personally, through my experiences at work, multiple conferences, IRC and newsgroups, I’m convinced that the Clojure community is a melting pot of innovation from many walks of developers.  I have confidence that there won’t be a more relevant language for me for at least five years.  Given trends in hardware, concurrency will become more of a driving force in language decisions, and I want to be on the cusp.  For larger numbers of cores and distributed systems in the future, clojure is making its way into message-passing, and will certainly have a solid offering.  I can stop worrying about languages for a while, and I can instead focus on the JVM platform itself and general systems problems until we hit a point where the tradeoffs are no longer a match for the systems that need to be built.

What they say about lisp is true, I really feel like I’m learning the truths of computing without getting bogged down by the act of expression.  After I got over the initial hump, I now spend 99% my time thinking about the problems I’m trying to solve instead of fussing with the tools.  When I have to learn something new about the language by doing a deep-dive, I always feel it’s a worthwhile exercise due to the readability of concise, idiomatic, well-composed code.

The right tool for the job

The ‘right tool for the job’ might be a more relevant thing to say in the material world, where you have to go somewhere to pay money for tools, and repurposing them would be too costly.  In open-source tech, we are able to trade our time and speculation to improve our own tools.  Taking advantage and contributing back to someone else’s tools is freely encouraged. A 50-year hammer has an opportunity to influence the design of a future toolmaker’s swiss army knife in combination with other tools built by experts from other disciplines.  I hope I’ve been persuasive that it’s more interesting to talk about actual design tradeoffs and their implications.

In conclusion, Clojure’s a more right tool for more jobs than one might think.  By making opinionated yet cautious choices, Clojure allows the user enough breathing room to compose and extend constructs in whichever way is appropriate, while maintaining a set of standards that promote safe, performant, and beautiful code.

Categories: Clojure, Observations

Clojure environment state of the union, install speedrun screencast

December 19, 2012 6 comments

I’ve been speaking to a colleague about simplifying my workstation setup, minimizing the number of OS instance configurations floating around in my head. I have multiple computers at home and at work, and I need to be able to minimize the time it takes to be productive from point-zero. I thought this would be a great time to try and make a speedrun of getting up and running with a clojure environment, starting with a freshly downloaded ubuntu ISO image and using it as a VM.

All-in-all, it takes 24 minutes, about 8 minutes is waiting for stuff to download. I could practice and get it down to ~15, but this should be a good order-of-magnitude estimate of what it takes. Minutes 16-24 are hassling with emacs, elpa, auto-complete, and order-of-installation issues.

This also serves as a demonstration of what can go wrong when even a full-time clojure developer has to recreate a working environment, and what kind of knowledge of workarounds may be needed. I had some problems with emacs that I can’t explain, I just found a workaround through trial-and-error and my minimal experience :-). Having remembered what it was like to not know these secrets, I know that this hassle can expand into days of digging.

As a community, we should aim to minimize this sort of hassle for beginners. I know there have been ill-fated efforts to do so, with out-of-date documentation and half-usable but well intended tools, and I don’t have an answer for that, but here’s a data point.

Categories: Clojure, Uncategorized Tags:

Separation of concerns, or let-map, birth of a clojure macro

June 11, 2012 8 comments

Motivation

I’ve been doing substantial work in clojure recently, and it’s created opportunities for me to think about how I program more deeply. Of course, in itself that would be a useless diversion, but in fact the nature of lisp’s abstraction tools affects my workflow significantly. There is truth to the notion of growing your own language to do the job. I really get the sense that I am learning deeper secrets of expressivity and programming by having so much power over my code, and I spend very little time actually repeating myself. There are of course adjustments to be made coming from an imperative, statically typed background, and even when I thought I understood the language from about a year of study, books and toy programs, I’m finding there’s a lot more to think about when you use it full-time. Here, I provide one such real-world example and a walkthrough of one of my first proper user-defined syntactic abstractions (macro).

Consider the case of a ‘main’ method, the initial entry-point to a command line program. Here’s the simplest example:

(defn -main
  [& args]
  (do-things args))

Clojure’s runtime will pack the java String array that normally takes the role of a variadic arglist into a clojure Seq named args. This simple case of a main method is simply passing the args to a function that can handle it. However, this is suboptimal. It’s taking the burden of argument-parsing and placing it deeper in your application code. Really, if I saw this in the wild, my first impulse would be to separate the concerns as much as possible. The business code should be concerned with data on its own terms, and any adapters and bridges from other domains (shell-land, the internet, etc.) should live outside as meager servants. Core application/library code should be general and pure. Consider the case that do-things is of this form:

(defn do-things
  "Does the things"
  [{:keys [things-to-do when-to-do-them how-to-do-them]}]
  (actually-do-things things-to-do when-to-do-them how-to-do-them))

If you’re not familiar with the {:keys [..]} syntax, it’s a shortcut for pulling out values from associative maps, called destructuring. It’s a pervasive pattern in clojure binding forms, originating from common lisp. It hides the map lookup for each key behind syntax sugar, using a macro. We will take advantage of this ability later, but for now, think of ‘do-things’ as a function that takes a map of this form:

{:things-to-do ...
 :when-to-do-them ...
 :how-to-do-them ...}

The keys are clojure keywords, and the values are whatever they need to be. In my feeble mind, I think of this structure as a sort of anonymous type, a specification of a contract to the input of the function. Since clojure is so incredibly powerful at dealing with data, we can avoid writing code that would normally show up as api by taking advantage of rich declarative data. And code is data. Back to our -main function:

(defn -main
  [& args]
  (do-things (parse args)))

`

We’ve added an extra step, now there exists a ‘parse’ method. By separating out the parsing of the args, we can test parsing separately, however the parse method’s output is now coupled to the contract of the do-things function. In most cases, this is not a substantial improvement, but it’s nice to have the option and freedom. The -main function is becoming closer to being a simple glue binding data transformations together, where each piece can exist, vary, be reasoned upon and tested independently.

To fully decouple the parsing, -main has to have a transformation step, here’s a simple example of what our parse method could look like in this scenario. Our parse method is now simply concerned with the structure of the arguments and defines its own input/output contract. ‘some-magic-parsing-fn’ is some library function that provides a data-centric interface to the arguments, for instance a getopt-style command-line input can be parsed into a clojure map, where options and specs specify how the data is transformed. We take that raw transformation and apply some defaults and things in our custom parse method.

(defn parse 
   [& args]
   (let [parsed (some-magic-parsing-fn args)
         {:keys [in out munge]} parsed
         in (or in "default-in")
         out (or out (get-default-out in))]
     {:in in
      :out out
      :munge munge}))

Here we see a function that applies some defaults to the parsed arguments, even making the default of ‘out’ rely on a function of the ‘in’. So, we have our separately-concerned argument transformation, it’s independent from both our glue logic and our library code. All it’s really doing is munging, ‘destructuring’ and reassembling things into a new map form. But, something feels odd and a little repetitive, no? What the heck is this about?

{:in in       
 :out out
 :munge munge}

It’s just ugly! Why did I do that? It turns out, that having the ability to destructure in binding forms is hella convenient, and you can’t do anything similar inside a map literal. Neither can you reference values to key/value pairs that you’ve already defined inline. In my case I had to do this same sort of thing a few times, I thought it would be worth my while to make it pretty. Wouldn’t it be great to glue together a destructuring binding form that outputs as a map? That seemed totally like a useful thing, so I made it. It’s kind of a ‘restructuring’ operation.

Implementation

I asked around on IRC for some advice on how to do this and got some help. I didn’t come up with it, but I can explain it :-).

(defmacro let-map
  [pairs]
  (let [names (map first (partition 2 pairs))]
    `(let [~@pairs]
       (zipmap
        [~@(map (comp keyword name) names)]
        [~@names]))))

Defmacro is itself a macro in clojure. The clojure compiler provides a hook to run code as a macro, that is, at compile-time. The common use of a macro is to manipulate forms and spit out other code, that is, syntactic abstraction, though it can be used for lots of things. The structure of ‘pairs’ is expected to be a name (symbol) followed by a value, just like regular let. The first let extracts all the names from the bindings vector for later use. The backtick is a syntax-quote operator, it tells the reader to interpret the following s-exp as data, and it will not evaluate calls. The major difference over the normal ‘ quote operator is the addition of namespace-expanding any symbols, for instance:

user> '(let [a b c])
(let [a b c])
user> `(let [a b c])
(clojure.core/let [user/a user/b user/c])
user> `(let [a b c])

Since a, b, and c are not resolvable, they are defined to be within the current namespace.

So, now we spit out a proper let form. Within the bindings of the let we see this nonsense: ~@pairs. What is that?! The ~ (unquote) switches the next form back to value-mode, where symbols actually get evaluated into values. This is the mode we’re normally in when we write code. Since pairs is a seq of values, we have a problem. [[a b c d]] is not what we’re looking for. How do we un-nest the values from the inner vector? The @ operator is the key, when used in this context it splices (expands) the values from a seq out of the seq. So, we have our let-bindings, how do we output a map? We have to build up the keys and values. Zipmap is suitable for assembling a maps from seqs of keys and values. We create a seq of keys by applying ‘keyword’ to the name of each symbol, splicing them into a vector. We create a seq of values by simply splicing the names in value-mode. The compiler will automatically substitute values in-place of the symbols, as in regular code. So, that totally works, but we don’t yet have destructuring. Wouldn’t it be nice if we could make that happen? It’s not so difficult actually, with some caveats.

Let’s see what ‘destructure’ gives us:

user> (destructure '[[a b] [1 2]])
[vec__2314 [1 2] a (clojure.core/nth vec__2314 0 nil) b (clojure.core/nth vec__2314 1 nil)]

Hmm…. interesante. It’s generating symbols and function calls to pull out the values that we need.

What if we just run our bindings through destructure? Here’s the new let-map:

(defmacro let-map
  "Avoids having to build maps from let bindings manually"
  [pairs]
  (let [pairs (destructure pairs)
        names (map first (partition 2 pairs))]
    `(let [~@pairs]
       (zipmap [~@(map (comp keyword name) names)]
               [~@names]))))

user> (let-map [[a b] [1 2]
                {c :c d :d} {:c 3 :d 4}])
{:d 4, :c 3, :map__2495 {:c 3, :d 4}, :b 2, :a 1, :vec__2494 [1 2]}

There you have it, if you’re willing to tolerate some extra gensyms for the destructured maps, this will suffice, but I think we can do better. How about if we walk through our input and pick out the original symbols? Then we can assemble only the parts we want.

My first thought was to flatten everything, though ‘flatten’ only works on seqs and vectors. If we look at the code to flatten, it gives us a clue what to do.

(defn flatten
  "Takes any nested combination of sequential things (lists, vectors,
  etc.) and returns their contents as a single, flat sequence.
  (flatten nil) returns nil."
  {:added "1.2"
   :static true}
  [x]
  (filter (complement sequential?)
          (rest (tree-seq sequential? seq x))))

So, we want to walk the forms and identify any symbols. Destructuring works on vectors or maps, so it would make sense to expand only those. We look at the definition of tree-seq, which is a clever builder higher-order-function that creates a sequence based on a tree walk, branching when a predicate is true, and pulling out a seq of children by calling a function on the node. It’s the right tool for the job!

user> (tree-seq #(or (vector? %) (map? %)) identity [[5 6] [1 2] {7 :c 9 :d}])
([[5 6] [1 2] {7 :c, 9 :d}] [5 6] 5 6 [1 2] 1 2 {7 :c, 9 :d} [7 :c] 7 :c [9 :d] 9 :d)

So, we create a seq, then we simply filter it for symbols. Since I always wish smart guys would show their thought process so I can learn how to be like them, I will show you all my mistakes along the way :-).

(defmacro get-symbols
  [form]
  (->> (tree-seq #(or (vector? %) (map? %)) identity form)
       (filter symbol?)))
user> (get-symbols '[[a b] [1 2] {c :c d :d}])
()

Oops…

(defmacro get-symbols
  [form]
  `(->> (tree-seq #(or (vector? %) (map? %)) identity form)  ;hmmm, I might be forgetting something here
        (filter symbol?)))

user> (get-symbols '[[a b] [1 2] {c :c d :d}])

No such var: user/form
  [Thrown class java.lang.RuntimeException]

Oops.

(defmacro get-symbols
  [form]
  `(->> (tree-seq #(or (vector? %) (map? %)) identity ~form)
        (filter symbol?)))

user> (get-symbols '[[a b] [1 2] {c :c d :d}])
(a b c d)

Success! Now, let’s hook it up to let-map.

(defmacro let-map
  "Avoids having to build maps from let bindings manually"
  [bindings]
  (let [bindings (destructure bindings)
        names (get-symbols bindings)]  ; as long as it's in there, right?
    `(let [~@bindings]
       (zipmap [~@(map (comp keyword name) names)]
               [~@names]))))

user> (let-map [[a b] [1 2]
                {c :c d :d} {:c 3 :d 4}])
{:d 4, :c 3, :map__3140 {:c 3, :d 4}, :b 2, :a 1, :vec__3139 [1 2]}

…try again…

(defmacro let-map
  "Avoids having to build maps from let bindings manually"
  [bindings]
  (let [names (get-symbols bindings)   ; immutability decomplects time and value
        bindings (destructure bindings)]
    `(let [~@bindings]
       (zipmap [~@(map (comp keyword name) names)]
               [~@names]))))

user> (let-map [[a b] [1 2]
                {c :c d :d} {:c 3 :d 4}])
{:d 4, :c 3, :b 2, :a 1}

Shazam! And if you want to inline get-symbols, notice we don’t have to quote anything.

EDIT: we also want to only zip uniquely, and calling ‘name’ may be unnecessary
EDIT: only traverse the odd elements bindings for symbols, in case vals are quoted symbols

Final Code: let-map

(defmacro let-map
  "Avoids having to build maps from let bindings manually"
  [bindings]
  (let [names (->> (take-nth 2 bindings)
                   (tree-seq #(or (sequential? %) (map? %)) identity)
                   (filter symbol?)
                   (into #{}))  ; dumps it all into a set
        bindings (destructure bindings)]
    `(let [~@bindings]
       (zipmap [~@(map keyword names)]
               [~@names]))))

Here we can see the code the macro is producing:

user> (pprint (macroexpand-1 '(let-map [[a b] [1 2]
                                        {c :c d :d} {:c 3 :d 4}])))
(clojure.core/let
 [vec__3309
  [1 2]
  a
  (clojure.core/nth vec__3309 0 nil)
  b
  (clojure.core/nth vec__3309 1 nil)
  map__3310
  {:c 3, :d 4}
  map__3310
  (if
   (clojure.core/seq? map__3310)
   (clojure.core/apply clojure.core/hash-map map__3310)
   map__3310)
  c
  (clojure.core/get map__3310 :c)
  d
  (clojure.core/get map__3310 :d)]
 (clojure.core/zipmap [:a :b :c :d] [a b c d]))

TL;DR

What did we just do? We’ve put somewhat of a ‘design pattern’ into a single word. We can trade a little bit of effort for arbitrary expressivity. But we lose a little on having to grok the ever-more nested conceptual tree of abstractions. Personally, I think the tradeoff is worth it. Behold the power of lisp!

The final result, it would be much nastier if we had to explode out all the maps:

(defn parse 
   [& args]
   (let-map [parsed (some-magic-parsing-fn args)
             {:keys [in out munge]} parsed
             in (or in "default-in")
             out (or out (get-default-out in))]))

(defn do-things
  "Does the things"
  [{:keys [things-to-do when-to-do-them how-to-do-them]}]
  (actually-do-things things-to-do when-to-do-them how-to-do-them))

(defn -main
  [& args]
  (let [{:keys [in out munge]}]
    (do-things 
      (let-map [things-to-do (make-things in)
                when-to-do-them :now
                how-to-do-them (munging munge out)]))))
Categories: Clojure

Dynamic JVM language Complexity and Implementation Line Count

January 14, 2012 10 comments

Yesterday, I was hanging out in the #clojure IRC chat (pretty good bunch of guys).  A fella mentioned that he was impressed with how easy it is to read and understand clojure’s guts, and I totally identified with his sentiment.  I’ve always found it informative and enjoyable.

His words:

“im pretty sure i know more about the internals of clojures functions in a few years of dabbling than i know about python after many years of serious use”

Now, I don’t mean to python-bash, I myself was very interested in python in much the same way as I am in clojure now, in fact I volunteered on the AV crew at pyCon 2009 and learned a ton of good stuff. I recommend an experience like this to anyone that can do it.  Of course, I’ve come to love clojure for the same reasons I loved python, plus the extra concurrency features, metaprogramming and speed.

In python, I always had the sentiment that there was some magical __hack__() function you could mess with to do weird stuff.  In general, there felt to be a whole layer of that underpinning everything.  You get the sense that there’s some untouchable magic.  I pride myself on doing deep-dives into whatever I’m working on and understanding it thoroughly at every level of abstraction.  I like the security and agility that comes with knowing what my code actually does down to the machine-level.  It means nothing’s sacred.  I am comfortable changing my code around quickly and I can generally grok others’ code pretty well.  In the couple of years that I really cared about python, I never reached this level, even with the great talks I saw at pyCon that explained a lot of internals.   Maybe if I spent some more years on it, the situation would improve, but lately I’m just more interested in learning CS fundamentals and actually getting stuff done.  Still, I could work in python without knowing every detail, and it felt better than java.

Clojure doesn’t feel like that at all.  In fact, to me it’s like a thin veneer over java.  I think I understand the gist of the whole implementation (minus macros, crazy data structure stuff and STM implementation details).  All the features are pretty orthogonal.  Yet, when people talk about clojure, they say it is powerful, terse and clear (I think).

I decided to do a little line count of the language implementations for fun.  I chose jython and jruby instead of their C implementations as it seems more apples to apples.  I chose jruby for the contrast to jython, since I was initially under the impression that they’re pretty similar.  They all share a garbage collector and underlying java infrastructure to build from, so ideally the code should be representative of only the language’s syntax and semantics, along with the standard library.  I downloaded the latest version of each implementation from their github pages, and took care to only count code in the src/ subdirectories, not tests.  Hopefully this is some indication of relative complexity.

Too much talking.  Here’s results:

JRuby:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                          1230          35707          50104         191259
yacc                             3            394            121           3986
Ruby                            36            239            196           1452
HTML                             1              0              0             65
-------------------------------------------------------------------------------
SUM:                          1270          36340          50421         196762
-------------------------------------------------------------------------------

Jython:

--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Java                            436          15226          15396          94203
Python                           10            916           1032           3192
Bourne Again Shell                1             23             33            152
DOS Batch                         1             29             28            101
--------------------------------------------------------------------------------
SUM:                            448          16194          16489          97648
--------------------------------------------------------------------------------

Clojure:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                           146           5485          11086          35386
Lisp                            38           2286            828          14557
XML                              2              0              0             81
HTML                             2              6             58             71
-------------------------------------------------------------------------------
SUM:                           188           7777          11972          50095
-------------------------------------------------------------------------------

Now, what can we observe here?

Firstly, both jython and jruby are mostly implemented in java.  Clojure on the other hand has a disproportionate amount of high-level code.  I think this might be due to performance philosophy more than anything.  I bet if the implementors could get the same speed implementing ruby-in-ruby or python-in-python, they would (see PyPy and Rubinius).  Clojure has a strong performance focus, and clojure code is compiled as statically as possible, which means you can write more performant stuff in high-level code (I know JRuby has gotten much, much faster lately with the invokeDynamic work).  Also, since it’s a lisp, it bootstraps itself from meager beginnings incrementally. I encourage everyone to study core.clj to see how awesome this is.

Secondly, wtf, clojure is 1/2 the size of jython and 1/4 the size of jruby.  Why?  Well, it’s a lisp.  This means that a parser is pretty trivial with LispReader.java clocking in at 1k lines.  Comparatively, the jython antlr package is 6k, and the jruby parser is 19k of java (4k yacc).  Once it’s parsed, it’s all native data structures that can be worked on with the normal sequence functions.  The others have crazy AST tree-visitor things.

How do we explain the rest of the size difference?  I’m not really sure.  I think a lot of it is actual language semantics, perhaps a substantial amount of code could be in an expanded standard library, but I highly doubt that jruby’s library is THAT much more useful than jython’s.  Of course, both those languages have the burden of implementing a standard library that acts the same on native code and the jvm, so I imagine there’s a lot of wheel re-invention.  Also, clojure’s relatively new.  I have no doubt that it will expand, but I trust it will keep comparatively slim.  Fun fact: if you pretend clojure code is 10x as dense as java (some say that), you still get 181k , less than jruby :-).

So, I know relatively little about ruby, and a bit more about python.  The conclusion I want to put forth isn’t that they’re doing it wrong or we’re better.  I just want us to ask the question, is all that extra stuff really necessary for the power you get from it?  And, wouldn’t you want to know how your tools work, too, or is it just a weird fixation?  Do people like to program by faith, or is it really not so hard to hold those languages fully in your head?

EDIT:
Groovy:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Java                           828          20549          51344         103197
Groovy                         147           2899           5407          12944
HTML                            62            153            165           1829
XML                              6             23             42             79
CSS                              1              7              7             14
-------------------------------------------------------------------------------
SUM:                          1044          23631          56965         118063
-------------------------------------------------------------------------------

Groovy is representative of a dynamic language that is loyal to only the JVM.  It should be more comparable to clojure since it does not have the burden of multi-platform semantics.  The antlr package alone (parser) is 7k lines of java.
Scala (a little complicated):
main src directory:

--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Scala                          1507          31557          64947         136732
Java                            202           5734          14834          26034
Javascript                        7             97            114           1217
XML                              18             74             70            973
CSS                               4            195             98            866
HTML                              1             19             13            168
Bourne Again Shell                1             24             35            143
Bourne Shell                      1              1              4              3
--------------------------------------------------------------------------------
SUM:                           1741          37701          80115         166136
--------------------------------------------------------------------------------

just the compiler directory:

--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Scala                           456          14648          22478          77911
Javascript                        7             97            114           1217
CSS                               3            179             97            804
Bourne Again Shell                1             24             35            143
XML                               2              0              4             30
--------------------------------------------------------------------------------
SUM:                            469          14948          22728          80105
--------------------------------------------------------------------------------

Scala’s here just for fun. it’s a static language, so it doesn’t quite fit in with the rest.

Categories: Observations

A StringTemplate Spring MVC integration that doesn’t suck

July 20, 2011 4 comments

I’ve been experimenting with all this web stuff that’s going on these days, and my platform of choice is java.  I have some experience with Spring MVC (2.0 sucks but 3 is ok), but I didn’t feel like using JSP.  StringTemplate is a template engine invented by the ANTLR folks to do code generation, and is a general purpose, fast template engine.  It enforces clean separation of code from presentation.  StringTemplate 4 is pretty recent, and that’s what I’m using for this experimentation.  Anyway, here’s the shiny bits:

Firstly, set up your build environment.  I use maven to pull in dependencies.  Add this to your pom.xml:

1 <dependency>
2 	<groupId>org.antlr</groupId>
3 	<artifactId>stringtemplate</artifactId>
4 	<version>4.0.2</version>
5 </dependency>

Now you need to inform Spring that you’ll be using StringTemplate instead of the standard JSTLView.  Here’s where things get hairy.  Spring uses the InternalResourceViewResolver class to resolve views, and it will instantiate a class that you configure it to use.  What this means is, spring calls new(), and this class doesn’t participate in bean lifecycle processes.  If you want dependency injection, this is one case where you’re hosed unless you find a more elaborate way to do things.  This is as far as I’ve cared to take it.

1 <bean id="viewresolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver" >
2 	<property name="viewClass" value="com.garytrakhman.common.BeanView" />
3 	<property name="prefix" value="/templates/" />
4 	<property name="suffix" value=".st" />
5 	<property name="requestContextAttribute" value="rc" />
6 </bean>

The magic happens in my BeanView class. This class is configured to access a factory and get the real bean view resolver. This enables me to unit-test and configure the actual logic using dependency injection.

 1 public class BeanView extends InternalResourceView {
 2 	private IViewResolver viewResolver;
 3 	{
 4 		Config config = ConfigFactory.getInstance();
 5 		viewResolver = config.getViewResolver();
 6 	}
 7
 8 	@SuppressWarnings({ "rawtypes", "unchecked" })
 9 	@Override
10 	protected void renderMergedOutputModel(Map model,
11 			HttpServletRequest request, HttpServletResponse response)
12 			throws Exception {
13 		viewResolver.renderMergedOutputModel(model, request, response,
14 				getBeanName());
15 	}
16 }

Now the delegation logic has been set-up. We now need the actual StringTemplate loading logic. Spring uses reflection to instantiate the InternalResourceView class, then we override that single method to take control of view rendering.

 1 import java.io.PrintWriter;
 2 import java.util.Map;
 3
 4 import javax.servlet.http.HttpServletRequest;
 5 import javax.servlet.http.HttpServletResponse;
 6
 7 import org.springframework.beans.factory.annotation.Autowired;
 8 import org.springframework.stereotype.Component;
 9 import org.stringtemplate.v4.AutoIndentWriter;
10 import org.stringtemplate.v4.ST;
11 import org.stringtemplate.v4.STGroup;
12 import org.stringtemplate.v4.STGroupFile;
13
14 import com.garytrakhman.common.IViewResolver;
15 import com.garytrakhman.common.impl.URLMap;
16 import com.google.common.collect.Maps;
17
18 @Component
19 public class STViewResolver implements IViewResolver {
20
21 	// assume thread safe modifying access to templates, adaptors, attributes,
22 	// imports, and renderer access on STGroup/ST (they are synchronized).
23 	// http://www.antlr.org/wiki/display/ST4/Differences+between+v3+and+v4
24
25 	@Autowired
26 	private URLMap urlMap;
27
28 	private Map<String, STGroup> groups = Maps.newConcurrentMap();;
29
30 	@Override
31 	public void renderMergedOutputModel(Map<String, ?> model,
32 			HttpServletRequest request,
33 			HttpServletResponse response, String viewName) throws Exception {
34
35 		ST template = getView(viewName);
36
37 		template.add("model", model);
38 		template.add("url", urlMap.get());
39
40 		PrintWriter writer = response.getWriter();
41
42 		template.write(new AutoIndentWriter(writer));
43
44 		writer.flush();
45 		writer.close();
46 	}
47
48 	private STGroup getGroup(String key) {
49 		STGroup out = groups.get(key);
50 		if (out != null) {
51 			return out;
52 		}
53
54 		out = new STGroupFile("templates/" + key + ".stg");
55 		out.delimiterStartChar = '%';
56 		out.delimiterStopChar = '%';
57 		groups.put(key, out);
58 		return out;
59 	}
60
61 	private ST getView(String viewName) {
62 		STGroup group = getGroup(viewName);
63 		ST template = group.getInstanceOf(viewName);
64 		return template;
65 	}
66 }

What’s happening here is pretty simple.  The object sets up a concurrent cache of the template groups (each file is a group).  They are organized in my src/main/resources/templates directory, and it uses the view name as the key to load the file.  Once a template is requested a second time, it loads straight from the cache.  Since these templates are compiled and fast, it makes for some very speedy behavior.  In my testing with JMeter, I was able to perform ~9k requests/sec with 100 threads hammering on a simple view on a core i5 2600K, running on jetty.  I also set-up some default keys/values for the templates, mainly the model and url keys.  You could easily use the request objects or anything else.  StringTemplate provides flexibility by accessing your beans with reflection or letting you implement adapters to map template data to objects.  So far, I’m very happy with it.

Categories: Uncategorized

Update

April 3, 2011 Leave a comment

I’ve started a new challenging job. One way that it’s unexpectedly hard is the difficulty of learning an existing code-base. Previously I was writing mostly new code, but I think it’s highlighting a fundamental issue in the way we’ve designed software for the last 20 years or so.

The promise of OOP is modularity, reuse, and general lowering of complexity. But, I think there is a fundamental mismatch between the ideals of objects and types, their interactions and how we think. My approach so far has always been an imperative one, mainly I think about what it’s going to do as I build my mental model of a code-base.

So, the first point that I want to make is as developers we all aim for simplicity and conciseness. We have certain requirements that our code should meet. Static type systems like C++ and Java promise that we can encode those requirements as types and function signatures. So, we do that, and generally take it to an extreme for maximum effectiveness. Encapsulation promises that we can encode our contracts in types. Polymorphism gives us some flexibility later, but the main issue is we have to create huge infrastructures of code to encode our requirements, and polymorphism is necessary to make it not suck as much every time we change something.

As I’m reading over a code-base, I have to re-discover the original intent that was lost in the ceremony. There’s cruft all around, and I’m following trails and back-tracking with some sense that what I’m looking for might be just around the corner, and some kind of hunch for predicting whether I’m getting closer or not, all the while learning what the code is going to do and getting bogged down in implementation details.

I propose a few things that would help.

Firstly, if we keep going the way we are, only able to encode contracts by types, I want a tool that can index call stacks, then find a path through from one type to another. I want to know what the main entry points are, for instance do I have to create a factory that takes an inner class enum for it’s argument to get access to a transformation to get the type I want? I should easily be able to generate a chain of function calls to get to the type I want. This would also make it simple to find the main entry points and highlight the designer’s intent.

Secondly, I want some way to formally specify the inputs and outputs of everything that is succinct, easy and helpful to use, lives outside of implementation details, and would be tool-friendly.

Thirdly, I want the capability of performing a semantic code search using these formal constructs. We really should be past the point of searching by text.

Categories: Observations

Game engine, OOP thoughts

January 1, 2010 Leave a comment

I think I have learned something valuable recently in thinking about how I will construct my game objects, and I should share.  I was considering the implementation details of embedding a physics engine into my objects, and I felt considerable apprehension about using multiple inheritance, though it was the most obvious thing to do.  I also considered a templated mixin approach, since I had seen it when using the opensteer pathfinding library for a class project.  Eventually, this led me to what I think now is the right answer.  The downside of the obvious thing, a crazy inheritance hierarchy, is I’d be making it very hard to change my mind if  something were to come up later that I hadn’t planned on, such as a new functionality or ability.  I’d have to blow up my interface classes, or make some awful hacks to make it work right.  The correct thing to do is to use components and has-a relationships as much as possible within the definitions of classes.  Multiple sources reference Design Patterns:

(1) program to an interface and not to an implementation

(2) favor object composition over class inheritance

There must be a good reason these folks that are much smarter than me say this, and I think this commitment to complexity is the issue that made me feel so nervous about adding functionality through multiple inheritance.  Simplicity makes life easier.  My scene graph structure should not have any idea of the physics details (or sound or whatever), that’s something for the classes to figure out.  If I am stingy with my inheritance, I’m less likely to get stuck somewhere along the way or need to hack something.  My game objects (scene graph nodes) can have-a physics object just as easily as they can be-a physics object, but the implementation ends up less tied down if I use components where it makes sense.  I think it will also give less troubles when I start thinking about multi-threading.  For instance, I could have a separate data structures for each type of component (physics objects, sound sources, etc.), and I wouldn’t need to parse the scene graph all the time.  Each subsystem would only be exposed to what it needs to in order to do its job.  This feels more right.

Some articles I found:

A talking door broke everything:

http://forums.tigsource.com/index.php?topic=10112.0

Interesting read:

http://gamearchitect.net/Articles/GameObjects1.html

Categories: Observations
Follow

Get every new post delivered to your Inbox.