Code Goes In, Art Comes Out

This talk was given at Clojure/conj 2018, and focuses on some of the interesting questions around practicing generative art.

[A transcript, lightly edited for readability, follows]

Thank you so much for coming. I am thrilled to talk about a topic that I really love, which is artwork, and specifically, I'm going to be talking about artwork that is created through programming: what you might call generative artwork. This talk isn't going to be as much of a technical hands-on guide to making generative artwork, as it is an exploration of some of the interesting philosophical questions that come up around the practice. Generative art is a relatively new medium, and it's significantly different from some of the traditional art forms that preceded it, so there are a lot of new questions that I think are really interesting about it, and they're particularly pertinent to us as programmers. I will spend some time talking about my actual software setup, and the tools that I use, but before I do that, I want to give you just a taste of the artwork that I've created, so that you have an idea of where I'm coming from.

I make artwork that typically has no input. It's not any sort of data visualization or image processing. Everything that's needed to generate the image is contained within the program itself, and whatever the program outputs is the final product. Typically, I'm working with digital images as the final product, but I don't strictly do that. Sometimes I break into the physical world, and I create things like this, which is a plotter painting titled Bring the End.

This is an actual painting, but it was created with a plotter, which is sort of a simple robot, and I'll talk a little bit about that later. I have primarily worked with abstract imagery, but more recently I've been experimenting with integrating hand-drawn components, so I wrote a custom interface to allow me to use digitally drawn input via Wacom tablet as a starting point for the algorithms. In this case I drew the outline of a figure, and pass it into the algorithm which takes over from there, and breaks it into different shapes, and assigns all the colors, and the details, but primarily this [the final image] is the sort of stuff that I've been making over the last five years or so.

Here are the tools that I'm using: I'm using Quil, which is a Clojure library. It's a wrapper around another Java library called Processing, which has been around for, I think, 10 or 15 years at this point. Processing is kind of a basic graphics library. It has a relatively simple API for working with shapes, and lines, and images. Processing was originally written as a type of Java library with its own strange IDE, but you can wrap it with Clojure using Quil. Processing was also ported to Java Script, and Quil also wraps that through ClojureScript. You have your choice of which platform you want to work with.

This is just a very simple program to give you an idea of what it looks like to set this up. There's very little boiler plate. This program won't draw anything particularly interesting. It's just kind of a mess of lines around the image, but what's nice about this is you can dive straight into creating the image, and the APIs are relatively straightforward.

Down there at the bottom, I've got the (begin-shape), a bunch of calls to (vertex), and then (end-shape), and that's 90% of how I'm using these APIs. Occasionally, I'll do some fancier things like layers and masks, but this is the core of it. You can work with the system as either vector graphics or raster graphics, so you have access to an actual pixel array. You can pick different renderers to use. You can work in 2D or 3D, although I prefer to work in 2D myself. You can do animations with this, so up top I have that (no-loop) call that prevents it from doing this, but ordinarily it will call a draw repeatedly for each frame, so you can refresh, and redraw, and so people create animations that way. They also create interactive software that way. You can take keyboard input, mouse input, Wacom tablet, it's just like a fancy version of a mouse, so it's easy to create interactive software with these systems, as well.

So why did I pick Clojure for this? Once I figured out there were actually tools for this, it got a lot better. Unfortunately, I started out with Matplotlib, which is about as painful to use as you would imagine, but Clojure is a lot nicer, and I think the main reason is that creating visual imagery is really a type of data processing. You're working with these shapes, curves, points, pixels. All of these are very conveniently represented as simple data structures, and so it maps very nicely onto kind of the typical Clojure style of operations of setting up these data transformation pipelines over simple data structures. It applies to this domain very well, and it also applies very well to audio, so there are Clojure libraries like Overtone that allow you to do generative audio, and those work nicely, as well.

Of course, all of you in here know that Clojure is fast to write, that's one of the reasons why I think we all like it, but especially for artwork, you're not really pre-planning this architecture. You're in more of an exploratory experimental mode, just jamming things out, and running it to see how it looks. So, especially with the integration with the REPL, where you can reload a module, and rerun your code, it makes for a very tight feedback loop, which is perfect for this type of exploratory work. And of course, compared to other Lisp dialects, Clojure's big advantage is that you have access to the JVM, which is both an excellent piece of engineering, and an excellent ecosystem. There's all these Java libraries out there that you randomly need, and it's nice not to have to write everything from scratch every single time. Of course, the JVM is relatively performant. It does well with large heap sizes.

If you find yourself in a spot where you need to optimize a bunch of array operations, you can drop into Java, and do that very easily, although I find that very uncommon. And of course if you actually want to deploy to the browser, you can do that via ClojureScript. If you are making interactive software, you start to have to deal with shared state that might be operated on concurrently, so the standard Clojure tool kit of atoms and agents, especially agents, tends to be useful for this style of work, and you also want to be careful not to block the animation thread, and so tools like core.async are very handy for doing this style of work. This last point is a little bit more speculative, but I kind of feel that having a more flexible language puts you in a better mindset for doing this type of creative work. It just helps you to think out of the box a little bit, and of course, it's just a lot more fun to work in a language that you enjoy.

How do you make interesting code with art?

Okay. So, over these few years while I've been working on generative artwork, there's been a few big questions, that I spent a lot of time thinking about, and the answers to these are not really that straightforward. Maybe that's not surprising, but I think there's some interesting things that come out of them. So, the first one I want to talk about is how do you make interesting artwork through programming. I mean, this is really the crux of generative artwork. But, I need to be careful about what I'm saying here: "interesting art" is a loaded term. What does that mean exactly? It can mean a lot of things. There's a lot of ways that art can be interesting, but I think one of the relative attributes here is that good art does things that we didn't predict in some way. It has an ability to surprise or teach us something new or maybe view something in a different light. And this is kind of what separates good art from craft. But, we have this need to understand the artwork, as well.

We really don't like it if we can't ever wrap our heads around it. Some artwork may be difficult to digest, which is okay, but eventually we want to feel like there was some meaning or structure or organization behind this artwork. It can't just be randomly tied together. Even if we can't develop a rational understanding of it, we have to have some kind of intuitive understanding of the artwork in order to enjoy it. So, if we appreciate art that wasn't predicted, does that put us in a tough spot when we're working with programming? Programming, one of its strengths that makes it a useful tool in the first place, is that it is very predictable. We can write these systems where we understand what will happen in any given state, and barring any sort of bugs, it's an extremely predictable way of working.

This seems to conflict with our desire to make unpredictable artwork, but a lot of generative artists have tackled this problem very successfully, so I'm going to talk about a few approaches to that. My own approach, and an approach that several artists that I follow also seem to use, is really a style of programming that's more of a guideline rather than an exact description of what should be created. To give you a visual example here, these are three images created from the same program titled Holmberg:

Without any code changes, just running it multiple times, I get two pretty different results, but they still seem to be very aesthetically related in some way. Almost none of the pixels match, but somehow they're tied together. The way that this is accomplished is by using randomness very selectively, and very carefully, all the way from high level structure to fine details, and for elements like color selection.

This algorithm is based primarily on a type of recursive triangle subdivision, and randomness is used to do things like instead of splitting every triangle perfectly, it might split it a bit off center or it might use a curve instead of a straight line. It might recurse to one depth in certain areas, and not in others, and it tries to keep those distributed in an interesting way. Colors may be inherited from one parent triangle to its children, but it also may randomly switch to another color with various probabilities, so this really is... The program is me pushing in a certain direction, but not explicitly laying out what needs to happen. I'm still trying to leave room for the program to surprise me with unexpected results, just type of guided randomness.

There's another artist that I really enjoy (Manolo Gamboa Naon). His name is very hard for me to pronounce, so I'll avoid butchering it, but I have two images here, which I believe are from the same program:

Again, you can see that even though they're quite different they maintain this kind of aesthetic consistency, and so this is kind of a popular way of working. There's other artists you can find that seem to take this approach. There's also some very different approaches that tend to rely on more emergent properties, and emergent structures, so what I mean by emergence in this case is you have this very simple set of rules that doesn't look like it's going to do anything interesting, but if you run it repeatedly over time, these large super structures start to present themselves that you wouldn't be able to predict just from looking at the rules.

The basis of this is really related to chaos theory, and a very simple example of a chaotic system is a triple pendulum. This is basically three pendulums attached to each other. Even though these are very predictable objects on their own, when you combine them you end up getting systems that are chaotic, meaning that any small difference in the initial state results in a very large difference in the final outcome.

So, some really good artwork that I enjoy by Jonathan McCabe here. He's working with kind of an advanced form of cellular automata. A lot of you may know what that is, but for those of you who don't, perhaps you've heard of Conway's Game of Life, which is a form of cellular automata. Essentially you have a grid with cells, and those can be on or off, and you step through time, and there are simple rules governing when cells turn on and off like if they have a certain number of neighbors that are turned on, they'll be on in the next step.

This is more complicated system. This is what he calls a multi-scaled Turing pattern, so he's using cellular automata for, sort of one for the macro scale features, and one for more detailed features, but there's an incredible amount of variety in the output here, and every single output from this program looks completely different. It's really wild, and yet it still has this kind of coherent structure that we still somehow understand. That's pretty remarkable.

Another kind of common approach is sort of like cellular automata, but without the grid, and something a little more complex than just cells, so in this case we have what I called agents or actors that move within the system, and affect it in some way. And these tend to also interact with each other, so if two agents collide, perhaps they change the color that they're drawing. You can set these up with simple rules, and just like with the cellular automata, you run a lot of steps, and it builds up this complex imagery. This also works really well for animation, but this is just a still.

One of the more recent developments is this come out of machine learning. I think neurography is a good term for it. This is artwork that was made by Mario Klingemann, and I have a few works by him here. This artwork is made with something called a GAN, a Generative Adversarial Network. It's a type of neural network, and it's trained in a particular style, but really the two levers that the artist has when working with these are, first of all the training set, so the collection of images that the network is trained against. And their other real point of control is the objective function. So this is how the network grade itself or scores itself to tell if it's doing a good job, and how it should change to do a better job, but there's a crazy amount of variety that comes out of these. This is just the smallest glimpse of what he's done. I definitely recommend checking out more of it.

These styles of artwork are all quite a bit different, but I think the common thread that ties them together is that they do an excellent job of blending randomness and structure. They're able to keep this balance between the two to where it's still surprising, and unpredictable, but they still have some type of order that we understand even if it's only intuitively. And one image that kind of demonstrates that very cleanly for me, this is a work by John Greene, who makes tiling artwork:

Obviously a tiling system like this is extremely ordered, but he manages to also introduce randomness in ways that create a lot of really beautiful variety. So, I think we can satisfactorily answer the question of “how do you make interesting art with programming?”. We can say that it's possible. There's different approaches for creating surprising results through programming.

How can we turn our aesthetics into code?

So, the next real interesting question is more related to, I guess, the philosophy of generative artwork, and where it's going in the future. And it's can we turn our aesthetics into code, right? When I'm writing these programs, what I'm trying to do is in some way encode what I like generally into the program without making it super precise, but art is a very intuitive practice. It's guided by a lot of gut, intuition, and you might make spurious decisions about what to do, and programming is almost the opposite of that. It's very explicit. It's very ordered. It's very structured. And so, this is a really unusual mixture to try, and join these things. And so, a lot of generative artists start to wonder: "Can I make a program that generates an endlessly interesting stream of images that are all different?" How far can I take this?

I'm going to narrow that problem down to just color. Imagine your task is to create a program that picks colors for every shape in an image to produce something pleasing. It seems like a doable task. Color as a three-dimensional space is pretty straightforward. But as you dig into it, it turns out that color's extremely relative. Any one color is affected by all the colors that surround it, and it's affected by the proximity of those colors, and it's affected by the size of them. Not only that, but color has all these cultural connotations for us. We associate colors with different emotions. We attach colors to objects, and to locations, and even to periods of time. And really good artists will use colors in unexpected ways, or ironic ways, and so it becomes pretty clear that color is actually not just a three dimensional space. It's tied into all this extremely high dimensional contextual space, and if you want to write a program that can do this, what you're really trying to do is create a program that's a new artist itself.

I mean, it requires nothing short of strong AI to really do this, and at that point, the definition of art becomes even muddier than it already is. There is an interesting short story (The Death of the Artist, in Novelty Waves) by Matt Pearson that talks about somebody writing an AI artist. I recommend checking it out. So, I think, the answer to this question, can we turn our aesthetics into code, is we can go part way there, but not entirely. It has to be a collaboration between the artist, and the program, which is okay.

Why do this at all?

So, the last question, maybe the most important question, why do this at all? Why make art with code? I think the most straightforward answer is basically, it's fun. We spend all this time developing these extremely complex skills with a tool that's incredibly powerful, and yet we rarely use it in fun ways.

How many of you have actually sat down, and written code with no predetermined goal? Yeah... like five of you, and it's probably pretty rare for the few of you that have. It's just not something we do normally, and I think that we really should. It's kind of a waste to not do fun things with these skills from time to time. Apart from just the artistic challenges, there's a lot of interesting engineering challenges that come out of doing this type of work.

If you want to write really fast code, you can work on writing a real-time GPU shader, and these things have incredibly high throughput. If you like efficient compact code, there's demoscene artwork which is all about generating things out of incredibly small executables. This is an entire world that fits into 4 kilobytes executable. And this is just a still from a video. It's a whole terrain with mountains, and smell, and clouds, and oceans, which is pretty mind-blowing.

If you like robots, we have robots. Some people get their hands on these crazy nine axis industrial things, and draw portraits. I have to satisfy myself with a simple three access robot.

This is an AxiDraw. It's made by a small company out in California. I use it for drawings and paintings, but it's really nice to be able to do things in the physical world sometimes. This is really under-explored space, and so if you like to do things to be the first person to do something, it's really easy here. You walk almost any direction for a short amount of time, and you're in new fertile ground. And as far as motivation goes, this is really great. You get with just a few hours of work some immediate visual feedback. It might be bad feedback, but it's not always that way. You'll usually enjoy what you've created, and so it's not a project that takes you months or years to get to a satisfactory point. At least for my type of personality, having this kind of motivation is useful.

No Tests. No Documentation.

I think it goes without saying: you're not going to do either of these at all. I don't.

I touched on this a little bit, but honestly we deserve to be able to have fun with these skills that we've developed. There's not really a good reason for us not to. I think we should just get in the habit of loosening up a little bit more often. But, on a more serious note, I do feel that we actually owe it to society to spend some of our time and energy on these things. We have a certain understanding of technology, of programming, that most of the world does not. I mean, to most people it's a total mystery to them, and artwork can bridge that gap sometimes.

As an example, think about state-of-the-art machine learning. If I went to somebody on the street, and asked them "what are the strengths and weaknesses of state-of-the-art machine learning algorithms?" they'd have no idea. That might even be hard for some of you in here to answer. But, if I let them listen to a short piece of music that was composed by a machine learning algorithm, or I let them read a poem that was generated by one of these algorithms, or look at a piece of artwork, they would very quickly and intuitively understand what's going on with it. They would be able to hear. Okay, maybe it can get the details right. It can string together these short sequences of notes in pleasing ways, but it misses the bigger picture. It doesn't have any drama. There's no story. There's no tension. It doesn't seem to have any sort of creativity in the way that we think about it, and so that's a very useful understanding for that person to have of how the technology works.

And I think the same kind of intuitive understanding can happen not just in this machine learning case, but in all sorts of artwork that's created via programming. I think there's an educational aspect, a communication aspect of this that's important. Generative artwork also tends to explain things about the world to us, given that the universe itself is generative, it's based on laws, and probabilities. These result in certain patterns, emergent patterns, and generative artwork deals heavily in patterns. And so, either intentionally or not, generative artwork is often times stumbling across some of these same patterns that we discover in the world. So, one cool of example of that is a seashell. I don't know what kind of species it is, but the shell happens to, the pattern there happens to be very, very similar to a cellular automata that Stephen Wolfram created (or discovered) called Rule 30, except that it has some glitches in it. But it's really remarkable how this stuff is reflected in nature.

To dive off the philosophical deep end, I think generative artwork also says something important about us as humans. Art is fundamentally about being human in a lot of ways. And when we try to write rules about artwork, we're trying to make more explicit attempts to describe what it is that we appreciate about artwork. And this approach very quickly exposes its own limitations. It becomes really clear how complex, and highly dimensional artwork is, and the gap between how programs work, and how humans work becomes a lot more evident. And we start to see that maybe these programs can tackle the "how" of making artwork, but they can't really tackle the "why", and so it emphasizes the importance of the "why" in artwork, and how we behave as humans.

So, I think this approach of generative artwork, to me at least, is revealing a lot about the nature of human life. And the last part that I want to say about this, is any culture that has no artwork is a dystopia. You want artwork to be in any healthy culture. Art encourages discovery. It encourages reflection, and communication, and appreciation of life. Given that programming is basically swallowing the world -- every day it's more and more important, and we spend more and more of our lives wrapped up in technology -- we need to be sure that we embrace the importance of artwork, and we devote some of our time and energy to enjoying it and creating it if we want to make sure that we establish a healthy culture for us all as we move into the future.

That's about the best answers I have to these questions. I hope that some of you will feel like learning a little bit more about generative artwork or making some yourselves. And with that, I'll say thank you.