Ross Bencina

I’ve been reading and thinking about maths quite a bit lately. In the process I happened upon Bret Victor’s killmath posts on interactive dynamic systems, explorable explanations and usable math. Some of the killmath examples really got me thinking. I’ve actually started coding a few experiments around this, but for now I just want to summarise some reflections and comments I have about Bret’s research. There is a lot more to Bret’s site than I touch on here, so I really recommend you go check it out if you havn’t already.

Explorable (Visual) Explanations

To give a feel for the material: one of Bret’s posts in Explorable Explanations gives an interactive example of a digital state variable filter (it’s not actually Chamberlin’s topology as I had previously tweeted). The example presents different graphical and symbolic representations of the filter and allows you to manipulate the filter’s parameters and hear the result. One of Bret’s videos demonstrates an interactive interface for exploring the Predator/Prey dynamical system with multiple cross-linked interactive visualisations. I discuss both of these examples below.

If you’ve read Edward Tufte’s book Visual Explanations — images and quantities, evidence and narrative you’ll have an idea of some of the concepts and considerations involved. Pictures, diagrams, sounds, and dynamic time-based demonstrations can all tell stories in ways that numbers and formulae can’t — potentially allowing us to gain new insights, solve problems and draw conclusions that are not accessible (to the uninitiated, if at all) by simply looking at mathematical formulae or unstructured data. Explorable Explanations extend visual explanation with interactivity allowing the reader/user to engage, explore, play, and listen, in order to gain a deeper understanding of a system’s dynamics and behaviour under varying conditions.

Of course, animations, simulations and interactive examples in the form of Java applets and Flash interactives have been circulating for years. To give a random example: sorting algorithm visualisation applets from the field of algorithm visualisation. Often though these applets are demonstrative rather than truly interactive and explorable. Closer to the mark is a recent site from the U.K. Department of Energy and Climate Change: My2050, a Flash interactive that allows users to experiment with a parametric model of energy generation and consumption, lifestyle choices, and modes of transport to design their future world. By moving sliders on the screen, users explore the environmental effects of their choices while being challenged to achieve a workable set of parameters. It’s a pretty neat web site that illuminates some of the environmental trade-offs we face.Click here to try it

To me there is something implied by Bret Victor’s work that isn’t present in DECC’s My2050 or in simple illustrative Java applets. DECC’s My2050 site supports developing intuition about the relationships between energy consumption, production and carbon footprint, however the underlying mathematical model and data set are not visible. In contrast, it seems to me that Bret’s examples are not just about increasing user intuition about system outcomes. They also support increased user awareness and expression of explicit models of system structure and dynamics. For example, by simultaneously showing the various representations of the state variable filter, or by showing multiple overlaid trajectories in the Predator/Prey model. I think this is a really interesting direction, but as I discuss later, the mechanisms for illustrating structure and inter-relationship deserve further development.

Scrubbable numbers

This brings me to Bret’s recent post on the Scrubbing Calculator. One facet of the Scrubbing Calculator that caught my attention was scrubbable numbers — numbers that you can click and drag on to change their values (not unlike a spin box) to support interaction with typed mathematical expressions. Changing one number changes the others according to their context in the surrounding text. Try clicking and dragging up and down on the numbers in the sentence below and you’ll see what I mean:

1 plus 2 equals 3

The point here is that this functionality would be available as you type, not just in a pre-written document like this one. The system would also deal with solving multiple simultaneous equations. Text is no longer just text, it’s “active” text. Looking at it another way, “widgets” are integrated with the text, making creation of an interactive user interface as easy as typing a sentence. Since behaviour is implied by context, the Scrubbable Calculator seamlessly provides a layer of interactive mathematical functionality without the user needing to explicitly invoke a separate “math tool” (calculator).

This reminded me of an idea I had a few years ago for a live coding user interface. (Live coding is a performance practice where audio-visual art is coded on the fly in performance.) The as-yet unimplemented idea was for a live coding text editor where typing something like:

oscillator.setFrequency( Slider() );

Would be immediately expanded in-line, as soon as you type it, to:

oscillator.setFrequency( Slider() );

When the code is executed, the slider would be automatically linked to the live audio algorithm thus allowing interactive manipulation without needing to explicitly construct or code a GUI — while at the same time retaining the direct visual linkage between the slider-manipulated parameter and the code it’s related to (an important factor for audience comprehension in live coding). Personally I think this would be an improvement over the current situation in many live coding environments where you need to re-execute lines of text to change variable values — something I sometimes see when watching Andrew Sorensen’s performances with impromptu for example. Of course, similar dynamic linkage in a live coding editor could be done with scrubbable numbers or other types of widgets instead of sliders.

Bret’s scrubbable example doesn’t end there though. He also illustrates dynamic creation of constraints (relations) between values by dragging linking lines, a bit like visually authoring spreadsheet equations or routing patch-cords embedded in free-form text. And this, I think, points to the overall direction that Bret is working on: tools for making mathematical systems less abstract, more dynamic, visual, situated, understandable and usable.

More expression of relationships between representations please

One thing that struck me about Bret’s state variable filter explorable example is that it presents a disjoint set of visual representations and formulae. The user is left to deduce (or perhaps assumed to already understand?) the relationships between the various representations. A formula for the transfer function is given, along with a pole-zero plot and the frequency response, but there is no built-in way to explore the mathematical relationships between these pictures. The relationship between the frequency response and the pole-zero plot is not illustrated. It isn’t at all clear that the pole locations on the z-plane plot are are the roots of the transfer function polynomial. Not to mention that casual readers may not even realise that the variable z is a complex number. To me this seems like an “interactive representation jumble” rather than a coherent, explorable presentation of what is in-fact a crystaline structure of interrelated mathematical meanings.

To be fair, Bret calls the SVF diagrams an “Explorable Example” — it serves to provide an interactive illustration within a larger textual narrative rather than as a self contained “Explorable Explanation.” That said, completely self-explanatory explorabile illustration is desirable. The interested reader should be able to examine everything in-situ, potentially down to the workings of the number system. If the interface employs multiple visual representations their relationships should be clear. Visual linkage could be provided with animation or additional elements showing the cross-relationships between points on the various graphs (as in the Predator/Prey example). Mathematical relationships could be expanded (click to zoom) or contracted to show the big picture. Clicking on a complex polynomial could visualise each component and expand to provide further detail. Clicking on any node in the data flow graph could show the transfer function for that node. Et cetera.

My impression is that Bret is working on these ideas. The link-creation system in the Scrubbable Calculator explores explicit creation of relationships between values, and the Predator/Prey video includes clear indicators on each graph to show how data points relate to those on each of the other graphs. There are other clues in the texts.

I’ve found Bret’s Explorable Explanations fascinating and inspiring. Do go check them out.