One of the greatest user interface design minds in the world today - Alan Kay (About the author of magic ink)

I first encountered Bret Victor through the Future of Coding community and his influential talk "Inventing on Principle". His work challenges conventional thinking about software and user interfaces. Victor's work draws comparisons to that of Alan Kay, with Victor being referred to as Kay's protégé. In his career, he worked as part of the 'HID Proto' future-interface prototyping team at Apple, where he designed UI concepts for experimental hardware platforms.

Before his work at Apple, in 2006 he wrote a paper on a new approach to user interface design: "Magic Ink: Information Software and the Graphical Interface". This paper stemmed from his belief that "software has remained frustrating, and as the importance of software to society has grown, so too has the crisis. The crusade marches on, with believers rarely questioning the sacred premise—that software must be interactive in the first place. That software is meant to be 'used.'" It showed me a new perspective on UI I hadn't seen before. The article remains remarkably relevant, as it predicted recent trends - the idea of software anticipating user needs is now being realised through AI and machine learning.

'Magic Ink' in a Nutshell: Victor's Core Points

While Victor provides his own summary at the end of the article, this is my personal interpretation of his key points for learning purposes.

Victor begins by reframing software design as a combination of "drawing pictures" (graphic design) and creating "things to push" (industrial design). He then identifies three primary reasons people use software: to learn, create, and communicate. This leads to his classification of software:

• Information Software, for learning (internal model)
• Manipulation Software, for creating (external model)
• Communication software, for communicating (shared model)

That most software is a mix of information and manipulation software and also communication is clearly acknowledged: "Although manipulation is the focus, good manipulation software must provide superb visualization as well."^2.

Moreover, he argues that manipulation software is the hardest to design. Yet, most software is information software. Thus, most software is not a machine but a medium for visual communication and that interaction should be a last resort when designing software. He defines the design of information software as the design of “context-sensitive information graphics”. The introduction of context is another key idea of his, which is ubiquitous since mobile phones took over: "Print has one supreme flaw: ink is indelible. An ink-and-paper design is static—it must display all its data, all the time." He lays out how software back then (the article is from 2005) could be improved by utilizing context (environment, history, interaction). He was even suggesting using machine learning to infer historical context! Victor gives multiple examples in his text while being most into details with his widget BART, which is a train trip scheduler. For the detailed comparison, please refer to the original paper.

His ideas like context-sensitive information graphics have been realised to a great extent. For example, Google Maps and other map applications can accurately guess the location you will likely explore by using your device's GPS. The minimizing interaction aspect has become a core concern when designing software^1. On the other hand, there is room for improvement in many applications. For instance, his critique of email applications remains relevant even today:

Most current designs revolve around the manipulation of individual messages—reading them one-by-one, searching them, sorting them, filing them, deleting them. But the purpose of reading email has nothing to do with the messages themselves. I read email to keep a complex set of mental understandings up-to-date—the statuses of personal conversations, of projects at work, of invitations and appointments and business transactions and packages in the mail. That this information happens to be parceled out in timestamped chunks of text is an implementation detail of the communication process. It is not necessarily a good way to present the information to a learner.

The email apps I know that move slightly more in Victor's direction are Gmail, with its categorization of messages ("Primary," "Social," and "Promotions"), and Superhuman, which automatically triages emails.

Glossary

• Internal model: “a mental representation of information”
• External model: "a model external to herself—a virtual object represented within the computer, or a remote physical object. Some examples include software for drawing, writing, music composition, architectural design, engineering design, and robot control"
• Shared model: "an internal model that is shared with others"
• Graphic Design: "is the art of conveying a message on a two-dimensional surface"
• Industrial Design: "is the art of arranging and shaping a physical product so it can be manipulated by a person."
• Context-sensitive information graphics: Dynamic graphics that go beyond a static image using who the user is and what they are trying to learn

Quotes

Software and Sorcery

A computational process is indeed much like a sorcerer’s idea of a spirit. It cannot be seen or touched. It is not composed of matter at all. However, it is very real. It can perform intellectual work. It can answer questions. It can affect the world by disbursing money at a bank or by controlling a robot arm in a factory. The programs we use to conjure processes are like a sorcerer’s spells.

Abelson and Sussman, Structure and Interpretation of Computer Programs (1984)

The Ideal of a PC as seen by Licklider, Bush, Engelbart

For Licklider and other early visionaries such as Vanevar Bush and Doug Engelbart,** See Bush’s paper “As We May Think” (1945) and Engelbart’s paper “Augmenting Human Intellect” (1962). the ideal of the then-hypothetical personal computer was a brain supplement, enhancing human memory and amplifying human reasoning through data visualization and automated analysis. Their primary concern was how a machine could help a person find and understand relevant knowledge. Although they were generally discussing scientific and professional work, their prescience fully applies in the modern home.

• https://worrydream.com/MagicInk/#:~:text=For Licklider and,the modern home.

Magic Ink

Liberating us from the permanence of publication is the undersung crux of the computer—the dynamic display screen. Its pixels are magic ink—capable of absorbing their context and reflecting a unique story for every reader. And the components surrounding the display—CPU, storage, network, input devices—are its peripherals for inferring context.

• https://worrydream.com/MagicInk/#:~:text=Liberating us from,inferring context.

Design of Information Software as Graphic Design Project

I suggest that the design of information software should be approached initially and primarily as a graphic design project. […] The designer must start by considering what the software looks like, because the user is using it to learn, and she learns by looking at it.

• https://worrydream.com/MagicInk/#:~:text=I suggest that the design,learns by looking at it.

Interaction harmful

Unless it is enjoyable or educational in and of itself, interaction is an essentially negative aspect of information software. […] Interaction should be used judiciously and sparingly, only when the environment and history provide insufficient context to construct an acceptable graphic.

• https://worrydream.com/MagicInk/#:~:text=Unless it is,an acceptable graphic.

Good Information Software

Good information software reflects how humans, not computers, deal with information.

• https://worrydream.com/MagicInk/#:~:text=Good information software reflects how humans%2C not computers%2C deal with information.

Information Software as Art

The hero of tomorrow is not the next Steve Wozniak, but the next William Playfair. An artist who redefines how people learn. An artist who paints with magic ink.

• https://worrydream.com/MagicInk/#:~:text=The hero of tomorrow is not the next Steve Wozniak%2C but the next William Playfair. An artist who redefines how people learn. An artist who paints with magic ink.

Settling for a better standard

People constantly settle for ugly, clunky software, but demand informative, professionally-designed books, newspapers, magazines, and—ironically—brochures, ads, and manuals for that very software.

• https://worrydream.com/MagicInk/#:~:text=People constantly settle for ugly%2C clunky software%2C but demand informative%2C professionally-designed books%2C newspapers%2C magazines%2C and—ironically—brochures%2C ads%2C and manuals for that very software.

Random Thoughts & Ideas

• Are ads the perfect information software? They are graphical, have no interaction and are context-dependent.
• Did we go too far with context? For example, filter bubbles
• How much closer did we get to Victor’s vision? Mobile took over and augmented reality is developing further
• What about large language models?

References From the Paper

• Abelson and Sussman, Structure and Interpretation of Computer Programs (1984)
• Alan Cooper’s The Inmates Are Running The Asylum (1999)
• Don Norman’s Design Of Everyday Things (2002)
• Jef Raskin’s The Humane Interface (2000)
• Cooper’s About Face (2003)
• Edward Tufte, The Visual Display of Quantitative Information (2001)
• Clay Shirky’s essays
• Social Software and the Politics of Groups (2003)
• J.C.R. Licklider, “Man-Computer Symbiosis” (1960)
• Bush’s paper “As We May Think” (1945)
• Engelbart’s paper “Augmenting Human Intellect” (1962)
• Howard Wainer’s book Graphic Discovery (2005)
• Raph Koster's Theory of Fun for Game Design (2004)
• James Paul Gee's What Video Games Have To Teach Us About Learning and Literacy (2003)
• Salen and Zimmerman's Rules of Play (2003)
• Chris Crawford's Art of Interactive Design (2003)
• Chris Crawford on Game Design (2003)
• Budzik and Hammond's paper User Interactions with Everyday Applications as Context for Just-in-time Information Access (2000)
• Laurie J. Flynn, Like This? You'll Hate That. New York Times, Jan. 23, 2006
• Ali and van Stam's paper TiVo: Making Show Recommendations Using a Distributed Collaborative Filtering Architecture (2004)
• Crawford Fundamentals of Interactivity (1993)
• Early History of Smalltalk (1993)
• Crawford Representation Versus Depiction (1994)
• Philip Meggs's superb History of Graphic Design (2005)
• Peter Drucker's fascinating essay The Next Information Revolution (1998)
• Henry Dreyfuss's recently rereleased autobiography Designing for People (1955)
• Raymond Loewy's autobiography Industrial Design (1979)
• A Mathematical Theory of Communication (1948), p2
• The Emperor's Old Clothes Turing Award lecture (1980), p81
• Watch What I Do (1993, available online)
• Henry Lieberman (ed.), Your Wish Is My Command (2001)
• David Kurlander's Chimera (1991)
• Ivan Sutherland's seminal Sketchpad (1963)
• Michael Gleicher's Briar (1993)
• Tom Mitchell's book Machine Learning (1997)
• Russell and Norvig's book Artificial Intelligence: A Modern Approach (2003)
• John Backus, Can Programming Be Liberated from the von Neumann Style?, 1978); C and UNIX
• Richard Gabriel, The Rise of "Worse is Better", 1991
• There is a recommended reading section on the original article: https://worrydream.com/MagicInk/#:~:text=the%20world%20again.-,Recommended%20reading,-%23