What Can a Robot Do?

Besides being a writer, an inventor, an engineer and a researcher Ken Goldberg is also an artist, surely one of the frontrunners in the field of robotics, automation and Artificial Intelligence. His research has touched upon very relevant themes such as the the relationship between nature and technology, expanding upon the realm of plants, the concept of Gaia, in the broader context of the Anthropocene. Valentino Catricalà has met him during the last Maker Faire – The European Edition and started a conversation that continued online, specifically on his works Telegarden and Alphagarden.

Valentino Catricalà: You have been a pioneer in  engineering and also in art. A hybrid figure ante litteram. Today it is more common seeing artists and engineers, or engineers and artists. This connection is part of the work of younger artists, due to the growing presence of technology in society. I think that when you started you did not have many references, perhaps you felt as someone who was discovering a new world. 

Ken Goldberg: Thank you Valentino. My mother was a psychologist, she was very interested in art, and my father was an engineer, and he was also interested in art. They would take us to museums, and there were many discussions about art in my family when I was a boy. I was very interested in being an artist, but because we had a difficult financial situation when I was growing up, my mother said to me “you can be an artist, but first become an engineer.” When I went to college, I studied engineering and maintained an interest in art. I discovered robotics which raised philosophical questions about what is human and what it means to be a human. I fell in love with that field.

When I was in grad school at Carnegie Mellon, I had some friends in the art department, and we were sitting around one night and painting, and someone said “Hey, can a robot do this?” The following night I went into the lab and programmed the robot to do some paintings with brushes, because I was interested in the contrast between the digital motions of the robot and the flow of the brush as it painted. I’ve spent many years experimenting with different kinds of paintings the robot could make, noticing how the brushstrokes were different every time.What was interesting to me as a graduate student, was the gap between digital and physical reality. I’m still fascinated by this question. 

Just before I finished my PhD in 1990, I was invited to present all the paintings in a solo exhibition at the CMU Forbes Art Gallery and I invited friends from Engineering and Art and musicians, we had a blast…  When later on I became a professor at USC in 1991, I was invited to develop an exhibit with painting professor Margaret Lazzari at the USC campus gallery involving a robot and large-scale paintings. It required months and months of work to put it together. The exhibit was only open for 3 weeks and I wanted to reach a bigger community.  

So, the next year in 1993, my grad student ran into my office “you have to see this new thing.” He  showed me how I could connect to all these places in the world, in the World Wide Web. My students and I were thrilled: what if we could connect a robot to the Internet? We stayed up most of the night to set up a robot in a sandbox (the Mercury Project). Long story short, it got a lot of attention: people online really liked the idea that they could operate a robot and move things around remotely. Then we said—let’s make this a full art project, and we came up with the idea of a garden.

That was the birth of Telegarden. A pioneering work which in my opinion has anticipated many current themes of the current contemporary art world—such as the garden, the idea of the Earth as Gaia, the relationship between technology and nature, and the new notion of the Anthropocene. In my head that was very difficult to make. 

For Telegarden we installed an industrial robot in a living garden. It was also intended as a critique of technology. While I am very excited about technology and its potential, I’m also critical of what often is an exaggeration of technology, both in terms of potential, and negative aspects. The reality is really very different, and so I feel that, as an engineer, it’s really important for me to also convey the nuances of the underlying questions around technology. 

We thought of the Telegarden as an absurdist application, as if gardening would be the last thing that people want to do over the internet, because a garden is so tactile and immediate and natural. Remember, this was in 1995, in the very early days of the Internet. The word spread very quickly: first we got dozens and then hundreds and thousands of people coming to visit the garden, again and again, because the idea wasn’t something you just go in and look at and then move on. It was something you had to return to, and had to spend time with there. What started as a critique evolved into a study of our human response to nature in the digital context.

Since the garden, you had already touched very contemporary themes, foregoing what is now called New Media Art. The first attempts were very focused on technology but in your case, you pushed the whole thing forward into the realm of sculpture. As I said, the creation process opened up to and anticipated some very contemporary themes such as social media, the environment, and questions of representation, what is real and what is illusion. Nowadays there’s a big hype around the theme of Anthropocene, and the relationship between networks and technology…

We certainly weren’t thinking about Facebook nor Donald Trump in 1995. But we were considering the implications of our experience becoming more digital. That tension is really interesting to me. I was reading Walter Benjamin, asking what constitutes an original object and the type of relationship we have with these objects, which is once again physical. One aspect that came out of this project was philosophical.

A student wrote to me about a year after we launched the Telegarden asking: “How do I know there is really a garden?” It was a fascinating question. What was obvious to us working with this garden on plumbing to keep it all functioning, wasn’t at all obvious for someone else. The student was quite rightly being skeptical. The question was particularly interesting because in a way we could be faking it. The question of what was real and what isn’t is very subtle when things are mediated. So, I really began questioning that aspect itself—what is the status of something that’s distant from you? Imagine, you’ve been tending a garden for weeks or months and it turns out that there was no garden, that it was just an illusion created with photographs, etc. This is what I became interested in, and it was around the time I moved to Berkeley.

I was sitting in my studio one day just reading a book about artificial intelligence. Looking at the back cover and reading about the author Hubert Dreyfus being on the faculty at UC Berkeley, I remember almost falling off my chair. I sent him an email and surprisingly he immediately said “Let’s meet, please come to my office.”  We had this really exciting meeting, and I explained this question and he said, it is really a question of epistemology, of knowledge itself. With his encouragement, we started working together and the result was a book that came out from MIT Press in 2000, The Robot in the Garden. It focuses on the distinction between virtual reality, where you have a fictional environment that’s synthesized, and distal reality, where you have a real environment that is distant and your perception of it is mediated. We invited 24 artists and engineers and philosophers to write a chapter each.

This question has become very relevant in 2021, as virtual reality has become much more sophisticated, and there is so much deception online today with deepfakes and conspiracy addicts like QAnon in the US. It’s very hard to tell the difference between what’s real and what’s not. Digital reality has become such a big part of our lives since the 1990s. And during the Covid-19 pandemic, so many kids and office workers are using teleconferencing. 

Yes, exactly. I remember reading your book when I was a student. I knew you were a scholar, an engineer, but also an artist—which is something that somehow is always closer to the younger generations of artists. Today the term “future” is dominant in the academic and journalistic debate. Future is a matter of the human capacity of imagining future events from the present one. It seems that we are now unable to make predictions for the future. I really think that artists today are more important than ever, but maybe we have to renew the idea of how artists can be helpful for the future and our society?

Well, there’s an essay that influenced me by C.P. Snow titled The Two Cultures, written in 1959. He was at Oxford and had friends in the humanities and in physics, he said they are completely different cultures, they don’t mix. They both generally have a fair amount of mutual suspicion and misunderstandings. That is still very present today. These are really two distinct cultures. I’m lucky to have friends on both sides, and can relate to both viewpoints. There’s almost a Gestalt effect, for me—you probably can identify with as well.

When I interact with someone from the humanities, a scholar of philosophy or an artist, it seems as if one part of my brain is working and when I’m talking with engineers there’s a different part of my mind working. Each is distinct. I feel a responsibility to try to interpret between these two cultures and serve in some way as a bridge to connect them.

You’re absolutely right when you say this is increasingly common today, we have many students who are doing a major in computer science and art history. These interactions are growing and I think that’s fantastic, because the new generation is very comfortably working with computers and technology, and at the same time expanding their interests in the humanities and arts.

Coming back to the internet and the garden, that really opened the door to the next 10 years of projects that involved exploring how the internet could be a useful venue, or an artistic medium, and as you know there is a wonderful history of “Net Art” that began around that same time in the late 1990s, and there are many artists especially Eastern Europe that were experimenting, and I appreciated being part of that community. While engaging and experimenting with people, I became interested in the medium itself and how the perception shifted. Essentially, it allowed us to express something that one couldn’t express before. Largely it was because of its democratic potential, as not everyone but many people could access the internet and get access to artworks—that would be very unusual. 

In a networked art project, I could remain online 24 hours a day for years into the future. That doesn’t have to be kept in a museum. So, that continued to be an interest for me,  we did a number of projects after that, which involved exploring ideas about perception and embodiment.

Which is evident in your artworks as well…

Yes, these projects over the years, such as sound installations and others focusing on research. But let’s go back to the garden. I’ve always wanted to revisit the Telegarden. It has been online for nine years. We closed it down in 2004. It’s always been my favorite project. And I wanted to think about, somehow, bringing it back.

Anyway as you know, in 2012 there was a major breakthrough in Artificial Intelligence, namely deep networks and deep learning. The  combination of large amounts of data and large amounts of computation, a new class of algorithms that allows you to combine those elements together to learn—basically, to have a large network of values, to be able to be trained into approximate very complex functions. That has caused a revolution, that is basically the contemporary Artificial Intelligence, which my students and I have been embracing. We’ve been working closely with colleagues, building systems, testing them, writing papers on that topic for the last decade. It occurred to me that this still suggested new questions within the context of the garden. In particular, could a robot learn to tend a living garden?  

The question is—again coming back to what we were talking about earlier—the real garden. If you look at the complexity it is enormously subtle. Everyone was talking about artificial intelligence soon being able to surpass humans. There will be super-intelligence and so on—although I am  very skeptical, to be honest. Nonetheless, could we embody this question into a physical system, so we could actually contrast the complexity of the natural world? With this new technology of artificial intelligence and that’s when it occurred to me in the Fall of 2019—that we could design a new kind of garden, but this time the garden would be autonomous. That was the beginning of what we call the AlphaGarden an ongoing research project and an art installation. Alpha because of the connection with Google and DeepMind, AlphaGo and AlphaZero and AlphaFold. I admire them greatly, they’re really one of the leading labs doing research in these areas, and a garden is a very different Alpha.

This relates a bit to what you were saying earlier about diversity, since a polyculture garden is very different from a monoculture. Monoculture is how most food is produced today—how farming is done. Where you essentially have row after row of the same crop, same plant grown and space very precisely apart so that there’s minimal interference between plants, if you look at an industrial farm, everything looks very uniformed.

There’s a trend today toward polyculture farming. Originally, farms were based on polyculture with different types of plants growing together, in close proximity. There are many benefits for the  plants, in how their mutual resistance to disease and insects increases, as they can reinforce each other. They use less water, because they maintain and share root structures that share water. Luckily this is starting to be used again in farms, in order to reduce pesticides and water consumption.

But the problem is that this requires more human labor—so we tried to recreate a miniature polyculture garden and have the robot garden completely on its own. Actually a real policulture garden might be too complex, beyond the capability of today’s robots and artificial intelligence, but I still want to give it a try.

The garden is a very big issue today in terms of the idea of a new-found relationship between the human being and the natural world. It is very important to find a third way because I think that we are still stuck into the old dichotomies—good-bad-kind-of. I feel your work helps us reflect on our contemporary art, while it creates this new path.  

Right! We’re not just trying to maximize the amount of food or leaf material that could come from the garden. We want to maximize diversity as well. We have a measure of diversity based on entropy. We can measure this with cameras, we can look down, we can classify the different kinds of plants that are growing at any time. And we can come up with a measure of how diverse the garden is. That means we need to support plants that are slow growing, and prune back the plants growing faster, so that there’s a balance. That’s the challenge from an engineering perspective. It’s very subtle because the interaction between plants is very difficult to model, even for the growth of a single plant. It’s a nice research topic, how do you fit a distribution, and how the plant will grow over time. It is a statistical function, because we don’t know exactly what the germination time is, or the radius over time, and the shape of each plant varies. As soon as the plant comes in contact with another plant then there’s a mutual effect. There’s a shading of one plant in terms of light, but really importantly the root structure starts to connect. That can be a positive or negative effect of what we call companionship between two plants.

All  this is very subtle and complex. It also involves other elements such as insects, natural pollinators like bees or other insects that can cross pollinate. The soil itself is very complex—but in our case, what we’re trying to do is learn the model by experience.The big challenge is time, because the timescale for the natural world and for plants is very very long. So, to do a proper experiment requires months of time. 

What we’ve done is we’ve created a simulator that simulates the growth of plants at 10,000 times the speed of nature. This lets us tune all the parameters, until we discover something that will yield a high-diversity polyculture garden. The challenge is to get the simulator to accurately reflect the real garden. This is  known as the “reality gap”. There’s a distinction between what really happens and what we simulate, but you can start to tune these two things together, and we can observe the garden while we observe the simulator, we tune in the simulator, we observe the garden again, keep iterating.

Can we accelerate and advance the ways we connect people from different backgrounds from vastly different cultures and experiences in terms of socio-economic backgrounds, genders, different ways of thinking, viewpoints, different kinds of training?

Over the past year, we’re slowly building a more sophisticated model of plants and interaction. Then we are using that model to generate a planting schedule, a seed placement we’ve received, as it grows we tune in the model to better match what we’re observing.

The idea is that if we keep doing this sufficiently long, that will get to a point where the robot system will be able to anticipate how the garden will grow and to take actions that will steer it in a positive direction. This is the theory, but the practice—well, there are many. Here’s one example: right now as we speak in January 2021, there’s a camera watching our garden in Berkeley CA. It’s all based on the time of year so the amount of light varies. Now it’s fairly dark and days are very short.

When we run this experiment again in the summer, the behavior will be different, because temperature and light will be different. How those effects manifest is fascinating to me, and it’s part of all the nuances because it all goes back to what you’re saying about the Anthropocene, to what degree the natural world is controllable by us as humans. 

I mean, think about how much has changed for humans in the past 100 years, or 1000 years in terms of technology, with the discovery of electricity, computing and the internet, but the natural world has changed remarkably little. We’re literally eating the same foods as 1000 years ago mostly.

So what is that nuance of complexity, what is the continuity in terms of the natural world, how that hasn’t changed—in contrast to the incredible evolution of the digital and the technological world. Regarding artificial intelligence, the claims are that an AI system can beat humans in many different areas like chess and Go. But can it do something as basic and human as tending a garden? That’s a fundamental human skill, going back to Adam and Eve. But it is still very difficult for robots. So I look at the nuances of dexterity and  perception in contrast to where we are with robots and artificial intelligence.

Plants, animals, technology…with your work you have thrived on diversity for many years. This is why it is important to connect the Telegarden with Alphagarden, an update toward a new way to see us in the world. A chance to consider a new conception of the anthropocene toward a new relationship between human beings, technology and nature?

Exactly. One of the things that we found so interesting is that the garden, and gardens in general, thrive on diversity, on having combinations of different things. So, you know, the monoculture garden has certain benefits, it can be very very productive. But monoculture is vulnerable to any small change that can cascade throughout the garden because it’s all uniformed. It’s the same thing in terms of culture. If we have people thinking along the same lines that you have the same vulnerability. What’s really important is diversity.Because when you bring people together with different viewpoints from different cultures from different training, different instincts, they begin to cross-fertilize and coexist. Basically, thrive together. That is, I think, one of the biggest challenges we face today. As we mentioned earlier, a new generation of students is coming out now with degrees that combine skills in the arts and technology.

Can we accelerate and advance the ways we connect people from different backgrounds from vastly different cultures and experiences in terms of socio-economic backgrounds, genders, different ways of thinking, viewpoints, different kinds of training? It’s so important to understand the benefits of  what I call “complementarity” between different ways of thinking the different—different cultures, different backgrounds, different cognitive styles and approaches.