Photosynesthesia: How I Collaborated on Art with a Potted Plant

Introducing the home of our free-range, radical thinking and unbound experimentation from our development team. This is the first of several articles where our Art & Science creative alchemists showcase how they’ve delved into art, science and the edgy application of inspired ideas that may one day change the world.

A favourite part of my job is that I get to create things that I’m proud of. As a senior full-stack developer, that usually means I’m working with our incredible team to bring a website to life. But what really gets me excited is our physical, digital activations – the magical manifestation of technology and creativity that starts with a humble experiment.

Bringing one of these projects to life is just cool, and to see the real thing, working, is so rewarding for the brain power and physical labour invested. It becomes both a physical object and a singular experience.

This article showcases some work I’ve been experimenting with. I wanted to know: how can I use technology to connect with something as ubiquitous as a house plant? The experiment led me to explore facial detection algorithms, generative music, and observe the feelings evoked by the natural cycles of the Earth. I called it: Photosynesthesia (pronounced FOTO-si-nuh-STEE-zhuh)

The Inspiration: What About a Plant Can Be Amplified to Create a Sensory Experience?

Like many of us, my relationship to time and space drastically changed during pandemic lockdowns. Stuck at home, the daily rhythms of life became more important. I could do nothing but observe the subtle changes in my environment: the way natural light changes as the seasons change; each new leaf forming on my plants. When we went back to the office that observation came with me. Our office is full of plants, all in different stages of life. And because of some quirk of our building’s architecture, every day just before sunset, a patch of sunlight appears on the hardwood floor and gradually slides up the wall before disappearing.

Then I attended the Eyeo festival in June 2022 with some colleagues. Several talks touched on the things I was thinking about. Ersin han Ersin of Marshmallow Laser Feast discussed Treehugger, a project that mapped the flow of water inside trees. Matt Kenyon talked about Spore 1.1, which uses stock market data to control the watering schedule of a live plant. Grace Boyle discussed her work in creating immersive sensory experiences.

I ended up on these questions: what would it mean to collaborate on art with a plant? What about plants can be amplified or interpreted to create a sensory experience? If a plant could express itself, how would that sound? What if plants could sense as we do? And how could I foster that connection?

The Execution: Giving the Plant Webcam Eyes

Photosynesthesia is a plant that makes music when someone pays attention to it. My pothos plant can see — and when it sees you, it sings you a song. It’s generative music, so no part of the composition ever repeats. Situated in the changing light of a west-facing window, the plant looks different each time you lock eyes.

Take a look (and a listen)

To bring this to life, I used face detection, and generative music. Behind the scenes, these are both managed with software called Max (MSP).

There are many ways to use technology to detect the presence of a person. I experimented with motion detectors and distance sensors, but their methods are too coarse — they don’t tell you if someone is engaging with a plant, just when they walk by. I realized the plant needed to identify attentive faces, not just a presence. So that’s why the plant has a webcam as its eyes. The webcam image is processed by a face detection algorithm, so the plant can precisely pinpoint the number of people looking at it.

The music is based on that camera input. I created the parameters for the composition, and it’s algorithmically generated in real time. I use a pentatonic scale to constrain what notes are played, so they all sound pleasant together. And I designed the logic for sweeping, building chords that form a slow melody. If someone pays attention to the plant for a while, or if multiple people are there, the music becomes more complexly layered.

The Impact: Making Space for Beauty, Sound and the Rareness of Pure Calm

All this technology goes to work to create a calming interlude in daily life. Time slows, and there’s a chance to just be. Looking at a plant and hearing its response makes me think about where I put my attention. Do I spend my life staring at my little rectangular phone, making myself sad? Why do that when I could be engaging with the life that’s everywhere around me, hearing beautiful sounds?

In the early evening, when the light hits just right, it’s easy to get absorbed. The music is never the same, so it doesn’t get old. It’s in a cozy corner of the office, just being, waiting for someone to appreciate its lush green soundscapes.

&More

So what did I learn? I feel like I successfully explored one way to collaborate with a plant. I was able to bring a sensory soundscape to one corner of the Art & Science office.

The joy of exploring work like this is seeing all the other openings. Once you go through one, you can’t help but see the many other possibilities that could lead to further experiments.

Here are some:

• What if there was a whole orchestra of plants, each with a unique, but concordant voice?

• What if the sounds changed depending on other sensed factors, like lighting conditions or humidity – amplifying qualities that plants already depend on?

• What if there was another element that the plant could control? Lighting is another quality of space that is often overlooked, but has a big impact.

• What if I concealed the webcam instead of centring it, to create a total illusion of a responsive plant?

• Could I further explore the concept of attention, where it goes, and how we use it?

Could I collaborate directly with the relationship between the earth and the sun, and the way that relationship shapes the light in our lives?

For now, I’ll leave you with this interlude from my musical plant.

Kat Hartog

Senior Developer