Cmd⚡Ctrl: My Prototype of a Universal Prototyping Tool

This is another example of our free-range, radical thinking and unbound experimentation from our development team. Here, Art & Science creative alchemists showcase how they’ve delved into art, science and the edgy application of inspired ideas that may one day change the world.

As developers and creatives, we build a variety of projects that go outside of work that we’re hired to complete. Sometimes we build video games; or AR/VR experiences; sometimes they’re applications requiring multiple modes of interaction both physical and digital; other times we build wild experiential pieces and experiments to prove a theory and just have some fun.

Often we use websockets–a real-time, two-way communication channel between a client (like a web browser) and a server–mixed with physical prototypes to control our fun projects and do initial testing. Think of it like building a model car before making the real one to make sure all the parts fit together and work properly. Prototyping tools help us identify and fix any issues in the concept before we commit to the final forms and logic.

Traditionally, we build prototyping tools and set up local servers for each project and then discard them at the end. This can be a waste of effort and materials. The Cmd / Ctrl box is a way of having something more universal that could be reused during the early phases of projects. But I also wanted to make it fun, satisfying and easy to use.

The Cmd / Ctrl Box

The idea of Cmd / Ctrl is that inputs can be mapped to anything that can connect to the Internet and use web sockets, such as Internet of Things devices, home automation devices, microcontrollers, websites, and more. As long as Cmd / Ctrl is connected to a Wi-Fi network, it can trigger changes in applications, both locally and remotely, physically and digitally, even changing real physical objects across the globe.

For the look and feel, I was inspired by 60s-style, sci-fi control panels. Think of that retro classic movie, 2001: A Space Odyssey. Those clunky, big buttoned control rooms have always looked so fun to me. And so satisfying to use with all of their switches, light up buttons and sliders. In my wildest dreams, I would have an entire room filled with buttons and switches to press at will, but for this project I had to settle for one on a smaller scale.

The Physical Interface

In terms of the actual control options, it was important that it covered a variety of interactions. So I could test different kinds of effects in different ways.

Two Buttons

Acts as instantaneous triggers or long presses.

On/Off Toggle

Tests interactions requiring a boolean state change.

Potentiometer

For applications that change based on a range of numbers.

How it Works

For people who love the technical details, this section explains what I built and how it works. For anyone who might find this a bit heavy on developer language, just skip to the next section, which talks about how it works in practice.

The physical box inputs are powered by a SparkFun ESP32 ThingPlus microcontroller, which can be connected to the Internet to facilitate the communication over web sockets using a Socket.IO package.

A Heroku-hosted Node.js server that acts as the Box listener and also the broadcaster of events. For example, when Button 1 is pushed, it sends a message to the server saying, “hey, Button 1 is pushed down.” The server receives this message, and then broadcasts a message back out to any listeners to act upon as they wish, saying something like, “hey, if there’s anybody out there listening to me, Button 1 was pushed down!”

It’s ultimately up to the listeners to interpret this message how they want. But because this listening and broadcasting is done through web sockets, this communication back and forth is done very, very quickly, and provides an almost instant reaction. And because anything can listen to the messages, it makes it a very useful tool for prototyping experiences, testing out theories and debugging.

The Experiments

I set up four different experiments to prove the variety of uses the box has, and its ease of use. My goal was to ensure that no single experiment took less than an hour to complete.

Button 1: Control an existing experiential piece… remotely

At Art & Science, one of our favourite pieces is something called The Orpheum, an experiential project which now lives in our office. You can find a virtual version of the piece to learn more (on desktop only). This audiovisual musical instrument has various melodic scenes and colour modes. The modes change when a user has stopped interacting with it for a brief period of time.

I set the Orpheum application to listen to Button 1 being pressed and change the mode at will. Now I can change something about the Orpheum colours even when I’m at home, making everyone at the office a little curious about why it keeps coming to life.

Time invested: 30 minutes.

Button 2: Send inspirational text messages at will

I wanted to test something that felt a bit more magical and unexpected. I set up a tiny local server on my computer that listens to Button 2’s messages. When it receives the message, it uses the Twilio API to send a random inspirational message to any phone number I want.

I liked this as an example of showing that the buttons can create surprising effects. Nobody expects to press a button and then get something back on their personal device without even opting into the experience. Perhaps this is an example of a more evil use-case, but it was fun and easy to do nonetheless. These are experiments of could–not should!

Time invested: 15 minutes.

On/Off Switch: Let there be light

Along the theme of magical and unexpected effects, I wanted to test out an IoT (Internet of Things) solution and get physical results. In my home, I use Philips Hue lights, which conveniently have an API for controlling them. Like the text messages experiment, this just required a tiny local node server to run at home to listen to the events.

Now, when I flip the on/off switch, my light at home turns on to a random colour (even if the box is somewhere else in the world). When I flip the switch off, the light turns off. Think of the possibilities. You too could make your friends feel like they have a ghost in their home!

Time invested: 30 minutes.

Potentiometer: A one-to-many shared experience

For a website-based experiment, I wanted to use the rotational potentiometer to create a shared experience where the input happens singularly, but anyone can experience the same effect. If you’re unfamiliar with the term potentiometer, think of a dial for volume control or a knob to change the radio station; it’s a component that allows you to move between a range of values.

As a proof of concept, this was done in the form of a simple website that displays a colour gradient. When the potentiometer is changed, the background colour transitions. No matter who is on the website, when they join, or what device they’re on, they will see the same change at the same time without having to reload the page.

While this kind of thing could absolutely be done fully digitally, there is something special about being able to cause that kind of change with a physical device. I could see applications like this working particularly well for experiential pieces or in-person events like concerts or conferences.

Time invested: 30 minutes.

Was it successful?

I would say yes! The Cmd / Ctrl box was very easy to integrate with. It drastically cut down the time I would have spent having to build each experiment from scratch. And now that the tool exists, I can continue to do experiments with it. I can easily make the inputs do something else; have the whole box affect a single thing; mix and match; make a button affect five different, unrelated things; or whatever! It’s very flexible and has no true purpose by itself, and that’s the point.

My four experiments were just a small example of what the Box could potentially affect. If I had unlimited time and money, I would love to build a larger box (or even a room) so I could test more complex applications. It could be used for things like home automation, connecting to other IoT devices, controlling games, VJing a cool light show, and who knows what else!

Jordana Harrison

Senior Full Stack Developer