Molecular vibration technology

Alright, first things first. Audio is nothing more than vibrating air. A speaker creates sound by pushing and pulling a magnet to create these vibrations, and a microphone works in reverse, that is, the magnet gets pushed and pulled by the sound, and the microphone records its movements.

Conceptually, sound can be thought of as a wave. Even when multiple sounds are happening at the same time, it is one singular wave. This wave can then be deconstructed back into a set of sine waves, which is the "purest" wave, and this is done by throwing enough mathematics at it (Fourier transforms, usually).

Sample rates and audio nodes

The way microphones work, loosely speaking, is they record the position of the vibrating magnet at a certain fixed interval. That interval is called the "sample rate". The higher the sample rate, the more precise the recording.

The wave in the graph above is the sum of two pure frequencies. However, if we were to record and play back the sound from this wave, we'd run into a problem! One of the two frequencies is so high that its samples coincide with those of a lower frequency, called an "alias".

It turns out, this "aliasing" happens for any frequency higher than half the sample rate. This maximum safe frequency is known as the "Nyquist frequency". In practice, the Nyquist frequency is not at all relevant to my project, both because modern microphones have clever anti-aliasing techniques, and because they have very high sample rates (usually 44,100 or 48,000 Hz), far above what I need.

Audio nodes and which ones I used

The Web Audio API works by chaining together "audio nodes". There are a variety of different node types, like an OscillatorNode for generating a basic waveform, DelayNode for introducing delays, a StereoPannerNode for changing the perceived direction of the sound, and many more. Each node has some input channels, output channels, and parameters, the amounts of each depend on the node type. The channels represent an audio waves, but in practice they are lists of numbers, i.e. the samples. The audio nodes transform these samples in one way or another, or generate new ones all on their own.

The nodes used for the Stello-Matic are an OscillatorNode to generate the sine waves (the beeps), connected to a GainNode for adjusting their volume. The Decho-Matic creates a MediaStreamAudioSourceNode, which passes the audio from the microphone to a custom AudioWorkletNode. The custom node looks for the 5 predefined frequencies and emits events to indicate how strong the signals are. More specifically, I implemented Goertzel's algorithm to do this. Initially, before learning about Goertzel, I tried using a native AnalyzerNode, but this uses a Fast Fourier Transform to measure all frequencies, ultimately making it less precise for the specific frequencies I needed. So much so, that it wasn't a good choice for this project. Goertzel's algorithm proved much more accurate, and in theory is more resource-efficient (but I did not test that). I also won't pretend I understand how Goertzel's algorithm works, but at least I can vouch for the fact that it does, in fact, work.

Gain or volume?

When learning about the basic audio nodes, I was a bit confused about why I need a GainNode instead of being able to adjust something like a .volume property. Well, gain is used to decrease or increase the amplitude of an audio wave. Generally speaking, that is equivalent to changing its volume, but volume has an upper bound, whereas gain does not. If you added more and more gain to a sine wave, it would go louder and louder until the speaker's limit was reached, after which the sine wave would start "clipping". Effectively, that means it gets flat bits where it would otherwise exceed the limits, causing it to start sounding more like a square wave.

This is why gain and volume are usually analogous, but not always; if there is clipping at play, gain only changes the way something sounds, not its volume.

I had learned about CSS linear() back in 2021, from watching an episode of the now-discontinued show HTTP203: bringing bounce and elastic easing to CSS. At that time, it was only a proposal, so I had banished it to the corner of my mind where all the unsupported up-and-coming web features live.

When implementing the hover effect for the devices in the poster, I started with a simple "up" on hover and "down" on un-hover. But then I remembered about linear(), and thought it could be cool if I could get the devices to bounce when they came back down. Sure enough, linear() is now widely supported, so I could make this happen!

CSS linear() works as follows. You define some key points ("stops", if you will), and everything in between is linearly interpolated, meaning every pair of adjacent points are connected by straight lines.

Syntactically, the points are defined very similarly to how a linear-gradient() is defined, except that gradients have colors and easing functions have numbers. You define points by their y-coordinates. Optionally (like with linear gradients) you may provide a "progress" percentage value, which you can think of as the x-coordinate. The points without an x-coordinate are spaced evenly between their neighbours.

#to-animate {
	/* Equivalent to linear(0 0%, .8 50%, 1 100%); */
	transition: left .5s linear(0, .8, 1);
}

Most animations are quick, so not many key points are needed for them to look good. But in theory, you could get any level of accuracy approximating a smooth curve, by adding more and more key points to the linear() expression.

Constructing such an expression for a bounce is a bit of a hassle to do by hand, though. Luckily for me, Jake Archibald (the same guy as in the video!) has made a linear easing generator that lets you convert an SVG path, which I find much easier to create manually, into a reasonably short linear() expression. I used the tool for the bouncing animation, and later for the cartridge clicking into the Stello-Matic, as well as the Decho-Matic's ticket printing animation. Thanks, Jake!

Initially, I created a PNG file for each asset, and linked to them as usual. But by the time the Stello-Matic was finished, I realized that the number of requests was climbing; over 20 images, most of which 3kB or smaller. I figured I could benefit from inlining them, both to prevent flooding the server with requests as well as to avoid the (comparatively not insignificant) overhead of response headers.

While inlined data URLs save the requests and their response headers, I do have my own image format, SN, which is about 30% better than PNG. I cannot inline SN files as data URL, though, because the service worker decoding them to PNG cannot intercept "requests" for data URLs. However, I can still utilize the SN format for the larger images. The 30% size advantage will just have to outweigh the cost of response headers, which it does, from a certain size onwards.

Naturally, there is a mix of inlining and using the SN format that optimizes the overall number of bytes transferred (headers included). This optimum, I guessed, would lie on the side of "only inline the really small images", but a large part of the reason for inlining was to reduce server strain, so instead I chose a cutoff on the larger side. Only images 3kB and up would be served through the SN format, anything smaller inlined. While writing this essay, I did wonder how much I saved compared to the two extremes (everything as a separate request and everything inlined). So I did some testing, and here are the results:

Worth the squeeze? Objectively: probably not. But for me, as someone having fun messing around with a hobby project: absolutely!

The Stello-Matic shows the time it takes for a certain payload to finish playing (both before and during transmission) on a small screen. Since this screen is only capable of showing a handful of different characters, I figured I could draw them by hand and somehow make that work. I needed the numbers, a decimal, the "s", "m" and "h" for seconds, minutes and hours respectively, and I decided to add a U+274C (CROSS MARK) as well as an error state (like if the payload is too large). Creating the PNGs for the individual characters was easy enough, but now I had to integrate them into the Stello-Matic screen. I could've just written some JavaScript that generates image elements, or drawn the characters onto a <canvas>. But I figured it must be possible to turn them into a font. How hard could it be?

Well, turns out, if you know nothing about fonts, it's not that easy. I'm not a fan of installing new things on my computer, but I must've installed at least three font editors before giving up on trying to do it the proper way. It was time to whip out the truly wicked tricks!

I had known about FontStruct: a web app that gives you a grid to create your font with. Unfortunately, this a very manual process. It does not let you load an image onto said grid. And I wasn't about to copy the characters pixel-by-pixel! But FontStruct is a web app, so I reckoned I could probably uhm, use some force to get it to do what I wanted.

Thus, I opened the dev tools, and reverse-engineered the editor's code enough to be able to programmatically fill in specific squares in the grid. Then I loaded a character image into a separate <canvas> element, iterated over the pixels, and for each filled pixel, I generated a line of code that fills the square at that exact coordinate in the FontStruct editor. Copy all that code, paste it in the devtools for FontStruct, and voilà! I can load a character from a PNG into FontStruct. Rinse and repeat for all the characters I needed, and that was that! I had obtained my font file! I then compressed it using Font Squirrel's webfont generator, and the final product weighs in at an acceptable 4.5kB. I'm not proud of my methods here, but sometimes, you just have to put aside any notion of "elegance" and brute force things.

While I had made a hand-written font for the Stello-Matic, the Decho-Matic would need to be able to display arbitrary text, both on the bottom screen, as well as on the printed ticket. I wasn't so keen on compiling a handwritten font that supports that many characters, but on the other hand, the sharp edges on regular fonts looked out-of-place on the Decho-Matic. To fix this, I created an SVG filter that creates some minor distortion in the text, so that it blends much better into its hand-drawn environment. The filter itself isn't too complex:

<filter id="decho-text-filter">
	<feTurbulence
		type="turbulence"
		baseFrequency=".1"
		numOctaves="2"
		result="turbulence" />
	<feDisplacementMap
		in="SourceGraphic"
		in2="turbulence"
		scale="3"
		xChannelSelector="R"
		yChannelSelector="G" />
</filter>

The <feTurbulence> element creates a map of some colorful splotches according to some noise functions, and the <feDisplacementMap> then uses that image to slightly shift the pixels in the text. The result? A hand-written feel to a bog-standard font!

The second filter is responsible for the slight curl in the printed ticket.

To achieve the curl, I used another <feDisplacementMap>, but this time I had to carefully craft the displacement image. To create a curling effect, the displacement map needs to be a gradient, ranging from gray at the top (no displacement) to orange-ish at the bottom left (displacement mostly to the left, and a bit up). But this can't just be a gradient from gray to orange, because then the displacement would be linear, instead of a curve. To fix this, I instead started the gradient at a transparent gray, with the end being opaque orange, and then putting a gray rectangle behind it. The result still starts and ends the same, but what happens in between is different. To visualize this, think of what happens to the pixel in the middle of the gradient: originally this would be halfway through gray and orange. Now, it is halfway through gray and orange, but at 50% opacity. Then it blends with the gray background, pulling it more towards gray. The result is that, the further along the gradient you go, the "steeper" the incline to orange gets, resulting in the beautiful curving curl I wanted.