The Making of a Dumpster Fire

A few weeks ago we launched a new marketing project for HEY.com at dumpsterfire.email. If you haven’t seen it yet, it’s a flaming dumpster with a printer and conveyor. You email [email protected], it prints out your email, and drops it into the rolling flames on a livestream. Simple, right?

What follows is far more than you ever want to learn about building an internet-connected dumpster fire.

The Contraption

We started with the simple concept of “a flaming dumpster you can email.” The idea was that your email would cause a dumpster somewhere in the world to erupt with flame and consume your email in a moment of remote catharsis. We wanted anyone in the world to be able to play and not have to sign up for anything.

(Real quick: It goes without saying that this is a highly dangerous project with many exotic ways it could go very wrong. We worked with professionals to build this thing and would not have attempted it without them. This is not a complete manual on how to build a safe propane-powered flaming dumpster and this article should only be enjoyed as entertainment. Please don’t build this.)

Fire

Fuel selection was one of our first decisions. We needed a large flame that was instant, controllable, and predictable. This meant a gaseous flame we could turn on and off instead of a solid fuel like wood that would burn uncontrollably. Natural gas is plentiful in cities but delivered at too low of a pressure (~2 psi) to make a convincing, rolling flame. We wanted a real “burning garbage” look to deliver that 2020 essence.

We landed on propane because it’s also readily available but comes out of the tank at 100-200 psi depending on temperature. Propane is also a relatively “clean burning” fuel with the only byproducts being carbon dioxide and water vapor. We rented a 500 gallon liquid propane tank and connected it to a 25 psi regulator before a super heavy duty rubber hose that leads to the dumpster.

I reached out to my good friend Josh Bacon to help us with the flame effect portion of this project. He’s built a number of fire-enabled contraptions and works annually as a lead fire safety inspector at Burning Man. Leaning on existing knowledgeable people is good when you don’t have the time and burn cream to learn the hard way.

Two 120v industrial solenoids, or electronically-controlled valves, control the flow of propane at both the tank and at the back of the dumpster. When our microcontroller, a Raspberry Pi 4, calls for fire it opens the valves allowing the flow of propane from the tank. These valves are designed to “fail closed” meaning if we remove power the valves will snap shut by default.

Aboard the dumpster the propane flows to 3 loops of heavy copper tubing that acts as our flame effect. Tiny pinholes are drilled into the underside of the tubing to function as jets for the burning propane. We put these on the underside of the tubing so the flames would have a natural rolling look. The entire effect is wired with stainless steel safety wire to the underside of an expanded metal mesh grate that sits in a custom-fabricated ⅛” steel plate tray. The entire tray sits on welded supports so we can lift it out with a forklift for maintenance.

The propane is ignited by two redundant hot surface igniters – or HSIs – screwed into the effect tray. These HSIs remain on all of the time so we don’t have to manage a separate ignition circuit that may fail. If the igniters somehow failed and the propane was allowed to flow freely it would only be for short 30-second bursts into open atmosphere. As a safeguard we have an operator running the dumpster that’s tasked with overseeing the operation and hitting the “emergency stop” if anything goes awry.

Printer

With our dumpster largely sorted and connected, we looked towards the printer. Since this was going to be outdoors we couldn’t use an inkjet for fear the ink itself would freeze. A laser printer is warm by design so I searched for a commercial-grade printer that could deal with the duty cycle of running 8-10 hours per day. I settled on an HP M255dw as it was both affordable and toner was in wide availability. There were likely dozens of better options, but done is better than perfect.

The problem with most laser printers is they print “face down.” We needed our prints “face up” so you can see them on camera. Rather than trying to chase down the perfect printer we simply feed it a 2-page PDF with a blank first page so the output is “face up.” More on that later.

One of the more challenging parts of this project was getting this printer to successfully eject the paper onto the conveyor using gravity alone. We continually increased the printer-to-conveyor angle until the paper reliably slipped out naturally onto the belt. In hindsight a printer that ejected out of the front instead of the top would’ve been significantly easier to design around.

The printer is driven by the Raspberry Pi over ethernet using CUPS on Debian.

Conveyor

If you were trying to get this project done quickly and easily you wouldn’t use a conveyor. Getting this done the efficient way would probably involve the printer being located above the dumpster with a ramp leading to the flames. A ramp can never fail, works every time, and requires no code or power.

There is only one good reason to do it the way we did it: Conveyors are way cooler.

The conveyor we’re using is a Dorner that came out of a pharmaceutical manufacturing plant. In its former life it was part of a cap sorting machine that put pill bottle caps into neat little rows so they could be applied with great care. We removed all of that carefully-calibrated accessory structure and attached a giant steel leg to boost it up to dumpster height.

The conveyor was originally controlled by an intensely complicated industrial control system located in the box beneath the printer. After spending about 6 hours gazing into this abyss of relays and wires I decided it was far easier to splice into the control box on the side instead and simulate the start-stop buttons with relays. When we call for “start” our relay closes the “start” circuit at the button on the conveyor and the magic begins. To stop the conveyor we open the “emergency stop” circuit momentarily which stops the conveyor. It’s not the “proper” solution but it works which makes it right.

Enclosure

This mess of wires, propane, dumpster, and fire needed to be protected from the elements. It couldn’t be indoors because the fire would set off the fire alarm – among other obvious complications. We could’ve potentially figured out ways to vent the dumpster with a restaurant grill hood or similar, but that would’ve turned into a multi-thousand dollar engineering project with more electrical systems, fans, cinder block modifications and other high-effort things. So outside it went.

It was our industrial designer friend Eric Froh that suggested we hack the side off of a 20ft shipping container to make a somewhat weather-resistant enclosure. We hired Ben Wolf of Ferrous Wolf to modify our shipping container to fit the bill. Ben and his compatriots cut the side off of the container and reinforced it with structural steel. They custom-fabricated the chimney from the former container sides and reinforced it with angle iron and expanded metal. The whole operation was completed in about a week.

The container shields most of the project from the weather with a covered chimney out of the top. We enclosed the printer and electronics in a plexiglass and aluminum box to keep the rest of the weather off of the sensitive pieces.

The interior of the container was a drab beige color so we enlisted the help of our designer friend Monica Dubray to both choose and paint the container and the dumpster. The rest of the color is added by W2812b LEDs driven by a Pixelblaze.

Cameras and Streaming

The project is streamed during operating hours on 3 networked cameras. The main camera is a Panasonic CX350 while the other two are Panasonic CX10s. I chose these cameras because they can natively stream over hardwired ethernet via your choice of RTMP – the most common streaming protocol originally created by Macromedia – or NDI. NDI allows you to stream up to 4k over a local network and control camera functions remotely.

We ended up using NDI to feed the cameras into Open Broadcaster Software – OBS – which enabled us to create a picture-in-picture display showing all 3 cameras on 1 stream. OBS is running on a 2019 Macbook Pro on the network.

We send our composited stream out to Restream.io so it can simultaneously broadcast it to Twitch, IBM Video and Youtube. We originally had it just going to IBM Video (formerly Ustream) which was embedded on hey.science but we quickly shot beyond our allotted 5,000 viewer hours in the first few minutes of the project. At approximately 47k viewer hours, or $9,400, we made a fast switch to Twitch to avoid another giant hosting bill.

We still pipe video to IBM Video because that’s where our clips are made and sent to email submitters after their message gets torched. More on that below.

Controller

The microcontroller that runs the whole dumpster-side operation is a standard Raspberry Pi 4 8GB with a relay hat. The relay hat directly switches both of our 120v solenoids for gas control and the low voltage controls to manipulate the conveyor. It has been astoundingly robust and workable to our needs.

The Code

While I’m now knowledgeable in most things dumpster it’s our Lead Systems Administrator Nathan Anderson that developed all of the backend code that makes the dumpster work. I asked him for some brief words on what makes it tick:

“Despite running HEY.com, our amazing new email product, we were unwilling to run this experiment through our production servers. So we used postfix to forward [email protected] over to an AWS SES endpoint. There, we would do a preliminary screening, rejecting mail that failed DKIM, SPF, DMARC, SPAM, and VIRUS checks. We would also check the headers against our moderation rules (custom sender/domain exclusion checks). From there, we’d drop the email into a S3 bucket.

The S3 bucket is configured to send an SNS notification whenever an object is written to the bucket. This notification triggers another Lambda function that does several more steps:

1. Screen for size. Since the SES Lambda action does not receive information about the total size of the email, we have to check the object size in S3. SES allows email up to 10MB in size, but we limit the dumpsterfire to 5MB.
2. Render the email. We use the mail gem to extract text parts or images. This detection is brittle, as all email parsing is. I initially wanted to extract our email processing/rendering routines from Hey, but our time constraints ruled that out, and we went with a naive approach.
3. Write the simplified job object (sender, rendered content, Hey status) to S3, and pop a message into the screening queue, and another message on our reply queue.

After the emails are rendered, they’re almost ready to print. We have secondary screening queues where our rules are re-applied. This queue is designed to re-scan the jobs based on our moderation rules. After the screener lambda processes the jobs, they’re put into the moderation queue.

From here, we’re using node-red to handle the environment on the Raspberry Pi. We’ve got a moderation page that shows the Operator the content of the email, and provides them with buttons to:

• Print – Puts the message into a print queue. (VIP/Regular)
• Print Now – Puts the message into the Alpha queue.
• Skip – Drops the job, and deletes the content from S3.
• Skip and Block Sender/Domain – Drops the job, and adds the sender/domain to our moderation rules.

If the Operator notices there is a stream of messages from a single email address or domain, they’re able to add that sender/domain to our moderation rules, and we can re-scan the moderation queue, by sending the jobs in the moderation queue back through the screener.

The printing happens in a loop, and pulls messages from the 3 priority-based queues, Alpha -> VIP -> Normal. When the RaspberryPi pulls the job, it reads the body from S3, and then processes the email into a printable PDF via img2pdf or paps. Due to the way most printers output paper, we need to abuse the duplex function and generate a 2 sided PDF with a blank first page so it comes out with the text facing up.

Once we have the PDF, we use zuzel-printer, a node.js wrapper around `lp` to send the job to the printer, and tell us when it’s finished printing. After `lp` is done, we use onoff + setTimeout, to handle triggering our relays for the belt start/stop and triggering the fire.

In a perfect world, we’d use a sensor (machine vision detection, optical switch, etc…) to just keep the belt on until the paper was in the correct spot, but again, time constraints prevented us from using that method, and again, we used the naive approach of timing the belt speed. (Please don’t tell my FLL kids that I relied on timing for this…😅)

We’re also using those same timings to drive our video api. After we stop the fire, and then stop the recording, we tag the recording we just made with the job id, and store the video id in our job object that is written into the Completed Queue in AWS.

This queue handles cleanup in S3, and puts the final message on the complete-reply queue.

Both the reply and complete-reply queues are consumed by processes in our datacenter to send email replies out via [email protected].”

If you’d like to get the actual code for this project we made it available via a public repository here: https://github.com/basecamp/dumpsterfire-2020

Why

Most built objects are rarely as simple as they seem on the surface – even a dumpster fire. I could easily write a 600-page epic on the things we’ve learned behind each individual system on this project. 

You’re likely asking yourself, “Why go into excruciating detail about propane, shipping container architecture, and conveyor belt control schema – aren’t you working at a software company?”

The truth is the vast majority of these sorts of marketing activations are done by 3rd party marketing agencies. Some people somewhere at Brand X have a half dozen long meetings with Agency Y and talk about this big attention-grabbing thing they’re going to do for user growth. The agency finds anonymous, knowledgeable builders to put the thing together, drags it out into the public square, and slaps the decals for Brand X on it. Invoices go out, impressions are garnered, rinse repeat. You could swap out the stickers and make it about Brand Z next week. Nobody would know the difference.

A project like our dumpster fire would be nearly impossible to pitch in-house at a giant publicly-held company. You can hear it now: “It involves actual fire? That’s too risky. We don’t know how to do that. I don’t even know where we’d start. I don’t get it.”

We built this at Basecamp because it was fun. We got to work with physical parts, build and refine unpredictable machines, and make a few hundred thousand people laugh along the way. Is this going to be a good business move? We’ll see. Right now we know for sure that it was entertaining for both us and our audience. You can’t say the same thing about banner ads.

We used our in-house skills and passions to make this big, silly thing happen. The parts we weren’t great at – fabrication, flame effects, industrial design – were hired out to local artists directly at a thriving wage.

Why build an email-connected dumpster fire? Why not.

Credits:

Metal Fabrication: Ben Wolf and Ferrous Wolf

Industrial Design: Eric Froh

Colors and Paint: Monica Dubray

Flame Effects: Josh Bacon

HERL Website: Adam Stoddard, Marketing Designer at Basecamp

Developer: Nathan Anderson, Lead Systems Administrator at Basecamp

Photography: Derek Cookson

Chief Cook and Bottle Washer: Andy Didorosi, Head of Marketing at Basecamp

PS: I did a tech walkthrough of the project you can watch below: