Writing a Shell in Go

I wrote a shell. Not a wrapper around bash. Not a configuration layer. A shell. Parser, pipeline executor, job control, signal handling, the works. Twenty-two thousand lines of Go that can run my dotfiles and not crash immediately.

The sane response to “I wish my shell did X” is to write a shell function. The insane response is to write a new shell. I chose the second one.

The Problem with Pipes

Unix pipes are one of the best ideas in computing. Take the output of one program, feed it to the next. Simple, composable, elegant. Except the data flowing through those pipes is unstructured text. Every program in the pipeline has to parse whatever the previous program decided to print, and they all decide differently.

I’ve written enough awk '{print $3}' followed by sed 's/,//g' followed by sort -n to know that this is fine for quick tasks and miserable for anything complex. The moment you need to filter JSON, aggregate by a field, or join two data sources, you’re reaching for Python or jq or giving up and opening a Jupyter notebook.

I wanted a shell where structured data could flow through pipes as naturally as text. Records with fields, not lines with whitespace. Something that still felt like a shell when you needed to ls and grep and cd, but could handle real data processing without switching to a different tool.

Records Through Pipes

The core idea is record streams. When a command produces structured output, it emits JSON records tagged with a binary marker so the next command in the pipeline knows it’s getting records, not text. Commands that understand records can filter, transform, and aggregate them. Commands that don’t just see normal text output. Backward compatibility preserved.

In practice it looks like this: you pipe the output of a command that produces records into a transformation, and the transformation operates on fields by name instead of column positions. No more counting whitespace to figure out which field you want. No more sed gymnastics when a column contains spaces.

The record system was the second big feature I built. The first was the Lisp.

The Lisp Inside the Shell

The shell embeds an interpreter for a language I’ve been building called M28, a Lisp with Python-like syntax. It handles the data processing side of things: list comprehensions, lambda functions, generators. When you need to do something more complex than a simple pipeline, you can drop into M28 inline.

The integration goes both ways. M28 can call shell commands and capture their output. Shell pipelines can include M28 transformations. The parser has heuristics to figure out whether a parenthesized expression is a shell subshell or a Lisp expression, which was a more interesting problem than I expected. A lot of shell syntax looks like valid Lisp if you squint.

This is where the project gets opinionated. Most modern shell alternatives pick one approach: Nushell has its own expression language, PowerShell has .NET objects, Oil has its own syntax extensions. I went with embedding an existing language because I wanted the shell to be a shell and the programming language to be a programming language. Keep the strengths of both and let them talk to each other.

Everything Else a Shell Has to Do

Building the exciting features is fun. Building the boring ones is where the time goes.

Job control with proper signal handling. Background processes that get their own process groups. pushd and popd with a directory stack. trap for signal handlers. set -e for strict mode. Here-documents. Command substitution. Extended test expressions with regex support. Tab completion backed by a SQLite database that learns which arguments you use with which commands.

Each of these features has edge cases that only become apparent when you try to use the shell for real work. cd - should take you to the previous directory, obviously. But what about cd with no arguments when HOME isn’t set? What about CDPATH lookups? What about symlink resolution? Every builtin has a story like this, a seemingly simple behavior with a long tail of special cases that bash handles and you didn’t know about until yours didn’t.

I have new respect for the bash maintainers. They’ve been handling these edge cases for decades.

Where It Stands

The shell is functional enough that I use it for development sessions. Parser handles the common cases. Pipelines work with proper file descriptor management. Job control is solid. The record stream system is new and still finding its shape. M28 integration works but the boundary between shell and Lisp could be smoother.

It’s not a replacement for bash. It might never be. But it’s a useful tool for the specific thing I built it for: processing structured data without leaving the command line. And like all side projects that involve building something from scratch, the real value is in what I learned along the way. How shells actually work is different from how you think shells work, and the only way to find out is to build one.

The code is on GitHub.