How Telelogs works

Context

I have a bad habit when something breaks. I open a terminal, docker logs -f one container, open another tab, kubectl logs a pod, open another tab, lose track of which tab is which, and by the time I find the error the pod has restarted and the logs are gone. Pretty annoying.

The “proper” fix is a full observability stack, and honestly, for a lot of my stuff that feels like hiring a moving company to carry one chair. I wanted something smaller: one app where all my container logs show up, and somewhere cheap to keep them for longer than a container lives.

So I started building Telelogs. It’s a self-hosted server that tails your Docker containers and Kubernetes pods, a native desktop app to read them, and every line archived into a bucket that belongs to you. It’s written in Rust and it’s still early (v0.2.0 as I write this), so take everything here as “how it works right now”.

Here’s the whole thing in one picture. Press play, and turn the sound on if you like little clicks.

Play 0:00 / 0:00

The labels under the diagram match the sections below, so you can jump to whichever part you care about.

The pieces

The repo is a Cargo workspace with five crates, and each one has one job:

Crate	What it does
telelog-core	The shared log model: LogRecord, Target, Filter
telelog-proto	The gRPC API and the conversions to and from core types
telelog-sources	Where logs come from: Docker and Kubernetes, VMs next
telelog-server	Collects logs, keeps them in memory, archives them, serves them
telelog-app	The desktop client, built with GPUI

Splitting it this way made me answer “what actually is a log line here?” once, in telelog-core, instead of three slightly different times. Every source turns whatever it reads into the same LogRecord: a timestamp, where it came from, stdout or stderr, a level, the body and some labels.

tail

The server sits next to the things you want logs from. For Docker it talks to the daemon over the socket (through bollard), tails every container, and follows new ones as they start. Stopped and removed containers are marked, not dropped, which matters later.

docker run -d --name telelog-server \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -p 127.0.0.1:7070:7070 \
  ghcr.io/tolaniverse/telelog-server --allow-unauthenticated

Kubernetes came in v0.2.0. With --kubernetes, the server uses your kubeconfig’s current context (or its service account when it runs inside the cluster) and reads every container of every pod. Each one becomes a source named namespace/pod/container, pods are picked up the moment they start, and containers get followed again after a restart.

telelog-server --kube-context prod --namespace shop,payments \
  --selector app.kubernetes.io/part-of=shop

A small thing I’m weirdly proud of: on Docker Desktop or OrbStack, Kubernetes runs its containers on the same Docker daemon, so a naive server would read every pod twice. Telelogs reads those once, through Kubernetes, and skips the pause containers. I tested it against a real AKS cluster with 4 nodes and 99 containers, and it was ready in about 3 seconds while sitting around 40 MB of memory.

stream

The server keeps the newest lines in memory and streams them to the app over gRPC (tonic on both sides). The whole API is five calls:

service LogService {
  rpc ListTargets(ListTargetsRequest) returns (ListTargetsResponse);
  rpc Tail(TailRequest) returns (stream LogRecord);
  rpc WatchTargets(WatchTargetsRequest) returns (stream Target);
  rpc Query(QueryRequest) returns (stream LogRecord);
  rpc GetStorage(GetStorageRequest) returns (StorageInfo);
}

Tail is the live stream. WatchTargets tells the app whenever a container starts, stops or disappears, so the source list stays honest without polling. Query and GetStorage are for the bucket, more on that in a bit.

filter

The app is one live stream across everything, with text, level and time filters, plus a JSON view for structured lines.

Levels were trickier than I expected. Most apps print something like INFO or level=warn, cool, easy. Kubernetes components use klog (I1002 23:47:01 ..., W..., E...), so those needed their own parsing. Stack traces were worse: a Java exception is one ERROR line followed by twenty lines of at com.something... with no level at all. Telelogs now gives those continuation lines the level of the line that started them, so filtering by Error shows you the whole trace and not just its first line. Python tracebacks get the same treatment.

(Also, fun bug: it used to crash if a log line had a multi-byte character, like an accent or an emoji, sitting right around its 64th byte. Rust strings and byte offsets humbled me again.)

secure

On 127.0.0.1 the server runs open, because it’s just you on your machine. The moment you ask it to listen anywhere else, it refuses to start without a token:

export TELELOG_TOKEN=$(telelog-server gen-token)
telelog-server --listen 0.0.0.0:7070 --tls-cert cert.pem --tls-key key.pem

The app sends the same token, and if you’re using a self-signed cert you hand it the CA with --ca-cert. I went back and forth on whether this was too strict for a hobby tool, but logs leak everything (emails, tokens, the odd password someone printed “just for debugging”), so I’d rather the default be annoying than open.

archive

This is the part I actually care about most. Memory is fast but it forgets, so give the server a bucket and it archives every line there:

AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=eu-west-1 \
telelog-server --archive-url s3://acme-logs/prod --retention-days 90

It works with S3, Cloudflare R2, MinIO, Google Cloud Storage, or just a local folder (file:///path), all through the object_store crate. Every minute by default, the server writes what it has as zstd-compressed JSON lines under:

<prefix>/v1/<YYYY-MM-DD>/<HH>/

and each file is named after the time range it covers. That naming is doing a lot of quiet work. To find logs for a time range, you list one or two hour folders and read the files whose names overlap. No index, no database, just the path.

Two more things keep this sane:

• A _checkpoint.json records the newest archived line. When the server restarts, it reads the checkpoint and asks Docker for everything since then, so there’s no gap and no duplicates.
• Once an hour, the server deletes whole days older than your retention. --retention-days 0 keeps everything, if you like paying for storage.

docker rm

This is why the archive exists. Someone removes a container, and normally its logs go with it. Here, they were already flushed to the bucket a minute ago, so the history is still there. You can query a container that no longer exists, which still feels slightly magical to me every time it works.

query

Pick a time range in the app, say last Tuesday between 13:00 and 14:00, and the app calls Query. Whatever the in-memory buffer doesn’t hold, the server reads back from the bucket.

The fiddly part was overlap. The app already has some lines live, and the bucket has some of the same lines archived, so a naive merge shows things twice. QueryRequest has a skip_from map: for each source, “I already have everything from this timestamp onwards, don’t send it”. Sources the app doesn’t mention come back in full, and that’s exactly how lines from removed containers show up.

One honest limitation: a bucket search loads at most the newest 20,000 matching lines per query. That’s fine for “what happened in this hour”, and not fine for “grep my whole year”. I don’t know yet if the fix is smarter paging or a small index, and I’m trying not to build a database by accident.

yours

The app has a Storage screen that shows the bucket’s size, oldest day, last write and any errors. That’s it. The logs sit in your bucket, in a boring documented format, and if Telelogs disappeared tomorrow you could still read them with zstd -d and jq.

That was kind of the whole point. I didn’t want another place my logs live. I wanted a nicer way to read the place they already live.

What’s next

• VMs, probably through OpenTelemetry.
• The app is macOS only for now, and it isn’t notarized yet, so the first launch needs Open Anyway in System Settings.
• Better querying of the bucket, once I figure out what “better” means without overbuilding it.

The server is AGPL-3.0, and the app and libraries are Apache-2.0. If you try it and it breaks, please open an issue, I genuinely want to know.

Thank you for staying