·2 min read

Taktile is a top-k contributor to duckdb-iceberg

Taktile supports open source. We support React Flow, a Berlin-built library at the core of our decision flow editor, and we believe the tools we all depend on get better when the people using them help build them.

Recently we have gone from paying for open source to contributing to it. Our latest work is on duckdb-iceberg, DuckDB’s Apache Iceberg extension, where we contributed a patch that makes top-k queries (ORDER BY … LIMIT k) much cheaper. Those are exactly the operational queries that humans and agents need to get the latest data from large Iceberg tables.

At DuckDB Paris, organised by Altertable, Tom argued that duckdb-iceberg still has plenty of low-hanging fruit: optimizations that anyone willing to dig in can reach. Our top-k patch reuses an optimizer hook that DuckDB core already had. If you are looking for a way into a high-impact open source project, this is a good place to start.

Why contribute? Everyone wins

Contributing to open source is a win for everybody:

  • For you. A code artifact you can point to regardless of where you work.
  • For your employer. Your company gets a better tool and bragging rights.
  • For the community. Nobody else has to solve the same problem again.

What we contributed so far

Pull requests merged into duckdb-iceberg:

  1. Order iceberg_scan and prune matching: for ORDER BY … LIMIT k queries, sorts data files by their per-file column bounds and skips files that cannot contain a top-k row. Resolves #746. Contributed by Tom Larkworthy.
  2. Prune all-null data files for constant comparisons: WHERE col = 'x' now skips files where col is always null, without breaking null-safe comparisons such as IS NOT DISTINCT FROM. Contributed by Tom Larkworthy.
  3. Fix equality deletes leaking across non-matching partitions: fixes a correctness bug where an equality delete was applied to data files in other partitions. Contributed by Jack Goldsworth.

We also reported bugs that the maintainers then fixed: #940, #941 and #950 by Tom, and #660 by Ted Papathanasiou.

Thanks to the DuckDB maintainers, especially Tom Ebergen and Thijs, for careful reviews; to Sylvain Utard and Altertable for organizing the meetup; to Algolia for hosting; and to everyone who came out in Paris. See you at the next one!