A Filter Panel That Narrows a Million Rows in the Browser

I wrote before about how Updog's canvas grid renders a million rows without slowing down. That post ended on the real point underneath it. Rendering a million rows is not the same as working with them. A person importing a file does not want to scroll through a million rows looking for the dozen that need attention. They want the grid to hand those dozen rows over.

Updog is a browser CSV importer and spreadsheet editor, so those two problems, rendering the rows and letting a person work through them, sit inside the same product. The first post about Updog's browser-first approach already made this promise. Exact filters, validation states, error views, search, column transforms, and bulk actions are what make a large import manageable. I touched on the filter worker again in the canvas grid post, cheapest checks first, row data mirrored into typed arrays, only matching row IDs returned. This post picks up the rest of that story. It covers what the panel looks like, why the order of its checks matters, and what a person pays in milliseconds when they use it.

Why filtering lives on the surface

Spreadsheets filter one column at a time. Open the header, click the dropdown, pick a value, and repeat for the next column. On a wide sheet half the columns sit scrolled out of view, so finding the right header is its own small hunt before the click happens. Once a few filters are on, there is no single place that shows which ones are active, only a scroll back through each header to check. On a file where several things need checking at once, that becomes a bit inconvenient. Updog keeps filtering on the surface instead, in its own panel next to the grid, where search, error status, change status, and per-column filters sit next to each other and a person combines them rather than opening one menu after another.

Say the review needs every row where the country is Canada, the role is Support, and the email field has an error. Column by column, that means three separate menus and three things to remember. In the panel, it is three clicks, and the grid narrows right away.

None of this exists because manual cleanup is fun. Column matching, value matching, and validation catch most of what a file gets wrong, but not everything, and the leftover rows are why manual import exists as its own kind of work. A person still has to look at some of them and decide. Updog cannot make that step disappear. It can make it hurt less.

Cheapest checks first, text search last

Every filter a person turns on runs against the same rows in a single pass, but the checks do not cost the same. Checking whether a row belongs to a hidden data source is one Set lookup. Checking whether a row's text contains a search term means scanning every character of every field in that row. Updog's filter engine orders its checks by cost, starting with source visibility, then change status, then validation status, then empty cells, then per-column values, then numeric and date ranges, and text search last.

The order changes what a person feels when they use the panel. Turn on a text search after two filters that already narrow the data down to a couple thousand rows, and the extra cost is close to nothing, because the substring check only runs on the rows that already passed the cheaper filters. Turn on that same search with nothing else active, and it has to scan the full dataset, since there is nothing cheaper to filter with first. In one measured run against close to a million rows, adding a search term on top of two already-narrow filters added no measurable time at all.

What keeps a million rows fast to search

The panel's speed also comes down to what the filter engine keeps in memory and how it updates it. Filtering runs in a dedicated Web Worker, off the main thread, so typing in the search box never competes with the browser for the frame the grid is trying to paint. The worker holds the dataset as flat, parallel arrays rather than row objects.

let rowIds: TRowId[] = [];
let rowTexts: string[] = [];      // NUL-delimited, formatted, for display and search
let rowRawTexts: string[] = [];   // NUL-delimited, unformatted, for number and date compares
let errorBitmask: Uint32Array = new Uint32Array(0);
let editedBitmask: Uint32Array = new Uint32Array(0);
let newFlags: Uint8Array = new Uint8Array(0);
let deletedFlags: Uint8Array = new Uint8Array(0);
let sourceIds: string[] = [];
let rowIndexMap: (number | undefined)[] = [];   // rowId -> array index, O(1) lookup

Cell values live as one delimited string per row rather than an object with a key for every field. A row's error and edited state live as bits inside a Uint32Array, thirty-two columns packed into each word.

const columnMask = new Uint32Array(this._wordsPerRow);
if (filterColumns) {
  for (const col of filterColumns) {
    const idx = this.fieldOrder.indexOf(col);
    if (idx >= 0) {
      const w = idx >>> 5;
      if (w < this._wordsPerRow) columnMask[w] |= 1 << (idx & 31);
    }
  }
} else {
  columnMask.fill(0xffffffff);
}

Checking whether a row has an error in a column the person is filtering on becomes one bitwise AND instead of a loop over every field. On a sheet with forty columns, that is eight bytes of state per row for full error tracking.

None of it gets rebuilt on every keystroke. Editing a single cell sends the worker one small message carrying only that row's new text.

case "UPDATE_TEXT": {
  const idx = rowIndexMap[msg.rowId];
  if (idx !== undefined) {
    rowTexts[idx] = msg.rowText;
    rowRawTexts[idx] = msg.rowRawText;
  }
  break;
}

The worker updates one entry in a lookup table and moves on. The cost of an edit stays flat no matter how many rows sit behind it.

The last piece is what happens when a person types faster than the worker can answer. Every filter request carries a number that increases each time one goes out. If a newer request leaves before an older one comes back, Updog throws the old reply away instead of showing a result that no longer matches what the person typed.

private onFilterResult(
  filteredIds: TRowId[] | null,
  baseFilteredIds: TRowId[] | null,
  generation: number,
): void {
  if (generation !== this._filterGeneration) return;
  // ...apply filteredIds, notify subscribers
}

That single check is what keeps fast typing in the search box from flashing a stale result on screen before the real one arrives.

Several ways to narrow the data at once

The panel covers more ground than a single filter usually does. Search matches whole words or an exact cell, with an option to scope it to specific columns. A checkbox for rows with errors expands into the individual error messages behind it, so a person can isolate the one validation rule failing on four hundred rows without wading through every other kind of error at the same time. Separate checkboxes cover new rows, edited rows, empty cells, and rows a merge left in the wrong place. Per-column filters add select, number-range, and date-range controls wherever you define one on a column.

{
  id: "salary",
  title: "Salary",
  editor: { type: "number", decimalPlaces: 2 },
  filter: { type: "number-range", label: "Salary" },
},
{
  id: "role",
  title: "Role",
  editor: { type: "select", options: ["Admin", "Editor", "Viewer"] },
  filter: { type: "select", label: "Role", multiple: true },
},

That configuration lives in the same schema you already write for Updog's CSV importer SDK, next to validators and editors, with nothing extra to wire up. Every filter type, whatever it checks, feeds into the same ordered pipeline described above. There is no special case for one kind of filter at the engine's boundary, which is part of why adding a new way to narrow the data never costs a second pass over the rows.

What it costs in milliseconds

Numbers matter more than a description of the design, so here they are. The playground run behind these numbers held exactly a million rows across fifteen columns, with a filter defined on every non-text column. Against that dataset, a single filter interaction, a search term, an error toggle, a column filter, or a combination of them, round-tripped in 30 to 250 milliseconds. The slowest single case was a search with no matches at all, at just under 240 milliseconds, still nowhere near a wait a person would notice.

That is one measured session, worth treating as an honest sample rather than a guarantee. Sorting the full dataset is a separate operation from filtering and cost closer to 760 milliseconds in the same session, since it has to place every row in order rather than check each one against a condition once. Text columns add a further cost on top of that. Updog sorts them through Intl.Collator, the browser API that places accented letters, an umlaut among them, in the position a person reading that language expects, and each comparison during the sort costs more than a plain less-than check. Number and date columns skip the collator and compare values directly, which is part of why a text column takes longer to sort than a numeric one at the same row count. Filtering and sorting do different work, and this post is about the one that runs on every keystroke.

None of it matters at a few thousand rows. A plain array filter, no worker, no bitmask, runs fine at that size, and a dedicated panel is more interface than a small file needs. It starts to earn its cost once a real file pushes past a hundred thousand rows and a person needs a way to work through it that does not mean scrolling by hand.

The canvas grid post ended with rendering solved. The harder question was always what a person does once a million rows sit on screen. This is that answer. A panel with every way to narrow the data in one place, still fast enough that finding the dozen rows that matter feels instant, whether you are evaluating Updog as an embedded CSV importer or already have it running in production.