One Index, Many Truths

Most multi-tenancy questions in a search cluster are about hiding documents. Tenant A sees these companies, tenant B sees those, and you solve it with a filter on a field.

This was the other kind. Every tenant could see the same company. They just could not agree on what its revenue was.

The problem

The platform held financial data for Indian companies, most of it derived from statutory filings. That gives you one obvious set of numbers per company: the public one.

But some client organisations maintained their own curated version. An analyst had gone through the filings, reconciled a subsidiary, restated a segment, and produced a revision that their organisation should see instead of the public numbers. Not in addition to it, not on a separate screen. When someone from that organisation searched for companies with revenue over a billion rupees, their curated figure was the one that had to decide whether the company matched.

So: one company, several simultaneously valid sets of financials, and which one is true depends entirely on who is asking.

A flat document cannot express that. Revenue is one field and holds one number. The two obvious escapes both fail on contact:

• An index per tenant. Hundreds of nearly identical copies of the whole company index, every reindex multiplied by the tenant count, and a filings update that has to fan out everywhere. No.
• A document per tenant, per company. The document count multiplies by the number of tenants, and every non-financial value (name, address, directors, products, all the expensive crawled and classified data) gets duplicated alongside it, for data that is identical across every copy.

The thing that varies is a small subset of fields. Only that subset should multiply.

The shape

The financial fields move off the top level of the document and into a nested array. Each entry in that array holds one complete, internally coherent set of financials, tagged with two things: which tenants it belongs to, and which curation run produced it.

Everything that does not vary by tenant stays exactly where it always was, stored once. One company is still one document.

The query side

This is where the real work was, and it is almost entirely invisible from the outside, which was the goal. Callers send the same filters they always sent. Four things happen to those filters on the way down.

The tenant filter is applied. Any query that touches a financial field automatically gains a restriction to the sets belonging to the current user’s organisation. Automatically is the important word here. If this were something a developer had to remember, someone would eventually forget, and forgetting means showing one client another client’s curated numbers. It belongs in one place that every query has to pass through.

Filters are rewritten. A caller still asks for revenue above a billion, because a caller should not have to know how the index is laid out. The query layer knows which fields now live inside the nested array, rewrites the filter to point at them, and asks Elasticsearch to report back which entry matched.

Sorting gets the same treatment. Sorting on a nested field needs the same rewriting, and it is easy to miss, because sorting is usually handled a long way from filtering.

Results are flattened again. The matching entry comes back nested, and before the response leaves the API its values are lifted back to the top level of each result. The caller sees revenue sitting exactly where it always sat. Nothing downstream, no serializer, no export, no chart, had to learn that anything had changed.

That last step is what made the migration finishable. The nesting exists between the query layer and the index, and nowhere else in the system.

The write side

The other reason nesting wins here is partial updates.

When an analyst re-curates one organisation’s numbers, only one entry in the array has changed. Rebuilding the whole document to write it would mean fetching the directors, the products, the addresses and everything else again, none of which have changed. That is expensive, and it is a race waiting to happen if a filings update lands at the same moment.

So a write to a single set goes through a script that edits the array in place: find the entry for those tenants, update it if it is already there, add it if it is not, and drop any entry that no longer belongs to anyone. The rest of the document is written separately, and both go out in the same bulk call.

One organisation’s numbers change. Nothing else on the document is touched.

Proving it

The migration was the frightening part. Financial data on a platform where people make investment decisions is not somewhere you want to find a subtle regression six weeks later.

So we did not diff the code. We indexed the same companies both ways, old shape and new shape side by side, and compared every field of every document in batches of several thousand, writing out one row per divergence: which record, which field, the old value, the new value, the verdict.

The first run produced a couple of thousand mismatches, and the useful discovery was that almost none of them were bugs.

The old shape fetched each financial field on its own, each one taking the most recent record it could find. The new shape fetched a whole coherent set at once. So where a company had a balance sheet for 2023 but a cash flow statement only up to 2022, the old document had quietly been placing a 2023 figure next to a 2022 one and presenting the result as a single year. The new document returned nothing instead. That is not a regression. That is the new index refusing to invent a number the old one had been inventing for years.

Which is the actual argument for building the harness. Diffing field by field did not just tell us the migration was safe. It surfaced a class of data defect that had been sitting in production, invisible, because nothing had ever compared those fields against each other before.

The rows that mattered, genuine mismatches rather than expected blanks, were a small minority, and each one was a real fix. Triage was the work. The diff was only what made triage possible.

What it cost

I would do this again, and I would still be honest about the bill.

Nesting is not free. The indexing code has to hold two ideas at once, the ordinary fields and the ones that vary by tenant, and the query layer permanently knows about both kinds, so every new query feature has to ask which kind it is dealing with. Anyone joining the team meets that distinction on their first day and carries it forever after.

The alternative was hundreds of duplicate indices or hundreds of duplicate documents. For a small set of fields that genuinely differ by tenant, on a document where everything else is identical and expensive to build, nesting is the right trade. But it is a trade, and the cost lands on whoever maintains the query layer next.