Per-field aggregation

per-field is the aggregation service behind federated analytics. Where average combines everybody's model layers one way — a weighted mean — per-field combines each named field of the upload with its own operation, and you choose which.

That choice is the point. If four sites each report a target_spacing of [1.0, 0.8, 0.8] or thereabouts, should the federation take the smallest, the largest, or the mean? Only you know what your pipeline needs downstream. The platform will not guess, and this page is largely about what that means in practice: there is a step where you must act, and until you do, the branch will not aggregate.

This page is for the person setting up the branch. For what federated analytics is and why the numbers are what they are, read Federated analytics, by hand first.

The flow

Five steps. The third is the one people lose work to.

Configure the branch      →  service = per-field
Leaves upload             →  the round is REFUSED, and your field
                                names are discovered from it
YOU ASSIGN AN OPERATION   →  one per field  —  AND SAVE
   TO EVERY FIELD, AND SAVE      (nothing is stored until you save)
Leaves upload again       →  this round aggregates
Results return            →  every site downloads the same combined values

Step 2 failing is not a mistake you made. Read on.

1. Configure the branch

Set the branch's aggregation service to per-field, along with the usual branch settings. Like any change to the whole branch configuration, this requires the run to be stopped.

You do not need to know your field names yet. You cannot know them yet — that is what step 2 is for.

2. Leaves upload, and the platform discovers the fields

Bring your leaves up and let them upload once. Each upload is a set of named entries rather than a positional list of layers, and BranchKey records the names, shapes and types it finds across all participants.

This round will be refused, because none of those fields has an operation yet. That is the designed behaviour and is covered in full below; the point of the round is to find out what your payload contains.

That discovered list is what you are shown in step 3. It is the union across every participant, so a field that only one site sent still appears rather than being quietly dropped.

A field only some leaves send is a fault, not an option

If a name in that list did not come from every leaf, do not assign it an operation. Assigning it makes every subsequent round fail, because an assigned field must be present at every participant. Fix the site so that all leaves send the same field names. See Every leaf must send the same fields.

3. Assign an operation to every field — and save

This step is not optional, and neither is the save

Open the branch's aggregation settings. You will see every discovered field with an operation selector beside it. Choose an operation for each one, then press Save settings (or Save N change(s) in the bar that appears once you have unsaved edits).

Choosing an operation does not store it. Navigating away without saving discards every choice you made on that screen — all of them. If you have twenty fields to assign, that is twenty selections gone.

The page does try to stop you: it marks each unsaved row, shows a pinned bar counting your unsaved changes, and asks for confirmation if you navigate away or close the tab. Treat those as a backstop rather than the plan. Save deliberately, before you go anywhere, and reload the page afterwards if you want to confirm it took — when everything is stored the page says "All changes saved."

Some rows arrive pre-filled. Fields whose names follow the save_analytics convention — score_n, score_sum, score_sumsq, score_min, score_max, score_nan — already have an operation selected, because the client library produced those names and their combination is arithmetic rather than opinion (see How the six combine).

A pre-filled row is still an unsaved row. It is filled in for you; it is not stored until you save. The settings page shows those rows distinctly for exactly this reason.

Every other field — anything your own code produced — starts blank. That is deliberate. A confident wrong default is worse than a blank, because nobody re-examines a control that already looks answered, and min, max and mean on a field like target_spacing each produce a perfectly plausible number and a completely different preprocessing plan.

4. The next round aggregates

Once every field has a saved operation, have your leaves upload again. That round proceeds normally: each field is combined across sites by the operation you assigned, and the result is packaged as one aggregated file.

The round that was refused in step 2 does not come back — it was terminated, not held, so its uploads produce nothing. It bought you your field names, which is what it was for.

5. Results return to every site

Every participant is notified and downloads the same aggregated .npz, containing the combined value of each field under the name it was sent under. It is the same wait-and-download mechanism as a training round — client.queue.get(...) then client.file_download(...) — worked through in full under A complete analytics run below.

The first round is refused, and that is correct

If any field in the upload has no operation assigned, BranchKey refuses the round. It does not pick something sensible, and it does not aggregate the fields it does understand and skip the rest. It produces nothing.

On the very first round of a new branch, every field is new, so this is what normally happens. A refused first round is expected. It is not a fault, and nothing is broken.

The refusal names the fields, which is the whole reason it is worth reading rather than retrying:

aggregation blocked: no primitive is assigned to 'intensity_p995' (sent by leaf
4f2c9e1a-…); 'patch_size' (sent by leaf 8b91d33c-…, leaf 4f2c9e1a-…). Every field
present in the upload must have an entry in 'field_primitives' before this branch
can aggregate; refusing to produce a result for an unconfigured field

Two things to take from that message: which fields need a decision, and which leaves sent them (as leaf UUIDs, not the friendly names used above). If a field arrived from some leaves and not others, read Every leaf must send the same fields before you assign it — assigning it is the wrong fix and will fail every round.

The refused round is lost. It is terminated, not held: nothing is aggregated, and no result is produced from those uploads.

The branch is also paused. Any refusal on this page pauses the run — not just the must_match disagreements described below. That is deliberate: a branch that cannot aggregate should stop accepting uploads rather than keep collecting files it will refuse for the same reason every round.

So the recovery is three steps, and the third is easy to miss:

Assign the named fields.
Save.
Press start. The run is paused, so without this there is no next round — your leaves will sit idle waiting, and it will look as though saving did nothing.

The next round is the one that aggregates.

You do not need to stop the run to assign the operations, and you should not. That is a deliberate exception: everywhere else, changing a branch's configuration requires the run to be stopped, but stopping a run invalidates the branch's lanes and purges the uploaded files. If this setting were gated the same way, the act of fixing the problem would destroy more than it recovered. The exception exists to avoid making things worse, not to rescue the refused round.

Why refuse rather than default

An unconfigured aggregation would still return a number. It would look like an answer, it would flow into your normalisation constants, and nothing downstream would ever indicate that the platform had guessed. Refusing costs you one round. Guessing costs you the result and does not tell you.

Where you actually see the refusal

This is worth knowing before it happens, because the leaf client is told nothing. A refusal is not sent to the participants. client.queue.get(...) simply never fires, so a leaf script waiting on it blocks until its timeout and then reports nothing more useful than a timeout. If your job appears to hang after uploading, this is the first thing to check.

The refusal reaches you in two places:

The branch's audit log, as an aggregator.schema_validation.failed entry. This carries the message above, the reason, the field names and the leaves involved. It is the authoritative record.
The branch's aggregation settings page, which lists the fields still awaiting a decision in a banner — "N field(s) need a decision. Aggregation is blocked until every field present in an upload has an operation: …". In the common case this is the same list, and it is the more convenient one, because you are already on the page where you fix it.

If the settings page shows no fields at all, the platform has not recorded a field list for this run yet — recording is best-effort and can be skipped. Take the names from the refusal message in the audit log and add them by hand.

Every leaf must send the same fields

Three further conditions, all terminal, all checked before any arithmetic runs. After the missing assignment, these are the most likely way a round dies.

Every assigned field must be present at every leaf. If a field is in the assignment map but missing from any one upload, the round is terminated. The message is about the round as a whole, not about one leaf and one field: it lists every field that failed to arrive everywhere, and for each it names both the leaves that did not send it and the leaves that did.

aggregation blocked: the participants disagree about which fields they send. 6
field(s) assigned in 'field_primitives' did not arrive from every participant:
'ratio_max', 'ratio_min', 'ratio_n', 'ratio_nan', 'ratio_sum', 'ratio_sumsq' —
not sent by leaf 92aa0031-…, sent by leaf 18715147-…. A field can only be combined
across the sites that sent it, which is not the statistic the branch asked for and
would not be labelled as anything else. The remedy is at the SITES, not in the
configuration: make every participant send the same field set. Taking the field
back out of 'field_primitives' does not work either — a field present in an upload
with no assignment is refused as 'unassigned_field'

Both sides are named because which side to change is the question you are actually asking, and it cannot be answered from one of them. The fields are grouped: six fields that share the same pair of leaf lists — one site with an extra ratio column, which is one column but six bundle entries — are attributed once, rather than repeating the same two UUIDs six times.

This is why a field that only some of your leaves send is a trap rather than a choice. Assign it and every round fails with field_missing; leave it unassigned and every round fails with the refusal above. There is no configuration that makes it work, and the message says so. The fix is at the site — make every leaf produce the same set of field names — not in the settings page.

If a field is assigned but arrives from no leaf at all, the message is a different one: nobody is disagreeing, the map simply names a field this run's payloads do not contain, and there the fix is in the configuration — remove it from the map, or fix the sites to send it.

Shapes must agree. The first upload in the round is the reference, and every other leaf is measured against it:

leaf 8b91…: field 'target_spacing' has shape (2,), run schema expects (3,)

Types must agree, too. This is the subtle one:

leaf 8b91…: field 'num_channels' has dtype float64, run schema expects int64

A site writing "num_channels": 4 and another writing "num_channels": 4.0 disagree, because one is an integer and the other is a float. Nothing about the numbers is wrong; the round dies anyway. Fix the value's type in your client code so every site produces the same one.

save_analytics cannot hit the type trap

save_analytics fixes the types itself — the two counts, _n and _nan, are always 64-bit integers and the other four are always 64-bit floats, whatever the dtype of the column you handed it. The shape and type conditions above bite on save_fields, where the value is whatever you built.

The operations

Set one of these per field. sum, min and max cover almost every analytics job.

Every operation combines element-wise across sites: for a field holding three numbers, the first result comes from everyone's first number, the second from everyone's second, and so on. That single rule serves both a scalar like score_min and a vector like target_spacing.

Operation	What it does	When to use it
`sum`	Adds the participants' values.	Counts, class frequencies, the `_n` / `_sum` / `_sumsq` fields that make a pooled mean and variance exact, and `_nan`.
`min`	The smallest value across participants.	Pooled minimums. Exact: the smallest value in the federation is the smallest of the per-site minimums.
`max`	The largest value across participants.	Pooled maximums. Exact, for the same reason.
`mean`	Arithmetic mean over participants — each site counts once.	Only when you genuinely want to average the sites.
`weighted_mean`	Σ(wᵢxᵢ)/Σwᵢ using each leaf's declared `weighting` — federated averaging applied to one field.	Learning-side payloads.
`must_match`	Asserts every participant sent an identical value, and returns it.	Configuration-like fields: channel counts, label sets, modality lists.
`pooled_variance`	Exact population variance of the combined data, from each site's (n, Σx, Σx²).	A single field whose value is those three statistics, in that order. Not for `save_analytics` output — see below.
`median_of_values`	Median of the participants' values.	See the warning below.

Each of the four operations below carries a condition that is easy to miss.

`mean` is not a pooled mean

mean averages the sites, not the records. Four sites reporting means of 0.2, 0.4, 0.6 and 0.8 give 0.5, no matter whether one of them holds 100 rows and another holds 800.

If you want the mean over all the data — which is almost always what you want — send n and Σx as separate fields, apply sum to both, and divide. That is exact and correctly weighted by row count. save_analytics does this for you.

`pooled_variance` does not fit the `save_analytics` bundle

pooled_variance expects one field whose value is the three statistics stacked together — shape (3,), with n first, Σx second and Σx² third. save_analytics does the opposite: it emits age_n, age_sum and age_sumsq as three separate one-element fields. There is no way to point pooled_variance at them.

If you sent your columns with save_analytics, do not reach for this operation. Give the three fields sum, as the settings page already suggests, and compute the variance yourself from the combined result — it is Σx²/n − (Σx/n)², worked through in Federated analytics, by hand. pooled_variance is for the case where your own code packs the three statistics into a single array before uploading.

`weighted_mean` needs a `weighting`

It reads each leaf's declared weighting, which an analytics payload does not carry. Applied to an analytics upload it fails rather than quietly falling back to an unweighted mean — which is the right behaviour, but it means this operation belongs on the learning side.

`must_match` disagreements pause the branch

A must_match field is one where sites disagreeing means somebody is misconfigured, not that the data varies. When it fires, three things happen: the round is terminated, an event is published naming the differing leaves and their values, and the branch's run is set to pause.

Every refusal pauses, not just this one

Pausing is described here because must_match is the clearest case, but it is not special. Any terminal refusal on this page — an unassigned field, a missing field, a shape or dtype mismatch, an unknown operation — pauses the run in the same way, for the same reason and with the same recovery.

must_match violation on 'num_channels': leaf 8b91d33c-… = 4, leaf 4f2c9e1a-… = 1

The pause is intentional, and it is specifically pause rather than stop. Both block further uploads, but only stop advances the run number — so pausing leaves the run, and everything recorded about it, intact for you to come back to. Fix the misconfigured site, then set the run back to start from the dashboard or the API, the same control described under Runs: start, pause, stop.

Note that the error reports the actual per-site values, because those are what tell you which site to correct. This is a configuration disagreement, not a statistic; reducing it to "two sites disagree" would withhold the only useful part.

`median_of_values` is a median of medians

median_of_values answers a different question

Each site contributes one value per field, so a median taken across sites is a median of per-site values — not the median of the pooled data. The two can differ substantially.

It is deliberately not called median for that reason. It never errors and never warns: it returns a plausible number that is subtly the wrong answer, which is the most expensive kind of wrong. If you need a true pooled median, you cannot get it here — see below.

What is not available

True pooled medians and percentiles. They cannot be derived from a fixed bundle of summary numbers. This is a property of the mathematics, not a missing feature: the information is not in what the sites sent, and no combining operation will recover it.

Say your preprocessing clips to percentile bounds — nnU-Net's 0.5/99.5 intensity clipping is exactly this pattern. That step cannot be federated with per-field aggregation as it stands. Minimum and maximum are available and exact; percentiles between them are not. If this is on your critical path, contact us before you design around it.

Per-site figures. The aggregated result contains combined values only. It does not carry any individual site's statistics, and no user is shown them. The single exception is a must_match disagreement, described above, where the per-site values are the content of the error.

Reserved names

Two entries in every upload archive belong to the platform, not to you:

weighting — the aggregation weight, used by the learning services
__bk_payload_kind__ — the archive's declaration of what kind of payload it is

Neither appears in the discovered field list, and neither needs an operation. If your archive has 22 entries and the settings page offers you 20 rows, this is why. Assigning an operation to weighting is rejected outright; __bk_payload_kind__ is simply never offered to you, because the aggregator strips it before the field list is built.

You cannot use either name for a field of your own, either — the client library rejects both when you build the archive.

The client library side

Two methods produce a per-field payload.

save_analytics — you hold a raw column and want its statistics pooled across the federation. It reduces each column to six numbers before writing anything, so the raw values never leave the site, and the field names it produces are the ones the settings page can pre-fill:

file_path = client.save_analytics({
    "age": patients["age"].to_numpy(),
    "tumour_volume": patients["volume"].to_numpy(),
})
file_id = client.file_upload(file_path)

save_fields — your own code already computed the values, and you choose how each combines:

file_path = client.save_fields({
    "target_spacing": np.array([1.0, 0.8, 0.8]),
    "num_channels": 4,
    "n_cases": 312,
})
file_id = client.file_upload(file_path)

save_weights is untouched by any of this and continues to work exactly as before.

Availability

save_analytics and save_fields are newer than save_weights and are not yet in a published release of the branchkey package, so upgrading will not currently produce them. Check with hasattr(client, "save_analytics"), and contact us if you want to use per-field aggregation now — we will tell you how to get a build that has them.

A complete analytics run

The whole loop for one leaf: reduce the columns, upload, wait, download, derive. Unlike a training loop this runs once, not for a fixed number of rounds — there is no model and nothing to converge.

It uses the same client setup as Your first federation, which is worth reading first if you have not connected a leaf before.

import json
import numpy as np
import pandas as pd
from branchkey import Client, Credentials, APIConfig

with open("leaf-1.json") as f:
    credentials = Credentials.from_dict(json.load(f))

with Client(
    credentials=credentials,
    api_config=APIConfig(host="https://app.branchkey.com"),
    use_websocket=True,
) as client:

    # --- 1. Your local data. It does not leave this machine. ----------------
    records = pd.read_csv("site_data.csv")

    # --- 2. Reduce each column to its six combinable numbers. --------------
    #        The raw values are consumed here and never written.
    file_path = client.save_analytics({
        "age":           records["age"].to_numpy(),
        "tumour_volume": records["volume"].to_numpy(),
    })

    # --- 3. Upload. Blocks until the run is active. ------------------------
    file_id = client.file_upload(file_path)
    print(f"Uploaded: {file_id}")

    # --- 4. Wait for the aggregation notification. -------------------------
    #        If this times out, the round was probably refused because some
    #        field has no operation assigned. See "Where you actually see the
    #        refusal" above — the leaf is not told why.
    print("Waiting for aggregation...")
    aggregation_id = client.queue.get(block=True, timeout=600)

    # --- 5. Download the combined result. ----------------------------------
    aggregated_file = client.file_download(aggregation_id)
    print(f"Downloaded: {aggregated_file}")

    # --- 6. Derive the statistics you actually want. -----------------------
    #        The archive holds the combined six per column, under the same
    #        names they were sent under. Mean, variance and std are yours to
    #        compute — three lines, exact over the pooled data.
    with np.load(aggregated_file, allow_pickle=False) as archive:
        combined = {name: float(archive[name][0]) for name in archive.files}

    for column in ("age", "tumour_volume"):
        n     = combined[f"{column}_n"]
        total = combined[f"{column}_sum"]
        sumsq = combined[f"{column}_sumsq"]

        mean     = total / n
        variance = max(sumsq / n - mean ** 2, 0.0)   # clamp float noise at zero
        std      = variance ** 0.5

        print(
            f"{column}: federation n={n:.0f} "
            f"min={combined[f'{column}_min']:.4f} "
            f"max={combined[f'{column}_max']:.4f} "
            f"mean={mean:.4f} std={std:.4f}"
        )
        # Use these as your normalisation constants — the SAME numbers at
        # every site, which is the entire point of running this.

Two details worth noting. client.queue.get is the same call whichever transport you use, and it returns an aggregation_id, which is what file_download takes. And file_download returns the path it wrote to, under the client's output_dir — there is no fixed aggregated.npz filename.

A runnable reference client

There is a complete four-site federated analytics demo client, with synthetic data whose answers you can verify by hand, deliberately built so you can check the platform rather than trust it. Contact us for access.

Every field must have an operation — the rule, restated

It is the one thing that catches everybody, so it is worth stating once more without the surrounding detail:

Every field present in an upload needs an operation.
Fields your own code produced start blank.
Blank means the round is refused, and the refusal names the blanks.
Choosing an operation is not saving it. Save.
Every leaf must send the same field names, shapes and types, or the round dies too.
The refusal reaches the audit log and the settings page. It never reaches the leaf.

The flow​

1. Configure the branch​

2. Leaves upload, and the platform discovers the fields​

3. Assign an operation to every field — and save​

4. The next round aggregates​

5. Results return to every site​

The first round is refused, and that is correct​

Where you actually see the refusal​

Every leaf must send the same fields​

The operations​

mean is not a pooled mean​

pooled_variance does not fit the save_analytics bundle​

weighted_mean needs a weighting​

must_match disagreements pause the branch​

median_of_values is a median of medians​

What is not available​

Reserved names​

The client library side​

A complete analytics run​

Every field must have an operation — the rule, restated​

See also​