Federated analytics, by hand

Everything else in this section is about federated learning: many sites train the same model, and BranchKey combines the numbers inside those models. Federated analytics is the other half. No model is trained. You ask a question about the data itself — how many records are there across the whole federation? what is the smallest value anywhere? what is the mean, the spread? — and BranchKey answers it over every site's data at once, while every record stays exactly where it is.

This page explains what that means, why the platform asks each site for six particular numbers rather than for the answer, and works a real federation through by hand so you can check the arithmetic yourself.

For the practical side — selecting the service, choosing an operation per field, and the step where you must save — see Per-field aggregation.

The use case: normalising before you train

Almost every training pipeline begins by putting the data on a common scale. You subtract a mean and divide by a standard deviation, or you clip to a range and rescale to 0–1. Those constants have to come from somewhere.

If each site computes them from its own data, each site normalises differently, and the model sees four different worlds. Site A's "1.0" is not site B's "1.0". The model spends its capacity learning the sites apart rather than learning the task.

The fix is to normalise every site against the same constants — the federation's constants. Which means you need the federation's mean, spread, minimum and maximum before the first round of training. That is a federated analytics job, and it is usually a short one:

1. Run an analytics job. Every site reports summary statistics; the platform combines them and returns the federation-wide figures to everyone.
2. Every site normalises its own data using those shared constants.
3. Now start federated learning, with all sites speaking the same units.

Step 1 is what this page is about, and it is short: a handful of rounds, not the hundreds a training job takes.

On a brand-new branch, expect it to take at least two. The first round is refused by design, because the platform has to see your uploads before it can show you your field names and ask what each one means. That refusal is part of the flow rather than a fault — the sequence, and the step where you must act, are on Per-field aggregation.

Analytics runs are not training runs

An analytics job uses the same entities and the same loop as a training job — leaves upload, BranchKey aggregates, everybody downloads — but there is no model, no weighting, and no convergence to wait for. Use a separate branch for it. A branch runs one aggregation service at a time, so pointing your training branch at analytics would break the leaves already on it.

What actually leaves the site

Nothing that resembles a record.

The client library takes your raw column — every patient's age, every measurement, the whole thing — and reduces it to exactly six numbers before it writes a single byte to disk:

What is sent	Symbol	Meaning
age_n	n	how many values there were
age_sum	Σx	those values added up
age_sumsq	Σx²	each value squared, then added up
age_min	min	the smallest
age_max	max	the largest
age_nan	nan	how many values were missing and left out

file_path = client.save_analytics({"age": patients["age"].to_numpy()})
file_id = client.file_upload(file_path)

The archive that goes over the wire holds age_n, age_sum, age_sumsq, age_min, age_max, age_nan and nothing else. Not one patient's age is in it.

This is worth being precise about, because it is a stronger claim than it looks. It is not an instruction to you to be careful with your data. It is a property of the library: save_analytics consumes the raw values and returns aggregates, and there is no argument to it that turns that off. Raw values cannot leave by mistake, because there is no code path along which they leave at all.

Missing values

Real columns have gaps, and what happens to them changes what the answer means.

By default (nan_policy="omit") a NaN is dropped before the reduction. The count n then counts only the values that were actually there, so the mean and variance stay exact — but exact over the values that were present, not over every record you think you sent. A column that is 80% missing yields a confident-looking federation mean computed from a fifth of the data.

That is what the sixth number is for. age_nan counts the values that were dropped, and it adds up across sites exactly as age_n does, so the result you get back carries both halves:

n + nan       = how many records the federation actually held
n / (n + nan) = how complete the column was

Before that field existed, the only trace of a dropped value was a smaller n — and a smaller n is indistinguishable from a smaller cohort. Now the returned numbers say which one it was, and you do not have to remember what you expected in order to notice.

The library also warns when it drops anything, naming the column and the count. That warning is a Python UserWarning, which in a notebook or a container log is easy to miss entirely; the nan counts come back with the result whether anyone saw the warning or not. So if a missing value means "this record is incomplete" and you are happy summarising what remains, the default is right, and the result will tell you how much was left out.

age_nan is always sent, even when it is zero

Even under nan_policy="raise", where it can only ever be zero, the field is still there. That is deliberate. Which fields a site sends has to be a property of the library, never of that site's data. If the bundle grew a field only when a column happened to have gaps, two sites in the same round would disagree about what they send, and the round would be refused whichever way you configured it: tell the platform to expect age_nan and the site with clean data is rejected for not sending it; leave it out and the site with gaps is rejected for sending something nobody asked for. A column of zeroes is the price of a bundle whose shape does not depend on the data.

If a missing value instead means the export is wrong, make it an error:

file_path = client.save_analytics(
    {"age": patients["age"].to_numpy()},
    nan_policy="raise",     # any NaN is an error, naming the column and the count
)

Either way a NaN never reaches Σx, and that matters more than it looks: NaN propagates through addition, so one missing value at one site would turn that column's federation sum, mean, variance and standard deviation into NaN for every participant, with nothing to say which site caused it. Infinities are rejected outright under both policies, for the same reason.

min and max are real records

n, Σx and Σx² describe no individual. min and max do: the federation's minimum age is some actual person's age. That is deliberate — it is exactly what makes range-based normalisation possible — but it should be a decision you have made, not something a data-protection review discovers for you. If your columns are such that an extreme value identifies someone, say so before you run the job.

Why those six, and not the answer?

The obvious design is for each site to compute its own mean and variance and send those. It does not work, and the reason is worth understanding, because it is the whole justification for the six numbers.

Sums combine across sites. Statistics do not.

Take two sites with three values each:

	values	mean	variance
Site A	[10, 10, 10]	10	0
Site B	[90, 90, 90]	90	0
Pooled	[10, 10, 10, 90, 90, 90]	50	1600

Each site is internally uniform, so each reports a variance of zero. Pool the six values and the variance is 1600. There is no way to get 1600 out of two zeros — not by averaging them, not by weighting them, not by any combination at all. The pooled variance depends on how far apart the site means are, and a per-site variance cannot express that, because from inside site A there is no site B.

Now do the same thing with the six numbers. Sum the counts, sum the sums, sum the sums of squares:

n   = 3 + 3       = 6
Σx  = 30 + 270    = 300
Σx² = 300 + 24300 = 24600

and the answers fall straight out:

mean     = Σx / n              = 300 / 6      = 50
variance = Σx²/n − mean²       = 4100 − 2500  = 1600

Exactly 1600. Not an estimate of it.

So the rule is: Σx makes the mean combinable, and Σx² makes the variance combinable. That is why the platform asks for sums instead of statistics. Everything you actually want — count, sum, minimum, maximum, range, mean, variance, standard deviation — comes back out of those six, exactly, over the pooled data.

How the six combine

Each of the six has one correct way to combine across sites, and it is not a choice anyone makes:

Field	Combined by	Why there is no alternative
_n	sum	Counts add.
_sum	sum	Sums add.
_sumsq	sum	Sums of squares add.
_min	minimum	The smallest value in the federation is the smallest of the per-site minimums.
_max	maximum	The largest is the largest of the per-site maximums.
_nan	sum	Counts add — dropped values are counted the same way present ones are.

Note the shape of that table: min is combined with min, not with mean. Averaging the sites' minimums would produce a number that describes nothing — not the federation's minimum, and not any site's. The arithmetic decides, not the operator.

A worked federation

Two sites, one column. This is a real run of BranchKey's federated-analytics demo client, with values chosen so you can check every step without trusting any code.

Each site holds an evenly spaced ramp of values, so its minimum is exactly the bottom of its range, its maximum exactly the top, and its mean exactly the midpoint:

	rows	score spans	site mean
Site 1	100	0.0 – 0.4	0.2
Site 2	200	0.2 – 0.6	0.4

Step 1 — each site reduces its column

save_analytics computes six numbers per site. Nothing else is written:

	score_n	score_sum	score_sumsq	score_min	score_max	score_nan
Site 1	100	20.000000	5.360269	0.0	0.4	0
Site 2	200	80.000000	34.693467	0.2	0.6	0

Site 1's score_sum of 20 is 100 values averaging 0.2. Site 2's 80 is 200 values averaging 0.4. Neither column has a gap in it, so both score_nan are zero — and both are still sent.

Step 2 — BranchKey combines them

Sum, sum, sum, minimum, maximum:

score_n     = 100 + 200            = 300
score_sum   = 20 + 80              = 100
score_sumsq = 5.360269 + 34.693467 = 40.053736
score_min   = min(0.0, 0.2)        = 0.0
score_max   = max(0.4, 0.6)        = 0.6
score_nan   = 0 + 0                = 0

These six combined values are what comes back down to every participant, under the same names they were sent under. The aggregated .npz holds score_n, score_sum, score_sumsq, score_min, score_max, score_nan.

Step 3 — you derive the statistics

The platform returns the six combined values; the mean, variance and standard deviation are one line of arithmetic each, and you do them wherever you are working:

count    = n                    = 300
sum      = Σx                   = 100
min      = 0.0
max      = 0.6
range    = max − min            = 0.6
records  = n + nan              = 300 + 0           = 300

mean     = Σx / n               = 100 / 300         = 0.333333
variance = Σx²/n − mean²        = 0.133512 − 0.111111 = 0.022401
std      = √variance            = √0.022401         = 0.149671

In Python, straight off the downloaded archive. The result arrives the same way a training result does — wait on the client's queue for an aggregation_id, then download it:

import numpy as np

aggregation_id  = client.queue.get(block=True, timeout=600)
aggregated_file = client.file_download(aggregation_id)

with np.load(aggregated_file, allow_pickle=False) as archive:
    n     = float(archive["score_n"][0])
    total = float(archive["score_sum"][0])
    sumsq = float(archive["score_sumsq"][0])

mean     = total / n
variance = sumsq / n - mean ** 2
std      = variance ** 0.5

print(n, mean, variance, std)
# 300.0 0.3333333333333333 0.022401344545398143 0.1496707872144666

The full script — credentials, upload, wait, download, derive — is A complete analytics run.

Check it by hand

The federation mean is exactly 100/300 = 0.3333…. Confirm that it is the right answer and not a coincidence: 100 values averaging 0.2 contribute 20, and 200 values averaging 0.4 contribute 80, so the mean over all 300 rows is (20 + 80)/300 = 1/3.

Notice what that is not. It is not either site's mean. And it is not the average of the two site means, which would be (0.2 + 0.4)/2 = 0.3. The federation mean is weighted by rows, so site 2 counts twice as much as site 1 because it holds twice as much data — which happens automatically, because Σx and n both add. There is no weight to configure and no way to get it wrong.

Why the aggregator returns six numbers rather than eight

It would be no extra work for the platform to divide score_sum by score_n and hand you the mean. It deliberately does not. The same machinery serves people who are not sending a bundle at all — someone combining their own preprocessing tool's output field by field — and the moment the aggregator starts recognising that score_sum and score_n are related, it stops being a general mechanism and starts being a special case. Three divisions in your own notebook is the cheaper side of that trade.

What you get, and what you cannot

Everything in this list is exact over the pooled data, not an approximation:

count · sum · min · max · range · mean · variance · std · completeness

("Pooled data" means every value that was actually summarised. If a column had missing values and you left them to be dropped, these describe the values that were present — and nan tells you how many were not, so n + nan is what the federation held — see Missing values.)

And the honest other half:

Medians and percentiles are not available

A pooled median cannot be recovered from any fixed set of summary numbers. Neither can a percentile. This is not a gap in the implementation — the information genuinely is not in the bundle, and no amount of combining will put it there.

This matters concretely if your preprocessing clips to percentile bounds. nnU-Net, for example, clips CT intensities to the 0.5th and 99.5th percentiles of the foreground; that step cannot be federated with the six-number bundle. Percentile-based preprocessing needs an approach this feature does not currently offer, so plan around it rather than discover it mid-project. Contact us if it is on your critical path.

There is an operation named median_of_values, and it is not the exception to that. It takes the median of the sites' values — a median of medians — which is a different quantity that happens to look like an answer. See the operations table before you reach for it.

Values your own code computed

Not every analytics job starts from a raw column. If you run a preprocessing planner at each site and it produces derived values of its own — a target voxel spacing, a channel count, a case count — there is no column left to reduce and the six-number bundle does not apply. You send those values as they are, with save_fields, and you choose how each one should be combined across sites.

file_path = client.save_fields({
    "target_spacing": np.array([1.0, 0.8, 0.8]),
    "num_channels": 4,
    "n_cases": 312,
})
file_id = client.file_upload(file_path)

That is the same mechanism. The six-number bundle is simply the case where the platform already knows how each field should be combined, because it produced the names itself.

save_fields sends what you give it

save_analytics guarantees that raw values never leave the site, because it does the reduction. save_fields makes no such guarantee: whatever you hand it is what is uploaded. If the values came out of a planner they are already aggregates and that is fine — but the responsibility for that is yours on this path, not the library's.

It is also the path where every site must agree: the same field names, the same shapes and the same types at every leaf, or the round is rejected. 4 at one site and 4.0 at another is a disagreement. See Every leaf must send the same fields.

Where to go next

• Per-field aggregation — the flow end to end, the operations you can choose, and the save step that releases the round
• Configuring a branch — selecting the service and its settings
• Federated averaging, by hand — the same treatment for the learning side