Skip to main content

FAQ & Troubleshooting

Answers to the questions we hear most often, and fixes for the errors you're most likely to meet. If your question isn't here, contact us.

Concepts & getting started​

What am I actually sending to BranchKey — is it my data?​

No. You send your model weights (numbers the model learned), never your raw data. Your images, scans, or records stay on your machine. See What are model weights?.

Do I need to be a Python expert?​

No, but the client library is Python, so you will write a small amount of it. The Concepts section assumes no prior Python-ML background and the quick start gives you a complete, copy-able loop. If you'd rather not write the loop at all, the Docker template runs it for you.

What is the difference between a Tree, a Branch, and a Leaf?​

A Tree is a project/collection, a Branch is one federation within it (one model, one aggregation algorithm), and a Leaf is a single participating client. See Platform Entities.

What is the difference between a "run" and an "aggregation"?​

A run is a whole training session on a branch, identified by a run number; you start, pause, and stop it. An aggregation is a single averaging round within a run — it fires each time enough leaves have uploaded their weights. One run contains many aggregations. See Runs and aggregations.

How many participants do I need before an aggregation happens?​

An aggregation fires when the number of uploaded files reaches the branch's files_per_aggregation setting. If it's set to 5, the platform waits for 5 leaves' uploads before averaging and notifying everyone. See Configuring a branch.

How do I choose the weighting value?​

Usually your number of local training samples, so participants with more data count proportionally more. weighting=1 gives everyone an equal say. Only relative sizes matter. See Extracting & reloading weights.

Credentials & access​

I lost my leaf's session token — how do I get it back?​

You can't — the session_token is shown only once, when the leaf is created. Delete the leaf and create a new one to get fresh credentials.

Which token format do I use where?​

The SDK handles this automatically. If you call the REST API directly:

  • Authentication API (login, entities): Authorization: Bearer <access_token>
  • API Gateway (upload, download, run, metrics) as a leaf: Authorization: Bearer <leaf_id>:<session_token> plus header LEAF: true

Login fails with the API​

The login route is POST /v2/users/login (served at /auth/v2/users/login) with body {"identifier": "<username-or-email>", "password": "..."}. There is no /v2/auth/login endpoint. See REST API Setup.

Uploading & downloading weights​

My upload fails with "run is blocked"​

The branch's run is not active. file_upload only succeeds while client.run_status == "start". Start the run from the dashboard, or construct the client with RunConfig(wait_for_run=True) so it waits for the run to start instead of raising:

from branchkey import Client, RunConfig
client = Client(credentials, run_config=RunConfig(wait_for_run=True, check_interval_s=15))

My leaves upload but no aggregation ever happens (the most common misstep)​

Almost always this is files_per_aggregation set higher than the number of leaves actually participating. An aggregation only fires once that many files have arrived. So if files_per_aggregation = 3 but you are running 2 leaves, the threshold is never reached and nothing happens — and in blocking mode each leaf uploads once and then waits forever for a result that can't come.

Fix: set files_per_aggregation to the number of leaves you want in each round — e.g. 2 for two participating leaves, 10 for ten. It must not exceed the number of leaves that actually upload, or the round deadlocks. This is the single most common configuration mistake. See files_per_aggregation.

validate_weight_shape error on upload​

The client validates your .npz before sending it. This error means the file is malformed — usually one of:

  • Missing weighting field — you must build the file with client.save_weights(...), which adds it. Don't hand-assemble the .npz.
  • No layer arrays found — your parameters list was empty. Extract the weights before saving (see Extracting & reloading weights).
  • A layer is not a NumPy array — convert framework tensors to NumPy first (PyTorch: param.detach().cpu().numpy()).

I get a shape-mismatch error​

Every leaf in a branch must upload the same model architecture, so each layer_N has the same shape for everyone. A mismatch means one participant is using a different network, a different layer order, or a different input size. Agree the architecture up front and make sure every leaf builds it identically. See Shapes, layers & the NPZ format.

The downloaded file is XML, not an NPZ / won't load​

You downloaded an error page from the storage layer instead of the file (a presigned-URL or integrity issue). Check that the aggregation_id you passed to file_download came from client.queue.get() and hasn't expired (aggregated results have an expiry set on the branch). Retry the download; if it persists, contact us with the aggregation_id.

My aggregated weights loaded but the model got worse​

Almost always a layer-order problem. save_weights numbers layers layer_0, layer_1, … in upload order; on reload you must visit your model's parameters in the same order, and sort layer keys by their integer suffix (not alphabetically, or layer_10 lands after layer_1):

layer_keys = sorted(npz.files, key=lambda k: int(k.split("_")[1]))

Federated analytics (per-field)​

My first round was refused: "no primitive is assigned to …"​

This is expected on a new per-field branch, and nothing is broken. Every field in an upload must have an operation assigned to it, and on the first round every field is new. The platform refuses rather than guessing, because a guessed operation returns a number that looks like an answer.

The message names the fields that need a decision, and which leaves sent them. Assign an operation to each one in the branch's aggregation settings, save, and then press start — the refusal also paused the run, so without that last step there is no next round and your leaves will sit idle. The next round is the one that aggregates. The refused round itself is lost; it is terminated, not held.

Do not stop the run in order to make the change. You do not need to, and stopping purges the branch's uploaded files. See Per-field aggregation.

My analytics leaf uploaded and then hung with no error​

client.queue.get(...) blocks until an aggregation notification arrives, and a refused round sends no notification — the leaf is told nothing at all, so it waits until your timeout and then reports only a timeout.

Look in the branch's audit log for an aggregator.schema_validation.failed entry, which carries the real reason, or open the branch's aggregation settings, which banners the fields still awaiting an operation. The usual causes, in order of likelihood: a field with no operation assigned; a field assigned but missing from one leaf's upload; two leaves disagreeing on a field's shape or type. See Where you actually see the refusal.

A per-field round fails with "the participants disagree about which fields they send"​

Every field with an operation assigned must be present in every leaf's upload. When one is not, the round is terminated with field_missing, and the message covers the whole round: it lists every field that failed to arrive everywhere and, for each, names both the leaves that did not send it and the leaves that did — because which side to change is the question you are actually asking.

This usually means one of your sites produces a field the others do not. Assigning it guarantees this failure on every round; leaving it unassigned guarantees the refusal above (unassigned_field) instead. Neither setting works, and the message says so: "The remedy is at the SITES, not in the configuration: make every participant send the same field set." Note that one extra column from save_analytics is six fields, not one, so expect six names.

If the field arrived from no leaf at all, you get a different message — "assigned field(s) are absent from every upload in this round" — and that one really is a configuration fix: remove the field from the map, or fix the sites to send it.

The same applies to shapes and types: the first upload in a round is the reference, and a leaf sending 4 where another sent 4.0 is an int64 against a float64 and is rejected. See Every leaf must send the same fields.

I set the operations for all my fields and they were gone when I came back​

They were not saved. Choosing an operation in the settings page puts it in a working copy; navigating away without pressing Save discards the lot. Assign, save, and reload to confirm. Rows that arrive pre-filled from the field name are also unsaved until you save them. See Assign an operation to every field — and save.

Can I get a median or a percentile across the federation?​

No. Not a true pooled one. Each site sends a fixed set of six summary numbers per column, and a pooled median or percentile genuinely cannot be recovered from them — this is arithmetic, not a missing feature. Count, sum, minimum, maximum, range, mean, variance and standard deviation are all available and all exact.

median_of_values exists but answers a different question: it is the median of the sites' values, not of the pooled data, and it will not tell you it has done so. If percentile-based clipping is part of your preprocessing, contact us before designing around it.

Transport & networking​

Should I use WebSocket or RabbitMQ?​

Use WebSocket (use_websocket=True) unless you have an existing reason not to. It needs only port 443. RabbitMQ is the legacy transport and additionally requires outbound port 5671.

WebSocket connection fails​

Confirm the client can reach the API host over HTTPS at all (curl -I https://app.branchkey.com). The WebSocket uses the same host and port (443) as the REST API. If a strict proxy blocks WebSocket upgrades, fall back to RabbitMQ with use_websocket=False.

RabbitMQ connection fails / "Connection refused"​

Ensure you're using the TLS port 5671 (not 5672) and ssl=True, and that outbound 5671 to rabbitmq.branchkey.com is open. Or switch to the WebSocket transport to avoid the extra port. See RabbitMQ transport.

SSL: CERTIFICATE_VERIFY_FAILED​

Your system doesn't trust BranchKey's (Let's Encrypt) certificate — rare on up-to-date systems. Update your CA bundle:

# Debian / Ubuntu
sudo apt-get update && sudo apt-get install ca-certificates

Never disable SSL verification in production to "fix" this.

Notifications never arrive​

The platform only emits aggregation notifications while a run is active and once files_per_aggregation uploads have accumulated. Check client.run_status == "start" and that enough leaves are uploading.

Docker template​

Permission denied on mounted files (UID mismatch)​

The container runs as a non-root user with UID 1000 (appuser) — a deliberate security choice. Docker maps users by their numeric UID, not their name, so a mounted host file is only readable inside the container if UID 1000 has permission on it. A frequent real-world case: your host files are owned by UID 1001 (or any UID ≠ 1000), so the container gets Permission denied even though the files look fine when you ls them on the host.

Quick fix — make the files group/other-readable (fine for non-secret config):

chmod 644 application.yaml    # readable by anyone, including UID 1000

Robust fix — share a group between the host owner and the container's UID. This keeps credentials off world-readable permissions while still letting the container read them: give the mounted files a shared group, grant that group access, and run the container as a member of it with --group-add.

# 1. Create a shared data group (pick an unused GID, e.g. 2000)
sudo groupadd -g 2000 docker-data

# 2. Give the mounted files/dir to that group, with group read (+ write if needed)
sudo chgrp -R docker-data /path/to/your/mounted/dir
sudo chmod -R g+rwX /path/to/your/mounted/dir

# 3. Run the container as a member of that group, so its UID 1000 inherits access
docker run --rm \
  --group-add 2000 \
  -v /path/to/your/mounted/dir/application.yaml:/app/application.yaml:ro \
  -v /path/to/your/mounted/dir/secret/leaf-1.json:/app/credentials.json:ro \
  registry.gitlab.com/branchkey/demo-applications/dockerised-implementation:latest

The principle: Docker matches by UID/GID numbers. Either make the files readable by UID 1000, or put the host file-owner and the container in a common group and grant that group access. For Docker Compose, add the group under the service's group_add: key. See SECURITY.md in the Docker template repository for the full treatment.

Configuration file not found / credentials.json not found​

Check your volume mounts. The container looks for /app/application.yaml and /app/credentials.json; mount your files to those paths:

docker run --rm \
  -v $(pwd)/application.yaml:/app/application.yaml:ro \
  -v $(pwd)/secret/leaf-1.json:/app/credentials.json:ro \
  registry.gitlab.com/branchkey/demo-applications/dockerised-implementation:latest

Docker build fails: ../client_application not found​

You're building the client image from a context that expects the client library source alongside it. Use the pre-built registry image (no build needed) as shown in the Docker template guide, or run from the repository root with the expected directory layout.

Still stuck?​

Email [email protected] with your leaf name, branch ID, and the full error output (set LOGGING_LEVEL: DEBUG for more detail). See Contact & Support.