cashubtc/cdk

Fork 0

feat: optimize SQL balance calculation #1152

Merged

vnprc merged 6 commits from optimize_get_balance_2 into main

2025-10-06 08:29:57 +00:00

vnprc commented

2025-10-02 15:05:22 +00:00

(Migrated from github.com)

replace proof-fetching approach with SUM aggregation

add get_balance() method to Database trait
implement SQL SUM aggregation in cdk-sql-common
update total_balance() to use get_balance() instead of get_unspent_proofs()
redb impl maintains existing behavior

Description

Hashpool produces and consumes mint quotes at a rapid pace. I recently implemented a dashboard that queries for the wallet balance every three seconds. I noticed this message in the logs:

WARN 
  total_balance:get_unspent_proofs:get_proofs{mint_url=Some(MintUrl("http://localhost:3338")) unit=Some(Hash)}: cdk_sql_common::common: 
  [SLOW QUERY] Took 21 ms: SELECT
                  amount,
                  unit,
                  keyset_id,
                  secret,
                  c,
                  witness,
                  dleq_e,
                  dleq_s,
                  dleq_r,
                  y,
                  mint_url,
                  state,
                  spending_condition
              FROM proof

This query, which scans the entire proof table, was being called every time the wallet balance was requested. This PR speeds up this process by performing the sum operation in db logic instead of application logic.

Here are some log messages demonstrating the 100x speedup of the balance query:

2025-09-23T21:17:49.992802Z  INFO translator_sv2::lib: 🕐 Proof sweeper loop #8 starting
2025-09-23T21:17:49.994467Z  INFO total_balance: cdk_sql_common::wallet: Balance query completed in 0.18ms: 9561
2025-09-23T21:17:49.994486Z  INFO translator_sv2::lib: 💰 Current wallet balance: 9561 diff
2025-09-23T21:17:49.994516Z  INFO translator_sv2::lib: 📋 Found 16 quotes in Paid state with mintable amount
2025-09-23T21:17:50.055000Z  INFO translator_sv2::lib: Minted 452 ehash from 16 quotes
2025-09-23T21:17:50.055115Z  INFO total_balance: cdk_sql_common::wallet: Balance query completed in 0.10ms: 10013
2025-09-23T21:17:50.055122Z  INFO translator_sv2::lib: 💰 Wallet balance after sweep: 10013 diff

Notes to the reviewers

No test changes were needed because integration tests already cover balance calculations.

Suggested CHANGELOG Updates

CHANGED

ADDED

REMOVED

FIXED

Checklist

I followed the code style guidelines
I ran just final-check before committing

BTW final check is broken because it depends on a running postgres db, which leads just final-check to return false negative results for the following tests:

failures:
    test::add_and_find_proofs
    test::add_duplicate_proofs
    test::add_mint_quote
    test::add_mint_quote_only_once
    test::get_proofs_by_keyset_id
    test::kvstore_functionality
    test::read_mint_from_db_and_tx
    test::register_payments
    test::reject_duplicate_payments_diff_tx
    test::reject_duplicate_payments_same_tx
    test::reject_over_issue_different_tx
    test::reject_over_issue_same_tx
    test::reject_over_issue_with_payment
    test::reject_over_issue_with_payment_different_tx
    test::state_transition

replace proof-fetching approach with SUM aggregation - add get_balance() method to Database trait - implement SQL SUM aggregation in cdk-sql-common - update total_balance() to use get_balance() instead of get_unspent_proofs() - redb impl maintains existing behavior ### Description Hashpool produces and consumes mint quotes at a rapid pace. I recently implemented a dashboard that queries for the wallet balance every three seconds. I noticed this message in the logs: ``` WARN total_balance:get_unspent_proofs:get_proofs{mint_url=Some(MintUrl("http://localhost:3338")) unit=Some(Hash)}: cdk_sql_common::common: [SLOW QUERY] Took 21 ms: SELECT amount, unit, keyset_id, secret, c, witness, dleq_e, dleq_s, dleq_r, y, mint_url, state, spending_condition FROM proof ``` This query, which scans the entire proof table, was being called every time the wallet balance was requested. This PR speeds up this process by performing the sum operation in db logic instead of application logic. Here are some log messages demonstrating the 100x speedup of the balance query: ``` 2025-09-23T21:17:49.992802Z INFO translator_sv2::lib: 🕐 Proof sweeper loop #8 starting 2025-09-23T21:17:49.994467Z INFO total_balance: cdk_sql_common::wallet: Balance query completed in 0.18ms: 9561 2025-09-23T21:17:49.994486Z INFO translator_sv2::lib: 💰 Current wallet balance: 9561 diff 2025-09-23T21:17:49.994516Z INFO translator_sv2::lib: 📋 Found 16 quotes in Paid state with mintable amount 2025-09-23T21:17:50.055000Z INFO translator_sv2::lib: Minted 452 ehash from 16 quotes 2025-09-23T21:17:50.055115Z INFO total_balance: cdk_sql_common::wallet: Balance query completed in 0.10ms: 10013 2025-09-23T21:17:50.055122Z INFO translator_sv2::lib: 💰 Wallet balance after sweep: 10013 diff ``` ----- ### Notes to the reviewers No test changes were needed because integration tests already cover balance calculations. ----- ### Suggested [CHANGELOG](https://github.com/cashubtc/cdk/blob/main/CHANGELOG.md) Updates  #### CHANGED #### ADDED #### REMOVED #### FIXED ---- ### Checklist * [x] I followed the [code style guidelines](https://github.com/cashubtc/cdk/blob/main/CODE_STYLE.md) * [x] I ran `just final-check` before committing BTW final check is broken because it depends on a running postgres db, which leads `just final-check` to return false negative results for the following tests: ``` failures: test::add_and_find_proofs test::add_duplicate_proofs test::add_mint_quote test::add_mint_quote_only_once test::get_proofs_by_keyset_id test::kvstore_functionality test::read_mint_from_db_and_tx test::register_payments test::reject_duplicate_payments_diff_tx test::reject_duplicate_payments_same_tx test::reject_over_issue_different_tx test::reject_over_issue_same_tx test::reject_over_issue_with_payment test::reject_over_issue_with_payment_different_tx test::state_transition ```

crodas (Migrated from github.com) reviewed

2025-10-02 15:17:32 +00:00

thesimplekid (Migrated from github.com) approved these changes

2025-10-06 08:29:37 +00:00

github-actions[bot] commented

2025-10-27 23:12:55 +00:00

(Migrated from github.com)

Backport failed for v0.13.x, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin v0.13.x
git worktree add -d .worktree/backport-1152-to-v0.13.x origin/v0.13.x
cd .worktree/backport-1152-to-v0.13.x
git switch --create backport-1152-to-v0.13.x
git cherry-pick -x 1a493d61f821a044d1a82869a40f111413c61e8c

Backport failed for `v0.13.x`, because it was unable to cherry-pick the commit(s). Please cherry-pick the changes locally and resolve any conflicts. ```bash git fetch origin v0.13.x git worktree add -d .worktree/backport-1152-to-v0.13.x origin/v0.13.x cd .worktree/backport-1152-to-v0.13.x git switch --create backport-1152-to-v0.13.x git cherry-pick -x 1a493d61f821a044d1a82869a40f111413c61e8c ```

github-actions[bot] commented

2025-10-27 23:29:47 +00:00

(Migrated from github.com)

Backport failed for v0.13.x, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin v0.13.x
git worktree add -d .worktree/backport-1152-to-v0.13.x origin/v0.13.x
cd .worktree/backport-1152-to-v0.13.x
git switch --create backport-1152-to-v0.13.x
git cherry-pick -x 1a493d61f821a044d1a82869a40f111413c61e8c

No reviewers

thesimplekid

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

cashubtc/cdk!1152

No description provided.

Rows
Columns