Repository navigation
Conversation
computeBuckets() rescanned the whole event list once per day, and cloned it
once per day on top of that:
const dayEvents = events.filter(...) // full scan
const lastBeforeDay = [...events] // full clone
.filter((e) => e.timestamp < dayStartSec) // full scan
.at(-1); // ...to read one element
With days up to 365 (the schema's max) that is three full traversals and one
full array allocation per day. A further array pair was allocated per day to
build `allStatuses` just to run `.every()` over it.
The events are sorted by (timestamp, id) at the top of the function and the
day windows advance monotonically, so one forward cursor yields each day's
slice and the carried-in status without any rescanning. The `allStatuses`
check is done in place with an early exit.
getBatchedStatusHistory() separately grouped events with
`events.filter(...)` and `lastKnownEvents.find(...)` per entity, i.e.
O(entities x events). That is now a single grouping pass into Maps.
Measured against current main (median of 7 runs), computeBuckets for one
entity:
days events main now speedup
30 500 0.43ms 0.12ms 3.7x
90 2000 4.42ms 0.34ms 13.2x
365 5000 36.81ms 1.31ms 28.2x
365 20000 237.76ms 2.35ms 101.3x
Output is unchanged: fuzzed against the current implementation over 20,000
generated workloads (shuffled and sorted input, same-second ties, timezone
offsets, prior status) with the clock frozen, with identical results.
The new tests sit alongside the existing same-second ordering test and cover
the properties the cursor has to preserve, notably that a day inherits the
status of the newest event from any earlier day.
aithal007
requested review from
miloschwartz and
oschwartz10612
as code owners
November 12, 2026 17:39
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
## Community Contribution License Agreement
By creating this pull request, I grant the project maintainers an unlimited,
perpetual license to use, modify, and redistribute these contributions under any terms they
choose, including both the AGPLv3 and the Fossorial Commercial license terms. I
represent that I have the right to grant this license for all contributed content.
AI Disclosure
An AI coding assistant helped write the rewrite, the tests and the benchmarks, and to fuzz the new implementation against the current one. The equivalence and benchmark numbers below come from running them locally.
Description
Replaces #3802 (rebased onto latest
main; the old PR's red check was the repo's ESLint setup, not this change).Summary
computeBuckets()inserver/lib/statusHistory.tsbuilds one bucket per day for the status-history endpoints. For each day it rescanned the entire event list, and cloned it:That's three full traversals and one full array allocation per day, with
daysgoing up to 365 (the schema's own max). The clone exists only to read its last element. A further pair of arrays was allocated per day to buildallStatusesjust to call.every()on it.Separately,
getBatchedStatusHistory()grouped events per entity withevents.filter(...)andlastKnownEvents.find(...)inside a loop over entities, which is O(entities x events).What changed
computeBucketsalready sorts its input by(timestamp, id)(the same-second ordering fix for #3994), and the day windows advance monotonically, so a single forward cursor yields each day's slice and the carried-in status with no rescanning and no per-day clone. TheallStatusescheck became an in-place scan with an early exit, and the batched grouping became a single pass intoMaps.No behaviour change: this is only how the same result is computed. The same-second ordering regression test from #3994 is kept alongside the new tests.
Impact
This is synchronous work on the request path, so it's time the process can't serve anything else.
computeBuckets, single entity (median of 7 runs, against currentmain):mainBatched grouping (300 events per entity, measured on the original base before the rebase; that grouping code is unchanged):
The 10-entity grouping row is a wash: building the
Maps costs about what the scan did at that size. The win starts around 50 entities. ThecomputeBucketsimprovement applies at every size.Verification
Output equivalence: 20,000 generated workloads, zero differences against the current
computeBucketsonmain. The fuzz coversdays(1-365), event counts (0-1000), shuffled and sorted input, many same-second and same-day-boundary timestamps,priorStatus, andtzOffsetMinutes(including-720and+840), comparing fullJSON.stringifyoutput with the clock frozen so both versions see the same "now". (An earlier run without a frozen clock showed 2 of 6,000 differences, both the final window ending one second later because the wall clock ticked between the two calls.)Tests (
server/lib/statusHistory.test.ts) pin the properties the cursor must preserve, most importantly that a day inherits the status of the newest event from any earlier day, which is the part a naive cursor gets wrong.How to test?
npx tsx server/lib/statusHistory.test.ts: all assertions pass.npx tsc --noEmit: clean (SQLite and Postgres drivers).npm test: the new test file passes.Separately, I've emailed a report to security@pangolin.net. Could someone from the team take a look when you get a chance?