Repository navigation
feat(segmentation): GPU-resident Cellpose-SAM eval + v4 API migration - #46
Conversation
…l API Cellpose 4 removed the `Cellpose` class and the `model_type`/`omni`/ `channels` arguments. Rewrite `cellpose_eval`/`cellpose_segment` to use `CellposeModel(gpu=True, pretrained_model="cpsam")` with `channel_axis`, and bump the cellpose extra to `>=4.0` (plus `torch`, needed by the SAM stack). Regenerate uv.lock accordingly (cellpose 4.1.1 + torch/torchvision). The wrapper's import-error test is made deterministic by monkeypatching `_CELLPOSE_AVAILABLE`, so it exercises the guard regardless of whether cellpose is installed in the running environment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_resident) Add a GPU-resident inference path for Cellpose-SAM that uploads the input image to the GPU exactly once and returns only the final integer mask to the host. The cellpose network and its torch dynamics kernels are reused unchanged; the numpy/scipy/scikit-image/cv2/fastremap glue is replaced with cubic's device-agnostic wrappers (cubic.scipy -> cupyx, cubic.skimage -> cuCIM), so the flow field never leaves the device. - cuda.py: `to_torch` (cupy<->torch zero-copy via the CUDA Array Interface) and `ascupy_f32` (guards the silent bfloat16 -> |V2 corruption, since cupy has no bf16); both lazy-import torch so base cubic still imports. - cellpose_sam_gpu.py: device-aware tiling/normalization/resize + forward (run_net/run_3D), and the `segment_cpsam_resident` orchestrator. GPU-only by contract (clear precondition); `download=False` keeps dP/cellprob as CuPy, `download=True` returns the stock-eval tuple. Lists/5D inputs are processed per image. - cellpose_dynamics.py: dynamics, flow QC, hole-fill/size-filter and 3D stitching, all on the GPU. Validated against stock CellposeModel.eval: average precision = 1.0 at IoU 0.5/0.7/0.9 for 2D, 3D and stitch on synthetic data; residency test confirms a single device->host transfer. tile-norm/sharpen normalization sub-paths raise NotImplementedError for now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reviewer's GuideAdds a GPU-resident Cellpose-SAM segmentation path that keeps all intermediate tensors on the GPU, introduces CuPy↔torch zero-copy bridge helpers, and migrates existing Cellpose wrappers/tests and dependencies to the v4 SAM API while preserving a CPU-safe fallback path. Sequence diagram for GPU-resident Cellpose-SAM segmentation pathsequenceDiagram
actor User
participant segment_cpsam_resident
participant CUDAManager
participant CellposeModel
participant _run_net_gpu
participant compute_masks_resident
User->>segment_cpsam_resident: call with model, x (image)
segment_cpsam_resident->>CUDAManager: get_cp
CUDAManager-->>segment_cpsam_resident: CuPy handle
segment_cpsam_resident->>CellposeModel: access net, backbone, device
segment_cpsam_resident->>segment_cpsam_resident: convert_image
segment_cpsam_resident->>segment_cpsam_resident: normalize_img_gpu
segment_cpsam_resident->>segment_cpsam_resident: ascupy (host→device upload once)
segment_cpsam_resident->>_run_net_gpu: call with model.net, x
_run_net_gpu->>_run_net_gpu: run_net_gpu / run_3D_gpu
_run_net_gpu-->>segment_cpsam_resident: dP (CuPy), cellprob (CuPy), styles (CuPy)
segment_cpsam_resident->>compute_masks_resident: call with dP, cellprob
compute_masks_resident-->>segment_cpsam_resident: masks (CuPy)
alt download=False
segment_cpsam_resident->>segment_cpsam_resident: asnumpy(masks) (device→host once)
segment_cpsam_resident-->>User: masks (NumPy), [None, dP, cellprob], styles (CuPy)
else download=True
segment_cpsam_resident->>segment_cpsam_resident: asnumpy(dP, cellprob, styles)
segment_cpsam_resident-->>User: masks, [dx_to_circ(dP), dP, cellprob], styles (NumPy)
end
Flow diagram for new cellpose_eval / cellpose_segment v4 SAM APIflowchart LR
A[cellpose_segment] --> B[downscale_and_filter]
B --> C[cellpose_eval]
C --> D["CellposeModel(gpu=True, pretrained_model='cpsam')"]
D --> E["model.eval(image, channel_axis, diameter, do_3D, batch_size, flow_threshold, cellprob_threshold)"]
E --> F[masks]
F --> G[remove_touching_objects]
G --> H[clear_xy_borders]
H --> I[final masks]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
segment_cpsam_resident, the list/5D-image handling recursively calls itself for each element while passing through many parameters; consider extracting a small internal helper for the per-image path to reduce duplication and risk of future parameter drift. - The GPU precondition checks are split between
_require_cellpose,_check_gpu_precondition, and_cp; consolidating or clearly documenting their distinct responsibilities would make the error paths and invariants easier to reason about. - Some of the longer functions in
cellpose_sam_gpu.pyandcellpose_dynamics.py(e.g._run_net_gpu,_resize_and_compute_masks) are quite dense; factoring out small, named helpers for repeated patterns (e.g. 2D vs 3D resize branches, normalization/parameter resolution) would improve readability and maintainability.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `segment_cpsam_resident`, the list/5D-image handling recursively calls itself for each element while passing through many parameters; consider extracting a small internal helper for the per-image path to reduce duplication and risk of future parameter drift.
- The GPU precondition checks are split between `_require_cellpose`, `_check_gpu_precondition`, and `_cp`; consolidating or clearly documenting their distinct responsibilities would make the error paths and invariants easier to reason about.
- Some of the longer functions in `cellpose_sam_gpu.py` and `cellpose_dynamics.py` (e.g. `_run_net_gpu`, `_resize_and_compute_masks`) are quite dense; factoring out small, named helpers for repeated patterns (e.g. 2D vs 3D resize branches, normalization/parameter resolution) would improve readability and maintainability.
## Individual Comments
### Comment 1
<location path="cubic/cuda.py" line_range="173-182" />
<code_context>
+def to_torch(array: Any, device: Any = None, dtype: Any = None) -> Any:
</code_context>
<issue_to_address>
**issue:** Avoid defaulting CuPy inputs to hard-coded 'cuda' device in to_torch.
In the CuPy→torch path, `device` currently falls back to the hard-coded string `'cuda'` when `None`. This can break when users run on non-default CUDA devices (e.g. `cuda:1`) or alternative backends (MPS/ROCm), and may diverge from `model.device`. Since `_forward_gpu` passes `device=net.device` but other callers might not, this can cause subtle cross-device issues. Prefer leaving `device=None` and letting `torch.as_tensor` infer, or derive the default from `array.device`/`model.device` instead of hard-coding `'cuda'`.
</issue_to_address>
### Comment 2
<location path="cubic/cuda.py" line_range="218" />
<code_context>
+ return tensor.to(dtype) if dtype is not None else tensor
+
+
+def ascupy_f32(array: Any) -> object:
+ """Move (or keep) an array on the GPU as a CuPy array, fp32-safe at the bridge.
+
+ Identical to :func:`ascupy` except that torch tensors are first cast to
+ ``float32`` **on-device**. This guards the silent-corruption failure mode
+ where ``cupy.asarray`` on a torch ``bfloat16`` tensor yields an opaque
+ ``|V2`` array (CuPy has no ``bfloat16``). Use at the network-output
+ boundary, where activations may be ``bfloat16``.
+
+ Non-torch inputs are forwarded to :func:`ascupy` unchanged.
+ """
+ if _is_torch_tensor(array):
+ import torch # lazy
+
+ array = array.to(torch.float32)
+ return ascupy(array)
</code_context>
<issue_to_address>
**suggestion:** Consider matching the return type annotation of ascupy for ascupy_f32.
`ascupy_f32` currently returns `object`, while `ascupy` returns `Any`. Since this function is just a fp32-enforcing wrapper around `ascupy`, using `Any` here too would keep the signatures aligned and prevent downstream code from inferring an unnecessarily generic `object` type.
```suggestion
def ascupy_f32(array: Any) -> Any:
```
</issue_to_address>
### Comment 3
<location path="cubic/segmentation/cellpose_sam_gpu.py" line_range="353-362" />
<code_context>
+def run_3D_gpu(
</code_context>
<issue_to_address>
**suggestion:** Check that 3D style aggregation matches Cellpose expectations.
The 3D path currently initializes `style = None` and then replaces it with the `style` from only the last orthogonal sweep. In stock Cellpose, styles from multiple views/volumes are aggregated (e.g., averaged per-image). If callers expect `styles` to reflect all sweeps/planes, this change may subtly alter behavior. Please consider aggregating styles across the three permutations (e.g., taking the mean) instead of just using the final sweep’s style.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
`_remove_bad_flow_masks` recomputes flows from the predicted masks on the GPU. For very large masks that recomputation can exceed free VRAM, so mirror cellpose's `dynamics.remove_bad_flow_masks` guard: when `masks.size > 1e8` and the estimate (size*32) exceeds free GPU memory, fall back to cellpose's CPU flow-QC (warn, route through asnumpy) instead of OOM-ing. This is the one place a derived array may leave the GPU, and only to avoid an out-of-memory error on a huge image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codify the 3D (do_3D=True) and stitch (stitch_threshold>0) parity that was previously only checked with throwaway scripts, so regressions in run_3D_gpu / _stitch3D / the 3D mask path are caught. Both assert average precision >= 0.95 @ IoU 0.5 vs stock CellposeModel.eval on small synthetic fixtures and run in a few seconds on a GPU. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a GPU-resident Cellpose-SAM (cellpose v4) evaluation/segmentation path to reduce CPU↔GPU transfers, while migrating existing Cellpose integration from the removed v3 API to the v4 CellposeModel API and expanding test coverage around optional dependencies and GPU-only contracts.
Changes:
- Migrate
cubic.segmentation.cellposewrappers to Cellpose v4 (CellposeModel,pretrained_model,channel_axis, updated thresholds/batch args). - Introduce GPU-resident inference + GPU-resident dynamics/postprocessing modules (
cellpose_sam_gpu.py,cellpose_dynamics.py) and export the new orchestrator. - Add/adjust tests to be deterministic in optional-dep environments and to validate GPU residency and parity vs stock
CellposeModel.eval; updatecellposeoptional deps.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/segmentation/test_cellpose.py | Makes import-error test deterministic via monkeypatch. |
| tests/segmentation/test_cellpose_sam_gpu.py | Adds GPU-resident Cellpose-SAM API/behavior/residency/parity tests with clean skips when deps/GPU are missing. |
| pyproject.toml | Updates cellpose extra to require cellpose v4 and torch. |
| cubic/segmentation/cellpose.py | Migrates wrapper functions to v4 CellposeModel API and new eval parameters. |
| cubic/segmentation/cellpose_sam_gpu.py | Adds GPU-resident tiling/normalize/resize + forward path and the segment_cpsam_resident orchestrator. |
| cubic/segmentation/cellpose_dynamics.py | Adds GPU-resident dynamics, flow QC, hole fill/size filtering, and stitching. |
| cubic/segmentation/init.py | Exposes cellpose_eval and segment_cpsam_resident from the public segmentation package. |
| cubic/cuda.py | Adds CuPy↔torch zero-copy bridge (to_torch) and bf16-safe CuPy casting helper (ascupy_f32). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Drop the `_resident` suffix from the public entry point (and the internal `compute_masks_resident` -> `compute_masks`); the docstrings already explain the GPU-resident behavior. `segment_cpsam` is cubic's recommended default Cellpose-SAM segmentation method going forward. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…32 as Any
- to_torch: when no device is given for a CuPy input, derive it from the
array's own device (`cuda:{array.device.id}`) instead of a hard-coded
"cuda", so arrays on cuda:1 (etc.) are not silently moved to cuda:0.
- ascupy_f32: return `Any` to match `ascupy` and avoid an unnecessarily
generic `object` inference downstream.
Addresses Sourcery review on PR #46.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cellpose_eval` is exported from `cubic.segmentation` but, unlike `cellpose_segment`, did not check `_CELLPOSE_AVAILABLE`. In a no-cellpose environment it raised a `NameError` for `models` instead of a clear `ImportError`. Add the same guard for a deterministic error message. Addresses Copilot review on PR #46. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- _fill_holes_and_size_filter: assign through the slice view with `masks[slc] = np.where(filled, i+1, masks[slc])` instead of chained indexing `masks[slc][filled] = ...`, which is fragile; add a regression test that an interior hole is actually filled. - _get_masks: cast `pt` to int64 for `torch.sparse_coo_tensor`, which requires LongTensor indices (`pt` arrives as int32 from `.int()`). Addresses Copilot review on PR #46. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n_net_gpu CPSAM emits zero style vectors and stock `CellposeModel.eval` likewise returns zeros (its style aggregation is disabled), so the per-tile `stylea` buffer was computed and never used. Drop it; `styles` stays the allocated zeros, matching stock behavior. Addresses Copilot review on PR #46. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…PU path The GPU-resident Cellpose-SAM path previously raised NotImplementedError for the non-default normalizations (tile_norm_blocksize>0 and sharpen_radius/smooth_radius>0), since they rely on cv2.resize, scipy.ndimage, torch.from_numpy and np.array(list-of-arrays) on the host. Port both leaves device-agnostically: - _smooth_sharpen_img_gpu mirrors transforms.smooth_sharpen_img but bridges to torch via to_torch (zero-copy from CuPy), reuses cellpose's pure-torch gaussian_kernel, keeps the FFT body on torch, and writes into a fresh tensor so the input buffer is never mutated. - _normalize99_tile_gpu mirrors transforms.normalize99_tile with xp.zeros buffers, xp.stack for the per-z lists (np.array does not dispatch on a list of CuPy arrays), cubic.scipy.ndimage.gaussian_filter1d, and resize_gpu for the percentile maps (channel-preserving, so the cv2 singleton re-add is dropped). Fixes the upstream norm3D=False/Lz>1 typo (x99 assigned from x01_rsz) that forces x99==x01 and always raises ZeroDivisionError. normalize_img_gpu is restructured to apply sharpen first, then tile-norm or the standard percentile path (still delegated to cellpose.transforms.normalize_img, which dispatches on cupy unchanged); channel-last (axis=-1) is assumed. The orchestrator passes device=model.device through to both normalize calls. Add GPU parity tests vs stock CellposeModel.eval (tile-norm 2D, tile-norm 3D with norm3D, sharpen 2D; all AP>=0.95 @ IoU 0.5) and extend the CPU building-block test to exercise both leaves on NumPy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Adds a GPU-resident Cellpose-SAM inference path to cubic —
cubic.segmentation.segment_cpsam— so that, on a CUDA box, the input image is uploaded to the GPU exactly once and only the final integer mask returns to the host. Today stockCellposeModel.evalround-trips the image across PCIe several times (preprocessing on CPU, per-batch upload/download in the network forward, re-upload for dynamics, re-upload for flow QC). The new path keeps the flow field on the device end-to-end and is intended as cubic's default Cellpose-SAM segmentation method.cellpose itself is not modified: its network and torch dynamics kernels are reused as-is; only the numpy/scipy/scikit-image/cv2/fastremap "glue" is replaced with cubic's device-agnostic wrappers (
cubic.scipy→ cupyx,cubic.skimage→ cuCIM), bridging cupy↔torch zero-copy via the CUDA Array Interface.Changes
v4 (SAM) dependency + wrapper migration
cellpose>=4.0(+torch) in thecellposeextra; regenerateuv.lock(cellpose 4.1.1 + torch/torchvision).cellpose_eval/cellpose_segmentoff the removed v3 API (Cellpose,model_type/omni/channels) toCellposeModel(pretrained_model="cpsam")+channel_axis.GPU-resident eval
cuda.py:to_torch(cupy↔torch zero-copy) andascupy_f32(guards the silent bf16→|V2corruption — cupy has no bfloat16); both lazy-import torch so base cubic still imports without it.cellpose_sam_gpu.py: device-aware tiling/normalization/resize + forward, and thesegment_cpsamorchestrator — GPU-only by contract (clear precondition).download=False(default) keepsdP/cellprobas CuPy;download=Truereturns the stock-evaltuple. Lists / 5D inputs are processed per image.cellpose_dynamics.py: dynamics, flow QC, hole-fill/size-filter, and 3D stitching, all on the GPU. The flow-QC step falls back to cellpose's CPU path for very large masks (masks.size > 1e8and insufficient free VRAM) to avoid OOM.Validation
CellposeModel.eval: average precision = 1.0 @ IoU 0.5/0.7/0.9 for 2D, 3D, and stitch on synthetic data (100% foreground agreement).download=False;dP/cellprobstay CuPy.Notes / limitations
tile_norm/sharpennormalization sub-paths raiseNotImplementedErrorfor now (non-default; standard percentile normalization is supported).order=0vs cv2INTER_NEARESTdiffer at half-pixel ties; validated by IoU/AP).uv.lockdiff also folds in pre-existingexamples-extra (jupyter) drift that was already uncommitted onmain.🤖 Generated with Claude Code