Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,20 @@ the importance of a given rule or guideline:
## Repository Structure

`inkless` is a fork of Apache Kafka that adds Diskless Topics (KIP-1150). Most
of the tree is upstream Kafka. Inkless-specific changes live in dedicated
locations, which are listed in [`inkless-sync/config/protected-patterns.txt`](inkless-sync/config/protected-patterns.txt).
When a task is Inkless-related, work in these first.
of the tree is upstream Kafka. Which paths are inkless-owned — and which are
upstream files carrying inkless edits — is recorded in
[`INKLESS_OWNERSHIP`](INKLESS_OWNERSHIP). When a task is Inkless-related, work in the
**OWNED** paths first (`@aiven/inkless`).

Everything else (`clients/`, `metadata/`, `streams/`, `connect/`, `raft/`, etc.)
is upstream Kafka. Follow upstream conventions when editing these and keep
Inkless changes minimal and isolated.
Note that some diskless logic is **interleaved** into upstream broker classes rather
than confined to an inkless package — e.g. `DisklessFetchOffsetRouter.scala` and
`InitDisklessLog*.scala` are net-new under `core/src/main/scala/kafka/server/`, and
`ReplicaManager.scala`/`KafkaConfig.scala`/`BrokerServer.scala` carry heavy inkless
edits (the dual-owner INTERLEAVED entries in the manifest). Don't assume all inkless
Scala lives under `io/aiven/inkless/`.

Everything else is upstream Kafka. Follow upstream conventions when editing those and
keep Inkless changes minimal and isolated.

## Documentation (Load On-Demand)

Expand Down
61 changes: 61 additions & 0 deletions INKLESS_OWNERSHIP
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Inkless Ownership Manifest
#
# Single source of truth for what is inkless-owned in this fork, and in what mode.
# Format is CODEOWNERS-like: <glob> <owner...> # <description>
#
# This is a DOCUMENTARY manifest, not a live .github/CODEOWNERS: @apache/kafka is
# not a GitHub team in this repo, so nothing here triggers review routing. It exists
# to answer, in one place, "who owns this path and how do we treat it on upstream sync":
#
# @apache/kafka upstream default. Follow upstream conventions; keep
# inkless edits out unless unavoidable.
# @aiven/inkless OWNED: net-new, inkless-only. Auto-resolved "ours" on
# upstream merge conflicts (see inkless-sync/main-sync.sh).
# @aiven/inkless @apache/kafka INTERLEAVED: inkless edits woven into an upstream file.
# NOT auto-resolved -- these need manual conflict review
# on every upstream sync.
#
# CODEOWNERS last-match-wins: the `*` default is first; specifics below override it.
# The sync tooling derives its auto-resolve set from the OWNED lines (single-owner
# @aiven/inkless) in this file -- INTERLEAVED (dual-owner) lines are excluded on purpose.
# Consumers: inkless-sync/main-sync.sh, AGENTS.md.

# ── Upstream default ─────────────────────────────────────────────────────────
* @apache/kafka

# ── OWNED — net-new, entirely inkless ────────────────────────────────────────
storage/inkless/** @aiven/inkless # core Java: control_plane, storage_backend, produce/consume, cache, metadata
core/src/main/scala/io/aiven/inkless/** @aiven/inkless # Scala broker integration in its own package (e.g. consolidation)
core/src/test/scala/io/aiven/inkless/** @aiven/inkless # tests for the above
core/src/main/scala/kafka/server/*Diskless*.scala @aiven/inkless # net-new diskless files in the upstream package (e.g. DisklessFetchOffsetRouter)
core/src/main/scala/kafka/server/InitDisklessLog*.scala @aiven/inkless # net-new: InitDisklessLogManager / State / BatchQueue
core/src/test/java/kafka/server/Inkless*.java @aiven/inkless
core/src/test/scala/kafka/server/metadata/Inkless*.scala @aiven/inkless
docs/inkless/** @aiven/inkless
config/inkless/** @aiven/inkless
docker/inkless/** @aiven/inkless
docker/examples/docker-compose-files/inkless/** @aiven/inkless
inkless-benchmarks/** @aiven/inkless
inkless-sync/** @aiven/inkless
tests/kafkatest/tests/inkless/** @aiven/inkless
tests/kafkatest/services/inkless/** @aiven/inkless
.github/workflows/inkless*.yml @aiven/inkless
Makefile @aiven/inkless # inkless-specific targets
INKLESS_OWNERSHIP @aiven/inkless # this manifest itself
AGENTS.md @aiven/inkless
Comment thread
jeqo marked this conversation as resolved.
Resolved
CLAUDE.md @aiven/inkless
.ai-agents/** @aiven/inkless
.claude/** @aiven/inkless
.cursor/** @aiven/inkless
.opencode/** @aiven/inkless
.github/copilot-instructions.md @aiven/inkless

# ── INTERLEAVED — upstream files carrying inkless edits (both own; manual sync) ──
core/src/main/scala/kafka/server/ReplicaManager.scala @aiven/inkless @apache/kafka # heaviest: diskless fetch/append routing
core/src/main/scala/kafka/server/ControllerApis.scala @aiven/inkless @apache/kafka # diskless topic create/alter/delete API paths
core/src/main/scala/kafka/server/KafkaConfig.scala @aiven/inkless @apache/kafka # inkless.* config definitions
core/src/main/scala/kafka/server/BrokerServer.scala @aiven/inkless @apache/kafka # SharedState wiring, reporter scheduling
core/src/main/scala/kafka/server/DelayedFetch.scala @aiven/inkless @apache/kafka # diskless fetch completion
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala @aiven/inkless @apache/kafka # dynamic inkless config reconfig
core/src/main/scala/kafka/server/ConfigHandler.scala @aiven/inkless @apache/kafka # topic-config propagation
core/src/main/scala/kafka/server/ControllerServer.scala @aiven/inkless @apache/kafka # controller-side diskless wiring
1 change: 1 addition & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,7 @@ if (repo != null) {
'core/src/main/scala/io/aiven/inkless/**',
'core/src/test/scala/io/aiven/inkless/**',
// Inkless coding agent configurations
'INKLESS_OWNERSHIP',
'AGENTS.md',
'CLAUDE.md',
'.ai-agents/**',
Expand Down
33 changes: 13 additions & 20 deletions inkless-sync/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,12 +192,13 @@ inkless-sync/
├── RELEASE-SYNC-ACTION-PLAN.md # Action plan for release syncs
├── sessions/ # Committed sync session logs
│ └── SESSION-2025-11-21.md # Example: main sync session
├── lib/
│ └── common.sh # Shared utility functions
└── config/
└── protected-patterns.txt # Files to protect during merge
└── lib/
└── common.sh # Shared utility functions
```

Merge protection is driven by the repo-root [`INKLESS_OWNERSHIP`](../INKLESS_OWNERSHIP)
manifest (see below), not a file under `config/`.

During sync, a `.inkless-sync/` working directory (gitignored) is created with:
```
.inkless-sync/
Expand All @@ -222,24 +223,16 @@ mv .inkless-sync/SESSION-YYYY-MM-DD.md inkless-sync/sessions/

## Configuration

### Protected Patterns (`config/protected-patterns.txt`)
### Protected patterns (from `INKLESS_OWNERSHIP`)

Files matching these patterns are auto-resolved with "ours" (inkless version) during conflicts.
Only truly inkless-owned files should be listed here; version files and core Kafka files
with inkless modifications are intentionally excluded so they fall into manual review:
The auto-resolve set is derived from the repo-root [`INKLESS_OWNERSHIP`](../INKLESS_OWNERSHIP)
manifest — there is no separate patterns file. The sync extracts the **OWNED** globs
(entries whose sole owner is `@aiven/inkless`) via the `owned_patterns()` helper in
[`lib/common.sh`](lib/common.sh) and auto-resolves those with "ours" during conflicts.

```
# Inkless-owned directories (safe to auto-resolve)
storage/inkless/**
docs/inkless/**
docker/inkless/**
config/inkless/**
tests/kafkatest/tests/inkless/**
.github/workflows/inkless*.yml

# Inkless test files
core/src/test/java/kafka/server/Inkless*.java
```
**INTERLEAVED** entries (dual-owner `@aiven/inkless @apache/kafka` — upstream files carrying
inkless edits, e.g. `ReplicaManager.scala`) are intentionally excluded, so they fall into
manual review. To change what is protected, edit `INKLESS_OWNERSHIP`.

### Adding Apache Remote

Expand Down
47 changes: 0 additions & 47 deletions inkless-sync/config/protected-patterns.txt

This file was deleted.

16 changes: 16 additions & 0 deletions inkless-sync/lib/common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,19 @@ count_commits() {
local to="$2"
git rev-list --count "$from..$to" 2>/dev/null
}

# Emit the OWNED auto-resolve globs from the INKLESS_OWNERSHIP manifest: entries whose
# sole owner is @aiven/inkless. Interleaved (dual-owner @aiven/inkless @apache/kafka)
# and the upstream default are excluded so they fall into manual conflict review.
# Parses CODEOWNERS fields: strips inline comments, then requires exactly one owner
# token equal to @aiven/inkless (so an inline comment mentioning @apache/kafka on an
# owned line does not misclassify it).
owned_patterns() {
local manifest="$1"
awk '
{ sub(/#.*/, "") } # drop inline comment
{ n = split($0, f); if (n == 0) next }
f[1] == "" { next } # blank after trimming
n == 2 && f[2] == "@aiven/inkless" { print f[1] }
' "$manifest"
}
63 changes: 34 additions & 29 deletions inkless-sync/main-sync.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/lib/common.sh"

CONFIG_DIR="${SCRIPT_DIR}/config"
PROTECTED_PATTERNS="${CONFIG_DIR}/protected-patterns.txt"
# Ownership manifest at the repo root (inkless-sync/ is one level down).
OWNERSHIP_MANIFEST="${SCRIPT_DIR}/../INKLESS_OWNERSHIP"
Comment thread
jeqo marked this conversation as resolved.
Resolved
SYNC_DIR=".inkless-sync"

# Use shared upstream remote detection from lib/common.sh
Expand Down Expand Up @@ -195,23 +195,25 @@ phase_prepare() {
# Identify protected files that might conflict
log_info "Identifying protected inkless files..."
PROTECTED_COUNT=0
if [[ -f "$PROTECTED_PATTERNS" ]]; then
local temp_patterns
temp_patterns=$(grep -v "^#" "$PROTECTED_PATTERNS" | grep -v "^$" || true)
if [[ -n "$temp_patterns" ]]; then
# Disable globbing so patterns like "storage/inkless/**" are not
# expanded by the shell before being passed to `find`.
set -f
while IFS= read -r pattern; do
local count
count=$(find . -path "./.git" -prune -o -path "./${pattern}" -print 2>/dev/null | wc -l | tr -d ' ')
PROTECTED_COUNT=$((PROTECTED_COUNT + count))
done <<< "$temp_patterns"
# Re-enable globbing after processing patterns.
set +f
fi
log_info "Protected files identified: ${PROTECTED_COUNT}"
if [[ ! -f "$OWNERSHIP_MANIFEST" ]]; then
error "Ownership manifest not found: $OWNERSHIP_MANIFEST (required to protect inkless files during sync)"
return 1
fi
local temp_patterns
temp_patterns=$(owned_patterns "$OWNERSHIP_MANIFEST")
if [[ -n "$temp_patterns" ]]; then
# Disable globbing so patterns like "storage/inkless/**" are not
# expanded by the shell before being passed to `find`.
set -f
while IFS= read -r pattern; do
local count
count=$(find . -path "./.git" -prune -o -path "./${pattern}" -print 2>/dev/null | wc -l | tr -d ' ')
PROTECTED_COUNT=$((PROTECTED_COUNT + count))
done <<< "$temp_patterns"
# Re-enable globbing after processing patterns.
set +f
fi
log_info "Protected files identified: ${PROTECTED_COUNT}"

# Determine sync branch name
SYNC_BRANCH="sync/upstream-$(date +%Y%m%d)"
Expand Down Expand Up @@ -243,18 +245,21 @@ phase_prepare() {
git diff "$MERGE_BASE"..HEAD --name-only > "${SYNC_DIR}/manifest-files.txt"
git diff "$MERGE_BASE"..HEAD --stat > "${SYNC_DIR}/manifest-stats.txt"

# Write protected patterns
if [[ -f "$PROTECTED_PATTERNS" ]]; then
grep -v "^#" "$PROTECTED_PATTERNS" | grep -v "^$" > "${SYNC_DIR}/protected-patterns-clean.txt" || true

> "${SYNC_DIR}/protected-files.txt"
# Disable globbing so patterns are passed to find, not expanded by shell
set -f
while IFS= read -r pattern; do
find . -path "./.git" -prune -o -path "./${pattern}" -print 2>/dev/null >> "${SYNC_DIR}/protected-files.txt" || true
done < "${SYNC_DIR}/protected-patterns-clean.txt"
set +f
# Derive the auto-resolve globs from the ownership manifest (see owned_patterns
# in lib/common.sh; INTERLEAVED dual-owner entries are excluded for manual review).
if [[ ! -f "$OWNERSHIP_MANIFEST" ]]; then
error "Ownership manifest not found: $OWNERSHIP_MANIFEST (required to derive auto-resolve patterns)"
return 1
fi
owned_patterns "$OWNERSHIP_MANIFEST" > "${SYNC_DIR}/protected-patterns-clean.txt"

> "${SYNC_DIR}/protected-files.txt"
# Disable globbing so patterns are passed to find, not expanded by shell
set -f
while IFS= read -r pattern; do
find . -path "./.git" -prune -o -path "./${pattern}" -print 2>/dev/null >> "${SYNC_DIR}/protected-files.txt" || true
done < "${SYNC_DIR}/protected-patterns-clean.txt"
set +f

# Save sync metadata
cat > "${SYNC_DIR}/sync-info.txt" << EOF
Expand Down
Loading