Skip to content

v2: single-pass lexer, functional options, v1 parse parity - #425

Open
unknwon wants to merge 27 commits into
v2from
v2-next
Open

unknwon wants to merge 27 commits into
v2from
v2-next

Conversation

@unknwon

@unknwon unknwon commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

This is v2 of the library, at the new module path unknwon.dev/ini/v2. It has a new single-pass lexer, a functional options API, no package-level globals, and loading that takes linear time. With no options, v2 parses every input exactly as v1 does. A differential test suite checks this against the released gopkg.in/ini.v1 v1.67.3.

Speed

BenchmarkLoad10kKeys, 10,000 keys, Go 1.26, median of 3 runs:

Layout v1.67.3 v2 Allocations, v1 → v2
one section, inline comments 150 ms 6.0 ms 50.2k → 10.1k
1,000 sections of 10 keys 9.3 ms 4.5 ms 58.1k → 18.1k
mixed syntax (quotes, """, continuation, -, CRLF) 120 ms 6.6 ms 57.6k → 18.8k
  • Quadratic duplicate checks (perf: NewKey() and NewSection() use O(n) linear scan for existence check #377). Section.NewKey and File.NewSection scanned the whole key or section list on every insert to find duplicates. They now look the name up in the maps that already exist.
  • The parser is now a lexer. It reads the input once and takes names, values and comments as slices of it, so it no longer allocates for every line.
  • Leaner storage. A File keeps its sections in one ordered slice plus a name index, and a Section its keys the same way, replacing v1's parallel name lists and index slices. Shadow values are a []string on the key instead of a whole *Key each, and the second keysHash map is gone. This cut another 7–15% of memory per 10,000-key load.

Parity with v1

parity_test.go loads the same input twice into one File, with v1 and with v2. It compares everything both stored, reading v1's unexported fields by reflection and v2's directly, so it covers state the public API doesn't expose:

  • section names, comments and raw bodies
  • key values, comments, flags, shadows and nested values
  • otherwise the exact error message (v2's is unwrapped from ParseError)

It covers:

  • Seeds: every testdata/*.ini file, the examples from the docs site and GoDoc, and inputs taken from 155 parser-related issues and PRs (420 scanned).
  • FuzzParseParity: fuzzes the input together with 20 option bits. A final 3-minute run made 3.31 million comparisons with no difference, and earlier runs during development made about 11 million more.
  • TestParseParityRandom: 50,000 generated documents built from fragments that sit on feature boundaries, plus every seed under 500 random option sets.

I also planted behavior changes in the lexer on purpose; the parity tests caught every one that changed behavior.

The public API is compared the same way. api_parity_test.go calls v1 and v2 on the same files and values. It covers:

  • every Key accessor: the lenient, Valid and Strict lists, In, Range and Must*;
  • Write;
  • MapTo and StrictMapTo into a struct with every field kind v1 maps, and ReflectFrom of the result;
  • the struct API with targets of every shape: T, *T, **T, nil and slices.

Where v1 doesn't panic, v2 must return the same, except for the deliberate changes listed in the README. TestAPIParity, TestAPIParityShapes and FuzzAPIParity run it. It found that allowshadow slices un-escaped backslashes, which v1 never did, and that is fixed. Planting the earlier ReflectFrom(&ptr) regression back makes it fail.

The one deliberate parsing difference (go-ini/ini PR 210): v1 read a Python-style multiline value only as far as one read buffer (4 KiB by default). A longer value was cut mid-line and then failed with key-value delimiter not found. v2 reads the whole value. TestParsePythonMultilineLongerThanBuffer covers this.

API changes

  • Loading

    • Load(source, opts ...LoadOption) and LoadSources([]any{...}, opts...) take functional options; the LoadOptions struct is gone.
    • LooseLoad, InsensitiveLoad and ShadowLoad become the IgnoreMissingFiles(true), CaseInsensitive(true) and AllowShadows(true) options.
    • Every boolean option takes a bool, as in encoding/json/v2 and jsontext, so it can be set from a config flag (ini.AllowShadows(cfg.Shadows)) and a later option can turn off an earlier one.
    • Every v1 option keeps its name and meaning. DebugFunc becomes Debug(fn).
    • ReaderBufferSize is removed, because the lexer has no read buffer to size.
  • Writing

    • The package variables LineBreak, DefaultHeader, PrettySection, PrettyFormat, PrettyEqual and DefaultFormatLeft/Right, plus the KeyValueDelimiterOnWrite load option, become per-call WriteOptions of File.Write and File.SaveTo. One caller can no longer change another caller's output. The defaults produce the same output as v1.
    • WriteToIndent and SaveToIndent become the Indent option.
    • The pretty-printing switches are named for what they do: PrettyFormat → AlignDelimiters(bool), PrettySection → BlankLineBetweenSections(bool). PrettyEqual is gone; DelimiterSpaces(" ", " ") writes the same bytes. The names follow encoding/json/jsontext (SpaceAfterColon, Multiline). WriteTo stays, for io.WriterTo.
  • Default section: DefaultSection is now a constant. DefaultSectionName("default") renames the default section per file, for example for AWS config files (Allow for lower case [default] section #287).

  • Removed: the deprecated DEFAULT_SECTION, AllCapsUnderscore and Section.Haskey.

  • Clearer names:

    • SnackCase → SnakeCase.
    • SectionStrings/KeyStrings/KeysHash → SectionNames/KeyNames/KeyValues.
    • ValueWithShadows → Values, IsBooleanType → IsBoolean.
    • Insensitive* → CaseInsensitive*, Loose → IgnoreMissingFiles, ShortCircuit → FirstSourceOnly, UnparseableSections → RawSections, PreserveSurroundedQuote → PreserveSurroundingQuotes, KeyValueDelimiterOnWrite → WriteDelimiter.
    • The File.BlockMode field becomes the Locking(bool) load option (Locking(false) for v1's BlockMode = false), so locking can't be switched off while the file is in use.
  • No unlocked public fields:

    • Section.Comment and Key.Comment become Comment() and SetComment(c), which take the file's lock.
    • The File.NameMapper and File.ValueMapper fields become the NameMapping(m) and ValueMapping(fn) load options, so they can't change while other goroutines read values.
  • Must* only read. k.MustInt(3) returns 3 for a missing or invalid value but no longer stores it in the key. In v1 the default showed up in later reads and in Write output, so saving a config wrote out every default the program had asked for. Code that reads a default back later calls k.SetValue. Grafana's whole parsed config and Gogs' output are unchanged; Gitea relies on the write-back in the 45 places it marks INI-MUST-SIDE-EFFECT and needs SetValue there.

  • Errors: the ErrDelimiterNotFound/ErrEmptyKeyName structs and their IsErr* functions become sentinel errors checked with errors.Is. Messages are unchanged.

  • Fewer entry points: MapToWithMapper and StrictMapToWithMapper become a NameMapping(m) load option. The package-level ReflectFrom and ReflectFromWithMapper are dropped in favor of File.ReflectFrom.

  • Generic accessors: 34 typed Key methods become generic functions with the same behavior:

    • Ints, ValidInts, StrictInts and the same for the other types → ini.List[T], ini.ValidList[T] and ini.StrictList[T].
    • The In* methods → ini.In(k, def, candidates).
    • The Range* methods → ini.Range(k, def, lo, hi), which also takes time.Time.

    They parse with the same code as struct mapping, so every Scalar type works, including int8, time.Duration and named types. Internally, the eight list parsers and the exported Parser type that boxed every element are gone. String, Int, Bool, Duration, Strings, the Must* methods and the custom-layout time methods (TimesFormat, InTimeFormat, RangeTimeFormat and friends) stay on Key.

  • Import path: unknwon.dev/ini/v2; v1 stays at gopkg.in/ini.v1. v2 versions come from v2.x.y tags.

  • Requires Go 1.26, the oldest supported Go release; CI tests 1.26 and 1.27.

The README has a migration table.

New features (all opt-in; defaults unchanged)

Hardening

These fix v1 bugs outside the parser, so parse parity is unaffected.

  • Write output loads back to the same file. Each key line is checked with the lexer before it is written. v1's quoting is tried first, so ordinary output is byte-identical to v1 (the golden test passes unchanged). If v1's quoting reads back differently, other quotings are tried. If none works, Write returns an error instead of writing a corrupt file. Fixed along the way:

    • values ending in \, containing """, or starting with a delimiter
    • empty shadow values were dropped, losing systemd-style resets
    • comments of keys written on several lines were lost on reload
    • nested values were written after a non-empty value, and boolean keys that also had values were written wrongly
    • with Insensitive, an explicit [DEFAULT] section was written with the wrong section's keys
    • a raw body with no final newline ran into the next section header
    • Indent together with Python multiline or nested values indented lines that then joined the previous value

    FuzzWriteRoundTrip and TestWriteRoundTripRandom load, write and reload under random option sets and compare everything the File holds. The final 3-minute fuzz run made 1.38M executions with no failure. Write speed is unchanged: 8.8 ms vs 10.2 ms before for 10k keys, same allocations.

  • Reload replaces the contents (Reload retains keys removed from the source file #418, ShadowLoad和Reload问题 #273). Keys and sections removed from the sources are gone, and shadows no longer pile up. *Section and *Key pointers held across a reload see the new values. If loading fails, the file is unchanged.

  • Append reads only the new sources (Append() get file already closed when trying to append a second io.Reader #324). In-memory changes survive an Append. io.Reader sources are read once and kept, so a later Append or Reload no longer fails on an exhausted reader.

  • %(name)s expansion stops (Key.String() overflows the stack when two keys reference each other #409). Keys that reference each other overflowed the stack in Key.String, a fatal error that recover can't catch. Expansion is now limited by nesting depth, substitution count and total output size.

  • API misuse no longer panics or hangs. These calls now behave:

    • Key.Strings("")
    • negative section indexes
    • Section.Key("")
    • Section("") after deleting the default section with Insensitive
    • DeleteKey with Insensitive
    • ChildSections with non-unique names
  • Struct mapping returns errors instead of panicking. MapTo, StrictMapTo, Unmarshal, ReflectFrom and Marshal handle these types:

    • struct, named and pointer fields
    • nil targets
    • non-unique slices of pointers
    • recursive types

    Silent wrong values are fixed too: overflowing numbers, invalid slice elements in strict mode, zero and negative durations, *time.Time, sub-second times, unescaped slice elements, nil pointers, allowshadow comments, and uint8. The README lists every difference valid input can show.

  • Concurrent use is race-free. v1 locked only some writes, even with BlockMode on (the default). Reading a key while another goroutine called Reload, Append, SetValue or Write was a data race. Public reads now take the read lock and every mutation takes the write lock. The parser uses lock-free helpers on a file no one else can see yet, so loading is slightly faster. TestFile_Concurrency runs readers alongside every kind of writer under -race: the previous code had 40 races in it and the new code has none. A locked read costs about 18 ns more.

  • Unmarshal reads back what Marshal writes. A new FuzzStructMapping test loads arbitrary input, calls every accessor, maps it into a struct with every supported field kind, and checks the round trip. It found that allowshadow slices lost repeated and empty elements (an element containing the delimiter now makes Marshal return an error, as v1 split it), and that nonunique sections were merged when read back. Both are fixed.

Not in this PR

The v1 behaviors that users report as bugs are intentionally unchanged, so parity holds. They include # inside quotes (#393), even numbers of trailing backslashes (#347), comments at the end of the file (#362), and blank lines in Python multiline values (#205). Each one would be an opt-in option or a v2-default decision in a follow-up, and the parity suite shows exactly what each change affects.

Section.NewKey and File.NewSection scanned keyList and sectionList on
every insert to find duplicates, which made loading quadratic in the
number of keys per section (#377). Look them up in the maps that already
exist instead. Loading 10,000 keys in one section drops from about
143 ms to 9 ms.

inSlice has no callers left and is removed.
The lexer reads the whole input once and slices names, values and
comments out of it, so a typical key line allocates nothing. One switch
in lexer.next routes each line to a section, key, comment, raw body or
nested value; readValue applies v1's value rules in v1's order, which
its doc comment states as the contract.

Behavior is unchanged with one exception: a Python-style multiline
value is no longer cut at the reader buffer (4 KiB by default) and no
longer fails with "key-value delimiter not found" (#210). Multi-line
values are built in a strings.Builder instead of with += per line.

parity_test.go loads every input with the released gopkg.in/ini.v1 and
with this package and compares everything both stored, by reflection
over the fields the two versions share. Its seeds cover testdata, the
documented examples and the inputs from GitHub issues and pull
requests; FuzzParseParity fuzzes inputs and options together.

At 10,000 keys, Load allocates 4-5 times less than with the v1 parser.
Every section kept a second map that copied each value only to serve
KeysHash, and growing it was the largest cost left when loading. Each
Key now keeps that value in a hashValue field, updated where keysHash
was: on creation, on overwrite in NewKey, and in SetValue. It can't
reuse value, because Must* writes defaults into value but never
changed KeysHash; the new test pins that.

Loading 10,000 keys gets 25-33% faster.
Load and LoadSources take ...LoadOption instead of a LoadOptions struct,
and LooseLoad, InsensitiveLoad and ShadowLoad become the Loose,
Insensitive and AllowShadows options. Without options, parsing is
exactly v1's Load. Every v1 option keeps its name and meaning.
ReaderBufferSize is gone: the lexer has no read buffer to size.

The writer settings that were package variables (LineBreak,
DefaultHeader, PrettySection, PrettyFormat, PrettyEqual,
DefaultFormatLeft/Right) and KeyValueDelimiterOnWrite are per-call
WriteOptions of File.Write and File.SaveTo, so one caller can no longer
change another's output. WriteToIndent and SaveToIndent become the
Indent option; WriteTo stays for io.WriterTo.

New options that v1 had no way to express:
- DefaultSectionName renames the default section per file, e.g. AWS's
  [default] (#287), replacing the DefaultSection variable.
- CommentSymbols sets the comment characters (#350). The default "#;"
  keeps v1 parsing; the writer uses them for comments it adds.

Parse errors are wrapped in *ParseError with the 1-based line and
column where the failing line's content starts (#341). Its message is
"line L, column C: " followed by v1's message, and IsErrDelimiterNotFound
and IsErrEmptyKeyName see through it.

The parity tests map v1 options to v2 options and compare v1's error
messages with the error inside ParseError.
IsBooleanType reports whether a key was read as a boolean key (#279).

KeepEmptyShadowValues keeps empty values in ValueWithShadows and when
writing shadow keys (#366, #342). In systemd units an empty assignment
resets the values before it, which ValueWithShadows dropped since #316.
Without the option the result is unchanged.
They follow encoding/json's shape (#337): Unmarshal maps INI data to a
struct as MapTo does, and Marshal encodes a struct, or a pointer to one,
as ReflectFrom and Write do.
The v2 module is served from unknwon.dev/ini/v2 instead of
gopkg.in/ini.v2, so its versions come from v2.x.y tags rather than a
gopkg.in branch. v1 stays at gopkg.in/ini.v1.
CI doesn't upload coverage to Codecov, so the config has no effect.
Go supports its latest two releases, 1.26 and 1.27; the module now
requires the older of them instead of 1.22, which CI never tested.
- CommentSymbols compared the first byte of a line with the symbols'
  bytes, so a key whose first character shared a lead byte with a
  multi-byte symbol was read as a comment and lost. Symbols are now
  compared as characters everywhere: line starts, SpaceBeforeInlineComment,
  UnescapeValueCommentSymbols, and the comment prefix the writer adds,
  which was invalid UTF-8.
- With CommentSymbols(""), the writer put ";" before comments, which
  read back as keys. Writing a comment now returns an error instead.
- Marshal dropped all but one value of allowshadow fields and all but
  one section of nonunique fields; it now reflects into a file that
  allows both.
- KeepEmptyShadowValues still deduplicated repeated empty values, losing
  every reset after the first. Empty values it keeps are never treated as
  duplicates.
- A renamed default section (DefaultSectionName) was written as an empty
  header when it had no keys. It is now skipped like the DEFAULT section;
  with keys, its header is still written, as v1 did.
- Marshal and Unmarshal panicked on nil and on pointers to non-structs,
  as ReflectFrom and MapTo did. They now return an error, and a nil
  struct-pointer field writes no section.
Write now checks every key line with the lexer and picks a quoting that
reads back to the same name and value, starting with v1's choice so
ordinary output is unchanged. When no quoting works it returns an error
instead of writing a file that loads differently.

Also fixed in the writer:
- Empty shadow values are written, so systemd-style resets survive.
- A key written on several lines gets its comment before its last line,
  where the parser looks for it.
- Nested values follow an empty value line, and boolean keys that also
  have values are written as they were read.
- With Insensitive, an explicit [DEFAULT] section writes its own keys.
- A default section with a comment keeps its header.
- A raw body without a final line break no longer runs into the next
  section header.
- Indent returns an error where an indented line would join the value
  before it.

FuzzWriteRoundTrip and TestWriteRoundTripRandom load, write and reload
files under random option sets and compare everything the File holds.
Reload parses every source into a fresh file and moves the result in,
so keys and sections removed from a source are gone (#418) and shadows
don't pile up (#273). Sections and keys that still exist keep their
identity, so pointers callers hold see the new values. On error the
file is unchanged.

Append reads only the sources it adds, so changes made in memory stay
and earlier sources aren't parsed twice. Readers are read once and
kept, so a later Append or Reload doesn't fail on a used-up reader
(#324).
Two keys referencing each other overflowed the stack in Key.String,
a fatal error recover can't catch (#409). Expansion now tracks how
deep references nest (99, as before per level), how many substitutions
it makes (10,000) and how many bytes they produce (1 MiB), and stops
at any limit, returning the value expanded so far.
- Key.Strings with an empty delimiter returned only after running out
  of memory; it now returns the whole value.
- A negative index panicked in SectionWithIndex and corrupted the
  section list in DeleteSectionWithIndex, which now returns an error.
  SectionWithIndex("", n) no longer returns nil.
- With Insensitive, Section("") after deleting the default section
  created a new section on every call; DeleteKey ignored the case.
- Section.Key("") returns a key that isn't added instead of nil.
- ChildSections listed non-unique sections once per occurrence.
- A default-section key referencing itself returns its raw value.
MapTo, StrictMapTo, Unmarshal, ReflectFrom and Marshal panicked on
struct, named, pointer and nil types, nil targets, non-unique slices of
pointers, and recursive struct types, and silently read or wrote wrong
values in several cases. Values now go through one parser and one
formatter, time.Time is recognised by its type, and recursive types are
followed one level deep.

Also fixed: overflowing numbers wrapped, StrictMapTo ignored invalid
slice elements, zero and negative durations were dropped, *time.Time
was written as a section, sub-second times were truncated, slice
elements with the delimiter or a trailing backslash didn't read back,
nil pointers were written as empty keys, a nonunique or reflector field
stopped later fields from being written, allowshadow lost the comment
and dropped values, uint8 didn't map, embedded pointers were replaced,
and a nonunique field without sections failed the whole MapTo.

The README lists the changes valid input can show.
With BlockMode, reads of keys, values, sections and bodies, and all of
Write, took no lock, and Append parsed into the shared file, so reading
while another goroutine reloaded, appended, set values or reflected was
a data race. Public reads now take the read lock, mutations the write
lock, and the parser and internal code use lock-free helpers, so Load
and Reload parse a private file and Append holds the lock. Reload no
longer rewrites the fields readers use to find the lock. Loading is
slightly faster; a locked read costs one RLock pair.

FuzzStructMapping loads arbitrary input under random options, calls
every accessor, maps it, and checks that Unmarshal reads back what
Marshal writes. It found that allowshadow slices lost repeated, empty
and delimiter-containing elements, and that nonunique sections were
merged on reading. Marshal now keeps repeated shadow values and escapes
elements; Unmarshal enables the options allowshadow and nonunique
fields need, and splits shadow values the way Key.Strings does.
A key with nested values that later became a boolean key was written
with the boolean line first, and the parser can't add nested values to
a boolean key, so the output failed to load. The nested values now go
before the boolean line, after an empty value.
Renames: SnakeCase, SectionNames, KeyNames, KeyValues, Values,
IsBoolean, CaseInsensitive*, IgnoreMissingFiles, FirstSourceOnly,
RawSections and WriteDelimiter. Section.Haskey is removed. File.BlockMode
becomes the NoLocking load option, so locking can't be switched off
while the file is in use. The error structs and IsErr* functions become
sentinel errors for errors.Is, with the same messages. The *WithMapper
functions become a NameMapping load option, and the package-level
ReflectFrom functions are dropped in favor of File.ReflectFrom.
interface{} is any throughout.

The In*, Range*, typed list, Valid* and Strict* list methods of Key,
37 in all, become ini.In, ini.Range and ini.List[T]. They parse with
the struct mapping parser, so every Scalar type works, including sized
and named types, with overflow checks. The eight list parsers and the
exported Parser type that boxed every element are gone.

Internal helpers are named for what they do: decodeValue, decodeSlice,
decodeStruct, decodeSections, encodeValue, encodeSlice, encodeStruct,
newDataSource, loadSource and expandValue.
List returns the parsed elements with the zero value for one that
doesn't parse, as v1's Ints did, so it needs no error handling.
ValidList and StrictList bring back ValidInts and StrictInts, for any
Scalar type. TimesFormat, ValidTimesFormat, StrictTimesFormat,
InTimeFormat and RangeTimeFormat stay on Key, as a layout can't be a
type parameter.

Range also takes time.Time, replacing RangeTime, and In compares times
as instants with Equal, as v1 did, instead of ==, which failed for the
same instant in another zone.
Section and Key comments become Comment() and SetComment(), which take
the file's lock like every other accessor. NameMapper and ValueMapper
move from embedded File fields to the NameMapping and ValueMapping load
options, so they can't change while other goroutines read values.

A File keeps its sections in one ordered slice plus a name index, in
place of sectionList, sectionIndexes and the map. A Section keeps its
keys the same way, in place of keyList. Shadow values are a []string on
the key instead of a *Key each, which drops the isShadow flag.

KeyValues now reports the value each key holds, including defaults
stored by Must* methods; v1's KeysHash left those out.

Loading 10,000 keys uses 7-15% less memory and is faster in every case.
Gogs calls Section.ReflectFrom(&I18n) with I18n a package-level
*i18nConf. v1 dereferenced the extra pointer; v2 had started returning
"not a pointer to a struct".
PrettyFormat(bool) becomes AlignDelimiters(bool) and PrettySection(bool)
becomes BlankLineBetweenSections(bool), naming the output literally as
encoding/json/jsontext does with SpaceAfterColon and Multiline.
PrettyEqual() is removed: DelimiterSpaces(" ", " ") writes the same
bytes. PreserveSurroundedQuote() becomes PreserveSurroundingQuotes(),
fixing the grammar and keeping the verb jsontext uses in
PreserveRawStrings.
AllowShadows(), CaseInsensitive(), DefaultHeader() and the other 18
boolean options become AllowShadows(true) and so on, matching
AlignDelimiters(bool) and BlankLineBetweenSections(bool). An option can
now be set from a config flag, as in AllowShadows(cfg.Shadows), and a
later option can turn off an earlier one.

KeepEmptyShadowValues becomes PreserveEmptyShadowValues, matching
PreserveSurroundingQuotes.
Locking(false) reads without a double negative and maps directly from
v1's File.BlockMode = false.

Unmarshal adds the options its allowshadow and nonunique tags need after
the caller's, so AllowShadows(false) or a config flag that defaults to
false can't silently drop slice elements and sections.
api_parity_test.go calls v1 and v2 on the same files and values: every
Key accessor, the lenient, Valid and Strict lists, In, Range and Must*,
Write, MapTo and StrictMapTo into a struct with every field kind v1
maps, ReflectFrom of the mapped struct, and the struct API with targets
of every shape (T, *T, **T, nil, slices). Where v1 doesn't panic, v2 must
return the same, except for the deliberate changes in the README.
Planting the old ReflectFrom(&ptr) bug back makes it fail.

It found that allowshadow slices un-escaped backslashes, which v1 never
did: a value such as C:\\dir mapped to a different string. They are read
as in v1 again, each value split at the delimiter, and Marshal and
ReflectFrom return an error for an element that contains the delimiter,
which v1 wrote and read back as two elements.
v1's Must* methods wrote the default into the key, so later reads, MapTo
and Write saw it, and saving a config wrote out every default the program
had asked for. They now only read; call SetValue to store a value.

The API parity test resets v1's value after each Must* call. Without the
write-back it also exposed two deliberate MapTo differences that the old
chained defaults hid (zero durations, out-of-range list elements), now
listed with the others.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant