KV3 text grammar and its traps

Question

What exactly does the decompiler's KV3 text output contain, and what does a parser have to handle that a line-based grep or regex gets wrong?

Summary

• The grammar is small: maps, arrays, scalars, strings, binary blobs, and four type-prefixed forms (subclass:, resource_name:, soundevent:, panorama:). tools/kv3.py parses all 99 decompiled documents without error.
• Booleans are not consistently typed. m_bDisabled alone appears as the bool true, the string "true", and the integer 1 in one file. This produced a wrong published figure — see note 0002.
• Keys are not identifiers. Real top-level keys include Impact.Asphalt, Can Heal, @all_heroes and weapon_alternative_rmb+lmb_activate. A ^\t[A-Za-z_][A-Za-z0-9_]* regex silently drops 162 of 1,989 entries.
• Keys can contain escapes. The decompiler writes "You\'re Welcome", so a parsed key does not match the raw text literally.
• Values are null in 35 places, and one document uses triple-quoted multi-line strings.

Findings

Grammar

document  := header? map
header    := '<!-- kv3 encoding:... format:... -->'
map       := '{' (key '=' value)* '}'
key       := bare | '"' escaped '"'
value     := map | array | blob | string | number | 'true' | 'false' | 'null' | prefixed
array     := '[' (value ','?)* ']'
blob      := '#[' hexbyte* ']'
prefixed  := identifier ':' value

Commas are separators and a trailing one is allowed, so a parser can treat them as whitespace. Blocks open on the line after the =, which is why the key line reads key = with a trailing space.

Type prefixes, by frequency in abilities.vdata + heroes.vdata

tools/kv3.py unwraps these to the inner value. This discards the type tag — fine for reading, not safe if the parser is ever used to write KV3 back.

Booleans have three spellings

Within scripts/abilities.vdata, m_bDisabled is written as:

Two failure modes follow, and note 0002 originally hit the first:

1. Searching for m_bDisabled = true misses the 11 quoted and integer spellings.
2. Plain truthiness then treats the string "false" as true.

kv3.as_bool handles both. Anything reading a boolean out of this data should use it.

Keys are not identifiers

Top-level keys that a [A-Za-z_][A-Za-z0-9_]* pattern rejects:

162 entries in total, 8% of the corpus. Non-identifier keys are written quoted; identifier keys are written bare, so a matcher must accept both.

Escapes appear in keys, not just values

	"You\'re Welcome" = 
	"I\'ll Clear Troopers" = 

The apostrophe does not require escaping, but the decompiler escapes it anyway. Any code that locates a parsed key back in the raw text must tolerate a backslash before any character — tools/vpkdb.py's split_kv3_entries does.

Other forms

• null — 35 occurrences.
• Triple-quoted strings — only scripts/tools/game_asset_tags.vdata, and its content contains an escaped \".
• Empty containers {} and [] occur.
• Key order is meaningful for display and is preserved by the parser.

Binary blobs

One document uses a binary blob literal — a #[ ... ] block of whitespace-separated hex:

	permutations = 
	#[
		00 00 51 00 A2 00 1B 00 6C 00 BD 00 36 00 87 00 D8 00 09 00 5A 00 AB 00
		...
	]

core/textures/dev/scrambled_halton.vdata is the only occurrence in build 6679, at 386,660 bytes. A parser that does not know the form fails on the whole document rather than one value. kv3.parse returns bytes; build_content.py serialises it as {"__kv3_blob_bytes": n} — a labelled summary, not the data, so nothing pretends the JSON is complete.

Reproduce

python tools/decompile.py --fetch        # populates out/<mount>/ -- all 99 vdata_c
python -m unittest tools.test_kv3 -v     # 44 tests, incl. the whole corpus

Note the decompiler must cover every mount with no path filter. Restricting it to citadel/scripts/ misses 22 of the 99 files — the 8 citadel ones under stats/, soundstacks/ and the archive root, plus all 14 in core.

Confirm the identifier-regex gap:

python -c "
import sys, re; sys.path.insert(0, 'tools')
from pathlib import Path
import kv3
OLD = re.compile(r'^\t([A-Za-z_][A-Za-z0-9_]*)\s*=')
old = new = 0
for p in Path('out').rglob('*.vdata'):
    text = p.read_text(encoding='utf-8', errors='replace')
    old += len({m.group(1) for m in (OLD.match(l) for l in text.split(chr(10))) if m})
    new += len(kv3.parse(text))
print('regex keys', old, ' parser keys', new)"

Expected at build 6679: regex keys 1827 parser keys 1989.

Confirm the boolean spellings:

python -c "
import sys, collections; sys.path.insert(0, 'tools')
from kv3 import parse
doc = parse(open('out/scripts/abilities.vdata', encoding='utf-8').read())
print(collections.Counter(
    repr(v.get('m_bDisabled')) for v in doc.values()
    if isinstance(v, dict) and v.get('m_eAbilityType') == 'EAbilityType_Item'))"

Gotchas

1. Never read a boolean with == "true" or plain truthiness. Use kv3.as_bool.
2. Never enumerate entries with an identifier regex. It drops 8% of them.
3. Parsed keys are unescaped; raw text is not. Round-tripping a key back to a text offset needs escape-tolerant matching.
4. Unwrapping prefixes loses type information. resource_name:"x" and a plain "x" become indistinguishable after parsing.
5. generic_data_type is a scalar at the top level, not an entry — so "top-level keys" and "entries" are different counts in every document.

Open questions

• Whether other boolean fields use mixed spellings; only m_bDisabled was audited.
• Whether binary KV3 (.vdata_c before decompiling) carries type information the text form loses. Not examined — everything here is about the decompiler's text output.
• Whether duplicate keys can occur in one map. None were found, and the parser would silently keep the last.
• The _include / _multibase / _class metadata keys are parsed as ordinary data; only _multibase is interpreted, by kv3.flatten.

Sources

Derived from the decompiled output of the local install at build 6679, produced with Source 2 Viewer CLI 19.2 (see note 0003).