Initial commit
CI / test (macos-latest) (push) Canceled after 0s
CI / test (ubuntu-latest) (push) Canceled after 0s
CI / test (windows-latest) (push) Canceled after 0s
CI / release (macos-latest, aarch64-apple-darwin) (push) Canceled after 0s
CI / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
CI / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s

This commit is contained in:
Torbjörn Molin committed 2026-10-21 08:52:46 +02:00
commit 0d94f2bf2e
28 files changed
+11076

No files matched your search

+249
View File
@@ -0,0 +1,249 @@
# Log reader implementation plan
Oct 21, 2026 · Torbjörn
The app is a single native Rust binary built on egui/eframe for Windows, macOS and Linux. It opens a log file of hundreds of MB in under a second, then indexes it in the background so the UI never blocks. Everything below is a plan, not code.
## Goals and scope
- **Large files:** open 100 MB to 2 GB files without loading them into memory; first screen visible in under 1 s, full index in seconds.
- **Entries, not lines:** group multi-line records (stack traces, JSON dumps) into one entry, split on a detected line-start pattern.
- **Editable pattern:** auto-detect the start-of-entry pattern, show it, and let the user edit it with a live preview.
- **Collapsed by default:** a multi-line entry shows its first line plus a "+N lines" badge and expands in place.
- **Line wrap:** a toggle between one row per line with horizontal scrolling and wrapped lines.
- **Filter:** hide (or show only) entries matching keywords.
- **Highlight:** color whole entries, or the matched words, by keyword rules.
- **Rule files:** import and export filter and highlight rules as files.
- **Search:** regex search with next/previous and a match count.
- **Folder grep:** search every file in a folder for matching lines, then open any hit in the viewer at that line.
Out of scope for v1: live tailing of a growing file, UTF-16 and other non-UTF-8 encodings, remote sources, compressed files (.gz), and multi-file merge. Each open file is its own tab, with its own index and workers.
## Architecture
```
+------------------------------+
| Background workers (rayon) |
+--------------------+ | +------------------------+ | +-----------------------------+
| Log file (mmap) | | | Indexer | | results | Shared results |
| memmap2, read-only |-->| | entry starts, lines | |---------->| entry index (offset, lines) |
| never copied | | +------------------------+ | | visible-entry bitset |
+--------------------+ | | Filter pass | | | search match offsets |
bytes | | aho-corasick, RegexSet | | +-----------------------------+
| +------------------------+ | | read each frame
| | Search pass | | v
| | regex over raw bytes | | +-----------------------------+
| +------------------------+ | | UI thread (egui) |
+------------------------------+ | view model, expand state |
^ | decodes visible rows only |
| | highlights, LayoutJob cache |
| +-----------------------------+
| |
+------------------------------------------+
pattern, filter, query + generation (cancels stale work)
```
The UI thread never touches more of the file than the rows on screen. Indexing, filtering and searching run as parallel passes over the mapped bytes, publish results as they go, and are cancelled by a generation number whenever the user changes the pattern, filter or query.
## File access and background indexing
The file is memory-mapped read-only and never copied; the only thing held in RAM is a compact index of where each entry starts. Every pass over the bytes (index, filter, search) runs on worker threads and publishes partial results, so the UI is usable while they run.
**Byte source.** A `ByteSource` trait with one method, `bytes() -> &[u8]`, implemented with `memmap2`. Keeping it behind a trait leaves room for a chunked `pread` fallback (see Risks: Windows file locking) and for tailing later.
**Entry index.** One `Vec<EntryRec>` where `EntryRec { offset: u64, lines: u32 }`, 12 bytes per entry. A 500 MB file with 100-byte lines and mostly single-line entries is about 5M entries, or 60 MB. The line count is free during the scan and makes expand-all and the "+N lines" badge cheap.
**How the scan works.**
1. Split the file into N chunks (N = CPU cores), each boundary moved forward to the next `\n`.
2. In each chunk, find line starts with `memchr`. For each line, test the entry-start regex against only the first ~128 bytes of that line.
3. A line that matches starts a new entry; a line that does not is a continuation and bumps the current entry's line count.
4. Because the start/continuation decision is local to each line, chunks are independent. Run them with `rayon`, then concatenate in order and fix up the one entry that straddles each chunk boundary.
5. Publish progress (bytes scanned) and the first chunk's entries as soon as they exist, so the first screen renders before the scan finishes.
**Cancellation.** Every background job carries a generation number. Changing the pattern, filter or search bumps the generation; workers check it between blocks of ~1 MB and drop stale work.
**Text decoding.** Files are treated as UTF-8, which also covers plain ASCII. Rows are decoded lossily when drawn (`String::from_utf8_lossy`), so a bad byte shows as a replacement character instead of failing.
## Detecting and editing the entry-start pattern
The pattern is a regex anchored at line start. Detection picks the best of a fixed set of known formats by scoring them on a sample, falls back to inferring a pattern from line shapes, and always shows the result so the user can correct it.
**Sample.** The first 2,000 lines plus 500 lines from the middle and 500 from the end of the file, so a header block or a single huge stack trace does not skew the result.
**Built-in candidates** (tried in this order; more can be added as data in a TOML file):
| Name | Pattern (anchored with `^`) | Example |
| --- | --- | --- |
| ISO 8601 timestamp | `\[?\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}` | `2026-10-21 05:06:48,123 INFO ...` |
| Syslog | `[A-Z][a-z]{2} [ \d]\d \d{2}:\d{2}:\d{2}` | `Oct 21 05:06:48 host sshd[42]: ...` |
| Apache / nginx access | `\S+ \S+ \S+ \[\d{2}/\w{3}/\d{4}:` | `10.0.0.1 - - [21/Oct/2026:05:06:48 ...` |
| Slash date | `\d{1,2}/\d{1,2}/\d{2,4}[ T]\d{1,2}:\d{2}` | `10/21/2026 5:06:48 AM ...` |
| Time only | `\[?\d{2}:\d{2}:\d{2}` | `05:06:48.123 [main] ...` |
| Level first | `\[?(TRACE\|DEBUG\|INFO\|WARN\|WARNING\|ERROR\|FATAL)\b` | `ERROR [db] connection lost` |
| Epoch seconds | `\d{10}(\.\d+)?\b` | `1792472808.123 ...` |
| JSON lines | `\{` | `{"ts":"...","msg":"..."}` |
(In the Level-first row, `\|` is only Markdown table escaping; the regex uses plain `|`.)
**Scoring.** For each candidate, over the sample lines:
- `coverage` = share of lines that match.
- `false_starts` = matching lines that look like continuations (start with whitespace, `at `, `Caused by`, `}` or `]`).
- `score` = coverage of non-indented lines minus a penalty for false starts. A candidate needs a score above 0.6 to win; ties go to the more specific (earlier) pattern.
**Fallback inference.** If no candidate wins, map the first 24 bytes of each line to a shape (digit to `d`, letter to `a`, punctuation kept), take the most common shape prefix that covers at least 30% of lines, and turn it into a regex. If that also fails, use "every line not starting with whitespace" (`^\S`), which handles most stack traces.
**Editing UI.** A bar above the log shows the active pattern by name, for example "Entry start: ISO 8601 timestamp", with an edit button that opens a side panel:
- A dropdown of the built-in candidates, each with its sample score, plus a free-text regex field.
- A live preview of the first ~40 sample lines with entry boundaries drawn as rules and continuation lines indented, plus "N entries in sample".
- Invalid regex shows the parser's error inline and disables Apply.
- Apply re-runs the indexer (cancelling the old run). The choice is saved per file path, and optionally per file-name glob such as `*.app.log`, so the next open skips detection.
## Virtualized view and expandable entries
Only the rows on screen are ever decoded or laid out. Every display row is one physical line of fixed height, which lets egui's `ScrollArea::show_rows` do the virtualization; expansion just changes how many display rows an entry occupies. Line wrap breaks the fixed-height assumption, so it gets its own scrolling mode (below).
**View model.**
- `visible: Vec<u32>` holds the entry indices that pass the current filter (all entries when no filter is set).
- `expanded: BTreeSet<u32>` holds expanded entries, plus an `expand_all` flag.
- A collapsed entry is 1 display row; an expanded one is `lines` rows.
- A prefix-sum array over `visible` maps display row to (entry, line within entry) with a binary search. It is rebuilt only when the filter or the expanded set changes; with few expansions an incremental Fenwick tree avoids a full rebuild on each click.
**Row rendering.**
- Each row is a `LayoutJob` with highlight colors applied (next section), cached in an LRU keyed by (entry, line, wrap width, rules generation).
- The first line of a multi-line entry gets a disclosure triangle and a muted "+N lines" badge; continuation lines are indented and drawn in a slightly dimmer color when expanded.
- An optional left gutter shows the entry number or line number.
- Lines longer than 64 KB are cut at that length with a "show full line" action that opens a detail pane, since egui slows down on very long galleys.
**Line wrap toggle.** A toolbar toggle (Alt+Z) switches wrapping per tab, and the choice is remembered per file.
- **Wrap off** (default): one fixed-height row per line, horizontal scrolling, `show_rows` virtualization as above.
- **Wrap on:** rows have different heights, and measuring every line in the file to get exact heights would mean a full pass. Instead the scroll position is an anchor, (entry, line, wrapped sub-row), and each frame lays out rows downward from the anchor until the screen is full. Wrapped row counts are only computed for lines that are drawn.
- The scrollbar in wrap mode is a custom widget mapped to byte offset in the file. Dragging it binary-searches the entry index for the new anchor; wheel and keys move the anchor by sub-rows.
- Width changes (window resize) invalidate the layout cache and are debounced by ~100 ms. The anchor entry stays at the top so the view does not jump.
**Interaction.** Click the triangle or press Space/Right to expand, Left to collapse; a toolbar has Expand all and Collapse all. Ctrl/Cmd+C copies the selected entry (all lines), and a context menu offers "copy line" and "filter out entries like this". Go-to-line maps a line number to its entry with a binary search on byte offsets.
**Scrollbar during indexing.** Until the index is complete, the total row count is estimated from bytes scanned so far, so the scrollbar is roughly right and settles when the scan finishes.
## Filter, highlight and search
All three match against the whole entry (every line), not just its first line. Filter and search run as cancellable background passes over the full file; highlighting runs only on the rows being drawn.
### Keyword filter
- A filter bar holds chips, each a rule: **Exclude** (hide entries containing it) or **Only** (show only entries containing it). Each chip has case-sensitive and regex toggles.
- Semantics: an entry is shown if it matches at least one Only rule (or there are none) and no Exclude rule.
- Plain keywords are compiled together into one `aho-corasick` automaton per rule kind, which scans at memory speed regardless of keyword count; regex chips use a `RegexSet`.
- The pass runs with `rayon` over entry ranges and writes a bitset; `visible` is rebuilt from it. Partial results stream in, and the status bar shows "12,408 of 1,203,551 entries (filtering 63%)".
- Typing is debounced by 150 ms; each change bumps the generation and cancels the previous pass.
- The scroll position is kept anchored to the entry at the top of the screen (or the nearest visible one) when the filter changes.
### Keyword highlighting
- An ordered list of rules: pattern (keyword or regex), color, and scope, which is either **row** (tint the whole entry's background) or **match** (color only the matched text).
- For a row tint the first matching rule wins; match colors from all rules are applied, later rules drawn under earlier ones.
- Defaults: `ERROR`/`FATAL` red row tint, `WARN` amber row tint. Colors come from a palette that has light and dark variants.
- Rules are evaluated only for visible rows while building the `LayoutJob`, so changing them is instant and needs no full-file pass.
- Rule sets are saved globally and can be overridden per file glob.
### Regex search
- A search box (Ctrl/Cmd+F) with toggles for regex, case sensitivity and whole word. Plain-text mode escapes the input.
- Uses `regex::bytes::Regex` over the mapped bytes in parallel chunks, collecting match byte offsets (capped at 1M with a "too many matches" note). Each offset maps to its entry with a binary search on the entry index.
- The status bar shows "match 37 of 1,204" and updates as the scan progresses; Enter / Shift+Enter (and F3) go to the next and previous match.
- Jumping to a match scrolls to its entry, auto-expands it if the match is on a continuation line, and draws the current match in a distinct color; other matches on screen are outlined.
- A match inside an entry hidden by the filter is skipped by default, with a toggle to "search hidden entries too" that temporarily reveals it.
- The `regex` crate has no look-around or backreferences. That is the right trade for guaranteed linear-time scans of large files; `fancy-regex` can be an opt-in later.
### Importing and exporting rules
- Filters and highlight rules export to a `.logrules.toml` file, which can also carry an entry-start pattern. TOML keeps it readable and easy to review when a team shares it in a repo.
- The file has a `version` field. Each rule keeps its kind (exclude, only, highlight), pattern, regex and case flags, color and scope.
- Export offers everything, filters only, highlights only, or the selected rules.
- Import comes from a menu item or by dropping the file on the window. It asks whether to **merge** (skip exact duplicates) or **replace**, and whether to apply to this tab or save as the global default.
- Every pattern is compiled on import. Invalid ones are listed with the regex error and skipped, so the rest still load.
- Named rule sets in settings (for example "Java service" or "nginx") are just saved rule files and can be switched from a dropdown.
## Folder grep
A separate "Find in files" tab searches every file under a folder and lists matching lines grouped by file. Double-clicking a hit opens that file in a viewer tab, scrolled to the entry that holds the line.
**Inputs.** Folder picker (or drag a folder in), pattern with regex, case and whole-word toggles, include and exclude globs (default `*.log;*.txt;*.out`), recursive toggle, "respect .gitignore" toggle, skip binary files, and a max file size.
**Engine.** Built on ripgrep's own libraries rather than a hand-rolled loop:
- `ignore` walks the folder in parallel and applies globs and ignore files.
- `grep-searcher` and `grep-regex` search each file, using mmap for large files, detecting binary files, and reporting line numbers and byte offsets.
**Results.** Results stream into a virtualized tree as they arrive: file (with path, match count and size), then its matching lines with the line number and the match highlighted. The header shows "1,284 matches in 37 of 2,910 files" and a cancel button. Matches are capped at 10,000 per file and 1M overall, with a note when a cap is hit. Results can be copied or exported as plain text in `path:line:text` form.
**Opening a hit.** Opening a result (double-click, or Enter) opens the file in a viewer tab, or reuses the tab if the file is already open. The viewer turns the byte offset into an entry with a binary search on the index, expands the entry if the match is on a continuation line, and fills its search box with the grep pattern, so next and previous keep working inside the file. If indexing has not reached that offset yet, the viewer waits for that chunk and then jumps.
## Crate choices
All of these are mature, pure-Rust (or with well-supported native bits) and build on all three platforms. Pin current versions when the project starts.
| Need | Crate | Why this one |
| --- | --- | --- |
| App shell and UI | `eframe` + `egui` | The requested stack; one binary, native windows on Win/macOS/Linux |
| Tabs, tables, extra widgets | `egui_extras` (+ `egui_dock` if tabs need docking) | Settings and rules panels, file and grep tabs |
| Memory-mapped file | `memmap2` | Maintained successor of `memmap`, cross-platform |
| Newline scanning | `memchr` | SIMD byte search, the fastest way to find line starts |
| Regex | `regex` (bytes API) | Linear-time guarantee, works on `&[u8]` without UTF-8 validation |
| Multi-keyword match | `aho-corasick` | One pass for any number of filter keywords |
| Parallel passes | `rayon` | Chunked parallel index, filter and search |
| Worker messages | `crossbeam-channel` | Progress and partial results from workers to the UI |
| Folder walk | `ignore` | ripgrep's parallel walker with globs and .gitignore |
| Folder grep | `grep-searcher` + `grep-regex` | ripgrep's search core: mmap, binary detection, line numbers |
| File dialogs | `rfd` | Native open dialogs on all three platforms |
| Settings and rule files | `serde` + `toml` + `directories` | Human-editable config and `.logrules.toml` files in OS-standard folders |
| Row layout cache | `lru` | Bounded cache of `LayoutJob`s |
| Errors and logging | `anyhow`, `thiserror`, `tracing` | Standard choices |
| Later: tailing | `notify` | File-change events for live tail |
| Later: encodings | `encoding_rs` | UTF-16 and legacy code pages |
| Packaging | `cargo-dist` or `cargo-bundle` | Installers, .app bundle, and a GitHub Actions release matrix |
Benchmarks use `criterion` with generated fixture files (100 MB, 1 GB; short lines, long lines, heavy stack traces) and a generated folder of 1,000 files for grep.
## Phased milestones
Nine phases, each ending in something usable. Phases 1 to 3 come first in order; phases 4 to 8 depend only on them and can be reordered, except that folder grep (8) reuses search (5).
1. **Skeleton and big-file viewing.** eframe app with a tab per open file; open via dialog, drag-and-drop and command-line argument; `memmap2` source; a plain line view (each line is an entry) using `show_rows`; recent files.
- Exit: a 1 GB file opens in under 1 s and scrolls at 60 fps on a mid-range laptop.
2. **Entry index and expandable entries.** Parallel background indexer with a hard-coded ISO-timestamp pattern, progress bar, cancellation; view model with expand/collapse, expand all, "+N lines" badges, copy entry.
- Exit: 1 GB indexed in under 5 s; UI stays responsive during the scan.
3. **Auto-detection and pattern editor.** Sampling, candidate scoring, shape-based fallback, side panel with live preview, per-path persistence.
- Exit: correct detection on a fixture set of at least 10 real log formats (app, syslog, nginx, Java with stack traces, .NET, JSON lines).
4. **Line wrap.** Wrap toggle with anchor-based scrolling, the byte-offset scrollbar, and debounced relayout on resize.
- Exit: wrapped scrolling through a 1 GB file stays at 60 fps.
5. **Regex search.** Search box, parallel scan, match list, next/previous with auto-expand, match coloring.
6. **Keyword filter.** Filter chips, aho-corasick and `RegexSet` pass, bitset, streaming results, scroll anchoring.
7. **Highlight rules, settings and rule files.** Rules panel, row and match scopes, light and dark palettes, settings file, `.logrules.toml` import and export with merge or replace, named rule sets.
8. **Folder grep.** Find-in-files tab on `ignore` + `grep-searcher`, streaming result tree, caps and cancel, open a hit in a viewer tab at the right entry.
- Exit: greps a 2 GB folder of 1,000 logs in under 5 s on SSD with a warm cache.
9. **Polish and release.** Keyboard map, go-to-line, error states (file deleted, permission denied), signed and notarized macOS build, Windows installer, Linux AppImage or tarball via a CI release matrix.
After v1, the likely next items are live tailing, UTF-16 and other encodings, opening .gz files, merging several files into one timeline, remote sources (SSH, cloud logs), and a minimap of highlight and match positions beside the scrollbar.
## Risks and open questions
| Risk | Impact | Mitigation |
| --- | --- | --- |
| Windows: a mapped file cannot be truncated by its writer | An app that rotates its log by truncation gets an error while the viewer has that file open | Acceptable for a viewer of finished logs; the `pread` source behind `ByteSource` stays available if it ever bites |
| Unix: file truncated while mapped | Reading past the new end raises SIGBUS and crashes the app | Check file size on focus and before each pass; on shrink, remap and reindex |
| Wrap mode on huge entries | Laying out a multi-MB entry stalls a frame | 64 KB line cap plus the detail pane; lay out only the rows on screen |
| Files with tiny lines (e.g. 20 bytes) | 1 GB becomes ~50M entries, 600 MB of index | Store offsets as u32 deltas per 4 GB block (8 bytes per entry), or page the index |
| Folder grep on huge trees | Long runs, too many results to hold | Streaming results, caps, cancel, and default include globs |
| Pattern detection picks wrong format | Entries merged or split | Always show the active pattern and make it one click to change |
| macOS distribution | Unsigned apps are blocked by Gatekeeper | Budget an Apple developer account and notarization in phase 9 |
Open questions:
- [x] Should grep results show only the matching line, or also the first line of the entry it belongs to? Answer: Matching line only.
- [x] Should rule files also be loadable from a fixed team folder or URL, not just by import? Answer: File import/export only.