Files
logreadlite/log-reader-implementation-plan.md
T
Torbjörn Molin 0d94f2bf2e
CI / test (macos-latest) (push) Canceled after 0s
CI / test (ubuntu-latest) (push) Canceled after 0s
CI / test (windows-latest) (push) Canceled after 0s
CI / release (macos-latest, aarch64-apple-darwin) (push) Canceled after 0s
CI / release (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Canceled after 0s
CI / release (windows-latest, x86_64-pc-windows-msvc) (push) Canceled after 0s
Initial commit
2026-10-21 08:52:46 +02:00

22 KiB

Log reader implementation plan

Oct 21, 2026 · Torbjörn

The app is a single native Rust binary built on egui/eframe for Windows, macOS and Linux. It opens a log file of hundreds of MB in under a second, then indexes it in the background so the UI never blocks. Everything below is a plan, not code.

Goals and scope

  • Large files: open 100 MB to 2 GB files without loading them into memory; first screen visible in under 1 s, full index in seconds.
  • Entries, not lines: group multi-line records (stack traces, JSON dumps) into one entry, split on a detected line-start pattern.
  • Editable pattern: auto-detect the start-of-entry pattern, show it, and let the user edit it with a live preview.
  • Collapsed by default: a multi-line entry shows its first line plus a "+N lines" badge and expands in place.
  • Line wrap: a toggle between one row per line with horizontal scrolling and wrapped lines.
  • Filter: hide (or show only) entries matching keywords.
  • Highlight: color whole entries, or the matched words, by keyword rules.
  • Rule files: import and export filter and highlight rules as files.
  • Search: regex search with next/previous and a match count.
  • Folder grep: search every file in a folder for matching lines, then open any hit in the viewer at that line.

Out of scope for v1: live tailing of a growing file, UTF-16 and other non-UTF-8 encodings, remote sources, compressed files (.gz), and multi-file merge. Each open file is its own tab, with its own index and workers.

Architecture

                         +------------------------------+
                         | Background workers (rayon)   |
+--------------------+   |  +------------------------+  |           +-----------------------------+
| Log file (mmap)    |   |  | Indexer                |  |  results  | Shared results              |
| memmap2, read-only |-->|  | entry starts, lines    |  |---------->| entry index (offset, lines) |
| never copied       |   |  +------------------------+  |           | visible-entry bitset        |
+--------------------+   |  | Filter pass            |  |           | search match offsets        |
       bytes             |  | aho-corasick, RegexSet |  |           +-----------------------------+
                         |  +------------------------+  |                        | read each frame
                         |  | Search pass            |  |                        v
                         |  | regex over raw bytes   |  |           +-----------------------------+
                         |  +------------------------+  |           | UI thread (egui)            |
                         +------------------------------+           | view model, expand state    |
                                        ^                           | decodes visible rows only   |
                                        |                           | highlights, LayoutJob cache |
                                        |                           +-----------------------------+
                                        |                                          |
                                        +------------------------------------------+
                               pattern, filter, query + generation (cancels stale work)

The UI thread never touches more of the file than the rows on screen. Indexing, filtering and searching run as parallel passes over the mapped bytes, publish results as they go, and are cancelled by a generation number whenever the user changes the pattern, filter or query.

File access and background indexing

The file is memory-mapped read-only and never copied; the only thing held in RAM is a compact index of where each entry starts. Every pass over the bytes (index, filter, search) runs on worker threads and publishes partial results, so the UI is usable while they run.

Byte source. A ByteSource trait with one method, bytes() -> &[u8], implemented with memmap2. Keeping it behind a trait leaves room for a chunked pread fallback (see Risks: Windows file locking) and for tailing later.

Entry index. One Vec<EntryRec> where EntryRec { offset: u64, lines: u32 }, 12 bytes per entry. A 500 MB file with 100-byte lines and mostly single-line entries is about 5M entries, or 60 MB. The line count is free during the scan and makes expand-all and the "+N lines" badge cheap.

How the scan works.

  1. Split the file into N chunks (N = CPU cores), each boundary moved forward to the next \n.
  2. In each chunk, find line starts with memchr. For each line, test the entry-start regex against only the first ~128 bytes of that line.
  3. A line that matches starts a new entry; a line that does not is a continuation and bumps the current entry's line count.
  4. Because the start/continuation decision is local to each line, chunks are independent. Run them with rayon, then concatenate in order and fix up the one entry that straddles each chunk boundary.
  5. Publish progress (bytes scanned) and the first chunk's entries as soon as they exist, so the first screen renders before the scan finishes.

Cancellation. Every background job carries a generation number. Changing the pattern, filter or search bumps the generation; workers check it between blocks of ~1 MB and drop stale work.

Text decoding. Files are treated as UTF-8, which also covers plain ASCII. Rows are decoded lossily when drawn (String::from_utf8_lossy), so a bad byte shows as a replacement character instead of failing.

Detecting and editing the entry-start pattern

The pattern is a regex anchored at line start. Detection picks the best of a fixed set of known formats by scoring them on a sample, falls back to inferring a pattern from line shapes, and always shows the result so the user can correct it.

Sample. The first 2,000 lines plus 500 lines from the middle and 500 from the end of the file, so a header block or a single huge stack trace does not skew the result.

Built-in candidates (tried in this order; more can be added as data in a TOML file):

Name Pattern (anchored with ^) Example
ISO 8601 timestamp \[?\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2} 2026-10-21 05:06:48,123 INFO ...
Syslog [A-Z][a-z]{2} [ \d]\d \d{2}:\d{2}:\d{2} Oct 21 05:06:48 host sshd[42]: ...
Apache / nginx access \S+ \S+ \S+ \[\d{2}/\w{3}/\d{4}: 10.0.0.1 - - [21/Oct/2026:05:06:48 ...
Slash date \d{1,2}/\d{1,2}/\d{2,4}[ T]\d{1,2}:\d{2} 10/21/2026 5:06:48 AM ...
Time only \[?\d{2}:\d{2}:\d{2} 05:06:48.123 [main] ...
Level first \[?(TRACE|DEBUG|INFO|WARN|WARNING|ERROR|FATAL)\b ERROR [db] connection lost
Epoch seconds \d{10}(\.\d+)?\b 1792472808.123 ...
JSON lines \{ {"ts":"...","msg":"..."}

(In the Level-first row, \| is only Markdown table escaping; the regex uses plain |.)

Scoring. For each candidate, over the sample lines:

  • coverage = share of lines that match.
  • false_starts = matching lines that look like continuations (start with whitespace, at , Caused by, } or ]).
  • score = coverage of non-indented lines minus a penalty for false starts. A candidate needs a score above 0.6 to win; ties go to the more specific (earlier) pattern.

Fallback inference. If no candidate wins, map the first 24 bytes of each line to a shape (digit to d, letter to a, punctuation kept), take the most common shape prefix that covers at least 30% of lines, and turn it into a regex. If that also fails, use "every line not starting with whitespace" (^\S), which handles most stack traces.

Editing UI. A bar above the log shows the active pattern by name, for example "Entry start: ISO 8601 timestamp", with an edit button that opens a side panel:

  • A dropdown of the built-in candidates, each with its sample score, plus a free-text regex field.
  • A live preview of the first ~40 sample lines with entry boundaries drawn as rules and continuation lines indented, plus "N entries in sample".
  • Invalid regex shows the parser's error inline and disables Apply.
  • Apply re-runs the indexer (cancelling the old run). The choice is saved per file path, and optionally per file-name glob such as *.app.log, so the next open skips detection.

Virtualized view and expandable entries

Only the rows on screen are ever decoded or laid out. Every display row is one physical line of fixed height, which lets egui's ScrollArea::show_rows do the virtualization; expansion just changes how many display rows an entry occupies. Line wrap breaks the fixed-height assumption, so it gets its own scrolling mode (below).

View model.

  • visible: Vec<u32> holds the entry indices that pass the current filter (all entries when no filter is set).
  • expanded: BTreeSet<u32> holds expanded entries, plus an expand_all flag.
  • A collapsed entry is 1 display row; an expanded one is lines rows.
  • A prefix-sum array over visible maps display row to (entry, line within entry) with a binary search. It is rebuilt only when the filter or the expanded set changes; with few expansions an incremental Fenwick tree avoids a full rebuild on each click.

Row rendering.

  • Each row is a LayoutJob with highlight colors applied (next section), cached in an LRU keyed by (entry, line, wrap width, rules generation).
  • The first line of a multi-line entry gets a disclosure triangle and a muted "+N lines" badge; continuation lines are indented and drawn in a slightly dimmer color when expanded.
  • An optional left gutter shows the entry number or line number.
  • Lines longer than 64 KB are cut at that length with a "show full line" action that opens a detail pane, since egui slows down on very long galleys.

Line wrap toggle. A toolbar toggle (Alt+Z) switches wrapping per tab, and the choice is remembered per file.

  • Wrap off (default): one fixed-height row per line, horizontal scrolling, show_rows virtualization as above.
  • Wrap on: rows have different heights, and measuring every line in the file to get exact heights would mean a full pass. Instead the scroll position is an anchor, (entry, line, wrapped sub-row), and each frame lays out rows downward from the anchor until the screen is full. Wrapped row counts are only computed for lines that are drawn.
  • The scrollbar in wrap mode is a custom widget mapped to byte offset in the file. Dragging it binary-searches the entry index for the new anchor; wheel and keys move the anchor by sub-rows.
  • Width changes (window resize) invalidate the layout cache and are debounced by ~100 ms. The anchor entry stays at the top so the view does not jump.

Interaction. Click the triangle or press Space/Right to expand, Left to collapse; a toolbar has Expand all and Collapse all. Ctrl/Cmd+C copies the selected entry (all lines), and a context menu offers "copy line" and "filter out entries like this". Go-to-line maps a line number to its entry with a binary search on byte offsets.

Scrollbar during indexing. Until the index is complete, the total row count is estimated from bytes scanned so far, so the scrollbar is roughly right and settles when the scan finishes.

All three match against the whole entry (every line), not just its first line. Filter and search run as cancellable background passes over the full file; highlighting runs only on the rows being drawn.

Keyword filter

  • A filter bar holds chips, each a rule: Exclude (hide entries containing it) or Only (show only entries containing it). Each chip has case-sensitive and regex toggles.
  • Semantics: an entry is shown if it matches at least one Only rule (or there are none) and no Exclude rule.
  • Plain keywords are compiled together into one aho-corasick automaton per rule kind, which scans at memory speed regardless of keyword count; regex chips use a RegexSet.
  • The pass runs with rayon over entry ranges and writes a bitset; visible is rebuilt from it. Partial results stream in, and the status bar shows "12,408 of 1,203,551 entries (filtering 63%)".
  • Typing is debounced by 150 ms; each change bumps the generation and cancels the previous pass.
  • The scroll position is kept anchored to the entry at the top of the screen (or the nearest visible one) when the filter changes.

Keyword highlighting

  • An ordered list of rules: pattern (keyword or regex), color, and scope, which is either row (tint the whole entry's background) or match (color only the matched text).
  • For a row tint the first matching rule wins; match colors from all rules are applied, later rules drawn under earlier ones.
  • Defaults: ERROR/FATAL red row tint, WARN amber row tint. Colors come from a palette that has light and dark variants.
  • Rules are evaluated only for visible rows while building the LayoutJob, so changing them is instant and needs no full-file pass.
  • Rule sets are saved globally and can be overridden per file glob.
  • A search box (Ctrl/Cmd+F) with toggles for regex, case sensitivity and whole word. Plain-text mode escapes the input.
  • Uses regex::bytes::Regex over the mapped bytes in parallel chunks, collecting match byte offsets (capped at 1M with a "too many matches" note). Each offset maps to its entry with a binary search on the entry index.
  • The status bar shows "match 37 of 1,204" and updates as the scan progresses; Enter / Shift+Enter (and F3) go to the next and previous match.
  • Jumping to a match scrolls to its entry, auto-expands it if the match is on a continuation line, and draws the current match in a distinct color; other matches on screen are outlined.
  • A match inside an entry hidden by the filter is skipped by default, with a toggle to "search hidden entries too" that temporarily reveals it.
  • The regex crate has no look-around or backreferences. That is the right trade for guaranteed linear-time scans of large files; fancy-regex can be an opt-in later.

Importing and exporting rules

  • Filters and highlight rules export to a .logrules.toml file, which can also carry an entry-start pattern. TOML keeps it readable and easy to review when a team shares it in a repo.
  • The file has a version field. Each rule keeps its kind (exclude, only, highlight), pattern, regex and case flags, color and scope.
  • Export offers everything, filters only, highlights only, or the selected rules.
  • Import comes from a menu item or by dropping the file on the window. It asks whether to merge (skip exact duplicates) or replace, and whether to apply to this tab or save as the global default.
  • Every pattern is compiled on import. Invalid ones are listed with the regex error and skipped, so the rest still load.
  • Named rule sets in settings (for example "Java service" or "nginx") are just saved rule files and can be switched from a dropdown.

Folder grep

A separate "Find in files" tab searches every file under a folder and lists matching lines grouped by file. Double-clicking a hit opens that file in a viewer tab, scrolled to the entry that holds the line.

Inputs. Folder picker (or drag a folder in), pattern with regex, case and whole-word toggles, include and exclude globs (default *.log;*.txt;*.out), recursive toggle, "respect .gitignore" toggle, skip binary files, and a max file size.

Engine. Built on ripgrep's own libraries rather than a hand-rolled loop:

  • ignore walks the folder in parallel and applies globs and ignore files.
  • grep-searcher and grep-regex search each file, using mmap for large files, detecting binary files, and reporting line numbers and byte offsets.

Results. Results stream into a virtualized tree as they arrive: file (with path, match count and size), then its matching lines with the line number and the match highlighted. The header shows "1,284 matches in 37 of 2,910 files" and a cancel button. Matches are capped at 10,000 per file and 1M overall, with a note when a cap is hit. Results can be copied or exported as plain text in path:line:text form.

Opening a hit. Opening a result (double-click, or Enter) opens the file in a viewer tab, or reuses the tab if the file is already open. The viewer turns the byte offset into an entry with a binary search on the index, expands the entry if the match is on a continuation line, and fills its search box with the grep pattern, so next and previous keep working inside the file. If indexing has not reached that offset yet, the viewer waits for that chunk and then jumps.

Crate choices

All of these are mature, pure-Rust (or with well-supported native bits) and build on all three platforms. Pin current versions when the project starts.

Need Crate Why this one
App shell and UI eframe + egui The requested stack; one binary, native windows on Win/macOS/Linux
Tabs, tables, extra widgets egui_extras (+ egui_dock if tabs need docking) Settings and rules panels, file and grep tabs
Memory-mapped file memmap2 Maintained successor of memmap, cross-platform
Newline scanning memchr SIMD byte search, the fastest way to find line starts
Regex regex (bytes API) Linear-time guarantee, works on &[u8] without UTF-8 validation
Multi-keyword match aho-corasick One pass for any number of filter keywords
Parallel passes rayon Chunked parallel index, filter and search
Worker messages crossbeam-channel Progress and partial results from workers to the UI
Folder walk ignore ripgrep's parallel walker with globs and .gitignore
Folder grep grep-searcher + grep-regex ripgrep's search core: mmap, binary detection, line numbers
File dialogs rfd Native open dialogs on all three platforms
Settings and rule files serde + toml + directories Human-editable config and .logrules.toml files in OS-standard folders
Row layout cache lru Bounded cache of LayoutJobs
Errors and logging anyhow, thiserror, tracing Standard choices
Later: tailing notify File-change events for live tail
Later: encodings encoding_rs UTF-16 and legacy code pages
Packaging cargo-dist or cargo-bundle Installers, .app bundle, and a GitHub Actions release matrix

Benchmarks use criterion with generated fixture files (100 MB, 1 GB; short lines, long lines, heavy stack traces) and a generated folder of 1,000 files for grep.

Phased milestones

Nine phases, each ending in something usable. Phases 1 to 3 come first in order; phases 4 to 8 depend only on them and can be reordered, except that folder grep (8) reuses search (5).

  1. Skeleton and big-file viewing. eframe app with a tab per open file; open via dialog, drag-and-drop and command-line argument; memmap2 source; a plain line view (each line is an entry) using show_rows; recent files.
    • Exit: a 1 GB file opens in under 1 s and scrolls at 60 fps on a mid-range laptop.
  2. Entry index and expandable entries. Parallel background indexer with a hard-coded ISO-timestamp pattern, progress bar, cancellation; view model with expand/collapse, expand all, "+N lines" badges, copy entry.
    • Exit: 1 GB indexed in under 5 s; UI stays responsive during the scan.
  3. Auto-detection and pattern editor. Sampling, candidate scoring, shape-based fallback, side panel with live preview, per-path persistence.
    • Exit: correct detection on a fixture set of at least 10 real log formats (app, syslog, nginx, Java with stack traces, .NET, JSON lines).
  4. Line wrap. Wrap toggle with anchor-based scrolling, the byte-offset scrollbar, and debounced relayout on resize.
    • Exit: wrapped scrolling through a 1 GB file stays at 60 fps.
  5. Regex search. Search box, parallel scan, match list, next/previous with auto-expand, match coloring.
  6. Keyword filter. Filter chips, aho-corasick and RegexSet pass, bitset, streaming results, scroll anchoring.
  7. Highlight rules, settings and rule files. Rules panel, row and match scopes, light and dark palettes, settings file, .logrules.toml import and export with merge or replace, named rule sets.
  8. Folder grep. Find-in-files tab on ignore + grep-searcher, streaming result tree, caps and cancel, open a hit in a viewer tab at the right entry.
    • Exit: greps a 2 GB folder of 1,000 logs in under 5 s on SSD with a warm cache.
  9. Polish and release. Keyboard map, go-to-line, error states (file deleted, permission denied), signed and notarized macOS build, Windows installer, Linux AppImage or tarball via a CI release matrix.

After v1, the likely next items are live tailing, UTF-16 and other encodings, opening .gz files, merging several files into one timeline, remote sources (SSH, cloud logs), and a minimap of highlight and match positions beside the scrollbar.

Risks and open questions

Risk Impact Mitigation
Windows: a mapped file cannot be truncated by its writer An app that rotates its log by truncation gets an error while the viewer has that file open Acceptable for a viewer of finished logs; the pread source behind ByteSource stays available if it ever bites
Unix: file truncated while mapped Reading past the new end raises SIGBUS and crashes the app Check file size on focus and before each pass; on shrink, remap and reindex
Wrap mode on huge entries Laying out a multi-MB entry stalls a frame 64 KB line cap plus the detail pane; lay out only the rows on screen
Files with tiny lines (e.g. 20 bytes) 1 GB becomes ~50M entries, 600 MB of index Store offsets as u32 deltas per 4 GB block (8 bytes per entry), or page the index
Folder grep on huge trees Long runs, too many results to hold Streaming results, caps, cancel, and default include globs
Pattern detection picks wrong format Entries merged or split Always show the active pattern and make it one click to change
macOS distribution Unsigned apps are blocked by Gatekeeper Budget an Apple developer account and notarization in phase 9

Open questions:

  • Should grep results show only the matching line, or also the first line of the entry it belongs to? Answer: Matching line only.
  • Should rule files also be loadable from a fixed team folder or URL, not just by import? Answer: File import/export only.