Files
reqlite/implementation-plan.md
T
2026-10-19 09:31:57 +02:00

23 KiB
Raw Blame History

Implementation plan: Rust + egui API client

This plan builds on tech-stack.md. It has three parts:

  1. Decisions that are fixed up front, so later work doesn't revisit them.
  2. Architecture: how the code is split up and how data flows through it.
  3. Milestones: small vertical slices, each ending in something you can run and test.

Working name: reqlite (rename freely).

Revision 3: async with Tokio and .http files as the storage format (revision 2), and no CI for now (revision 3). Quality checks run locally with just check, and release builds are made by hand. All three were requested by Torbjörn.


1. Decisions

Topic Decision Why
Language / edition Rust stable, edition 2021, pinned with rust-toolchain.toml Builds are reproducible on every machine.
UI eframe + egui + egui_extras (syntax highlighting, tables) Immediate mode keeps all state in plain structs, and one exe covers all platforms.
Renderer eframe's default wgpu backend, with glow as a feature flag glow is a fallback for old GPUs and VMs.
Async runtime tokio (multi-thread runtime on background threads) + tokio-util (CancellationToken) The UI thread stays egui's; all I/O (HTTP, file loading, watching) runs as Tokio tasks.
HTTP curl crate with features static-curl, http2, ssl (Schannel on Windows, OpenSSL/rustls elsewhere), behind an async trait It is faithful to how curl behaves, gives timings, and has a single binary with no DLLs. See section 2.3 for how it becomes async.
Async trait Native async fn in traits, plus async-trait only where dyn HttpClient is needed
Serialization Own .http parser/writer in core; serde_json for environments, JSON bodies and Postman import; toml for app settings
Storage format .http files (JetBrains HTTP Client / VS Code REST Client syntax), with environments in http-client.env.json Files are git friendly, human readable, and also open in JetBrains IDEs and VS Code. See section 2.4.
Errors thiserror in libraries, anyhow only in the binaries
Logging tracing + tracing-subscriber, logged to a file in the app data directory Bug reports can include the log.
Paths directories crate for config, data and cache dirs
Secrets http-client.private.env.json (git-ignored, the convention both IDEs use), optionally backed by keyring
File dialogs rfd (async API) Native dialogs on every platform.
Testing cargo test + #[tokio::test], insta (snapshots), wiremock (async HTTP mock server), egui_kittest (UI)
Lint / format cargo fmt --check, cargo clippy -- -D warnings, cargo deny (licenses, advisories), run locally through one check command (see below) There is no CI, so a single local command keeps the bar consistent.
License MIT or Apache-2.0 (your call) It is compatible with egui and curl.

2. Architecture

2.1 Workspace layout

reqlite/
  Cargo.toml                # [workspace], shared deps in [workspace.dependencies]
  rust-toolchain.toml
  deny.toml
  justfile                  # `just check` = fmt + clippy + test + deny; `just release` = static release build
  crates/
    core/                   # no UI, no network, no global state, no tokio dependency
      src/model.rs          # Request, Header, Body, Environment, Collection
      src/template.rs       # {{var}} parsing and resolution
      src/http.rs           # trait HttpClient (async), ResolvedRequest, Response, Timings, HttpError
      src/httpfile/         # .http lexer, parser, writer (lossless round-trip)
      src/env.rs            # http-client.env.json + private.env.json
      src/collection.rs     # scan a folder of .http files into a tree
      src/import/           # curl.rs, postman.rs
      src/export/           # curl.rs
    http-curl/              # impl HttpClient for libcurl (uses tokio)
    app/                    # egui binary
      src/main.rs           # builds the tokio runtime, starts eframe
      src/state.rs          # AppState (pure data)
      src/action.rs         # enum Action, fn update(&mut AppState, Action) -> Vec<Effect>
      src/effects.rs        # spawns Effects as tokio tasks; results come back as Actions
      src/ui/               # one module per panel; reads state, emits Actions
    cli/                    # `reqlite run api/users.http --env dev [--name create-user]`
  tests/fixtures/           # sample .http files, env files, Postman exports, curl commands

Dependency rule: app and cli depend on core and http-curl, and core depends on neither. Enforce this by keeping UI, curl and Tokio crates out of core's Cargo.toml. core only defines the async trait and stays runtime-agnostic.

2.2 Data flow in the app (keeps the UI testable)

 UI panels ──Action──▶ update(state, action) ──Effect──▶ effects runner (tokio::spawn)
     ▲                         │                            │ HTTP, disk, dialogs, file watching
     └──── reads AppState ◀────┘◀── Action via mpsc + ctx.request_repaint() ──┘

Runtime setup. main builds a multi-thread Tokio runtime and keeps its Handle in the effects runner. eframe keeps the main thread. Do not use #[tokio::main] around eframe, because winit needs the main thread on macOS.

Pure update. UI code only draws state and pushes Actions. update is a pure, synchronous function: it changes state and returns Effects such as Effect::SendRequest { tab_id, request, cancel }, Effect::SaveFile { .. } or Effect::LoadCollection(path). Most behaviour is tested by calling update directly.

Running effects. The effects runner does handle.spawn(async move { ... }) for each effect and sends the resulting Action back through a tokio::sync::mpsc::UnboundedSender. Each frame, the UI drains the matching receiver with try_recv (non-blocking). The task calls egui::Context::request_repaint() after sending, so the UI wakes immediately.

Cancellation. Each in-flight request gets a CancellationToken, and Action::CancelClicked calls .cancel(). The HTTP layer races the transfer against token.cancelled().

Concurrency. Several tabs can send at once. Requests are independent tasks, and a Semaphore limits concurrency (default 8).

2.3 Core types and the async HTTP layer

pub struct Request {
    pub name: Option<String>,          // from `# @name` or `### Title`
    pub method: Method,                // GET, POST, … or Custom(String)
    pub url: String,                   // may contain {{vars}}; query lives in the URL
    pub headers: Vec<Header>,          // Header { name, value, disabled }  (disabled = commented out)
    pub body: Body,                    // None | Text(String) | FileRef(PathBuf)  (`< ./file.json`)
                                       //   | Multipart(Vec<Part>)
    pub directives: Directives,        // @no-redirect, @no-cookie-jar, @timeout, @connection-timeout …
    pub source: SourceSpan,            // file + byte range, for lossless rewrite
}

pub trait HttpClient: Send + Sync {
    async fn send(&self, req: ResolvedRequest, cancel: CancellationToken)
        -> Result<Response, HttpError>;
}

pub struct Response {
    pub status: u16, pub http_version: String,
    pub headers: Vec<(String, String)>,
    pub body: BodyStore,               // InMemory(Bytes) under a threshold, else TempFile(PathBuf)
    pub timings: Timings,              // dns, connect, tls, ttfb, total
    pub size: u64,
}

Making libcurl async: two steps behind the same trait.

  1. v1: tokio::task::spawn_blocking around a curl Easy2 handle.
    • The async side does tokio::select! on the blocking join handle and cancel.cancelled().
    • On cancel, it sets an AtomicBool that curl's progress callback checks, which aborts the transfer.
    • This is simple, well understood, and works for the dozens of concurrent requests a desktop app needs.
  2. Later, if needed: a curl Multi driver. One dedicated thread runs Multi::perform/wait, takes new handles over a channel, and completes each request through a oneshot. This gives true event-driven I/O and fewer threads, and the trait signature does not change.

The ready-made option is isahc (an async client built on libcurl). Check its maintenance state and Schannel/static-build story in M1 before choosing it over the small wrapper above.

ResolvedRequest is the request after variable substitution. It is the only thing the HTTP layer sees, which keeps http-curl simple and makes "copy as curl" exact.

2.4 On-disk format: .http files

A collection is a folder, and each .http file is a group of related requests. The folder tree in the sidebar mirrors the folders and files on disk.

my-api/
  http-client.env.json            # committed: shared environments
  http-client.private.env.json    # git-ignored: secrets, same shape
  .reqlite/collection.toml        # optional: sidebar ordering, app-only settings
  users.http
  orders/
    create-order.http
    payloads/order.json
### List users
GET {{base_url}}/users?limit=20
Accept: application/json

### Create user
# @name create-user
# @timeout 30
POST {{base_url}}/users
Content-Type: application/json
Authorization: Bearer {{token}}

{ "name": "Ada" }

### Upload avatar
POST {{base_url}}/users/1/avatar
Content-Type: application/octet-stream

< ./payloads/avatar.png
// http-client.env.json
{ "dev":  { "base_url": "http://localhost:8080" },
  "prod": { "base_url": "https://api.example.com" } }

What is supported (the subset that JetBrains and VS Code REST Client largely agree on):

  • ### request separators with an optional title.
  • Request line METHOD URL [HTTP/version], with query parameters allowed on continuation lines that start with ? or &.
  • Headers, then a blank line, then the body.
  • {{var}} placeholders.
  • File-level variables (@base_url = …).
  • < ./file body references.
  • multipart/form-data with boundaries.
  • # @name and # @no-redirect, # @no-cookie-jar, # @timeout, # @connection-timeout.
  • # and // comments.

Lossless round-trip is the key rule. The parser keeps a concrete syntax tree, including comments, blank lines, unknown directives, and response-handler blocks (> {% … %} and > ./script.js). Editing one request in the UI rewrites only that request's byte span. A file the app does not fully understand still survives an edit untouched. These blocks are shown read-only in the UI, and scripts are not run.

Things the format doesn't have, and where they go instead:

  • Auth. Auth is just headers (Authorization: Bearer {{token}}). The Auth tab is a helper that writes the header. For example, Basic auth writes Authorization: Basic {{user}} {{password}}, which JetBrains encodes automatically, and reqlite does the same.
  • Inherited auth and default headers per folder. These are skipped in v1. Use environment variables instead.
  • Stable request IDs. The identity of a request is its file plus its @name or title, and app-side state (history, open tabs) refers to that.

2.5 App data (not in the collection)

App data lives in directories::ProjectDirs and holds:

  • recent collections
  • window and layout state (eframe persistence)
  • request history (an append-only JSON Lines file, capped)
  • logs

3. Milestones

Each milestone ends with just check passing and something you can demo. The size estimates are in "focused evenings" and are rough.

M0: Skeleton and local tooling (1–2)

  • Create the workspace, toolchain pin, deny.toml, and empty crates.
  • Add a justfile (or a small xtask crate if you prefer no extra tool):
    • just check runs fmt, clippy, all tests, and cargo deny.
    • just release makes a static release build.
    • Optionally add a git pre-push hook that runs just check.
  • Prove the static libcurl build on Windows now (static-curl needs a C compiler; the MSVC Build Tools are enough).
  • main starts a Tokio runtime next to eframe, and a dummy async task updates a label, which proves the repaint wiring works.
  • Done when: just check passes on your Windows machine, and the release exe opens an egui window fed by an async task when copied to another Windows machine without Rust installed. Linux and Mac builds are checked by hand when you get to them.

M1: .http parser and async HTTP, headless (4–6)

  • core::httpfile: a lexer and parser into a concrete syntax tree, a writer, and Request extraction.
  • core::env: load public and private env files, plus file-level variables.
  • http-curl:
    • HttpClient using spawn_blocking around a curl Easy2 handle, as in section 2.3.
    • Bodies, redirects, timeouts, TLS verify toggle, proxy, HTTP/2, timings.
    • Spill the body to a temp file above 5 MB.
    • Cancellation through CancellationToken.
  • cli run <file.http> [--name X] --env dev prints the status, headers and body. This makes the core usable before any UI exists.
  • Tests:
    • Parser:
      • a fixture corpus with snapshots of the parsed tree;
      • round-trip parse → write is byte-identical for every fixture;
      • fuzzing with cargo-fuzz on the parser (optional, cheap to add).
    • HTTP: wiremock integration tests for each method and body type, redirects, timeout, cancel mid-transfer, and 20 concurrent requests.
    • TLS: a test with a self-signed certificate.
  • Done when: reqlite run users.http --env dev works against real APIs on Windows, and the same file runs in JetBrains or VS Code.

M2: Minimal UI, one request (3–4)

  • Layout:
    • top bar with method, URL and Send/Cancel
    • tabs for Params, Headers, Body and Auth (the Auth helper writes a header)
    • a response panel with status, time, size, and tabs for Body, Headers and Timings
  • Use the AppState, Action, update and Effect pattern with Tokio effects from section 2.2.
  • The URL and the Params table stay in sync both ways.
  • Response body:
    • pretty-printed JSON with syntax highlighting
    • a Raw mode
    • images shown inline
    • a "Save to file" option for anything else
  • A Source tab shows the request as .http text and can be edited directly. Edits there and in the form views stay in sync through the parser.
  • Tests:
    • update() unit tests for URL and params sync, send and receive, and cancel.
    • One egui_kittest smoke test using a fake async HttpClient.
  • Done when: you can use it for day-to-day one-off requests.

M3: Collections (3–4)

  • Scan a folder asynchronously for .http files and build the sidebar tree: folder, then file, then requests.
  • Create, rename, delete and duplicate requests and files. Moving a request between files is a cut-and-insert of its span.
  • Multiple open tabs with a dirty marker, Ctrl+S to save (a span rewrite only), and a prompt on close when there are unsaved changes.
  • Watch the folder with notify, run through Tokio, to reload files changed outside the app, for example after git pull or an edit in VS Code. If the open tab has unsaved changes, show a conflict prompt.
  • Tests:
    • Editing one request leaves the rest of the file byte-identical.
    • Saving an unchanged collection writes nothing.
    • A watcher test with a temp directory.
  • Done when: a collection can be committed to git, diffs read cleanly, and the same files work in your IDE.

M4: Environments and variables (2–3)

  • template.rs parses {{name}} with clear errors for unknown variables or unclosed braces. It resolves in this order:
    1. private environment
    2. public environment
    3. file variables
    4. built-ins: {{$uuid}}, {{$timestamp}}, {{$randomInt}}, which match JetBrains' $-prefixed dynamic variables
  • An environment picker in the top bar, and an environment editor that writes the JSON files.
  • Values from the private file show as •••• in the UI, and the app offers to add the private file to .gitignore if it is missing.
  • Unresolved variables are highlighted in the URL and header fields, and hovering a variable shows its value.
  • Tests: table-driven tests for template.rs covering precedence, missing variables, escaping and built-ins.
  • Done when: the same collection works against dev and prod by switching environments.

M5: Import and export (3–4)

  • Import from curl:
    • Paste a curl command into the URL bar (detect curl at the start), and it becomes .http text.
    • Tokenize with shell-words, plus handling for Windows ^ line continuations.
    • Support -X, -H, -d/--data*, -F, -u, -k, -L, --compressed, and -b.
  • Copy as curl comes from the ResolvedRequest, and copy as .http is the raw request span.
  • Postman Collection v2.1 import:
    • Writes .http files plus http-client.env.json.
    • Folders become directories and requests become ### blocks.
    • Pre-request and test scripts are skipped, and the import summary lists them as warnings.
  • Tests:
    • A fixtures folder of real-world curl commands (copied from Chrome and Firefox devtools on Windows and Unix) with snapshot results.
    • A few Postman exports, with snapshots of the generated .http files.
  • Done when: you can migrate your existing Postman collections.

M6: Everyday features (3–4)

  • Auth helpers for Basic and Bearer. OAuth2 is post-v1.

  • Cookie jar per collection, held in memory, respecting @no-cookie-jar, with a viewer.

  • A history panel (last N requests sent), with a button to restore one into a new tab.

  • Settings: default timeout, proxy (system, none or manual), TLS (system store, custom CA file), and font size and theme.

  • Keyboard shortcuts:

    Shortcut Action
    Ctrl+Enter Send
    Ctrl+T New tab
    Ctrl+W Close tab
    Ctrl+L Focus the URL bar
    Ctrl+P Quick-open a request
  • Done when: you no longer need Postman for your normal work.

M7: Robustness and release (3–4)

  • Large responses:
    • Virtualized line viewer (ScrollArea::show_rows).
    • JSON is pretty-printed in a spawn_blocking task, with a "show raw" option.
    • Search in the response with Ctrl+F.
  • Memory check: measure idle and after-large-response RAM on Windows (the Tokio runtime adds a few MB at most), and note the binary size per release.
  • Windows packaging:
    • Embed an icon and version info with winres.
    • Choose a portable zip and/or an MSI via cargo-wix.
    • Code signing is optional; without it, SmartScreen shows a warning.
  • Linux and Mac: AppImage or tarball, and a .app bundle via cargo-bundle.
  • Releases: run just release on each OS by hand, tag the commit, and upload the files (for example to a GitHub Release). A release workflow can be added if CI comes later.
  • Done when: v0.1.0 is downloadable and runs on a clean Windows machine with no extra installs.

Total: roughly 25–35 focused evenings to v0.1.0. The .http parser adds a little to M1, but M3 gets simpler because there is no custom format to design.

Post-v1 backlog (in rough priority order)

  • A curl Multi driver behind the same HttpClient trait (see section 2.3), if many concurrent requests ever matter.
  • OAuth2 (authorization code + PKCE, client credentials) with token caching.
  • Per-folder default headers and auth (an app-side extension in .reqlite/).
  • Request chaining via {{create-user.response.body.$.id}}. This is VS Code REST Client syntax; JetBrains does chaining through scripts.
  • OpenAPI import (generate .http files from a spec).
  • WebSocket and Server-Sent Events (these fit naturally now that the app is async).
  • GraphQL body type with schema fetch.
  • Response diff between two history entries.
  • Command palette.

Deliberately out of scope: running response-handler scripts, test runners, mock servers and cloud sync.


4. Testing strategy summary

Layer How When
core parser and writer A fixture corpus, insta snapshots, byte-identical round-trips, optional fuzzing just check
core template, env, import/export Unit and table-driven tests, snapshots just check
http-curl #[tokio::test] against a wiremock server; cancel and concurrency tests; self-signed TLS test just check
App logic Call update(state, action) and assert on state and effects; run effects against a fake async HttpClient just check
UI A few egui_kittest smoke tests per panel just check
End to end cli run against fixture .http files and wiremock just check
Compatibility Fixture files that are checked to also run in JetBrains and VS Code REST Client (manual, per release) Per release
Manual A release checklist: Windows clean VM, corporate proxy, HiDPI/multi-monitor Per release

Rule of thumb: if a bug can be reproduced without a window, the regression test goes in core or in update(), not in the UI tests.


5. Risks and mitigations

Risk Mitigation
The static libcurl build breaks on Windows or Mac Proved in M0 on your own machine, and rechecked on each OS before a release. A fallback reqwest implementation of HttpClient (natively async) is possible thanks to the trait.
Mixing Tokio and the egui event loop The runtime runs on its own threads, the UI never blocks on futures, and results arrive over a channel plus request_repaint. This is proven in M0.
Cancellation not stopping curl promptly The progress callback checks the cancel flag every call. A test cancels a slow wiremock response and checks that the task ends within 200 ms.
.http dialect differences between JetBrains and VS Code Support the common subset, keep unknown syntax verbatim (lossless parser), and document which dialect features are recognized.
Lossless parsing is harder than parse-and-reformat A concrete syntax tree with byte spans from the start, and a byte-identical round-trip test over every fixture in just check.
egui text editing is too limited for large bodies Body editing is capped and large bodies are view-only. Evaluate a code-editor crate only if it actually hurts.
egui breaking changes between versions Pin versions and upgrade deliberately. UI code is isolated in app/src/ui.
Scope creep toward "Postman" The out-of-scope list above. Scripts in .http files are preserved but never run.

6. First concrete steps

  1. Create the repository, the workspace skeleton and the justfile (M0).
  2. Build a static Windows exe where an async task updates the egui window, and check that it runs on a machine without Rust installed.
  3. Collect 15–20 real .http fixture files (your own requests, plus examples from the JetBrains and VS Code REST Client docs), then write the parser test-first against them (M1).