Files
reqlite/implementation-plan.md
T

375 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Implementation plan: Rust + egui API client
This plan builds on [tech-stack.md](tech-stack.md). It has three parts:
1. **Decisions** that are fixed up front, so later work doesn't revisit them.
2. **Architecture**: how the code is split up and how data flows through it.
3. **Milestones**: small vertical slices, each ending in something you can run and test.
Working name: `reqlite` (rename freely).
**Revision 3:** async with Tokio and `.http` files as the storage format (revision 2), and no CI for now (revision 3). Quality checks run locally with `just check`, and release builds are made by hand. All three were requested by Torbjörn.
---
## 1. Decisions
| Topic | Decision | Why |
|---|---|---|
| Language / edition | Rust stable, edition 2021, pinned with `rust-toolchain.toml` | Builds are reproducible on every machine. |
| UI | `eframe` + `egui` + `egui_extras` (syntax highlighting, tables) | Immediate mode keeps all state in plain structs, and one exe covers all platforms. |
| Renderer | eframe's default `wgpu` backend, with `glow` as a feature flag | `glow` is a fallback for old GPUs and VMs. |
| Async runtime | `tokio` (multi-thread runtime on background threads) + `tokio-util` (`CancellationToken`) | The UI thread stays egui's; all I/O (HTTP, file loading, watching) runs as Tokio tasks. |
| HTTP | `curl` crate with features `static-curl`, `http2`, `ssl` (Schannel on Windows, OpenSSL/rustls elsewhere), behind an async trait | It is faithful to how curl behaves, gives timings, and has a single binary with no DLLs. See section 2.3 for how it becomes async. |
| Async trait | Native `async fn` in traits, plus `async-trait` only where `dyn HttpClient` is needed | |
| Serialization | Own `.http` parser/writer in `core`; `serde_json` for environments, JSON bodies and Postman import; `toml` for app settings | |
| Storage format | `.http` files (JetBrains HTTP Client / VS Code REST Client syntax), with environments in `http-client.env.json` | Files are git friendly, human readable, and also open in JetBrains IDEs and VS Code. See section 2.4. |
| Errors | `thiserror` in libraries, `anyhow` only in the binaries | |
| Logging | `tracing` + `tracing-subscriber`, logged to a file in the app data directory | Bug reports can include the log. |
| Paths | `directories` crate for config, data and cache dirs | |
| Secrets | `http-client.private.env.json` (git-ignored, the convention both IDEs use), optionally backed by `keyring` | |
| File dialogs | `rfd` (async API) | Native dialogs on every platform. |
| Testing | `cargo test` + `#[tokio::test]`, `insta` (snapshots), `wiremock` (async HTTP mock server), `egui_kittest` (UI) | |
| Lint / format | `cargo fmt --check`, `cargo clippy -- -D warnings`, `cargo deny` (licenses, advisories), run locally through one `check` command (see below) | There is no CI, so a single local command keeps the bar consistent. |
| License | MIT or Apache-2.0 (your call) | It is compatible with egui and curl. |
---
## 2. Architecture
### 2.1 Workspace layout
```
reqlite/
Cargo.toml # [workspace], shared deps in [workspace.dependencies]
rust-toolchain.toml
deny.toml
justfile # `just check` = fmt + clippy + test + deny; `just release` = static release build
crates/
core/ # no UI, no network, no global state, no tokio dependency
src/model.rs # Request, Header, Body, Environment, Collection
src/template.rs # {{var}} parsing and resolution
src/http.rs # trait HttpClient (async), ResolvedRequest, Response, Timings, HttpError
src/httpfile/ # .http lexer, parser, writer (lossless round-trip)
src/env.rs # http-client.env.json + private.env.json
src/collection.rs # scan a folder of .http files into a tree
src/import/ # curl.rs, postman.rs
src/export/ # curl.rs
http-curl/ # impl HttpClient for libcurl (uses tokio)
app/ # egui binary
src/main.rs # builds the tokio runtime, starts eframe
src/state.rs # AppState (pure data)
src/action.rs # enum Action, fn update(&mut AppState, Action) -> Vec<Effect>
src/effects.rs # spawns Effects as tokio tasks; results come back as Actions
src/ui/ # one module per panel; reads state, emits Actions
cli/ # `reqlite run api/users.http --env dev [--name create-user]`
tests/fixtures/ # sample .http files, env files, Postman exports, curl commands
```
**Dependency rule:** `app` and `cli` depend on `core` and `http-curl`, and `core` depends on neither. Enforce this by keeping UI, curl and Tokio crates out of `core`'s `Cargo.toml`. `core` only defines the async trait and stays runtime-agnostic.
### 2.2 Data flow in the app (keeps the UI testable)
```
UI panels ──Action──▶ update(state, action) ──Effect──▶ effects runner (tokio::spawn)
▲ │ │ HTTP, disk, dialogs, file watching
└──── reads AppState ◀────┘◀── Action via mpsc + ctx.request_repaint() ──┘
```
**Runtime setup.** `main` builds a multi-thread Tokio runtime and keeps its `Handle` in the effects runner. eframe keeps the main thread. Do not use `#[tokio::main]` around eframe, because winit needs the main thread on macOS.
**Pure update.** UI code only draws state and pushes `Action`s. `update` is a pure, synchronous function: it changes state and returns `Effect`s such as `Effect::SendRequest { tab_id, request, cancel }`, `Effect::SaveFile { .. }` or `Effect::LoadCollection(path)`. Most behaviour is tested by calling `update` directly.
**Running effects.** The effects runner does `handle.spawn(async move { ... })` for each effect and sends the resulting `Action` back through a `tokio::sync::mpsc::UnboundedSender`. Each frame, the UI drains the matching receiver with `try_recv` (non-blocking). The task calls `egui::Context::request_repaint()` after sending, so the UI wakes immediately.
**Cancellation.** Each in-flight request gets a `CancellationToken`, and `Action::CancelClicked` calls `.cancel()`. The HTTP layer races the transfer against `token.cancelled()`.
**Concurrency.** Several tabs can send at once. Requests are independent tasks, and a `Semaphore` limits concurrency (default 8).
### 2.3 Core types and the async HTTP layer
```rust
pub struct Request {
pub name: Option<String>, // from `# @name` or `### Title`
pub method: Method, // GET, POST, … or Custom(String)
pub url: String, // may contain {{vars}}; query lives in the URL
pub headers: Vec<Header>, // Header { name, value, disabled } (disabled = commented out)
pub body: Body, // None | Text(String) | FileRef(PathBuf) (`< ./file.json`)
// | Multipart(Vec<Part>)
pub directives: Directives, // @no-redirect, @no-cookie-jar, @timeout, @connection-timeout …
pub source: SourceSpan, // file + byte range, for lossless rewrite
}
pub trait HttpClient: Send + Sync {
async fn send(&self, req: ResolvedRequest, cancel: CancellationToken)
-> Result<Response, HttpError>;
}
pub struct Response {
pub status: u16, pub http_version: String,
pub headers: Vec<(String, String)>,
pub body: BodyStore, // InMemory(Bytes) under a threshold, else TempFile(PathBuf)
pub timings: Timings, // dns, connect, tls, ttfb, total
pub size: u64,
}
```
**Making libcurl async: two steps behind the same trait.**
1. **v1: `tokio::task::spawn_blocking` around a curl `Easy2` handle.**
- The async side does `tokio::select!` on the blocking join handle and `cancel.cancelled()`.
- On cancel, it sets an `AtomicBool` that curl's progress callback checks, which aborts the transfer.
- This is simple, well understood, and works for the dozens of concurrent requests a desktop app needs.
2. **Later, if needed: a curl `Multi` driver.** One dedicated thread runs `Multi::perform`/`wait`, takes new handles over a channel, and completes each request through a `oneshot`. This gives true event-driven I/O and fewer threads, and the trait signature does not change.
The ready-made option is `isahc` (an async client built on libcurl). Check its maintenance state and Schannel/static-build story in M1 before choosing it over the small wrapper above.
`ResolvedRequest` is the request after variable substitution. It is the only thing the HTTP layer sees, which keeps `http-curl` simple and makes "copy as curl" exact.
### 2.4 On-disk format: `.http` files
A collection is a folder, and each `.http` file is a group of related requests. The folder tree in the sidebar mirrors the folders and files on disk.
```
my-api/
http-client.env.json # committed: shared environments
http-client.private.env.json # git-ignored: secrets, same shape
.reqlite/collection.toml # optional: sidebar ordering, app-only settings
users.http
orders/
create-order.http
payloads/order.json
```
```http
### List users
GET {{base_url}}/users?limit=20
Accept: application/json
### Create user
# @name create-user
# @timeout 30
POST {{base_url}}/users
Content-Type: application/json
Authorization: Bearer {{token}}
{ "name": "Ada" }
### Upload avatar
POST {{base_url}}/users/1/avatar
Content-Type: application/octet-stream
< ./payloads/avatar.png
```
```json
// http-client.env.json
{ "dev": { "base_url": "http://localhost:8080" },
"prod": { "base_url": "https://api.example.com" } }
```
**What is supported** (the subset that JetBrains and VS Code REST Client largely agree on):
- `###` request separators with an optional title.
- Request line `METHOD URL [HTTP/version]`, with query parameters allowed on continuation lines that start with `?` or `&`.
- Headers, then a blank line, then the body.
- `{{var}}` placeholders.
- File-level variables (`@base_url = …`).
- `< ./file` body references.
- `multipart/form-data` with boundaries.
- `# @name` and `# @no-redirect`, `# @no-cookie-jar`, `# @timeout`, `# @connection-timeout`.
- `#` and `//` comments.
**Lossless round-trip is the key rule.** The parser keeps a concrete syntax tree, including comments, blank lines, unknown directives, and response-handler blocks (`> {% … %}` and `> ./script.js`). Editing one request in the UI rewrites only that request's byte span. A file the app does not fully understand still survives an edit untouched. These blocks are shown read-only in the UI, and scripts are not run.
**Things the format doesn't have, and where they go instead:**
- **Auth.** Auth is just headers (`Authorization: Bearer {{token}}`). The Auth tab is a helper that writes the header. For example, Basic auth writes `Authorization: Basic {{user}} {{password}}`, which JetBrains encodes automatically, and reqlite does the same.
- **Inherited auth and default headers per folder.** These are skipped in v1. Use environment variables instead.
- **Stable request IDs.** The identity of a request is its file plus its `@name` or title, and app-side state (history, open tabs) refers to that.
### 2.5 App data (not in the collection)
App data lives in `directories::ProjectDirs` and holds:
- recent collections
- window and layout state (eframe persistence)
- request history (an append-only JSON Lines file, capped)
- logs
---
## 3. Milestones
Each milestone ends with `just check` passing and something you can demo. The size estimates are in "focused evenings" and are rough.
### M0: Skeleton and local tooling (1–2)
- Create the workspace, toolchain pin, `deny.toml`, and empty crates.
- Add a `justfile` (or a small `xtask` crate if you prefer no extra tool):
- `just check` runs fmt, clippy, all tests, and `cargo deny`.
- `just release` makes a static release build.
- Optionally add a git pre-push hook that runs `just check`.
- **Prove the static libcurl build on Windows now** (`static-curl` needs a C compiler; the MSVC Build Tools are enough).
- `main` starts a Tokio runtime next to eframe, and a dummy async task updates a label, which proves the repaint wiring works.
- **Done when:** `just check` passes on your Windows machine, and the release exe opens an egui window fed by an async task when copied to another Windows machine without Rust installed. Linux and Mac builds are checked by hand when you get to them.
### M1: `.http` parser and async HTTP, headless (4–6)
- **`core::httpfile`:** a lexer and parser into a concrete syntax tree, a writer, and `Request` extraction.
- **`core::env`:** load public and private env files, plus file-level variables.
- **`http-curl`:**
- `HttpClient` using `spawn_blocking` around a curl `Easy2` handle, as in section 2.3.
- Bodies, redirects, timeouts, TLS verify toggle, proxy, HTTP/2, timings.
- Spill the body to a temp file above 5 MB.
- Cancellation through `CancellationToken`.
- **`cli run <file.http> [--name X] --env dev`** prints the status, headers and body. This makes the core usable before any UI exists.
- **Tests:**
- **Parser:**
- a fixture corpus with snapshots of the parsed tree;
- **round-trip `parse → write` is byte-identical for every fixture**;
- fuzzing with `cargo-fuzz` on the parser (optional, cheap to add).
- **HTTP:** `wiremock` integration tests for each method and body type, redirects, timeout, cancel mid-transfer, and 20 concurrent requests.
- **TLS:** a test with a self-signed certificate.
- **Done when:** `reqlite run users.http --env dev` works against real APIs on Windows, and the same file runs in JetBrains or VS Code.
### M2: Minimal UI, one request (3–4)
- Layout:
- top bar with method, URL and Send/Cancel
- tabs for Params, Headers, Body and Auth (the Auth helper writes a header)
- a response panel with status, time, size, and tabs for Body, Headers and Timings
- Use the `AppState`, `Action`, `update` and `Effect` pattern with Tokio effects from section 2.2.
- The URL and the Params table stay in sync both ways.
- Response body:
- pretty-printed JSON with syntax highlighting
- a Raw mode
- images shown inline
- a "Save to file" option for anything else
- A **Source** tab shows the request as `.http` text and can be edited directly. Edits there and in the form views stay in sync through the parser.
- **Tests:**
- `update()` unit tests for URL and params sync, send and receive, and cancel.
- One `egui_kittest` smoke test using a fake async `HttpClient`.
- **Done when:** you can use it for day-to-day one-off requests.
### M3: Collections (3–4)
- Scan a folder asynchronously for `.http` files and build the sidebar tree: folder, then file, then requests.
- Create, rename, delete and duplicate requests and files. Moving a request between files is a cut-and-insert of its span.
- Multiple open tabs with a dirty marker, Ctrl+S to save (a span rewrite only), and a prompt on close when there are unsaved changes.
- Watch the folder with `notify`, run through Tokio, to reload files changed outside the app, for example after `git pull` or an edit in VS Code. If the open tab has unsaved changes, show a conflict prompt.
- **Tests:**
- Editing one request leaves the rest of the file byte-identical.
- Saving an unchanged collection writes nothing.
- A watcher test with a temp directory.
- **Done when:** a collection can be committed to git, diffs read cleanly, and the same files work in your IDE.
### M4: Environments and variables (2–3)
- `template.rs` parses `{{name}}` with clear errors for unknown variables or unclosed braces. It resolves in this order:
1. private environment
2. public environment
3. file variables
4. built-ins: `{{$uuid}}`, `{{$timestamp}}`, `{{$randomInt}}`, which match JetBrains' `$`-prefixed dynamic variables
- An environment picker in the top bar, and an environment editor that writes the JSON files.
- Values from the private file show as `••••` in the UI, and the app offers to add the private file to `.gitignore` if it is missing.
- Unresolved variables are highlighted in the URL and header fields, and hovering a variable shows its value.
- **Tests:** table-driven tests for `template.rs` covering precedence, missing variables, escaping and built-ins.
- **Done when:** the same collection works against dev and prod by switching environments.
### M5: Import and export (3–4)
- **Import from curl:**
- Paste a curl command into the URL bar (detect `curl ` at the start), and it becomes `.http` text.
- Tokenize with `shell-words`, plus handling for Windows `^` line continuations.
- Support `-X`, `-H`, `-d`/`--data*`, `-F`, `-u`, `-k`, `-L`, `--compressed`, and `-b`.
- **Copy as curl** comes from the `ResolvedRequest`, and **copy as `.http`** is the raw request span.
- **Postman Collection v2.1 import:**
- Writes `.http` files plus `http-client.env.json`.
- Folders become directories and requests become `###` blocks.
- Pre-request and test scripts are skipped, and the import summary lists them as warnings.
- **Tests:**
- A fixtures folder of real-world curl commands (copied from Chrome and Firefox devtools on Windows and Unix) with snapshot results.
- A few Postman exports, with snapshots of the generated `.http` files.
- **Done when:** you can migrate your existing Postman collections.
### M6: Everyday features (3–4)
- Auth helpers for Basic and Bearer. OAuth2 is post-v1.
- Cookie jar per collection, held in memory, respecting `@no-cookie-jar`, with a viewer.
- A history panel (last N requests sent), with a button to restore one into a new tab.
- Settings: default timeout, proxy (system, none or manual), TLS (system store, custom CA file), and font size and theme.
- Keyboard shortcuts:
| Shortcut | Action |
|---|---|
| Ctrl+Enter | Send |
| Ctrl+T | New tab |
| Ctrl+W | Close tab |
| Ctrl+L | Focus the URL bar |
| Ctrl+P | Quick-open a request |
- **Done when:** you no longer need Postman for your normal work.
### M7: Robustness and release (3–4)
- **Large responses:**
- Virtualized line viewer (`ScrollArea::show_rows`).
- JSON is pretty-printed in a `spawn_blocking` task, with a "show raw" option.
- Search in the response with Ctrl+F.
- **Memory check:** measure idle and after-large-response RAM on Windows (the Tokio runtime adds a few MB at most), and note the binary size per release.
- **Windows packaging:**
- Embed an icon and version info with `winres`.
- Choose a portable zip and/or an MSI via `cargo-wix`.
- Code signing is optional; without it, SmartScreen shows a warning.
- **Linux and Mac:** AppImage or tarball, and a `.app` bundle via `cargo-bundle`.
- **Releases:** run `just release` on each OS by hand, tag the commit, and upload the files (for example to a GitHub Release). A release workflow can be added if CI comes later.
- **Done when:** v0.1.0 is downloadable and runs on a clean Windows machine with no extra installs.
**Total: roughly 25–35 focused evenings to v0.1.0.** The `.http` parser adds a little to M1, but M3 gets simpler because there is no custom format to design.
### Post-v1 backlog (in rough priority order)
- A curl `Multi` driver behind the same `HttpClient` trait (see section 2.3), if many concurrent requests ever matter.
- OAuth2 (authorization code + PKCE, client credentials) with token caching.
- Per-folder default headers (an app-side extension in `.reqlite/`; per-folder auth is done).
- Request chaining via `{{create-user.response.body.$.id}}`. This is VS Code REST Client syntax; JetBrains does chaining through scripts.
- OpenAPI import (generate `.http` files from a spec).
- WebSocket and Server-Sent Events (these fit naturally now that the app is async).
- GraphQL body type with schema fetch.
- Response diff between two history entries.
- Command palette.
Deliberately out of scope: running response-handler scripts, test runners, mock servers and cloud sync.
---
## 4. Testing strategy summary
| Layer | How | When |
|---|---|---|
| `core` parser and writer | A fixture corpus, `insta` snapshots, byte-identical round-trips, optional fuzzing | `just check` |
| `core` template, env, import/export | Unit and table-driven tests, snapshots | `just check` |
| `http-curl` | `#[tokio::test]` against a `wiremock` server; cancel and concurrency tests; self-signed TLS test | `just check` |
| App logic | Call `update(state, action)` and assert on state and effects; run effects against a fake async `HttpClient` | `just check` |
| UI | A few `egui_kittest` smoke tests per panel | `just check` |
| End to end | `cli run` against fixture `.http` files and `wiremock` | `just check` |
| Compatibility | Fixture files that are checked to also run in JetBrains and VS Code REST Client (manual, per release) | Per release |
| Manual | A release checklist: Windows clean VM, corporate proxy, HiDPI/multi-monitor | Per release |
**Rule of thumb:** if a bug can be reproduced without a window, the regression test goes in `core` or in `update()`, not in the UI tests.
---
## 5. Risks and mitigations
| Risk | Mitigation |
|---|---|
| The static libcurl build breaks on Windows or Mac | Proved in M0 on your own machine, and rechecked on each OS before a release. A fallback `reqwest` implementation of `HttpClient` (natively async) is possible thanks to the trait. |
| Mixing Tokio and the egui event loop | The runtime runs on its own threads, the UI never blocks on futures, and results arrive over a channel plus `request_repaint`. This is proven in M0. |
| Cancellation not stopping curl promptly | The progress callback checks the cancel flag every call. A test cancels a slow `wiremock` response and checks that the task ends within 200 ms. |
| `.http` dialect differences between JetBrains and VS Code | Support the common subset, keep unknown syntax verbatim (lossless parser), and document which dialect features are recognized. |
| Lossless parsing is harder than parse-and-reformat | A concrete syntax tree with byte spans from the start, and a byte-identical round-trip test over every fixture in `just check`. |
| egui text editing is too limited for large bodies | Body editing is capped and large bodies are view-only. Evaluate a code-editor crate only if it actually hurts. |
| egui breaking changes between versions | Pin versions and upgrade deliberately. UI code is isolated in `app/src/ui`. |
| Scope creep toward "Postman" | The out-of-scope list above. Scripts in `.http` files are preserved but never run. |
---
## 6. First concrete steps
1. Create the repository, the workspace skeleton and the `justfile` (M0).
2. Build a static Windows exe where an async task updates the egui window, and check that it runs on a machine without Rust installed.
3. Collect 15–20 real `.http` fixture files (your own requests, plus examples from the JetBrains and VS Code REST Client docs), then write the parser test-first against them (M1).