Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,17 +50,17 @@ jobs:
# Every database, or the ones the change reaches.
everything() {
echo engine=true
echo 'tests=-p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-oracle'
echo 'tests=-p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-engine -p reylo-oracle'
echo 'shards=["postgres","sqlite","mysql-1","mysql-2","mysql-slow"]'
}
if [ "$EVENT" != pull_request ]; then everything >> "$GITHUB_OUTPUT"; exit 0; fi
if ! git rev-parse -q --verify HEAD^2 > /dev/null; then everything >> "$GITHUB_OUTPUT"; exit 0; fi
# A renamed file counts at both paths, so a file moved out of a path that runs everything still does.
files=$(git diff --name-only --no-renames HEAD^1 HEAD)
echo "$files"
# Core, the oracle's own code, the build, the code lists the oracle reads, the validators' measurements
# the engine's tests read, and CI itself reach every database.
if grep -qE '^(engine/(core|build-support|wasm)/|engine/build\.sh$|oracle/src/|oracle/Cargo\.toml$|Cargo\.(toml|lock)$|rust-toolchain|src/error-codes-|wiki/json-schema-checks/|\.github/)' <<<"$files"; then
# Core, the crate the oracle answers through, the oracle's own code, the build, the code lists the
# oracle reads, the validators' measurements the engine's tests read, and CI itself reach every database.
if grep -qE '^(engine/(core|build-support|facade|wasm)/|engine/build\.sh$|oracle/src/|oracle/Cargo\.toml$|Cargo\.(toml|lock)$|rust-toolchain|src/error-codes-|wiki/json-schema-checks/|\.github/)' <<<"$files"; then
everything >> "$GITHUB_OUTPUT"
exit 0
fi
Expand Down
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,11 @@ Prefer nub over `node`, `bun`, npm, npx, pnpm and yarn: `nub install`, `nub add`

### Rust

One Cargo workspace at the root covers `engine/core`, `engine/postgres`, `engine/sqlite`, `engine/mysql`, `engine/build-support`, the entry crates in `engine/wasm/`, and `oracle/`: one `Cargo.lock` and one `target/`. It needs a rustup toolchain with the `wasm32-wasip1` target (`rustup target add wasm32-wasip1`; a distribution's `rustc` without it fails with "can't find crate for `std`") and wasi-sdk 34, named by `WASI_SDK_PATH` or unpacked under `engine/.wasi-sdk/`.
One Cargo workspace at the root covers `engine/core`, `engine/postgres`, `engine/sqlite`, `engine/mysql`, `engine/build-support`, `engine/facade` (`reylo-engine`, which a Rust host links: each database's engine behind a feature, and the answers to its needs over `tokio-postgres`, `rusqlite` or `mysql_async`), the entry crates in `engine/wasm/`, and `oracle/`: one `Cargo.lock` and one `target/`. It needs a rustup toolchain with the `wasm32-wasip1` target (`rustup target add wasm32-wasip1`; a distribution's `rustc` without it fails with "can't find crate for `std`") and wasi-sdk 34, named by `WASI_SDK_PATH` or unpacked under `engine/.wasi-sdk/`.

```sh
engine/build.sh --prepare # once per checkout, before any cargo command: cargo cannot resolve the workspace without it
cargo test -p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-oracle
cargo test -p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-engine -p reylo-oracle
engine/build.sh # rebuilds all three packages/<database>/engine.wasm
```

Expand Down Expand Up @@ -137,7 +137,7 @@ The docs are `site/content/docs/`: the main page `index.mdx`, which continues in
`.github/workflows/test.yml` runs every check on each push to `main` and on each release, and on a pull request the checks its change reaches, its run for an older commit cancelled:

- **test**, on every pull request: on Node 24 with throwaway Postgres 18 and MySQL 8.4, `nub install` at the root, in `packages/postgres`, `packages/mysql`, `lab/drivers` and `lab/jest`, the Jest lab's links, `nub run typecheck`, `nub run test` with none skipped, and `nub run build`.
- **engine**, on a pull request that changes `engine/`, `oracle/`, Cargo's files, `src/error-codes-*`, the JSON Schema validators' measurements the engine's tests read, or `.github/` (a renamed file at both its paths): `engine/build.sh --prepare`, `cargo build --workspace --all-targets` with no warning, and `cargo test` of the crates the change reaches (every one, `-p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-oracle`, where it changes core, the oracle's code, the build or CI; else the engine of each database it changes).
- **engine**, on a pull request that changes `engine/`, `oracle/`, Cargo's files, `src/error-codes-*`, the JSON Schema validators' measurements the engine's tests read, or `.github/` (a renamed file at both its paths): `engine/build.sh --prepare`, `cargo build --workspace --all-targets` with no warning, and `cargo test` of the crates the change reaches (every one, `-p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-engine -p reylo-oracle`, where it changes core, `engine/facade/`, the oracle's code, the build or CI; else the engine of each database it changes).
- **oracle**, on the same pull requests: `cargo run -q -p reylo-oracle -- --url <pg> --mysql-url <mysql> --strict-coverage <cases>`, in shards side by side (Postgres's cases, SQLite's, MySQL's in two, and `mysql_types`, the slowest, alone), each database's only where the change reaches it (`engine/<database>/` or one of its cases; everything for the files that run every engine check). Postgres 18 has pgvector, PostGIS and the five JSON Schema validators, which `.github/scripts/json-schema-validators.sh` builds and a cache keeps, with the packages of Postgres itself; MySQL is 8.4. A failure in the step that builds the validators is not the engine's.

Run what CI runs before you push. A failing check's log: `gh pr checks <number> -R colinhacks/reylo` names the run, and `gh run view <run id> -R colinhacks/reylo --log-failed` prints what failed. Pullfrog and Copilot review every pull request, and Vercel deploys a preview of the docs site.
Expand Down
18 changes: 15 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ resolver = "2"
members = [
"engine/build-support",
"engine/core",
"engine/facade",
"engine/postgres",
"engine/sqlite",
"engine/mysql",
Expand Down
3 changes: 2 additions & 1 deletion engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,15 @@ The engine is the SQL analysis: parsing, placeholders, types, nullability, neste
| `reylo-engine-sqlite` | `sqlite/` | SQLite's amalgamation patched with the engine's hooks (`csrc/`, compiled by `build.rs`), its resolved tree (`sqlite_tree`) and the analysis over it (`sqlite`) |
| `reylo-engine-mysql` | `mysql/` | Its own copy of `sqlparser` 0.63, cut to the statements it reads (`vendor/sqlparser`, whose README lists the changes), and the analysis over the server's prepare metadata (`mysql`, `mysql_shape`, `mysql_params`) |
| `reylo-engine-build` | `build-support/` | The source hash each crate's build script writes, which its fingerprint is made of |
| `reylo-engine` | `facade/` | For a Rust host: each database's engine behind a feature (`postgres`, `sqlite`, `mysql`), the replay loop of core's `replay`, and, behind a client feature (`tokio-postgres`, `rusqlite`, `mysql_async`), the answers to the engine's needs over that client and an `analyze` that runs a statement's whole replay on it |
| `reylo-engine-<database>-wasm` | `wasm/<database>/` | One line each: the ABI over one database, built to that database's artifact |

The `.sql` loader hosts it: `packages/<database>/engine.wasm`, a `wasm32-wasip1` reactor in the database's npm package `@reylosql/<database>`, which [src/engine.ts](../src/engine.ts) finds from the project and loads.

## Building

- `engine/build.sh --prepare` prepares the patched `pg_query` copy (`postgres/.patched/`) and the SQLite amalgamation (`sqlite/.sqlite/`). Every cargo command in the workspace needs it first.
- `cargo test -p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql` runs the engine's native tests.
- `cargo test -p reylo-engine-core -p reylo-engine-postgres -p reylo-engine-sqlite -p reylo-engine-mysql -p reylo-engine -p reylo-oracle` runs the engine's native tests (the oracle's dependency gives `reylo-engine` its three client features).
- `engine/build.sh` builds every database's artifact into `packages/<database>/engine.wasm` (`engine/build.sh postgres` builds one). It needs wasi-sdk 34 and a toolchain with the `wasm32-wasip1` target (rustup's has it; Homebrew's `rustc` does not).

The artifacts are committed so a checkout works without a Rust toolchain. Rebuild them with every change to the engine's sources: each crate's `build.rs` hashes its own sources, and a database's fingerprint combines core's hash with its own.
25 changes: 25 additions & 0 deletions engine/facade/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
[package]
name = "reylo-engine"
version = "0.0.0"
edition = "2021"
publish = false
description = "Reylo's analysis engine for a Rust host: each database's engine behind a feature, the replay loop that runs an analysis to its end, and the answers to the engine's needs over tokio-postgres, rusqlite or mysql_async."

[features]
postgres = ["dep:reylo-engine-postgres"]
sqlite = ["dep:reylo-engine-sqlite"]
mysql = ["dep:reylo-engine-mysql"]
# A client the engine's needs are answered over, which brings its database's engine.
tokio-postgres = ["postgres", "dep:tokio-postgres"]
rusqlite = ["sqlite", "dep:rusqlite"]
mysql_async = ["mysql", "dep:mysql_async"]

[dependencies]
reylo-engine-core = { path = "../core" }
reylo-engine-postgres = { path = "../postgres", optional = true }
reylo-engine-sqlite = { path = "../sqlite", optional = true }
reylo-engine-mysql = { path = "../mysql", optional = true }
serde_json = { version = "1", features = ["preserve_order", "unbounded_depth"] }
tokio-postgres = { version = "0.7", optional = true }
rusqlite = { version = "0.39", features = ["bundled"], optional = true }
mysql_async = { version = "0.37", default-features = false, features = ["minimal-rust"], optional = true }
10 changes: 10 additions & 0 deletions engine/facade/src/clients/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
//! The engine's needs answered over a database client: one module per client feature, each with
//! the [`crate::Answerer`] or [`crate::AsyncAnswerer`] for that client and an `analyze` that runs a
//! statement's whole replay on it.

#[cfg(feature = "mysql_async")]
pub mod mysql_async;
#[cfg(feature = "rusqlite")]
pub mod rusqlite;
#[cfg(feature = "tokio-postgres")]
pub mod tokio_postgres;
193 changes: 193 additions & 0 deletions engine/facade/src/clients/mysql_async.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
//! The MySQL engine's needs answered over a `mysql_async` connection: a describe from the
//! server's prepare, its column and parameter definitions as mysql2 reports them, and a catalog
//! query with its values written into the text, as mysql2's `query()` writes them.

use crate::{AnswerCache, Analysis, AsyncAnswerer, DbError, HostError, ReplayError};
use ::mysql_async::prelude::Queryable;
use ::mysql_async::{Conn, Value as My};
use reylo_engine_core::database::Database;
use reylo_engine_mysql::Mysql;
use serde_json::{json, Value};
use std::future::Future;

/// What a MySQL describe is answered with.
pub type Describe = <Mysql as Database>::Describe;

/// The analysis of `sql`, its needs answered over `conn`.
pub async fn analyze(conn: &mut Conn, sql: &str, cache: &mut AnswerCache<Describe>) -> Result<Analysis, ReplayError<HostError>> {
crate::replay::run_async::<Mysql, _>(sql, &mut Answers(conn), cache).await
}

/// The engine's needs, answered over a connection.
pub struct Answers<'a>(pub &'a mut Conn);

impl AsyncAnswerer<Describe> for Answers<'_> {
type Error = HostError;

fn describe(&mut self, sql: &str) -> impl Future<Output = Result<Result<Describe, DbError>, HostError>> + Send {
let conn = &mut *self.0;
async move {
match conn.prep(sql).await {
Ok(statement) => {
let columns: Vec<Value> = statement.columns().iter().map(|c| json!({ "name": c.name_str(), "type": c.column_type() as u8, "flags": c.flags().bits(), "charset": c.character_set(), "length": c.column_length(), "org_table": c.org_table_str(), "org_name": c.org_name_str() })).collect();
let params: Vec<Value> = statement.params().iter().map(|p| json!({ "type": p.column_type() as u8, "flags": p.flags().bits(), "charset": p.character_set() })).collect();
conn.close(statement).await.map_err(|e| HostError(e.to_string()))?;
serde_json::from_value(json!({ "columns": columns, "params": params })).map(Ok).map_err(|e| HostError(e.to_string()))
}
Err(::mysql_async::Error::Server(e)) => Ok(Err(DbError { message: e.message, code: error_name(e.code).map(String::from), position: None })),
Err(e) => Err(HostError(format!("the prepare did not reach the server: {e}"))),
}
}
}

fn query(&mut self, sql: &str, params: &[Value]) -> impl Future<Output = Result<Result<Vec<Vec<Value>>, DbError>, HostError>> + Send {
let conn = &mut *self.0;
async move {
let text = interpolate(sql, params).ok_or_else(|| HostError(format!("cannot write {params:?} into the catalog query\n {sql}")))?;
let mut result = conn.query_iter(text).await.map_err(|e| HostError(format!("catalog query failed: {}\n {sql}", message(&e))))?;
let types: Vec<u8> = result.columns_ref().iter().map(|c| c.column_type() as u8).collect();
let mut rows = Vec::new();
while let Some(row) = result.next().await.map_err(|e| HostError(format!("catalog query failed: {}\n {sql}", message(&e))))? {
rows.push(row.unwrap().into_iter().zip(&types).map(|(value, code)| decoded(value, *code)).collect());
}
Ok(Ok(rows))
}
}
}

/// A catalog query with its values written into the text, as mysql2's `query()` writes them
/// (`sqlstring`): each `?` in turn, a string quoted with its special characters escaped.
pub fn interpolate(sql: &str, params: &[Value]) -> Option<String> {
let mut out = String::with_capacity(sql.len());
let mut params = params.iter();
for c in sql.chars() {
if c != '?' {
out.push(c);
continue;
}
match params.next()? {
Value::Null => out.push_str("NULL"),
Value::Bool(b) => out.push_str(if *b { "true" } else { "false" }),
Value::Number(n) => out.push_str(&n.to_string()),
Value::String(s) => out.push_str(&quote(s)),
_ => return None,
}
}
params.next().is_none().then_some(out)
}

/// A string as a MySQL literal, by `sqlstring`'s escapes.
pub fn quote(s: &str) -> String {
let mut out = String::with_capacity(s.len() + 2);
out.push('\'');
for c in s.chars() {
match c {
'\0' => out.push_str("\\0"),
'\u{8}' => out.push_str("\\b"),
'\t' => out.push_str("\\t"),
'\n' => out.push_str("\\n"),
'\r' => out.push_str("\\r"),
'\u{1a}' => out.push_str("\\Z"),
'"' => out.push_str("\\\""),
'\'' => out.push_str("\\'"),
'\\' => out.push_str("\\\\"),
c => out.push(c),
}
}
out.push('\'');
out
}

/// One value of a text result as mysql2 hands it to the host: a number for the integer and
/// floating types, a string for the rest.
pub fn decoded(value: My, code: u8) -> Value {
match value {
My::NULL => Value::Null,
My::Bytes(bytes) => {
let text = String::from_utf8_lossy(&bytes).to_string();
match code {
1 | 2 | 3 | 8 | 9 | 13 => text.parse::<i64>().map(Value::from).or_else(|_| text.parse::<u64>().map(Value::from)).unwrap_or(Value::String(text)),
4 | 5 => text.parse::<f64>().map(Value::from).unwrap_or(Value::String(text)),
_ => Value::String(text),
}
}
My::Int(n) => json!(n),
My::UInt(n) => json!(n),
My::Float(f) => json!(f),
My::Double(f) => json!(f),
other => Value::String(other.as_sql(true)),
}
}

/// An error as a message: the server's with its code, any other as it prints itself.
pub fn message(e: &::mysql_async::Error) -> String {
match e {
::mysql_async::Error::Server(e) => format!("{} ({})", e.message, e.code),
other => other.to_string(),
}
}

/// The name mysql2 gives a server error (`err.code`), which the loader passes to the engine and
/// the engine prints: the errors a prepare of a statement answers with. An error not listed here
/// goes without a name, where the loader would have one.
fn error_name(code: u16) -> Option<&'static str> {
Some(match code {
1046 => "ER_NO_DB_ERROR",
1048 => "ER_BAD_NULL_ERROR",
1051 => "ER_BAD_TABLE_ERROR",
1052 => "ER_NON_UNIQ_ERROR",
1054 => "ER_BAD_FIELD_ERROR",
1055 => "ER_WRONG_FIELD_WITH_GROUP",
1056 => "ER_WRONG_GROUP_FIELD",
1060 => "ER_DUP_FIELDNAME",
1064 => "ER_PARSE_ERROR",
1066 => "ER_NONUNIQ_TABLE",
1093 => "ER_UPDATE_TABLE_USED",
1109 => "ER_UNKNOWN_TABLE",
1111 => "ER_INVALID_GROUP_FUNC_USE",
1136 => "ER_WRONG_VALUE_COUNT_ON_ROW",
1140 => "ER_MIX_OF_GROUP_FUNC_AND_FIELDS",
1146 => "ER_NO_SUCH_TABLE",
1210 => "ER_WRONG_ARGUMENTS",
1222 => "ER_WRONG_NUMBER_OF_COLUMNS_IN_SELECT",
1235 => "ER_NOT_SUPPORTED_YET",
1241 => "ER_OPERAND_COLUMNS",
1242 => "ER_SUBQUERY_NO_1_ROW",
1247 => "ER_ILLEGAL_REFERENCE",
1248 => "ER_DERIVED_MUST_HAVE_ALIAS",
1250 => "ER_TABLENAME_NOT_ALLOWED_HERE",
1267 => "ER_CANT_AGGREGATE_2COLLATIONS",
1295 => "ER_UNSUPPORTED_PS",
1305 => "ER_SP_DOES_NOT_EXIST",
1318 => "ER_SP_WRONG_NO_OF_ARGS",
1327 => "ER_SP_UNDECLARED_VAR",
1353 => "ER_VIEW_WRONG_LIST",
1356 => "ER_VIEW_INVALID",
1364 => "ER_NO_DEFAULT_FOR_FIELD",
1390 => "ER_PS_MANY_PARAM",
1582 => "ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT",
1583 => "ER_WRONG_PARAMETERS_TO_NATIVE_FCT",
1630 => "ER_FUNC_INEXISTENT_NAME_COLLISION",
3140 => "ER_INVALID_JSON_TEXT",
3143 => "ER_INVALID_JSON_PATH",
3554 => "ER_NO_SYSTEM_TABLE_ACCESS",
3579 => "ER_WINDOW_NO_SUCH_WINDOW",
3593 => "ER_WINDOW_INVALID_WINDOW_FUNC_USE",
_ => return None,
})
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn a_catalog_query_takes_its_values_as_mysql2_writes_them() {
assert_eq!(interpolate("select find_in_set(t, ?)", &[json!("users,it's\\")]).as_deref(), Some("select find_in_set(t, 'users,it\\'s\\\\')"));
assert_eq!(interpolate("select ?, ?", &[json!(1), Value::Null]).as_deref(), Some("select 1, NULL"));
assert_eq!(interpolate("select ?", &[]), None);
assert_eq!(decoded(My::Bytes(b"12".to_vec()), 8), json!(12));
assert_eq!(decoded(My::Bytes(b"12".to_vec()), 253), json!("12"));
assert_eq!(decoded(My::NULL, 253), Value::Null);
}
}
Loading
Loading