1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
       (‾‾)   /\   _                 (
        \ |  (  \ ( \.(               )                      _____
      \  \ \  `  `   ) \             (  ___                 / _   \
     (_`    \+   . x  ( .\            \/   \____-----------/ (☢️)   \_
    - .-               \+  ;          (  O                           \____
    (                       )          \_____________  `              \   /
    (__                +- .( -'.- <. - _  VVVVVVV VV V\                 \/
    (_____            ._._: <_ - <- _  (--  _AAAAAAA__A_/               |
      .    /./.+-  . .- /  +--  - .     \______________//_              \_______
      (__ ' /x  / x _/ (                                  \___'          \     /
     , x / ( '  . / .  /                                      |           \   /
        /  /  _/ /    +                                      /              \/
       '  (__/                                             /                  \

**[ Heavily WIP ]**

Tokamak is a cache coherent relational query execution and matrialization engine. Its purpose is to bridge the gap between high-latency, whole-dataset, background reactive denormalization pipelines and low-latency, request-oriented, foreground on-demand data retrieval. It is firmly oriented towards operating on a moderate amount of high value, relatively normalized, unwindowed, mutable data (most websites' primary databases) - not infinite streams or vast warehouses of immutable, isolated events or messages (as many systems in this space tend to be). It is conceptually inspired by [Dragon](https://engineering.fb.com/data-infrastructure/dragon-a-distributed-graph-query-engine/) and heavily structurally influenced by Presto.

Its primary goals and intended usecases include:
- Low-latency cache-accelerated on-demand retrieval of query results via HTTP given arbitrary per-request toplevel constraints for serving denormalized and transformed application data for frontend request processing
- Full-dataset materialized view creation and maintenance into target backends like Elasticsearch or Kafka
- Ability to freely mix fully-materialized and partially-materialized usecases with the same query plans through per-view configuration
- Reactive maintenance of all view data by receiving data change notifications of underlying source data from external sources like Kafka or simple HTTP POSTs
- Efficiency of reactive maintenance via storage of hidden internal intermediate state at multiple places in query plan
- Presto-like positioning as a storageless intermediate talking to configured external sources of truth on users' behalf
- Ability to use shared, space constrained storage for storing intermediate view state which can be selectively populated and evicted both reactively (due to source data change or storage pressure) and proactively (via external signals)

Its primary features [will] include:
- A primary query language of a simple subset of standard ANSI SQL
- Inline [Jmespath](http://jmespath.org/) expressions which are fully transparent first-class citizens to the planner
- Support for arbitrarily deeply nested relational joins, aggregations, and unnests via SQL
- Support for constructing, traversing, and modifying large, deeply nested document-like data via inline Jmespath
- UDF's from a variety of backends, including inline or preconfigured Java, JSR223, V8 (including support for bundled Browserify-ed NPM dependencies), local subprocesses, and remote HTTP calls - all usable in both SQL and Jmesepath
- Minor but powerful SQL semantic extensions like let-forms and concise edge table joins without requiring syntax changes (retaining external tool support)
- A type system supporting primitive types, Java collections, unions, type annotations, structs, structural types, and dynamic JSON
- Pluggable storage connectors with built-in support for JDBC, Redis, memcached, and Elasticsearch and bundled support for Kafka

Its planned future features include:
- Recursively defined named struct types
- Support for joining the current time as a single-row table
- Distributed execution of multiple nodes simultaneously servicing the same views
- Support for intra-query mutual recursion via CTE's
- Data-dependent state storage policies ('only cache users with more than 10 friends') and sharding policies
- Acceleration of large operations with Spark (as the core only depends on Guava, Guice, Jackson, and Antlr it is built to be shaded+relocateable)
- Support for online, in-place DDL

Additionally, the on-demand request handling side has some interesting and easily implemented capabilities:
- Support for multi-operation transactions via Websockets, with choice of read committed or repeatable read isolation semantics
- Ability to choose whether or not to persist any intermediate state built during the request (gaining in-request efficiency by not doing so at the expense of future queries)
- Ability to include, inline with each request, a set of invalidated source data id's and retrieve data with those invalidations considered (providing immediate read-after-write consistency), choosing whether or not to persist the results of those invalidations to shared state storage
- Pub/sub support for best-effort streaming realtime view updates via Websockets (in addition to the existing ability to output to Kafka)

Status:

Tokamak is under active development as a personal project of mine. It is currently capable of executing very simple SQL queries: all of parsing, planning, analyzing, rewriting, materializing, and maintaining state. Its implementation has an extremely broad scope and as such has dozens of subsystems each slowly being built up. It will not be ready for any production use for a long while but it is very much alive, and it has reached the point at which its test cases are written simply in SQL and executed against [TPCH data](https://github.com/prestosql/tpch).