214 lines
10 KiB
Markdown
214 lines
10 KiB
Markdown
# Event Log Based Store
|
|
|
|
All data rows are reconstructed exclusively from an event log. All interactions with data rows must occur via events in the log. For performance, data rows are cached for quick lookup.
|
|
|
|
Events are defined as:
|
|
type Event struct {
|
|
Seq int `json:"seq"` // Server-generated sequence number (applied order)
|
|
Hash string `json:"hash"` // Server-generated hash, guarantees event was processed
|
|
ItemID string `json:"item_id"` // Client-defined item identifier
|
|
EventID string `json:"event_id"` // Server-generated event identifier (uuid-v4)
|
|
Collection string `json:"collection"` // Client-defined collection/table name
|
|
Data string `json:"data"` // JSON array of RFC6902 patches
|
|
Timestamp time.Time `json:"timestamp"` // Server-generated timestamp (when processed)
|
|
}
|
|
|
|
When creating an event, only Data, Collection, and ItemID are required from the client. Hash, EventID, Seq, and Timestamp are computed server-side.
|
|
|
|
Server-side event processing:
|
|
- Retrieve the latest event for the collection.
|
|
- Assign the next sequence number (incremented from the latest).
|
|
- Generate a new EventID (uuid-v4).
|
|
- Assign the current timestamp.
|
|
- Compute the event hash as a function of the current event's data and the previous event's hash.
|
|
- Serialize the event manually (not via json.Marshal or %+v) to ensure field order for hashing.
|
|
- Apply the patch to the cached data row.
|
|
|
|
Event log compaction:
|
|
- Every 2 days, merge and compact the event log for each collection.
|
|
- All events older than 2 days are resolved, and a new minimal event log is generated that produces the same state.
|
|
- Sequence numbers (Seq) are never reset and always increment from the last value.
|
|
- Before merging or deleting old events, save the original event log as a timestamped backup file.
|
|
|
|
Client requirements:
|
|
- Must be able to apply patches and fetch objects.
|
|
- Must store:
|
|
- last_seq: sequence number of the last processed event
|
|
- last_hash: hash of the last processed event
|
|
- events: local event log of all processed events
|
|
- pending_events: locally generated events not yet sent to the server
|
|
- On startup, fetch new events from the server since last_seq and apply them.
|
|
- When modifying objects, generate events and append to pending_events.
|
|
- Periodically or opportunistically send pending_events to the server.
|
|
- Persist the event log (events and pending_events) locally.
|
|
- If the server merges the event log, the client detects divergence by comparing last_seq and last_hash.
|
|
- If sequence matches but hash differs, the server sends the full event log; the client reconstructs its state from this log.
|
|
|
|
If the server merges the event log and the client has unsent local events:
|
|
- Client fetches the merged events from the server.
|
|
- Applies merged events to local state.
|
|
- Reapplies unsent local events on top of the updated state.
|
|
- Resends these events to the server.
|
|
|
|
If a client sends events after the event log has been merged:
|
|
- The server accepts and applies these events as usual, regardless of the client's log state.
|
|
|
|
Merging the event log must not alter the resulting data state.
|
|
|
|
Required endpoints:
|
|
|
|
GET /api/<collection>/sync?last_seq=<last_seq>&last_hash=<last_hash>
|
|
- Returns all events after the specified last_seq and last_hash.
|
|
- If the provided seq and hash do not match the server's, returns the entire event log (client is out of sync).
|
|
|
|
PATCH /api/<collection>/events
|
|
- Accepts a JSON array of RFC6902 patch objects.
|
|
|
|
Server processing:
|
|
- As new events arrive, process the event log and update the cached state for the collection.
|
|
- The current state is available for clients that do not wish to process the event log.
|
|
- Only new events need to be applied to the current state; no need to reprocess the entire log each time.
|
|
- Track the last event processed for each collection (sequence number and hash).
|
|
|
|
On startup, the server must:
|
|
- Automatically create required collections: one for events and one for items (data state).
|
|
- Events must be collection-agnostic and support any collection; at least one example collection is created at startup.
|
|
- Ensure required columns exist in collections; if missing, reject PATCH requests with an error.
|
|
- Each collection maintains its own sequence number, hash, and event log.
|
|
|
|
|
|
---
|
|
|
|
|
|
## RFC6902
|
|
https://datatracker.ietf.org/doc/html/rfc6902
|
|
Operation objects MUST have exactly one "op" member, whose value
|
|
indicates the operation to perform. Its value MUST be one of "add",
|
|
"remove", "replace", "move", "copy", or "test"; other values are
|
|
errors. The semantics of each object is defined below.
|
|
|
|
Additionally, operation objects MUST have exactly one "path" member.
|
|
That member's value is a string containing a JSON-Pointer value
|
|
[RFC6901] that references a location within the target document (the
|
|
"target location") where the operation is performed.
|
|
|
|
The meanings of other operation object members are defined by
|
|
operation (see the subsections below). Members that are not
|
|
explicitly defined for the operation in question MUST be ignored
|
|
(i.e., the operation will complete as if the undefined member did not
|
|
appear in the object).
|
|
|
|
Note that the ordering of members in JSON objects is not significant;
|
|
therefore, the following operation objects are equivalent:
|
|
|
|
{ "op": "add", "path": "/a/b/c", "value": "foo" }
|
|
{ "path": "/a/b/c", "op": "add", "value": "foo" }
|
|
{ "value": "foo", "path": "/a/b/c", "op": "add" }
|
|
|
|
Operations are applied to the data structures represented by a JSON
|
|
document, i.e., after any unescaping (see [RFC4627], Section 2.5)
|
|
takes place.
|
|
|
|
## add
|
|
The "add" operation performs one of the following functions,
|
|
depending upon what the target location references:
|
|
|
|
o If the target location specifies an array index, a new value is
|
|
inserted into the array at the specified index.
|
|
o If the target location specifies an object member that does not
|
|
already exist, a new member is added to the object.
|
|
o If the target location specifies an object member that does exist,
|
|
that member's value is replaced.
|
|
|
|
The operation object MUST contain a "value" member whose content
|
|
specifies the value to be added.
|
|
For example:
|
|
{ "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] }
|
|
When the operation is applied, the target location MUST reference one
|
|
of:
|
|
o The root of the target document - whereupon the specified value
|
|
becomes the entire content of the target document.
|
|
o A member to add to an existing object - whereupon the supplied
|
|
value is added to that object at the indicated location. If the
|
|
member already exists, it is replaced by the specified value.
|
|
o An element to add to an existing array - whereupon the supplied
|
|
value is added to the array at the indicated location. Any
|
|
elements at or above the specified index are shifted one position
|
|
to the right. The specified index MUST NOT be greater than the
|
|
number of elements in the array. If the "-" character is used to
|
|
index the end of the array (see [RFC6901]), this has the effect of
|
|
appending the value to the array.
|
|
|
|
Because this operation is designed to add to existing objects and
|
|
arrays, its target location will often not exist. Although the
|
|
pointer's error handling algorithm will thus be invoked, this
|
|
specification defines the error handling behavior for "add" pointers
|
|
to ignore that error and add the value as specified.
|
|
|
|
However, the object itself or an array containing it does need to
|
|
exist, and it remains an error for that not to be the case. For
|
|
example, an "add" with a target location of "/a/b" starting with this
|
|
document:
|
|
{ "a": { "foo": 1 } }
|
|
is not an error, because "a" exists, and "b" will be added to its
|
|
value. It is an error in this document:
|
|
{ "q": { "bar": 2 } }
|
|
because "a" does not exist.
|
|
|
|
## remove
|
|
The "remove" operation removes the value at the target location.
|
|
The target location MUST exist for the operation to be successful.
|
|
For example:
|
|
{ "op": "remove", "path": "/a/b/c" }
|
|
If removing an element from an array, any elements above the
|
|
specified index are shifted one position to the left.
|
|
|
|
## replace
|
|
The "replace" operation replaces the value at the target location
|
|
with a new value. The operation object MUST contain a "value" member
|
|
whose content specifies the replacement value.
|
|
|
|
The target location MUST exist for the operation to be successful.
|
|
For example:
|
|
{ "op": "replace", "path": "/a/b/c", "value": 42 }
|
|
|
|
This operation is functionally identical to a "remove" operation for
|
|
a value, followed immediately by an "add" operation at the same
|
|
location with the replacement value.
|
|
|
|
## move
|
|
The "move" operation removes the value at a specified location and
|
|
adds it to the target location.
|
|
|
|
The operation object MUST contain a "from" member, which is a string
|
|
containing a JSON Pointer value that references the location in the
|
|
target document to move the value from.
|
|
|
|
The "from" location MUST exist for the operation to be successful.
|
|
For example:
|
|
{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" }
|
|
|
|
This operation is functionally identical to a "remove" operation on
|
|
the "from" location, followed immediately by an "add" operation at
|
|
the target location with the value that was just removed.
|
|
The "from" location MUST NOT be a proper prefix of the "path"
|
|
location; i.e., a location cannot be moved into one of its children.
|
|
|
|
## copy
|
|
The "copy" operation copies the value at a specified location to the
|
|
target location.
|
|
|
|
The operation object MUST contain a "from" member, which is a string
|
|
containing a JSON Pointer value that references the location in the
|
|
target document to copy the value from.
|
|
The "from" location MUST exist for the operation to be successful.
|
|
|
|
For example:
|
|
{ "op": "copy", "from": "/a/b/c", "path": "/a/b/e" }
|
|
|
|
This operation is functionally identical to an "add" operation at the
|
|
target location using the value specified in the "from" member.
|
|
|
|
## test
|
|
I think we don't care about this one
|
|
|