📝 docs: add idea for Redis-like volatile KV store in the server

main
_ 2021-01-03 04:45:14 +00:00
parent b62c1424fa
commit 075fd604ee
2 changed files with 82 additions and 0 deletions

View File

@ -0,0 +1,80 @@
# Fire-and-forget logs / status key-values
(DMX6CO4G)
## Motivation
This feature allows ptth_server to act as a
[sidecar](https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar)
process to apps and scripts written in C, C++, Bash, or other languages
where networking is difficult.
ptth_server will hold volatile data such as high-detail logs or status data
that is not worth saving to files. This data can then be retrieved by human
users or scrapers through ptth_relay.
This feature is similar to Redis, in some ways.
## Interface with client apps
The lifecycle is this:
- A client app connects to ptth_server via TCP
- The client app sends data to ptth_server
- The client app MAY react to messages from ptth_server
- The TCP connection is closed by ptth_server OR the client app
The client app can send data with these semantics:
- "Status" i.e. "In this key directory, in this key, store this value"
- "Log" i.e. "In this log directory, in this key, store this log string"
Logs and status values do not share namespaces or quotas. ptth_server MUST be
configured with quotas in order to enable this feature. There is no "unlimited"
quota option, but the quotas MAY be set to any 64-bit value.
Keys, values, and log strings are binary-safe, but SHOULD be valid UTF-8.
If they are not valid UTF-8, they MAY not appear in client apps, or they may
appear as errors. ptth_server MUST behave correctly if it receives non-UTF-8
data.
The data should be considered volatile. ptth_server MUST NOT save the data
to disk. The client app SHOULD NOT buffer data internally if ptth_server is
refusing connections or unavailable. Client apps will see the data as
write-only - They cannot retrieve data from ptth_server.
ptth_server does not guarantee that data will be stored for any length of time,
or at all. All data is subject to quotas based on time, byte count, key count,
etc. Logs WILL be evicted in oldest-first order. Status values
replace older values written to the same key. If a key directory reaches
its quota, old values MAY be evicted in any order, including oldest-first,
alphabetical, or random.
Quotas: ptth_relay is the source of truth for quotas. ptth_server SHOULD NOT
store the quotas on disk, it SHOULD request them from ptth_relay when connecting,
keep them in memory, and periodically refresh them. All quotas are set
per-key-directory. ptth_relay is the source of truth for the list of key
directories.
The quotas are all maximums. The quotas (with recommended value in parens) are:
- Keys in the directory (1,024)
- Bytes for 1 key (1,024)
- Bytes for 1 value (1,024)
- Total payload (i.e. keys and values) bytes (262,144)
It's hard to set a perfect limit on RAM usage, so the quotas have many
different dimensions. There is some upper bound on RAM usage, but it will
somewhere above the number of bytes needed to store keys and values alone.
ptth_server MAY send these messages to a client app:
- "Go" - i.e. "Everything is good on my end. If you send data I will try to
store it."
- "Stop" - i.e. "Something is wrong. If you send data I will ignore you."
Unlike HTTP, there is no synchronization or dependency between the two
halves of the TCP connection - ptth_server can send a message at any time,
and the client app can send data at any time. "Go" does not mean that the
server will store the next data sent from the client, because a "Stop" message
could be in-flight.

View File

@ -1,8 +1,10 @@
Interesting issues will get a unique ID with
`dd if=/dev/urandom bs=5 count=1 | base32`
- [DMX6CO4G](issues/2021-01Jan/status-DMX6CO4G.md) fire-and-forget logs / key-value status data
- ptth_tail
- Scraper `rsync -ru` example
- 'fax' like system similar to station307
- (WIP) Dark mode?
- [K5NPHQHP](issues/2020-12Dec/metrics-K5NPHQHP.md) API for metrics + instance data + recent logs on ptth_server
- API for remote mtime