📝 docs: add idea for Redis-like volatile KV store in the server

issues/2021-01Jan/status-DMX6CO4G.md

@@ -0,0 +1,80 @@
+# Fire-and-forget logs / status key-values
+
+(DMX6CO4G)
+
+## Motivation
+
+This feature allows ptth_server to act as a 
+[sidecar](https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar)
+process to apps and scripts written in C, C++, Bash, or other languages
+where networking is difficult.
+
+ptth_server will hold volatile data such as high-detail logs or status data
+that is not worth saving to files. This data can then be retrieved by human
+users or scrapers through ptth_relay.
+
+This feature is similar to Redis, in some ways.
+
+## Interface with client apps
+
+The lifecycle is this:
+
+- A client app connects to ptth_server via TCP
+- The client app sends data to ptth_server
+- The client app MAY react to messages from ptth_server
+- The TCP connection is closed by ptth_server OR the client app
+
+The client app can send data with these semantics:
+
+- "Status" i.e. "In this key directory, in this key, store this value"
+- "Log" i.e. "In this log directory, in this key, store this log string"
+
+Logs and status values do not share namespaces or quotas. ptth_server MUST be
+configured with quotas in order to enable this feature. There is no "unlimited"
+quota option, but the quotas MAY be set to any 64-bit value.
+
+Keys, values, and log strings are binary-safe, but SHOULD be valid UTF-8.
+If they are not valid UTF-8, they MAY not appear in client apps, or they may
+appear as errors. ptth_server MUST behave correctly if it receives non-UTF-8
+data.
+
+The data should be considered volatile. ptth_server MUST NOT save the data
+to disk. The client app SHOULD NOT buffer data internally if ptth_server is
+refusing connections or unavailable. Client apps will see the data as
+write-only - They cannot retrieve data from ptth_server.
+
+ptth_server does not guarantee that data will be stored for any length of time,
+or at all. All data is subject to quotas based on time, byte count, key count,
+etc. Logs WILL be evicted in oldest-first order. Status values
+replace older values written to the same key. If a key directory reaches
+its quota, old values MAY be evicted in any order, including oldest-first,
+alphabetical, or random.
+
+Quotas: ptth_relay is the source of truth for quotas. ptth_server SHOULD NOT
+store the quotas on disk, it SHOULD request them from ptth_relay when connecting,
+keep them in memory, and periodically refresh them. All quotas are set
+per-key-directory. ptth_relay is the source of truth for the list of key
+directories.
+
+The quotas are all maximums. The quotas (with recommended value in parens) are:
+
+- Keys in the directory (1,024)
+- Bytes for 1 key (1,024)
+- Bytes for 1 value (1,024)
+- Total payload (i.e. keys and values) bytes (262,144)
+
+It's hard to set a perfect limit on RAM usage, so the quotas have many
+different dimensions. There is some upper bound on RAM usage, but it will
+somewhere above the number of bytes needed to store keys and values alone.
+
+ptth_server MAY send these messages to a client app:
+
+- "Go" - i.e. "Everything is good on my end. If you send data I will try to
+store it."
+- "Stop" - i.e. "Something is wrong. If you send data I will ignore you."
+
+Unlike HTTP, there is no synchronization or dependency between the two 
+halves of the TCP connection - ptth_server can send a message at any time,
+and the client app can send data at any time. "Go" does not mean that the
+server will store the next data sent from the client, because a "Stop" message
+could be in-flight.

todo.md

@@ -1,8 +1,10 @@
 Interesting issues will get a unique ID with
 `dd if=/dev/urandom bs=5 count=1 | base32`
 
+- [DMX6CO4G](issues/2021-01Jan/status-DMX6CO4G.md) fire-and-forget logs / key-value status data
 - ptth_tail
 - Scraper `rsync -ru` example
+- 'fax' like system similar to station307
 - (WIP) Dark mode?
 - [K5NPHQHP](issues/2020-12Dec/metrics-K5NPHQHP.md) API for metrics + instance data + recent logs on ptth_server
 - API for remote mtime