3.6 KiB
Fire-and-forget logs / status key-values
(DMX6CO4G)
Bottom Line Up Front
A memory-only key-value store for ptth_server. Other processes on the same system can send data to ptth_server, and it will hold the data in memory and expose it to ptth_relay.
Motivation
This feature allows ptth_server to act as a sidecar process to apps and scripts written in C, C++, Bash, or other languages where networking is difficult.
ptth_server will hold volatile data such as high-detail logs or status data that is not worth saving to files. This data can then be retrieved by human users or scrapers through ptth_relay.
This feature is similar to Redis, in some ways.
Interface with client apps
The lifecycle is this:
- A client app connects to ptth_server via TCP
- The client app sends data to ptth_server
- The client app MAY react to messages from ptth_server
- The TCP connection is closed by ptth_server OR the client app
The client app can send data with these semantics:
- "Status" i.e. "In this key directory, in this key, store this value"
- "Log" i.e. "In this log directory, in this key, store this log string"
Logs and status values do not share namespaces or quotas. ptth_server MUST be configured with quotas in order to enable this feature. There is no "unlimited" quota option, but the quotas MAY be set to any usize value.
Keys, values, and log strings are binary-safe, but SHOULD be valid UTF-8. If they are not valid UTF-8, they MAY not appear in client apps, or they may appear as errors. ptth_server MUST behave correctly if it receives non-UTF-8 data.
The data should be considered volatile. ptth_server MUST NOT save the data to disk. The client app SHOULD NOT buffer data internally if ptth_server is refusing connections or unavailable. Client apps will see the data as write-only - They cannot retrieve data from ptth_server.
ptth_server does not guarantee that data will be stored for any length of time, or at all. All data is subject to quotas based on time, byte count, key count, etc. Logs WILL be evicted in oldest-first order. Status values replace older values written to the same key. If a key directory reaches its quota, old values MAY be evicted in any order, including oldest-first, alphabetical, random, or all at once.
Quotas: ptth_relay is the source of truth for quotas. ptth_server SHOULD NOT store the quotas on disk, it SHOULD request them from ptth_relay when connecting, keep them in memory, and periodically refresh them. All quotas are set per-key-directory. ptth_relay is the source of truth for the list of key directories.
The quotas are all maximums. The quotas (with recommended value in parens) are:
- Keys in the directory (1,024)
- Bytes for 1 key (1,024)
- Bytes for 1 value (1,024)
- Total payload (i.e. keys and values) bytes (262,144)
- Number of log events per key (2,048)
It's hard to set a perfect limit on RAM usage, so the quotas have many different dimensions. There is some upper bound on RAM usage, but it will somewhere above the number of bytes needed to store keys and values alone.
ptth_server MAY send these messages to a client app:
- "Go" - i.e. "Everything is good on my end. If you send data I will try to store it."
- "Stop" - i.e. "Something is wrong. If you send data I will ignore you."
Unlike HTTP, there is no synchronization or dependency between the two halves of the TCP connection - ptth_server can send a message at any time, and the client app can send data at any time. "Go" does not mean that the server will store the next data sent from the client, because a "Stop" message could be in-flight.