# Packet Data Stream

```c
#define FIO_STREAM
#include "fio-stl.h"
```

A packet-based byte stream for buffering data from memory buffers and file descriptors. It is useful when writes are partial: add packets to the stream, read the next contiguous chunk into or from a scratch buffer, write what you can, then advance by the number of bytes written.

For the low-level helpers commonly used with streams, see [`004 files.md`](https://facil.io/0.8.x/file-helpers/), [`004 sock.md`](https://facil.io/0.8.x/basic-socket-helpers/), and [`102 poll api.md`](https://facil.io/0.8.x/posix-portable-polling-api/).

---

## Configuration Macros

#### `FIO_STREAM_COPY_PER_PACKET`

```c
#define FIO_STREAM_COPY_PER_PACKET 98304
```

Maximum copied payload per packet. Larger copied buffers are split into multiple packets so memory can be released progressively. Override before including the implementation.

#### `FIO_STREAM_ALWAYS_COPY_IF_LESS_THAN`

```c
#define FIO_STREAM_ALWAYS_COPY_IF_LESS_THAN 116 /* 8 in DEBUG */
```

Small buffers are copied even when `copy_buffer == 0`, improving locality and avoiding tiny external references.

---

## Types

#### `fio_stream_s`

```c
typedef struct {
  fio_stream_packet_s *next;
  fio_stream_packet_s **pos;
  size_t consumed;
  size_t length;
} fio_stream_s;
```

Stream object. Treat the fields as private. Allocate on the stack with `FIO_STREAM_INIT` or on the heap with `fio_stream_new`.

#### `fio_stream_packet_s`

```c
typedef struct fio_stream_packet_s fio_stream_packet_s;
```

Opaque packed data node. Packets are prepared separately, then transferred to a stream with `fio_stream_add`.

---

## Lifecycle

#### `FIO_STREAM_INIT`

```c
#define FIO_STREAM_INIT(s) { .next = NULL, .pos = &(s).next }
```

Initializer for an in-place stream.

```c
fio_stream_s stream = FIO_STREAM_INIT(stream);
```

#### `fio_stream_new`

```c
fio_stream_s *fio_stream_new(void);
```

Allocates and initializes a heap stream.

**Returns:** stream pointer, or `NULL` on allocation failure.

Not declared when `FIO_REF_CONSTRUCTOR_ONLY` is defined.

#### `fio_stream_free`

```c
int fio_stream_free(fio_stream_s *stream);
```

Destroys `stream`, frees the stream object itself, and returns `0`.

Not declared when `FIO_REF_CONSTRUCTOR_ONLY` is defined.

#### `fio_stream_destroy`

```c
void fio_stream_destroy(fio_stream_s *stream);
```

Frees all queued packets and re-initializes the stream object. Safe to call with `NULL`.

---

## Packing Data

#### `fio_stream_pack_data`

```c
fio_stream_packet_s *fio_stream_pack_data(void *buf,
                                          size_t len,
                                          size_t offset,
                                          uint8_t copy_buffer,
                                          void (*dealloc_func)(void *));
```

Packs `len` bytes starting at `((char *)buf + offset)`.

- If `copy_buffer` is non-zero, or `len < FIO_STREAM_ALWAYS_COPY_IF_LESS_THAN`, bytes are copied into stream-owned packets.
- Otherwise the packet references `buf` and calls `dealloc_func(buf)` when the packet is freed, if `dealloc_func` is not `NULL`.
- Copied data larger than `FIO_STREAM_COPY_PER_PACKET` is split into multiple packets.

**Returns:** packet pointer, or `NULL` if `buf == NULL`, `len == 0`, the length is too large for the packet format, or allocation fails. If `dealloc_func` is provided, it is called on both success-after-copy and failure.

#### `fio_stream_pack_fd`

```c
fio_stream_packet_s *fio_stream_pack_fd(int fd,
                                        size_t len,
                                        size_t offset,
                                        uint8_t keep_open);
```

Packs bytes from an open file descriptor. Reads are performed later by `fio_stream_read` using `fio_fd_read`.

- `len == 0` auto-detects the remaining file size with `fio_fd_size(fd)`.
- `offset` is the starting file offset.
- If `keep_open == 0`, the descriptor is closed when the packet is freed, or on pack failure after ownership was accepted.
- If `keep_open != 0`, the caller remains responsible for closing `fd`.

**Returns:** packet pointer, or `NULL` on invalid input, size detection failure / oversized auto-detected size when `len == 0`, or allocation failure.

#### `fio_stream_add`

```c
void fio_stream_add(fio_stream_s *stream, fio_stream_packet_s *packet);
```

Appends `packet` to `stream` and transfers packet ownership to the stream. If `stream` or `packet` is `NULL`, the packet is freed.

This is not thread-safe.

#### `fio_stream_pack_free`

```c
void fio_stream_pack_free(fio_stream_packet_s *packet);
```

Frees a packet that was not added to a stream. Do not call this after `fio_stream_add` succeeds.

---

## Reading and Consuming

#### `fio_stream_read`

```c
void fio_stream_read(fio_stream_s *stream, char **buf, size_t *len);
```

Reads the next available bytes without consuming them.

Before the call, `*buf` must point to a scratch buffer with at least `*len` bytes. This buffer is required when data spans multiple packets or comes from a file descriptor.

After the call:

- On empty stream or error, `*buf == NULL` and `*len == 0`.
- Otherwise `*buf` either still points to the supplied scratch buffer or points directly into stream-owned memory.
- `*len` is updated to the number of readable bytes available at `*buf`.

Reset both `*buf` and `*len` before each read if reusing the same scratch buffer. This is not thread-safe.

#### `fio_stream_advance`

```c
void fio_stream_advance(fio_stream_s *stream, size_t len);
```

Consumes `len` bytes from the stream, freeing fully consumed packets. Usually pass the number of bytes actually written to the destination.

This is not thread-safe.

#### `fio_stream_any`

```c
uint8_t fio_stream_any(fio_stream_s *stream);
```

Returns non-zero when `stream` has pending packets. Returns `0` for `NULL` or empty streams. This is not truly thread-safe.

#### `fio_stream_length`

```c
size_t fio_stream_length(fio_stream_s *stream);
```

Returns the number of bytes waiting in the stream. Call with a valid stream object. This is not truly thread-safe.

---

## Ownership and Thread-Safety

Packing creates a packet owned by the caller. `fio_stream_add` transfers that ownership to the stream. `fio_stream_destroy`, `fio_stream_free`, and `fio_stream_advance` free packets as needed.

For referenced memory packets, the referenced buffer must remain valid until the packet is freed. Pass `copy_buffer != 0` when that lifetime is not guaranteed. For file descriptor packets, keep the descriptor readable until the packet is consumed or destroyed; `keep_open` controls who closes it.

Packing packets can be done before taking the stream lock, but mutating or consuming a single `fio_stream_s` (`add`, `read`, `advance`, and length checks) should be externally synchronized when shared across threads.

---

## Example

```c
#define FIO_STREAM
#include "fio-stl.h"
#include <stdio.h>
#include <string.h>

int main(void) {
  fio_stream_s stream = FIO_STREAM_INIT(stream);
  char scratch[4096];

  char msg[] = "hello, stream\n";
  fio_stream_packet_s *p = fio_stream_pack_data(msg, strlen(msg), 0, 1, NULL);
  fio_stream_add(&stream, p);

  while (fio_stream_any(&stream)) {
    char *buf = scratch;
    size_t len = sizeof(scratch);

    fio_stream_read(&stream, &buf, &len);
    if (!len)
      break;

    size_t written = fwrite(buf, 1, len, stdout);
    fio_stream_advance(&stream, written);

    if (written != len)
      break;
  }

  fio_stream_destroy(&stream);
  return 0;
}
```

### File Descriptor Packet

```c
#define FIO_STREAM
#define FIO_FILES
#include "fio-stl.h"

fio_stream_s stream = FIO_STREAM_INIT(stream);
int fd = fio_filename_open("./asset.bin", O_RDONLY);
fio_stream_packet_s *p = fio_stream_pack_fd(fd, 0, 0, 0); /* stream closes fd */
fio_stream_add(&stream, p);

/* ... read / write packets ... */
fio_stream_destroy(&stream);
```

------------------------------------------------------------