sgns::sgprocessing¶
Artifact and manifest binary serialization. More...
Classes¶
Types¶
| Name | |
|---|---|
| enum class uint8_t | TerminalState |
| enum class | UnmetRequirementCategory |
| enum class | RenderStage |
| enum class | MNNStage |
| enum class | ProcessingErrorStage |
| using std::function< void(CanExecuteResult)> | CanExecuteCallback Callback type for async CanExecute (D-03). |
| using sgns::sgprocessing::ProcessingProcessor | ProcessingProcessor |
Functions¶
| Name | |
|---|---|
| std::vector< uint8_t > | SerializeArtifact(const Artifact & artifact) |
| bool | DeserializeArtifact(const std::vector< uint8_t > & bytes, Artifact & out) |
| std::vector< uint8_t > | SerializeManifest(const ExecutionManifest & manifest) |
| bool | DeserializeManifest(const std::vector< uint8_t > & bytes, ExecutionManifest & out) |
| std::vector< uint8_t > | ComputeManifestHash(const ExecutionManifest & manifest) |
| void | ComputeArtifactIdentity(Artifact & artifact, const uint8_t * rawBytes, size_t byteCount) |
| bool | AddChunkHash(Artifact & artifact, const uint8_t hash[SHA256_HASH_SIZE]) |
| std::mutex & | VulkanInitMutex() |
| bool | HasUsableVulkanDevice() |
| bool | HasUsableVulkanDeviceCached() |
Attributes¶
| Name | |
|---|---|
| constexpr size_t | ARTIFACT_SERIALIZED_SIZE |
| constexpr size_t | MANIFEST_SERIALIZED_SIZE |
| constexpr size_t | MANIFEST_V2_SERIALIZED_SIZE |
| constexpr size_t | SHA256_HASH_SIZE SHA-256 digest size in bytes. |
| constexpr size_t | MAX_RESOURCE_NAME Max bytes for resource/pass/binding name strings (D-06) |
| constexpr size_t | MAX_MEDIA_TYPE Max bytes for media type string (D-06) |
| constexpr size_t | MAX_ARTIFACT_REFS Max input/output artifact hash references. |
| constexpr size_t | MAX_IDENTIFIER Max bytes for execution/attempt/task/subtask/pass ID strings. |
Detailed Description¶
Artifact and manifest binary serialization.
Pre-execution capability validation gate.
Capability validation data types.
Execution manifest data contract.
Artifact and terminal state data types.
Deterministic binary serialization for Artifact and ExecutionManifest structs.
Fixed-field, little-endian, fixed-offset binary layout (D-04, D-05, D-06). Every multi-byte value at a known, fixed offset; all integers in native little-endian byte order; variable-length data (strings, arrays) capped at maximum sizes with inline storage.
Byte-identical output across two runs with identical inputs (ARTF-05).
Artifact type system for Phase 08: Structured Artifacts & Execution Manifests.
Defines the TerminalState enum and Artifact struct — the data contracts for typed output records with content-hash-based identity, resource metadata, and per-chunk SHA-256 hashes. No protobuf — plain C++ structs per D-04/D-05.
Execution manifest for Phase 08: Structured Artifacts & Execution Manifests.
Defines the ExecutionManifest struct — a self-contained, fixed-field record that captures everything needed for hashing, signing, caching, and verification of an execution run (ARTF-04, D-13). All identity fields are inline; inapplicable identities use the sentinel zero-hash convention (D-14). ARTF-09 adds a trailing errorMessage field, appended without disturbing any existing field's offset.
Capability validation type system for Phase 06.
Defines the data contracts (UnmetRequirement, CanExecuteResult, CapabilitySnapshot) that all capability validation checks build against. No protobuf — plain C++ structs per D-05.
CapabilityValidator — pre-execution capability gate for SGProcessingManager.
Constructed internally by ProcessingManager (D-01, D-04). Builds a capability snapshot once at startup (D-09, D-12), then provides CanExecute() to validate jobs against the cached snapshot before claiming work from the network (D-02).
Uses PIMPL pattern to keep Vulkan headers out of transitive includes.
Header file for base class for processors. Derived classes will handle processing various types of AI/ML processing as needed. Give this to a ProcessingCoreImpl. Justin Church
Header file for processing bool inputs using MNN
Header file for processing buffer inputs using MNN
Header file for processing float inputs using MNN
Header file for processing int inputs using MNN
Header file for processing mat2 inputs using MNN
Header file for processing mat3 inputs using MNN
Header file for processing mat4 inputs using MNN
Header file for processing tensor inputs using MNN
Header file for processing texture1D inputs using MNN
Header file for processing textureCube inputs using MNN
Implementation of deterministic binary serialization for Artifact and ExecutionManifest structs. Fixed-field, little-endian layout per D-05.
Follows the same memcpy-based approach as SerializeRenderPassConfig() in ProcessingManager.cpp: pre-allocate zero-filled vector, then copy each field at its documented fixed offset.
CapabilityValidator implementation — BuildSnapshot and CanExecute.
BuildSnapshot: queries Vulkan device properties, MNN executor registry, disk space, and computes the deterministic executor identity hash. CanExecute: validates jobs against the cached snapshot across all five check categories (PassType, Vulkan, MNN, GPU memory, disk space).
Types Documentation¶
enum TerminalState¶
| Enumerator | Value | Description |
|---|---|---|
| Success | 0 | Normal completion — all outputs valid. |
| Cancelled | 1 | Cancellation token triggered (07 D-01/D-05) |
| Timeout | 2 | Deadline expired (07 D-02/D-09) |
| BudgetExceeded | 3 | Output byte budget exceeded (07 D-08/EXEC-03) |
| Error | 4 | All other failures (D-15: no error string in manifest) |
Terminal execution outcome for the manifest (D-15). Explicit uint8_t underlying type — maps directly to a single byte in the binary serialized manifest layout.
enum UnmetRequirementCategory¶
| Enumerator | Value | Description |
|---|---|---|
| VULKAN | 0 | Vulkan device/feature/limit insufficiency. |
| MNN | 1 | MNN model format or quantization not supported. |
| PASS_TYPE | 2 | No executor registered for the requested PassType. |
| RESOURCE | 3 | GPU memory or disk space insufficient. |
Category of unmet requirement for structured capability rejection (D-06). Follows same enum-prefix convention as ProcessingErrorStage in processing_processor.hpp.
enum RenderStage¶
| Enumerator | Value | Description |
|---|---|---|
| COMPILE | 0 | Shader compilation. |
| BUILD_PIPELINE | 1 | Pipeline creation. |
| DRAW | 2 | Draw submission. |
| READBACK | 3 | Readback from framebuffer. |
Standardized pipeline stages for RenderProcessor (D-12). Matches the four coarse checkpoints in D-04 for RenderProcessor.
enum MNNStage¶
| Enumerator | Value | Description |
|---|---|---|
| LOAD_MODEL | 0 | Model file loaded / MNN interpreter created. |
| CREATE_SESSION | 1 | MNN session created. |
| RUN | 2 | Inference executed. |
| READ_OUTPUT | 3 | Output tensor read. |
Standardized pipeline stages for MNN processors (D-12). Matches the coarse checkpoints in D-04 for MNN processors.
enum ProcessingErrorStage¶
| Enumerator | Value | Description |
|---|---|---|
| UNSPECIFIED | 0 | |
| CONTEXT_INIT_FAILED | ||
| RESOURCE_RESOLUTION | ||
| BUFFER_ALLOCATION | ||
| IMAGE_ALLOCATION | ||
| FORMAT_UNSUPPORTED | ||
| SHADER_MODULE_CREATION | ||
| PIPELINE_CREATION | ||
| RENDER_PASS_CREATION | ||
| DRAW_SUBMISSION | ||
| READBACK | ||
| DATA_TRANSFORM_UNSUPPORTED | ||
| CANCELLED | 12 | Processor cancelled via CancellationToken (D-05: distinct error code) |
| TIMED_OUT | 13 | Per-pass deadline expired (D-02, D-05) |
| BUDGET_EXCEEDED | 14 | Output artifact size exceeded max_output_artifact_bytes (EXEC-03) |
Per-stage failure classification for structured processor errors (D-25/D-26). Plain, non-outcome::result enum – StartProcessing()'s return type stays the concrete ProcessingResult, so the OUTCOME_HPP_DECLARE_ERROR_2 machinery is unnecessary here.
using CanExecuteCallback¶
Callback type for async CanExecute (D-03).
using ProcessingProcessor¶
Functions Documentation¶
function SerializeArtifact¶
Return: Vector of exactly ARTIFACT_SERIALIZED_SIZE bytes.
Serialize an Artifact to a fixed-size binary blob (ARTF-05).
function DeserializeArtifact¶
Return: true on success; false if input size != ARTIFACT_SERIALIZED_SIZE.
Deserialize a binary blob back into an Artifact struct.
function SerializeManifest¶
Return: Vector of exactly MANIFEST_V2_SERIALIZED_SIZE bytes (unchanged base region + schemaVersion+errorMessage trailer, ARTF-10).
Serialize an ExecutionManifest to a fixed-size binary blob (ARTF-05).
CRITICAL (D-04): The manifestHash field is zeroed before serialization and restored afterward so it does NOT participate in its own hash computation.
function DeserializeManifest¶
Return: false if input size < MANIFEST_SERIALIZED_SIZE (base region); trailer fields (schemaVersion, errorMessage) are read only when present and within bounds, defaulting to absent/empty otherwise.
Deserialize a binary blob back into an ExecutionManifest struct.
function ComputeManifestHash¶
Return: 32-byte SHA-256 hash.
Compute the manifest self-hash: SHA-256 of serialized manifest bytes with the manifestHash field zeroed (handled internally by SerializeManifest).
function ComputeArtifactIdentity¶
inline void ComputeArtifactIdentity(
Artifact & artifact,
const uint8_t * rawBytes,
size_t byteCount
)
Compute artifact identity from raw bytes (D-01, D-03). Delegates to sgprocmanagersha::sha256 for the actual hash. Fills both contentHash and artifactId in-place.
function AddChunkHash¶
Return: true on success, false if chunkHashCount >= 1024 (overflow guard).
Add a chunk hash to the artifact's chunk hash list (D-08).
function VulkanInitMutex¶
function HasUsableVulkanDevice¶
Runtime probe answering "does this host have at least one usable Vulkan device?", mirroring RenderProcessor::IsAcceptable's DISCRETE_GPU/ INTEGRATED_GPU device-type filter (D-32). Builds and immediately tears down its own throwaway VkInstance – it never creates a VkDevice and never touches RenderProcessor's own Vulkan state.
Callers MUST treat a false return as "skip GPU-dependent work" (e.g. via GTEST_SKIP()), never as a hard error – a GPU-less host is an expected, valid environment (D-34), not a failure condition.
Never throws.
function HasUsableVulkanDeviceCached¶
Process-lifetime cached variant of HasUsableVulkanDevice().
The uncached probe builds and destroys a whole VkInstance plus a full physical-device enumeration per call – far too heavy for per-session or per-chunk MNN backend selection (MNN_Volume creates a session per chunk). The device set does not change over a process's lifetime in any environment we care about, so the first caller's result is cached in a function-local static (thread-safe initialization guaranteed by C++11) and every later caller gets a plain bool read.
Intended use: MNN processors select MNN_FORWARD_VULKAN only when this returns true; on software-Vulkan-only hosts (llvmpipe/lavapipe in GPU-less CI containers) they select MNN_FORWARD_CPU instead – the native CPU backend is dramatically faster than Vulkan-on-lavapipe and restores the pre-WHOLEARCHIVE behavior these hosts always had (MNN's empty creator map silently CPU-fell-back then). Render passes keep their existing GTEST_SKIP policy instead.
Never throws.
Attributes Documentation¶
variable ARTIFACT_SERIALIZED_SIZE¶
Fixed total size of a serialized Artifact in bytes. resourceName[256] + artifactId[32] + passId[256] + outputBinding[256]
- dataType[64] + format[64] + width[4] + height[4] + depth[4] + byteSize[8]
- mediaType[128] + contentHash[32] + chunkHashCount[4] + chunkHashes[1024*32]
variable MANIFEST_SERIALIZED_SIZE¶
Fixed total size of a serialized ExecutionManifest in bytes. 5 * MAX_IDENTIFIER[256] + 6 * SHA256_HASH_SIZE[32] + inputArtifactCount[4]
- inputArtifactHashes[6432] + outputArtifactCount[4] + outputArtifactHashes[6432]
- startTimeUsec[8] + endTimeUsec[8] + terminalState[1] + gpuMemoryUsedBytes[8]
- outputBytesProduced[8] + wallClockUsec[8] + manifestHash[32]
variable MANIFEST_V2_SERIALIZED_SIZE¶
static constexpr size_t MANIFEST_V2_SERIALIZED_SIZE = MANIFEST_SERIALIZED_SIZE + sizeof( uint32_t ) + MAX_IDENTIFIER;
Trailer appended after the unchanged MANIFEST_SERIALIZED_SIZE base region (ARTF-10, schema evolution): schemaVersion[4] + errorMessage[256]. Expressed as an arithmetic expression (not a hardcoded literal) so it stays correct if MANIFEST_SERIALIZED_SIZE or MAX_IDENTIFIER ever change.
variable SHA256_HASH_SIZE¶
SHA-256 digest size in bytes.
variable MAX_RESOURCE_NAME¶
Max bytes for resource/pass/binding name strings (D-06)
variable MAX_MEDIA_TYPE¶
Max bytes for media type string (D-06)
variable MAX_ARTIFACT_REFS¶
Max input/output artifact hash references.
variable MAX_IDENTIFIER¶
Max bytes for execution/attempt/task/subtask/pass ID strings.
Updated on 2026-09-17 at 06:29:14 +0000