Skip to content
Goxe

v1.6.0

Improvements in Goxe v1.6.0

Section titled “Improvements in Goxe v1.6.0”

Version 1.6.0 marks a milestone in the flexibility of the Goxe processing engine. We have implemented a recursive JSON curation system that allows for breaking down and normalizing complex nested objects, in addition to standardizing the development environment using Nix Flakes.

Key Changes

Section titled “Key Changes”
  • Recursive JSON Normalization: A new aggregation engine that dynamically processes nested structures.
  • Nix Flakes Support: Inclusion of flake.nix to ensure reproducible builds and development environments.
  • Strict I/O Validation: The CLI now validates file extensions (.log, .txt, .json) to prevent ingestion errors.
  • Typing Optimization: Refactored the jsonManager using type assertions to improve performance during concurrency.

Log Processing with the CLI

Section titled “Log Processing with the CLI”

Goxe simplifies log reduction using the -b (or --brew) flag. Starting with this version, the engine automatically detects the file structure to apply recursion if necessary.

Command Usage

Section titled “Command Usage”

  1. Prepare the source: Ensure your file has a valid extension (.json, .log, or .txt).

  2. Execute reduction:

    Terminal window
    goxe -b path/to/your/file.json

  3. Result: Goxe will generate a normalized summary, processing even the JSON objects found within other fields.

Recursive JSON Normalization

Section titled “Recursive JSON Normalization”

The JsonManager is introduced. This implementation of the File interface is specifically designed for structured log processing, allowing for normalization that goes beyond plain text.

Structural Transformation Example

Section titled “Structural Transformation Example”

Input (Nested JSON):

{
 "level": "error",
 "metadata": {
     "service": "auth-api",
     "details": {
         "code": 500,
         "reason": "timeout"
     }
  }
}

Signature generated by Goxe (Analysis Pattern):

ORIGIN: [file-reader]
- Data: [level:error||metadata:details:code:<NUM>||reason:timeout||||service:auth-api|||] - Count: [1]

Normalization Logic: Goxe identifies volatile patterns and replaces them with placeholders to group similar logs:

  • Type Detection: float64, int, and bool values are normalized to <NUM> and <BOOL>.
  • Unique Identifiers: Through the isId function, hashes, UUIDs, or tokens (strings > 20 characters without spaces) are flagged as <ID>.
  • Determinism: Map keys are sorted alphabetically before generating the ID to ensure consistent patterns.

View the full changelog