CHORUS/docs/comprehensive/packages/shhh.md

SHHH Package Documentation

Overview

The SHHH (Secrets Handler for Hidden Hazards) package provides CHORUS with a comprehensive secrets detection and redaction system. SHHH prevents sensitive data (API keys, tokens, private keys, passwords) from being leaked through logs, telemetry, request forwarding, or other output channels. It operates as a runtime sentinel with composable rules, audit logging, and operational metrics.

Table of Contents

• Architecture
• Sentinel Engine
• Pattern Matching
• Redaction Mechanisms
• Audit Logging
• Finding Severity Levels
• Configuration
• API Reference
• Usage Examples

Architecture

Design Principles

1. Defense in Depth: Multiple detection rules covering various secret types
2. Minimal False Positives: High-signal patterns focused on real credentials
3. Performance: Efficient regex compilation and concurrent scanning
4. Composability: Custom rules, audit sinks, and finding observers
5. Operational Visibility: Comprehensive metrics and statistics

Core Components

┌─────────────────────────────────────────────────────────────┐
│                         Sentinel                            │
│  ┌────────────────────────────────────────────────────┐   │
│  │              Compiled Rules                         │   │
│  │  • Bearer Tokens    • Private Keys                 │   │
│  │  • API Keys         • OAuth Tokens                 │   │
│  │  • OpenAI Secrets   • Custom Rules                 │   │
│  └────────────────────────────────────────────────────┘   │
│                           │                                 │
│  ┌────────────────────────┴────────────────────────────┐  │
│  │                 Redaction Engine                     │  │
│  │  • Pattern Matching   • Content Hashing             │  │
│  │  • Text Replacement   • Finding Aggregation         │  │
│  └────────────────────────┬────────────────────────────┘  │
│                           │                                 │
│  ┌────────────────────────┴────────────────────────────┐  │
│  │              Audit & Observability                   │  │
│  │  • Audit Sink         • Finding Observers            │  │
│  │  • Statistics         • Metrics                      │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Sentinel Engine

Creating a Sentinel

// Default configuration with built-in rules
sentinel, err := shhh.NewSentinel(shhh.Config{})
if err != nil {
    log.Fatal(err)
}

// Custom configuration
sentinel, err := shhh.NewSentinel(shhh.Config{
    Disabled:             false,
    RedactionPlaceholder: "[***REDACTED***]",
    DisableDefaultRules:  false,
    CustomRules: []shhh.RuleConfig{
        {
            Name:                "internal-api-key",
            Pattern:             `(?i)(internal[_-]key\s*[:=]\s*["']?)([A-Za-z0-9]{16,})(["']?)`,
            ReplacementTemplate: "$1[REDACTED]$3",
            Severity:            shhh.SeverityHigh,
            Tags:                []string{"api", "internal"},
        },
    },
})

Sentinel Options

Configure sentinel behavior with functional options:

// With audit sink
auditSink := NewAuditLogger()
sentinel, err := shhh.NewSentinel(cfg,
    shhh.WithAuditSink(auditSink),
)

// With custom stats collector
stats := shhh.NewStats()
sentinel, err := shhh.NewSentinel(cfg,
    shhh.WithStats(stats),
)

// With finding observer
sentinel, err := shhh.NewSentinel(cfg,
    shhh.WithFindingObserver(func(ctx context.Context, findings []shhh.Finding) {
        for _, f := range findings {
            log.Printf("Found %d instances of %s", f.Count, f.Rule)
        }
    }),
)

// Combine multiple options
sentinel, err := shhh.NewSentinel(cfg,
    shhh.WithAuditSink(auditSink),
    shhh.WithStats(stats),
    shhh.WithFindingObserver(observer),
)

Runtime Control

Enable, disable, or modify sentinel behavior at runtime:

// Check if enabled
if sentinel.Enabled() {
    fmt.Println("Sentinel is active")
}

// Toggle sentinel on/off
sentinel.Toggle(false) // Disable
sentinel.Toggle(true)  // Enable

// Update audit sink at runtime
newAuditSink := NewDatabaseAuditSink()
sentinel.SetAuditSink(newAuditSink)

// Add finding observer after creation
sentinel.AddFindingObserver(func(ctx context.Context, findings []shhh.Finding) {
    // Process findings
})

Pattern Matching

Built-in Rules

SHHH includes carefully curated default rules for common secrets:

1. Bearer Tokens

Pattern: (?i)(authorization\s*:\s*bearer\s+)([A-Za-z0-9\-._~+/]+=*)

Example:

Input:  Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.secret
Output: Authorization: Bearer [REDACTED]

Severity: Medium Tags: token, http

2. API Keys

Pattern: (?i)((?:api[_-]?key|token|secret|password)\s*[:=]\s*["']?)([A-Za-z0-9\-._~+/]{8,})(["']?)

Example:

Input:  API_KEY=sk_live_1234567890abcdef
Output: API_KEY=[REDACTED]

Severity: High Tags: credentials

3. OpenAI Secrets

Pattern: (sk-[A-Za-z0-9]{20,})

Example:

Input:  OPENAI_KEY=sk-proj-1234567890abcdefghij
Output: OPENAI_KEY=[REDACTED]

Severity: High Tags: llm, api

4. OAuth Refresh Tokens

Pattern: (?i)(refresh_token"?\s*[:=]\s*["']?)([A-Za-z0-9\-._~+/]{8,})(["']?)

Example:

Input:  refresh_token="1/abc123def456ghi789"
Output: refresh_token="[REDACTED]"

Severity: Medium Tags: oauth

5. Private Key Blocks

Pattern: (?s)(-----BEGIN [^-]+ PRIVATE KEY-----)[^-]+(-----END [^-]+ PRIVATE KEY-----)

Example:

Input:  -----BEGIN RSA PRIVATE KEY-----
        MIIEpAIBAAKCAQEA...
        -----END RSA PRIVATE KEY-----

Output: -----BEGIN RSA PRIVATE KEY-----
        [REDACTED]
        -----END RSA PRIVATE KEY-----

Severity: High Tags: pem, key

Custom Rules

Define custom redaction rules for domain-specific secrets:

customRule := shhh.RuleConfig{
    Name:                "database-password",
    Pattern:             `(?i)(db[_-]?pass(?:word)?\s*[:=]\s*["']?)([^"'\s]{8,})(["']?)`,
    ReplacementTemplate: "$1[REDACTED]$3",
    Severity:            shhh.SeverityHigh,
    Tags:                []string{"database", "credentials"},
}

sentinel, err := shhh.NewSentinel(shhh.Config{
    CustomRules: []shhh.RuleConfig{customRule},
})

Pattern Syntax

Rules use Go's regexp package syntax:

• (?i) - Case-insensitive matching
• (?s) - Dot matches newlines (for multi-line patterns)
• ([^"'\s]{8,}) - Capture group: non-quote/space chars, min 8 length
• $1, $2 - Backreferences in replacement template

Best Practices:

1. Use capture groups to preserve context (prefixes, quotes)
2. Be specific to reduce false positives
3. Test patterns against real data samples
4. Consider minimum length requirements
5. Use anchors when appropriate (\b for word boundaries)

Redaction Mechanisms

Text Redaction

Redact secrets from plain text:

input := `
Config:
  API_KEY=sk_live_1234567890abcdef
  DB_PASSWORD=supersecret123
  Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.secret
`

// Labels provide context for audit logs
labels := map[string]string{
    "source": "config_file",
    "path":   "/etc/app/config.yaml",
}

redacted, findings := sentinel.RedactText(ctx, input, labels)

fmt.Println(redacted)
// Output:
// Config:
//   API_KEY=[REDACTED]
//   DB_PASSWORD=supersecret123
//   Authorization: Bearer [REDACTED]

fmt.Printf("Found %d types of secrets\n", len(findings))
for _, f := range findings {
    fmt.Printf("  %s: %d occurrences (severity: %s)\n",
        f.Rule,
        f.Count,
        f.Severity,
    )
}

Map Redaction

Redact secrets from structured data (in-place):

payload := map[string]any{
    "user": "john@example.com",
    "config": map[string]any{
        "api_key": "sk_live_1234567890abcdef",
        "timeout": 30,
    },
    "tokens": []any{
        "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.secret",
        map[string]any{
            "refresh": "refresh_token=abc123def456",
        },
    },
}

findings := sentinel.RedactMap(ctx, payload)

// payload is modified in-place
fmt.Printf("%+v\n", payload)
// Output:
// map[
//   user:john@example.com
//   config:map[api_key:[REDACTED] timeout:30]
//   tokens:[
//     [REDACTED]
//     map[refresh:refresh_token=[REDACTED]]
//   ]
// ]

// With base labels
baseLabels := map[string]string{
    "source": "http_request",
    "method": "POST",
}
findings := sentinel.RedactMapWithLabels(ctx, payload, baseLabels)

Nested Structure Traversal

SHHH recursively traverses nested structures:

1. Maps: Scans all string values, recurses into nested maps/slices
2. Slices: Scans all elements, handles mixed types
3. Strings: Applies all rules in order
4. Stringer Interface: Converts to string and scans

Path Generation:

• Map keys: parent.child.grandchild
• Array indices: parent[0], parent[1].child
• Root: Empty string or label-derived path

// Complex nested structure
data := map[string]any{
    "services": []any{
        map[string]any{
            "name": "api",
            "auth": map[string]any{
                "token": "Bearer secret123",
            },
        },
        map[string]any{
            "name": "worker",
            "auth": map[string]any{
                "token": "Bearer secret456",
            },
        },
    },
}

findings := sentinel.RedactMap(ctx, data)

// Findings include location paths
for _, finding := range findings {
    for _, loc := range finding.Locations {
        fmt.Printf("%s: %d occurrences at %s\n",
            finding.Rule,
            loc.Count,
            loc.Path,
        )
    }
}
// Output:
// bearer-token: 1 occurrences at services[0].auth.token
// bearer-token: 1 occurrences at services[1].auth.token

Audit Logging

Audit Events

Each redaction generates an audit event:

type AuditEvent struct {
    Rule     string            `json:"rule"`
    Severity Severity          `json:"severity"`
    Tags     []string          `json:"tags,omitempty"`
    Path     string            `json:"path,omitempty"`
    Hash     string            `json:"hash"`
    Metadata map[string]string `json:"metadata,omitempty"`
}

Fields:

• Rule: Name of the rule that matched
• Severity: Severity level (low, medium, high)
• Tags: Tags associated with the rule
• Path: Location in structure where secret was found
• Hash: SHA-256 hash of the secret value (for tracking)
• Metadata: Additional context (source, labels, etc.)

Implementing Audit Sinks

Create custom audit sinks to persist events:

// File audit sink
type FileAuditSink struct {
    file *os.File
    mu   sync.Mutex
}

func (f *FileAuditSink) RecordRedaction(ctx context.Context, event shhh.AuditEvent) {
    f.mu.Lock()
    defer f.mu.Unlock()

    eventJSON, _ := json.Marshal(event)
    f.file.Write(eventJSON)
    f.file.WriteString("\n")
}

// Database audit sink
type DBQuditSink struct {
    db *sql.DB
}

func (d *DBAuditSink) RecordRedaction(ctx context.Context, event shhh.AuditEvent) {
    _, err := d.db.ExecContext(ctx,
        `INSERT INTO audit_events (rule, severity, path, hash, metadata, created_at)
         VALUES ($1, $2, $3, $4, $5, NOW())`,
        event.Rule,
        event.Severity,
        event.Path,
        event.Hash,
        event.Metadata,
    )
    if err != nil {
        log.Printf("Failed to record audit event: %v", err)
    }
}

// Syslog audit sink
type SyslogAuditSink struct {
    writer *syslog.Writer
}

func (s *SyslogAuditSink) RecordRedaction(ctx context.Context, event shhh.AuditEvent) {
    priority := syslog.LOG_WARNING
    if event.Severity == shhh.SeverityHigh {
        priority = syslog.LOG_ERR
    }

    msg := fmt.Sprintf("SHHH: %s detected at %s (hash: %s)",
        event.Rule,
        event.Path,
        event.Hash,
    )

    s.writer.Write(priority, msg)
}

Secret Hashing

SHHH hashes detected secrets using SHA-256 for tracking without storing plaintext:

// Automatic hashing (internal)
secretValue := "sk_live_1234567890abcdef"
hash := sha256.Sum256([]byte(secretValue))
hashString := base64.RawStdEncoding.EncodeToString(hash[:])

// Hash in audit event
event := shhh.AuditEvent{
    Rule: "api-key",
    Hash: hashString, // "vJ8x3mHNqR8..."
}

Use Cases:

• Track repeated leaks of same secret
• Correlate incidents across systems
• Detect secret rotation failures
• Never store plaintext secrets in audit logs

Finding Severity Levels

Severity Enum

const (
    SeverityLow    Severity = "low"
    SeverityMedium Severity = "medium"
    SeverityHigh   Severity = "high"
)

Severity Guidelines

Low Severity

Impact: Minimal security risk

Examples:

• Development/testing credentials
• Non-production API keys
• Internal documentation tokens
• Temporary access codes

Response: Log and notify, no immediate action required

Medium Severity

Impact: Moderate security risk, limited blast radius

Examples:

• Access tokens (short-lived)
• Bearer tokens (limited scope)
• OAuth refresh tokens
• Session identifiers

Response: Log, notify, consider rotation

High Severity

Impact: Critical security risk, potential full compromise

Examples:

• Private keys (RSA, ECDSA, Ed25519)
• Master API keys
• Database passwords
• Service account credentials
• Production secrets

Response: Immediate alert, mandatory rotation, incident investigation

Finding Structure

type Finding struct {
    Rule      string     `json:"rule"`
    Severity  Severity   `json:"severity"`
    Tags      []string   `json:"tags,omitempty"`
    Count     int        `json:"count"`
    Locations []Location `json:"locations,omitempty"`
}

type Location struct {
    Path  string `json:"path"`
    Count int    `json:"count"`
}

Finding Observers

React to findings in real-time:

sentinel, err := shhh.NewSentinel(cfg,
    shhh.WithFindingObserver(func(ctx context.Context, findings []shhh.Finding) {
        for _, finding := range findings {
            switch finding.Severity {
            case shhh.SeverityHigh:
                // Immediate alert
                alerting.SendCritical(fmt.Sprintf(
                    "HIGH SEVERITY SECRET DETECTED: %s (%d occurrences)",
                    finding.Rule,
                    finding.Count,
                ))

                // Log with full context
                for _, loc := range finding.Locations {
                    log.Printf("  Location: %s (%d times)", loc.Path, loc.Count)
                }

            case shhh.SeverityMedium:
                // Standard notification
                log.Printf("MEDIUM SEVERITY: %s detected %d times",
                    finding.Rule,
                    finding.Count,
                )

            case shhh.SeverityLow:
                // Debug logging
                log.Printf("DEBUG: %s detected %d times",
                    finding.Rule,
                    finding.Count,
                )
            }
        }
    }),
)

Configuration

Config Structure

type Config struct {
    // Disabled toggles redaction off entirely
    Disabled bool `json:"disabled"`

    // RedactionPlaceholder overrides the default "[REDACTED]"
    RedactionPlaceholder string `json:"redaction_placeholder"`

    // DisableDefaultRules disables built-in curated rule set
    DisableDefaultRules bool `json:"disable_default_rules"`

    // CustomRules allows bespoke redaction patterns
    CustomRules []RuleConfig `json:"custom_rules"`
}

type RuleConfig struct {
    Name                string   `json:"name"`
    Pattern             string   `json:"pattern"`
    ReplacementTemplate string   `json:"replacement_template"`
    Severity            Severity `json:"severity"`
    Tags                []string `json:"tags"`
}

Configuration Examples

Minimal (Defaults)

cfg := shhh.Config{}
sentinel, err := shhh.NewSentinel(cfg)
// Uses default rules, "[REDACTED]" placeholder

Custom Placeholder

cfg := shhh.Config{
    RedactionPlaceholder: "***SENSITIVE***",
}
sentinel, err := shhh.NewSentinel(cfg)

Custom Rules Only

cfg := shhh.Config{
    DisableDefaultRules: true,
    CustomRules: []shhh.RuleConfig{
        {
            Name:                "credit-card",
            Pattern:             `\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b`,
            ReplacementTemplate: "[CARD-REDACTED]",
            Severity:            shhh.SeverityHigh,
            Tags:                []string{"pci", "payment"},
        },
        {
            Name:                "ssn",
            Pattern:             `\b\d{3}-\d{2}-\d{4}\b`,
            ReplacementTemplate: "[SSN-REDACTED]",
            Severity:            shhh.SeverityHigh,
            Tags:                []string{"pii", "identity"},
        },
    },
}
sentinel, err := shhh.NewSentinel(cfg)

Augment Default Rules

cfg := shhh.Config{
    CustomRules: []shhh.RuleConfig{
        {
            Name:                "internal-token",
            Pattern:             `(?i)(x-internal-token\s*:\s*)([A-Za-z0-9]{16,})`,
            ReplacementTemplate: "$1[REDACTED]",
            Severity:            shhh.SeverityMedium,
            Tags:                []string{"internal", "http"},
        },
    },
}
sentinel, err := shhh.NewSentinel(cfg)
// Uses default rules + custom rule

From JSON

{
  "disabled": false,
  "redaction_placeholder": "[***]",
  "disable_default_rules": false,
  "custom_rules": [
    {
      "name": "gitlab-token",
      "pattern": "(?i)(gitlab[_-]token\\s*[:=]\\s*[\"']?)([A-Za-z0-9_-]{20,})([\"']?)",
      "replacement_template": "$1[REDACTED]$3",
      "severity": "high",
      "tags": ["gitlab", "vcs"]
    }
  ]
}

var cfg shhh.Config
err := json.Unmarshal(configJSON, &cfg)
if err != nil {
    log.Fatal(err)
}

sentinel, err := shhh.NewSentinel(cfg)

Statistics

Stats Tracking

SHHH maintains comprehensive operational metrics:

type StatsSnapshot struct {
    TotalScans      uint64            `json:"total_scans"`
    TotalFindings   uint64            `json:"total_findings"`
    PerRuleFindings map[string]uint64 `json:"per_rule_findings"`
}

Retrieving Statistics

// Get snapshot
snapshot := sentinel.StatsSnapshot()

fmt.Printf("Total scans: %d\n", snapshot.TotalScans)
fmt.Printf("Total findings: %d\n", snapshot.TotalFindings)
fmt.Printf("Average findings per scan: %.2f\n",
    float64(snapshot.TotalFindings) / float64(snapshot.TotalScans),
)

fmt.Println("\nPer-rule statistics:")
for rule, count := range snapshot.PerRuleFindings {
    fmt.Printf("  %s: %d\n", rule, count)
}

Shared Stats Collector

Share stats across multiple sentinels:

// Create shared stats
stats := shhh.NewStats()

// Create multiple sentinels sharing stats
sentinel1, _ := shhh.NewSentinel(cfg1, shhh.WithStats(stats))
sentinel2, _ := shhh.NewSentinel(cfg2, shhh.WithStats(stats))

// Both sentinels contribute to same stats
sentinel1.RedactText(ctx, text1, nil)
sentinel2.RedactText(ctx, text2, nil)

// Get combined statistics
snapshot := stats.Snapshot()

API Reference

Core Types

// Sentinel - main redaction engine
type Sentinel struct { /* ... */ }

// Finding - detected secret information
type Finding struct {
    Rule      string
    Severity  Severity
    Tags      []string
    Count     int
    Locations []Location
}

// Location - where secret was found
type Location struct {
    Path  string
    Count int
}

// AuditEvent - audit log entry
type AuditEvent struct {
    Rule     string
    Severity Severity
    Tags     []string
    Path     string
    Hash     string
    Metadata map[string]string
}

Sentinel Methods

// NewSentinel creates a new secrets sentinel
func NewSentinel(cfg Config, opts ...Option) (*Sentinel, error)

// RedactText scans and redacts text
func (s *Sentinel) RedactText(ctx context.Context, text string, labels map[string]string) (string, []Finding)

// RedactMap scans and redacts map in-place
func (s *Sentinel) RedactMap(ctx context.Context, payload map[string]any) []Finding

// RedactMapWithLabels redacts map with base labels
func (s *Sentinel) RedactMapWithLabels(ctx context.Context, payload map[string]any, baseLabels map[string]string) []Finding

// Enabled reports if sentinel is active
func (s *Sentinel) Enabled() bool

// Toggle enables/disables sentinel
func (s *Sentinel) Toggle(enabled bool)

// SetAuditSink updates audit sink at runtime
func (s *Sentinel) SetAuditSink(sink AuditSink)

// AddFindingObserver registers finding observer
func (s *Sentinel) AddFindingObserver(observer FindingObserver)

// StatsSnapshot returns current statistics
func (s *Sentinel) StatsSnapshot() StatsSnapshot

Options

// WithAuditSink attaches audit sink
func WithAuditSink(sink AuditSink) Option

// WithStats supplies shared stats collector
func WithStats(stats *Stats) Option

// WithFindingObserver registers finding observer
func WithFindingObserver(observer FindingObserver) Option

Interfaces

// AuditSink receives redaction events
type AuditSink interface {
    RecordRedaction(ctx context.Context, event AuditEvent)
}

// FindingObserver receives aggregated findings
type FindingObserver func(context.Context, []Finding)

Usage Examples

Example 1: Basic Text Redaction

package main

import (
    "context"
    "fmt"
    "log"

    "chorus/pkg/shhh"
)

func main() {
    // Create sentinel with defaults
    sentinel, err := shhh.NewSentinel(shhh.Config{})
    if err != nil {
        log.Fatal(err)
    }

    // Sample text with secrets
    input := `
    API Configuration:
    - API_KEY=sk_live_1234567890abcdef
    - DB_PASSWORD=supersecret123
    - Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.payload
    `

    // Redact
    redacted, findings := sentinel.RedactText(
        context.Background(),
        input,
        map[string]string{"source": "config"},
    )

    fmt.Println("Redacted output:")
    fmt.Println(redacted)

    fmt.Printf("\nFound %d types of secrets:\n", len(findings))
    for _, finding := range findings {
        fmt.Printf("- %s: %d occurrences [%s]\n",
            finding.Rule,
            finding.Count,
            finding.Severity,
        )
    }
}

Example 2: HTTP Request Redaction

package main

import (
    "context"
    "encoding/json"
    "io"
    "log"
    "net/http"

    "chorus/pkg/shhh"
)

type Server struct {
    sentinel *shhh.Sentinel
}

func NewServer() (*Server, error) {
    sentinel, err := shhh.NewSentinel(shhh.Config{})
    if err != nil {
        return nil, err
    }

    return &Server{sentinel: sentinel}, nil
}

func (s *Server) HandleRequest(w http.ResponseWriter, r *http.Request) {
    // Read request body
    body, _ := io.ReadAll(r.Body)
    r.Body.Close()

    // Parse JSON
    var payload map[string]any
    if err := json.Unmarshal(body, &payload); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }

    // Redact secrets before logging
    baseLabels := map[string]string{
        "source":     "http_request",
        "method":     r.Method,
        "path":       r.URL.Path,
        "remote_ip":  r.RemoteAddr,
    }

    findings := s.sentinel.RedactMapWithLabels(
        context.Background(),
        payload,
        baseLabels,
    )

    if len(findings) > 0 {
        log.Printf("⚠ Redacted %d types of secrets from request", len(findings))
        for _, f := range findings {
            if f.Severity == shhh.SeverityHigh {
                log.Printf("  HIGH: %s (%d occurrences)", f.Rule, f.Count)
            }
        }
    }

    // Safe to log now
    safeJSON, _ := json.Marshal(payload)
    log.Printf("Request payload: %s", safeJSON)

    // Process request...
    w.WriteHeader(http.StatusOK)
}

func main() {
    server, err := NewServer()
    if err != nil {
        log.Fatal(err)
    }

    http.HandleFunc("/api/", server.HandleRequest)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Example 3: Structured Logging Integration

package main

import (
    "context"
    "fmt"

    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"

    "chorus/pkg/shhh"
)

// SecureLogger wraps zap.Logger with SHHH redaction
type SecureLogger struct {
    logger   *zap.Logger
    sentinel *shhh.Sentinel
}

func NewSecureLogger() (*SecureLogger, error) {
    logger, err := zap.NewProduction()
    if err != nil {
        return nil, err
    }

    sentinel, err := shhh.NewSentinel(shhh.Config{})
    if err != nil {
        return nil, err
    }

    return &SecureLogger{
        logger:   logger,
        sentinel: sentinel,
    }, nil
}

func (sl *SecureLogger) Info(msg string, fields ...zapcore.Field) {
    // Redact message
    redacted, _ := sl.sentinel.RedactText(
        context.Background(),
        msg,
        nil,
    )

    // Redact field values
    safeFields := make([]zapcore.Field, len(fields))
    for i, field := range fields {
        if field.Type == zapcore.StringType {
            redactedValue, _ := sl.sentinel.RedactText(
                context.Background(),
                field.String,
                nil,
            )
            safeFields[i] = zap.String(field.Key, redactedValue)
        } else {
            safeFields[i] = field
        }
    }

    sl.logger.Info(redacted, safeFields...)
}

func (sl *SecureLogger) Warn(msg string, fields ...zapcore.Field) {
    redacted, findings := sl.sentinel.RedactText(
        context.Background(),
        msg,
        nil,
    )

    if len(findings) > 0 {
        sl.logger.Warn("Secrets detected in log message",
            zap.Int("finding_count", len(findings)),
        )
    }

    sl.logger.Warn(redacted, fields...)
}

func main() {
    logger, err := NewSecureLogger()
    if err != nil {
        panic(err)
    }

    // Safe logging - secrets automatically redacted
    logger.Info("Connecting to API",
        zap.String("api_key", "sk_live_1234567890abcdef"),
        zap.String("endpoint", "https://api.example.com"),
    )

    logger.Warn("Authentication failed with token: Bearer eyJhbGci...")
}

Example 4: Audit Trail with Database Sink

package main

import (
    "context"
    "database/sql"
    "encoding/json"
    "log"

    _ "github.com/lib/pq"
    "chorus/pkg/shhh"
)

// DBAuditSink persists audit events to PostgreSQL
type DBAuditSink struct {
    db *sql.DB
}

func NewDBAuditSink(connStr string) (*DBAuditSink, error) {
    db, err := sql.Open("postgres", connStr)
    if err != nil {
        return nil, err
    }

    // Create audit table
    _, err = db.Exec(`
        CREATE TABLE IF NOT EXISTS shhh_audit (
            id SERIAL PRIMARY KEY,
            rule VARCHAR(100) NOT NULL,
            severity VARCHAR(20) NOT NULL,
            path TEXT,
            hash VARCHAR(64) NOT NULL,
            tags JSONB,
            metadata JSONB,
            created_at TIMESTAMP DEFAULT NOW()
        )
    `)
    if err != nil {
        return nil, err
    }

    return &DBAuditSink{db: db}, nil
}

func (d *DBAuditSink) RecordRedaction(ctx context.Context, event shhh.AuditEvent) {
    tagsJSON, _ := json.Marshal(event.Tags)
    metadataJSON, _ := json.Marshal(event.Metadata)

    _, err := d.db.ExecContext(ctx,
        `INSERT INTO shhh_audit (rule, severity, path, hash, tags, metadata)
         VALUES ($1, $2, $3, $4, $5, $6)`,
        event.Rule,
        event.Severity,
        event.Path,
        event.Hash,
        tagsJSON,
        metadataJSON,
    )

    if err != nil {
        log.Printf("Failed to record audit event: %v", err)
    }
}

func (d *DBAuditSink) GetRecentFindings(hours int) ([]shhh.AuditEvent, error) {
    rows, err := d.db.Query(`
        SELECT rule, severity, path, hash, tags, metadata, created_at
        FROM shhh_audit
        WHERE created_at > NOW() - INTERVAL '$1 hours'
        ORDER BY created_at DESC
    `, hours)
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    var events []shhh.AuditEvent
    for rows.Next() {
        var event shhh.AuditEvent
        var tagsJSON, metadataJSON []byte
        var createdAt string

        err := rows.Scan(
            &event.Rule,
            &event.Severity,
            &event.Path,
            &event.Hash,
            &tagsJSON,
            &metadataJSON,
            &createdAt,
        )
        if err != nil {
            continue
        }

        json.Unmarshal(tagsJSON, &event.Tags)
        json.Unmarshal(metadataJSON, &event.Metadata)

        events = append(events, event)
    }

    return events, nil
}

func main() {
    // Create audit sink
    auditSink, err := NewDBAuditSink("postgres://user:pass@localhost/chorus?sslmode=disable")
    if err != nil {
        log.Fatal(err)
    }

    // Create sentinel with audit sink
    sentinel, err := shhh.NewSentinel(
        shhh.Config{},
        shhh.WithAuditSink(auditSink),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Redact text (events automatically recorded)
    text := "API_KEY=sk_live_1234567890abcdef"
    sentinel.RedactText(
        context.Background(),
        text,
        map[string]string{
            "source": "user_input",
            "user_id": "user-123",
        },
    )

    // Query recent findings
    findings, err := auditSink.GetRecentFindings(24)
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("Found %d audit events in last 24 hours", len(findings))
    for _, finding := range findings {
        log.Printf("- %s: %s (hash: %s)", finding.Rule, finding.Severity, finding.Hash)
    }
}

Example 5: Real-time Alerting

package main

import (
    "context"
    "fmt"
    "log"

    "chorus/pkg/shhh"
)

// AlertingObserver sends alerts for high-severity findings
func AlertingObserver(ctx context.Context, findings []shhh.Finding) {
    for _, finding := range findings {
        if finding.Severity != shhh.SeverityHigh {
            continue
        }

        // Send alert
        alert := fmt.Sprintf(
            "🚨 HIGH SEVERITY SECRET DETECTED\n"+
            "Rule: %s\n"+
            "Count: %d\n"+
            "Tags: %v\n",
            finding.Rule,
            finding.Count,
            finding.Tags,
        )

        if len(finding.Locations) > 0 {
            alert += "Locations:\n"
            for _, loc := range finding.Locations {
                alert += fmt.Sprintf("  - %s (%d times)\n", loc.Path, loc.Count)
            }
        }

        // Send via Slack, PagerDuty, email, etc.
        sendAlert(alert)
    }
}

// MetricsObserver tracks findings in Prometheus
func MetricsObserver(ctx context.Context, findings []shhh.Finding) {
    for _, finding := range findings {
        // Increment Prometheus counter
        secretsDetectedCounter.WithLabelValues(
            finding.Rule,
            string(finding.Severity),
        ).Add(float64(finding.Count))
    }
}

func sendAlert(message string) {
    // Slack webhook
    log.Printf("ALERT: %s", message)

    // In production:
    // slack.PostMessage(message)
    // pagerduty.CreateIncident(message)
    // email.Send(message)
}

func main() {
    sentinel, err := shhh.NewSentinel(
        shhh.Config{},
        shhh.WithFindingObserver(AlertingObserver),
        shhh.WithFindingObserver(MetricsObserver),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Simulate secret detection
    text := `
    Production credentials:
    AWS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    PRIVATE_KEY=-----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEA...
    -----END RSA PRIVATE KEY-----
    `

    sentinel.RedactText(
        context.Background(),
        text,
        map[string]string{
            "source": "production_config",
            "environment": "prod",
        },
    )

    // Alerts triggered automatically via observers
}

Best Practices

Rule Design

1. Be Specific: Minimize false positives with precise patterns
2. Test Thoroughly: Validate rules against real data samples
3. Use Capture Groups: Preserve context around secrets
4. Consider Performance: Avoid overly complex regex patterns
5. Document Rules: Add clear names and tags

Integration

1. Redact Early: Apply redaction before logging or transmission
2. Audit Everything: Enable audit sinks in production
3. Monitor Metrics: Track detection rates and patterns
4. Alert on High Severity: Immediate alerts for critical secrets
5. Regular Reviews: Periodically review audit logs for patterns

Performance

1. Compile Rules Once: Create sentinel at startup, reuse across requests
2. Share Stats: Use shared stats collector for multiple sentinels
3. Batch Operations: Redact entire structures rather than individual fields
4. Minimize Rules: Only include necessary custom rules

Security

1. Never Log Plaintext: Always redact before logging
2. Hash for Tracking: Use audit event hashes to track without storing secrets
3. Rotate on Detection: Treat secret detection as potential compromise
4. Principle of Least Privilege: Restrict audit log access
5. Encrypt Audit Logs: Protect audit logs with encryption at rest

Integration Points

COOEE Logger Integration

// In COOEE logger initialization
sentinel, _ := shhh.NewSentinel(shhh.Config{})

func (l *Logger) Log(level, message string, fields map[string]interface{}) {
    // Redact message
    redacted, _ := sentinel.RedactText(context.Background(), message, nil)

    // Redact fields
    findings := sentinel.RedactMap(context.Background(), fields)

    if len(findings) > 0 {
        fields["_shhh_redactions"] = len(findings)
    }

    // Safe to log
    l.backend.Write(level, redacted, fields)
}

WHOOSH Search Integration

// Before indexing documents
func (idx *Indexer) IndexDocument(doc Document) error {
    // Redact sensitive fields
    findings := sentinel.RedactMap(context.Background(), doc.Fields)

    if len(findings) > 0 {
        log.Printf("Redacted %d secrets before indexing", len(findings))
    }

    // Safe to index
    return idx.backend.Index(doc)
}

CHORUS Agent Integration

// In agent message handling
func (a *Agent) SendMessage(msg Message) error {
    // Redact message content
    redactedContent, _ := sentinel.RedactText(
        context.Background(),
        msg.Content,
        map[string]string{
            "agent": a.ID,
            "channel": msg.Channel,
        },
    )

    msg.Content = redactedContent

    return a.transport.Send(msg)
}

Troubleshooting

High False Positive Rate

Problem: Rules matching non-secret content

Solutions:

• Make patterns more specific
• Add negative lookaheads to exclude known patterns
• Increase minimum length requirements
• Use word boundaries (\b)

Performance Issues

Problem: Slow redaction on large payloads

Solutions:

• Profile regex patterns for complexity
• Reduce number of custom rules
• Process in chunks for very large inputs
• Consider async redaction for non-critical paths

Missing Detections

Problem: Secrets not being caught

Solutions:

• Add custom rules for domain-specific secrets
• Review audit logs for patterns
• Test rules against known secret formats
• Consider case-insensitive matching ((?i))

See Also

• CHORUS Security Architecture
• COOEE Logging Package
• UCXL Package
• Audit Trail System