module LittleGhost::Support::OutputTruncation

OutputTruncation keeps large tool results within a predictable context budget without breaking UTF-8. Its byte-to-token estimate is deliberately approximate; use a provider tokenizer when exact accounting is required.

Constants

APPROX_BYTES_PER_TOKEN: Byte estimate used when no provider tokenizer is available.

Public Instance Methods

approx_bytes_for_tokens (tokens)

Source

# File lib/little_ghost/support/output_truncation.rb, line 20
def approx_bytes_for_tokens(tokens)
  Integer(tokens) * APPROX_BYTES_PER_TOKEN
end

Converts a token budget to its approximate byte budget.

approx_token_count (text)

Source

# File lib/little_ghost/support/output_truncation.rb, line 15
def approx_token_count(text)
  approx_tokens_from_byte_count(String(text).bytesize)
end

Estimates tokens from the UTF-8 byte length of text.

approx_tokens_from_byte_count (bytes)

Source

# File lib/little_ghost/support/output_truncation.rb, line 25
def approx_tokens_from_byte_count(bytes)
  (Integer(bytes) + APPROX_BYTES_PER_TOKEN - 1) / APPROX_BYTES_PER_TOKEN
end

Converts bytes to an approximate token count, rounded up.

truncate_middle_with_token_budget (text, max_tokens)

Source

# File lib/little_ghost/support/output_truncation.rb, line 31
def truncate_middle_with_token_budget(text, max_tokens)
  content = utf8_content(text)
  max_tokens = Integer(max_tokens)
  max_bytes = approx_bytes_for_tokens(max_tokens)
  return [content, nil] if max_tokens.positive? && content.bytesize <= max_bytes

  prefix, suffix = split_string(content, max_bytes / 2, max_bytes - (max_bytes / 2))
  removed_tokens = approx_tokens_from_byte_count([content.bytesize - max_bytes, 0].max)
  truncated = "#{prefix}…#{removed_tokens} tokens truncated…#{suffix}"
  [truncated, approx_token_count(content)]
end

Keeps text within budget or produces a middle-truncated UTF-8 string and the original approximate token count.