LombokSimHash

Near-duplicate document detection for RAG pipelines — SimHash, MinHash, LSH, SHA-256, and shingling.

LombokSimHash is library #02 of LombokRAGFrameworks — a Tier 4 Hub in the Lombok Ecosystem. It identifies redundant documents before they enter your vector store, preventing wasteful indexing of duplicate content.

• Zero dependencies — pure implementation in every language
• no_std compatible — Rust core runs in embedded/WASM environments
• Deterministic across languages — identical hash values in Rust, TypeScript, Python, Go, and PHP
• Streaming API — process large documents chunk-by-chunk

use lomboksimhash::{simhash64_text, hamming64};

let a = simhash64_text("the quick brown fox jumps over the lazy dog");
let b = simhash64_text("the quick brown cat jumps over the lazy dog");

let distance = hamming64(a, b);   // 8
let similar = distance <= 3;      // false — different enough

import { simhash64Text, hamming64, MinHasher, jaccardEstimate, LshIndex } from 'lomboksimhash';

const a = simhash64Text('the quick brown fox jumps over the lazy dog');
const b = simhash64Text('the quick brown cat jumps over the lazy dog');
console.log(hamming64(a, b));  // 8

// MinHash + Jaccard
const mh1 = new MinHasher(128);
mh1.addTokens(['the', 'quick', 'brown', 'fox']);
const mh2 = new MinHasher(128);
mh2.addTokens(['the', 'quick', 'brown', 'fox']);
console.log(jaccardEstimate(mh1.signature, mh2.signature));  // 1.0

from lomboksimhash import simhash64_text, hamming64, MinHasher, jaccard_estimate, LshIndex

a = simhash64_text("the quick brown fox jumps over the lazy dog")
b = simhash64_text("the quick brown cat jumps over the lazy dog")
print(hamming64(a, b))  # 8

# LSH index for approximate nearest neighbors
lsh = LshIndex(bands=20, rows=5)          # threshold ≈ 0.549
mh = MinHasher(num_perm=100)              # 20 * 5 = 100
mh.add_tokens(["the", "quick", "brown", "fox"])
lsh.insert("doc1", mh.signature)
print(lsh.query(mh.signature))            # ['doc1']

import "github.com/codinglombok/LombokSimHash"

a := lomboksimhash.SimHash64Text("the quick brown fox jumps over the lazy dog")
b := lomboksimhash.SimHash64Text("the quick brown cat jumps over the lazy dog")
dist := lomboksimhash.Hamming64(a, b)  // 8

use CodingLombok\LombokSimHash\LombokSimHash;

$a = LombokSimHash::simhash64Text('the quick brown fox jumps over the lazy dog');
$b = LombokSimHash::simhash64Text('the quick brown cat jumps over the lazy dog');
echo LombokSimHash::hamming64($a, $b);  // 8

$ lomboksimhash-cli simhash "hello world"
$ lomboksimhash-cli sha256 "hello world"
$ lomboksimhash-cli compare "text one" "text two"
Text 1 SimHash: ...
Text 2 SimHash: ...
Hamming distance: 12
Similarity: 81.3%

Document ──┬─→ SHA-256 ──────────→ exact duplicate?   → skip
           ├─→ SimHash64 ─────────→ Hamming ≤ 3?       → near-duplicate, skip
           └─→ Shingle → MinHash → LSH candidate + Jaccard ≥ 0.8? → skip
                                              ↓
                                        otherwise → chunk + embed

All ports produce byte-identical hash values. Verified against vectors/lomboksimhash-vectors-v1.json:

Guaranteed by fixed constants: FNV-1a (0xcbf29ce484222325 / 0x100000001b3), LCG seed 0x517cc1b727220a95, and Mersenne prime 2^61 - 1.

• Changelog — version history

Apache 2.0 © CodingLombok