A tokenised name, with everything the comparison needs precomputed. More...
Collaboration diagram for ls_tokens_t:Data Fields | |
| char | t [LS_MAX_TOKENS][LS_TOKEN_LEN] |
| unsigned | h [LS_MAX_TOKENS] |
| unsigned char | len [LS_MAX_TOKENS] |
| unsigned long long | bloom |
| int | n |
Detailed Description
A tokenised name, with everything the comparison needs precomputed.
The hash and the length are not an optimisation detail, they are most of the function's cost. Comparing two names is O(tokens x tokens), and doing that with strcmp() – calling strlen() inside the inner loop, no less – measured 610 ns per catalogue name, against 82 ns for SQLite to hand the row over. Scoring was 88% of a lookup.
bloom is the OR of (1 << (hash % 64)) over the tokens: if two names share no bit they share no token, so the whole quadratic comparison can be skipped on one AND.
Definition at line 743 of file lensserious_db.c.
Field Documentation
◆ bloom
| unsigned long long ls_tokens_t::bloom |
Definition at line 748 of file lensserious_db.c.
Referenced by _digest_load(), _tokenize(), and ls_db_match_lens().
◆ h
| unsigned ls_tokens_t::h[LS_MAX_TOKENS] |
FNV-1a of the token
Definition at line 746 of file lensserious_db.c.
Referenced by _digest_load(), _score_tokens(), and _tokenize().
◆ len
| unsigned char ls_tokens_t::len[LS_MAX_TOKENS] |
Definition at line 747 of file lensserious_db.c.
Referenced by _digest_load(), _score_tokens(), and _tokenize().
◆ n
| int ls_tokens_t::n |
Definition at line 749 of file lensserious_db.c.
Referenced by _digest_load(), _score_tokens(), _tokenize(), and ls_db_match_lens().
◆ t
| char ls_tokens_t::t[LS_MAX_TOKENS][LS_TOKEN_LEN] |
Definition at line 745 of file lensserious_db.c.
Referenced by _tokenize(), and ls_db_match_lens().
The documentation for this struct was generated from the following file:
