Rust implementation of the CVM algorithm for counting distinct elements in a stream
Add timing for exact count
author
Stephan Hügel
date
(May 29, 2024, 3:12 PM +0100)
commit
977a179e
977a179ecd9827d2dbb3ef5cc64d04317eb24fb9
parent
f9382627
f9382627048734a8b470de7bc4dfe36a3d590327
+1
-1
+1
-1
README.md
+1
-1
README.md
···
63
63
If you're thinking about using this library, you presumably know that it only provides an estimate (within the specified bounds), similar to something like HyperLogLog. You are trading accuracy for speed!
64
64
65
65
## Perf
66
-
Calculating the unique tokens in a [418K UTF-8 text file](https://www.gutenberg.org/ebooks/8492) using the CLI takes 7.2 ms ± 0.3 ms on an M2 Pro. Counting 10e6 7-digit integers takes around 13.5 ms. Run `cargo bench` for more.
66
+
Calculating the unique tokens in a [418K UTF-8 text file](https://www.gutenberg.org/ebooks/8492) using the CLI takes 7.2 ms ± 0.3 ms on an M2 Pro. Counting 10e6 7-digit integers takes around 13.5 ms. An exact count using the same regex and HashSet runs in around 18 ms. Run `cargo bench` for more.
67
67
68
68
## Implementation Details
69
69
The CLI app strips punctuation from input tokens using a regex. I assume there is a small performance penalty, but it seems like a small price to pay for increased practicality.