Sitelet https://web.archive.org/web/20201202192336/https://github.com/thanos-io/thanos/pull/3531
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

store: Optimized common cases for time selecting tiny amount series. Avoid looking up symbols. #3531

Open
wants to merge 1 commit into
base: master
from

Conversation

@bwplotka
Copy link
Member

@bwplotka bwplotka commented Dec 2, 2020 •

Key rationales: The majority of query latency comes from symbols.Lookup.

Also cleaned up the code.

Benchmarks

Baseline:

/tmp/___BenchmarkTelemeterRealData_Series_in_github_com_thanos_io_thanos_pkg_store -test.v -test.bench ^\QBenchmarkTelemeterRealData_Series\E$ -test.run ^$ -test.benchtime=1m -test.benchmem
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkTelemeterRealData_Series
Built index header; Starting
BenchmarkTelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12         	      43	1866550478 ns/op	395648807 B/op	 4348808 allocs/op
BenchmarkTelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12        	      50	1417560470 ns/op	160595177 B/op	 3017177 allocs/op
BenchmarkTelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12      	     392	 168130862 ns/op	49921099 B/op	  493076 allocs/op
BenchmarkTelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12           	      14	5295236741 ns/op	1083815895 B/op	11798226 allocs/op
BenchmarkTelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12          	      18	4149971387 ns/op	436568360 B/op	 8577264 allocs/op
PASS

Process finished with exit code 0

``

New

benchstat -delta-test=none _dev/bench/store_symb_a.txt _dev/bench/store_symb_b.txt
name                                                                old time/op    new time/op    delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        1.87s ± 0%     1.66s ± 0%  -10.85%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       1.42s ± 0%     0.25s ± 0%  -82.17%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12     168ms ± 0%     157ms ± 0%   -6.72%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          5.30s ± 0%     4.53s ± 0%  -14.53%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         4.15s ± 0%     0.68s ± 0%  -83.67%

name                                                                old alloc/op   new alloc/op   delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        396MB ± 0%     396MB ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       161MB ± 0%      95MB ± 0%  -40.95%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12    49.9MB ± 0%    49.9MB ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12         1.08GB ± 0%    1.08GB ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         437MB ± 0%     228MB ± 0%  -47.77%

name                                                                old allocs/op  new allocs/op  delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        4.35M ± 0%     4.35M ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       3.02M ± 0%     0.05M ± 0%  -98.36%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12      493k ± 0%      493k ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          11.8M ± 0%     11.8M ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         8.58M ± 0%     0.02M ± 0%  -99.78%

Cmp:

benchstat -delta-test=none _dev/bench/store_symb_a.txt _dev/bench/store_symb_b.txt
name                                                                old time/op    new time/op    delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        1.87s ± 0%     1.66s ± 0%  -10.85%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       1.42s ± 0%     0.25s ± 0%  -82.17%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12     168ms ± 0%     157ms ± 0%   -6.72%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          5.30s ± 0%     4.53s ± 0%  -14.53%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         4.15s ± 0%     0.68s ± 0%  -83.67%

name                                                                old alloc/op   new alloc/op   delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        396MB ± 0%     396MB ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       161MB ± 0%      95MB ± 0%  -40.95%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12    49.9MB ± 0%    49.9MB ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12         1.08GB ± 0%    1.08GB ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         437MB ± 0%     228MB ± 0%  -47.77%

name                                                                old allocs/op  new allocs/op  delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        4.35M ± 0%     4.35M ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       3.02M ± 0%     0.05M ± 0%  -98.36%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12      493k ± 0%      493k ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          11.8M ± 0%     11.8M ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         8.58M ± 0%     0.02M ± 0%  -99.78%

Code: https://gist.github.com/bwplotka/cbcbbcd1802181b7785da11dcc0f5cfd

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

…Avoid looking up symbols.

Key part: Majority of query latency comes from symbols.Lookup.

Also cleaned up the code.

## Benchmarks

Baseline:
```
/tmp/___BenchmarkTelemeterRealData_Series_in_github_com_thanos_io_thanos_pkg_store -test.v -test.bench ^\QBenchmarkTelemeterRealData_Series\E$ -test.run ^$ -test.benchtime=1m -test.benchmem
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkTelemeterRealData_Series
Built index header; Starting
BenchmarkTelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12         	      43	1866550478 ns/op	395648807 B/op	 4348808 allocs/op
BenchmarkTelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12        	      50	1417560470 ns/op	160595177 B/op	 3017177 allocs/op
BenchmarkTelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12      	     392	 168130862 ns/op	49921099 B/op	  493076 allocs/op
BenchmarkTelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12           	      14	5295236741 ns/op	1083815895 B/op	11798226 allocs/op
BenchmarkTelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E
BenchmarkTelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12          	      18	4149971387 ns/op	436568360 B/op	 8577264 allocs/op
PASS

Process finished with exit code 0
```

New
```
benchstat -delta-test=none _dev/bench/store_symb_a.txt _dev/bench/store_symb_b.txt
name                                                                old time/op    new time/op    delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        1.87s ± 0%     1.66s ± 0%  -10.85%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       1.42s ± 0%     0.25s ± 0%  -82.17%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12     168ms ± 0%     157ms ± 0%   -6.72%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          5.30s ± 0%     4.53s ± 0%  -14.53%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         4.15s ± 0%     0.68s ± 0%  -83.67%

name                                                                old alloc/op   new alloc/op   delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        396MB ± 0%     396MB ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       161MB ± 0%      95MB ± 0%  -40.95%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12    49.9MB ± 0%    49.9MB ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12         1.08GB ± 0%    1.08GB ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         437MB ± 0%     228MB ± 0%  -47.77%

name                                                                old allocs/op  new allocs/op  delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        4.35M ± 0%     4.35M ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       3.02M ± 0%     0.05M ± 0%  -98.36%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12      493k ± 0%      493k ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          11.8M ± 0%     11.8M ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         8.58M ± 0%     0.02M ± 0%  -99.78%
```
Cmp:

```
benchstat -delta-test=none _dev/bench/store_symb_a.txt _dev/bench/store_symb_b.txt
name                                                                old time/op    new time/op    delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        1.87s ± 0%     1.66s ± 0%  -10.85%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       1.42s ± 0%     0.25s ± 0%  -82.17%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12     168ms ± 0%     157ms ± 0%   -6.72%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          5.30s ± 0%     4.53s ± 0%  -14.53%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         4.15s ± 0%     0.68s ± 0%  -83.67%

name                                                                old alloc/op   new alloc/op   delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        396MB ± 0%     396MB ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       161MB ± 0%      95MB ± 0%  -40.95%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12    49.9MB ± 0%    49.9MB ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12         1.08GB ± 0%    1.08GB ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         437MB ± 0%     228MB ± 0%  -47.77%

name                                                                old allocs/op  new allocs/op  delta
TelemeterRealData_Series/alerts2w/01DN3SK96XDAEKRB1AN30AAW6E-12        4.35M ± 0%     4.35M ± 0%   -0.00%
TelemeterRealData_Series/alerts15s/01DN3SK96XDAEKRB1AN30AAW6E-12       3.02M ± 0%     0.05M ± 0%  -98.36%
TelemeterRealData_Series/subssyncs2w/01DN3SK96XDAEKRB1AN30AAW6E-12      493k ± 0%      493k ± 0%   +0.00%
TelemeterRealData_Series/subs2w/01DN3SK96XDAEKRB1AN30AAW6E-12          11.8M ± 0%     11.8M ± 0%   -0.00%
TelemeterRealData_Series/subs15s/01DN3SK96XDAEKRB1AN30AAW6E-12         8.58M ± 0%     0.02M ± 0%  -99.78%
```

Code: https://gist.github.com/bwplotka/cbcbbcd1802181b7785da11dcc0f5cfd


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
@bwplotka bwplotka requested review from kakkoyun, pracucci, brancz, squat and yeya24 Dec 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
You can’t perform that action at this time.