peb iterator #34
peb iterator #34
Conversation
disktail/disktail.go
Outdated
| itemHandler := handler | ||
|
|
||
| if skipDuplicates { | ||
| keys := make(map[string]struct{}) |
There was a problem hiding this comment.
This map could grow really big, but our main use case is iterating over a couple thousand users with a key size of 24, so this should not be a problem.
| time.Sleep(100 * time.Millisecond) | ||
| } | ||
| t.Fatalf("operation timed out") | ||
| } |
There was a problem hiding this comment.
would also add a test for the cleaner
|
|
||
| func (r *DiskTail[T]) Iterate(ctx context.Context, skipDuplicates bool, stats *IterStats, handler func(item *Item[T]) HandleResult) error { | ||
| if stats == nil { | ||
| stats = &IterStats{} |
There was a problem hiding this comment.
Locking on these stats doesn't make much sense? They're not visible to the caller.
There was a problem hiding this comment.
what would you suggest? Hide the stats behind an interface so in case it's not needed it won'T do anything? I don't want to have ifs everywhere just to check if the stats were provided..
frairon
force-pushed
the
peb-iterator
branch
6 times, most recently
from
785a995 to
fcfb727
Compare
disktail/example_test.go
Outdated
There was a problem hiding this comment.
Would not call it example_test.go since its not run via go test. Maybe either move that to an examples folder or create a runnable example and put into cmd/disktail_example/main.go.
disktail/disktail.go
Outdated
| if err != nil { | ||
| return fmt.Errorf("error creating iterator: %w", err) | ||
| } | ||
| r.store.NewBatch() |
There was a problem hiding this comment.
we ignore the returned Batch, can be removed probably
frairon
force-pushed
the
peb-iterator
branch
6 times, most recently
from
cb75a33 to
7992213
Compare
frairon
force-pushed
the
peb-iterator
branch
2 times, most recently
from
3f28190 to
929e3a2
Compare
This PR creates a disk-based ringbuffer updated by a kafka topic