Flaky rules test fix (#6549) #11207

Job	Run time
lint	4m 27s
build	4m 5s
test	16m 2s
test-no-race	7m 44s
CodeQL	7m 17s
integration-configs-db	2m 56s
integration (requires_docker)	7m 13s
integration (integration_alertmanager)	4m 2s
integration (integration_backward_compatibility)	10m 53s
integration (integration_memberlist)	7m 42s
integration (integration_querier)	15m 9s
integration (integration_ruler)	5m 48s
integration (integration_query_fuzz)	5m 33s
deploy_website	59s
deploy	3m 39s
	1h 43m 29s

Provide feedback