[bug]: cannot clear IMAP flags that were set by a Sieve script

[andreymal]

What happened?

I was playing with Sieve scripts and once I noticed I can no longer mark some (but not all) emails as unread.

How can we reproduce the problem?

I can reproduce the problem by doing the following steps:

1. Create and activate a Sieve script that uses setflag, I made this one (using the ManageSieve protocol):

   require "imap4flags";
   if address :contains "From" "andreymal"{
   setflag ["\\Flagged","\\Seen"];
   }

2. Receive an email that will automatically be marked as read by the Sieve script

3. Deactivate / remove the Sieve script to make sure it can no longer affect any emails

4. Try to mark the email as unread (i.e. remove the \Seen flag, I use Thunderbird if this is important)

5. Refresh the mailbox and notice the email is marked as read again for no reason

Version

v0.11.x

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

Internal

What operating system are you using?

Linux

Relevant log output

When I mark the email as unread, these lines appear in the log:

2025-01-06T20:26:00Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 54430, size = 33, contents = "181 uid store 12 -Flags (\\Seen)\r\n"
2025-01-06T20:26:00Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-06T20:26:00Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 7
2025-01-06T20:26:00Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 6
2025-01-06T20:26:00Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 54430, accountId = 41, mailboxId = 0, documentId = [51], type = "Clear", details = ["Seen"], elapsed = 0ms
2025-01-06T20:26:00Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 54430, size = 72, contents = "* 10 FETCH (FLAGS (\\Flagged \\Seen) UID 12)\r\n181 OK UID STORE completed\r\n"

So Stalwart understands that the \Seen flag should be removed but doesn't remove it for some reason

Code of Conduct

• I agree to follow this project's Code of Conduct

[mdecimus]

Does it work if you remove flags that were set by Thunderbird?

[andreymal]

Does it work if you remove flags that were set by Thunderbird?

Yes, all other emails can be flagged and unflagged just well

For comparison, logs from an email that was not affected by the Sieve script:

# Marking as read
2025-01-11T15:56:43Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 32, contents = "72 uid store 13 +Flags (\\Seen)\r\n"
2025-01-11T15:56:43Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-11T15:56:43Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 7
2025-01-11T15:56:43Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 6
2025-01-11T15:56:43Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, accountId = 41, mailboxId = 0, documentId = [10], type = "Add", details = ["Seen"], elapsed = 2ms
2025-01-11T15:56:43Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 79, contents = "* 11 FETCH (FLAGS (NonJunk \\Flagged \\Seen) UID 13)\r\n72 OK UID STORE completed\r\n"

# Marking as unread
2025-01-11T15:56:51Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 32, contents = "77 uid store 13 -Flags (\\Seen)\r\n"
2025-01-11T15:56:51Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-11T15:56:51Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 7
2025-01-11T15:56:51Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 6
2025-01-11T15:56:51Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, accountId = 41, mailboxId = 0, documentId = [10], type = "Clear", details = ["Seen"], elapsed = 0ms
2025-01-11T15:56:51Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 73, contents = "* 11 FETCH (FLAGS (NonJunk \\Flagged) UID 13)\r\n77 OK UID STORE completed\r\n"

[andreymal]

Noticed a fun fact: after marking the problem email as unread and back as read, the \Seen flag appears twice in the log

2025-01-11T16:03:40Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 33, contents = "165 uid store 12 +Flags (\\Seen)\r\n"
2025-01-11T16:03:40Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-11T16:03:40Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 7
2025-01-11T16:03:40Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 6
2025-01-11T16:03:40Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, accountId = 41, mailboxId = 0, documentId = [51], type = "Add", details = ["Seen"], elapsed = 0ms
2025-01-11T16:03:40Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 86, contents = "* 10 FETCH (FLAGS (\\Flagged \\Seen NonJunk \\Seen) UID 12)\r\n165 OK UID STORE completed\r\n"

[andreymal]

Does it work if you remove flags that were set by Thunderbird?

To clarify — flagging the problem email as Junk and NonJunk seems to work well. Only \Seen and \Flagged (flags that were set by the Sieve script) are broken

2025-01-11T16:28:26Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 35, contents = "254 uid store 12 -FLAGS (NonJunk)\r\n"
2025-01-11T16:28:26Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, accountId = 41, mailboxId = 0, documentId = [51], type = "Clear", details = ["Keyword("NonJunk")"], elapsed = 0ms
2025-01-11T16:28:26Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 83, contents = "* 10 FETCH (FLAGS (\\Flagged \\Seen Junk \\Seen) UID 12)\r\n254 OK UID STORE completed\r\n"

2025-01-11T16:28:29Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 32, contents = "258 uid store 12 -FLAGS (Junk)\r\n"
2025-01-11T16:28:29Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, accountId = 41, mailboxId = 0, documentId = [51], type = "Clear", details = ["Keyword("Junk")"], elapsed = 0ms
2025-01-11T16:28:29Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 86, contents = "* 10 FETCH (FLAGS (\\Flagged \\Seen NonJunk \\Seen) UID 12)\r\n258 OK UID STORE completed\r\n"

[andreymal]

For the sake of science, I did the opposite — added the Junk flag using the Sieve script:

require "imap4flags";
setflag ["Junk"];

And I can clear this flag without any problems (as well as \Seen and \Flagged because they were not set by this Sieve script), so only \Seen and \Flagged that were set by the Sieve script seem to be broken

2025-01-11T16:40:03Z TRACE Raw IMAP input received (imap.raw-input) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 32, contents = "343 uid store 18 -FLAGS (Junk)\r\n"
2025-01-11T16:40:03Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-11T16:40:03Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-11T16:40:03Z TRACE Data store iteration operation (store.data-iterate) elapsed = 0ms
2025-01-11T16:40:03Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 7
2025-01-11T16:40:03Z TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 4
2025-01-11T16:40:03Z DEBUG IMAP STORE command (imap.store) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, accountId = 41, mailboxId = 0, documentId = [8], type = "Clear", details = ["Keyword("Junk")"], elapsed = 0ms
2025-01-11T16:40:03Z TRACE Raw IMAP output sent (imap.raw-output) listenerId = "imaptls", localPort = 993, remoteIp = x.x.x.x, remotePort = 48476, size = 65, contents = "* 16 FETCH (FLAGS (NonJunk) UID 18)\r\n343 OK UID STORE completed\r\n"