Fix error after second start_chat() for StatefulLLMPipeline

[sbalandi]

Ticket: CVS-161902

[Wovchena]

You have a similar block below:

    if (!m_tokenized_chat_history.empty()) {
        reset_kv_state();
        m_history = {};
        m_templated_chat_history.clear();
        m_tokenized_chat_history.clear();
    }

I propose to change the condition to if (have_state) for the old block and reset everything there.

Should a similar thing be performed in the end of generate() if is_chat_conversation is false? It would be better to perform in the beginning of generate() to improve exception safety, but we already do that in the end everywhere, so stay consistent.

[ilya-lavrenov]

should start_chat call stop_chat first ?

[sbalandi]

updated
in chat scenario we rely on the existing attention_mask , and when is_chat_conversation is false new attention_mask is explicitly used, that why it doesn't crash for non-chat

[Wovchena]

I think the request to add a test is valid. You can take the idea from #1674 (comment)

[ilya-lavrenov]

I think we should also cover a case when user call next start_chat w/o stop_chat for current one.
IMO, it should automatically stop current chat

[sbalandi]

cases with system_massage, with double start_chat and with several chats one after another are added.
If i understand right tests work now for PA backend, should it be added run for SDPA too ?

[ilya-lavrenov]

If i understand right tests work now for PA backend, should it be added run for SDPA too ?

they will be added as part of CVS-159925
For now, you can check locally that tests are passing on SPDA branch by changing default backend here

openvino.genai/src/cpp/src/llm_pipeline.cpp

Line 62 in 1bdd4f9

std::string attention_backend = PA_BACKEND;

[ilya-lavrenov]

does a current PR fix #1663 ?

[sbalandi]

does a current PR fix #1663 ?

The reason looks like point which Vladimir mention above. If execution is interrupted and generation() does not complete correctly, this error will occur. I moved the resets to the beginning, this fixes similar behavior for phi-3.