ApplyAllConfiguredChangesToDatabaseAsync fails fast, what could be the reasons?

[ulf-melin-sveasolar]

Hi all, I am trying to implement a .net 6 console application that will upgrade the event store database schema but have run into problems. The process exits when I run "await ApplyAllConfiguredChangesToDatabaseAsync()" and it does not tell me why.

The following is my code, before this I have set up the store with the correct connection string and searchpath and then when I run this locally using F5 in Visual Studio it all works as expected, no problem. But when I build a docker image and execute it there the process dies without executing neither the catch- nor the finally-clauses. I have a global UnhandledErrorHandler also, but that does not get executed either.

So does any one have an idea on how to track down what is wrong with my code?

try
{
    await store.Storage.ApplyAllConfiguredChangesToDatabaseAsync();
    Console.WriteLine("Did ApplyAllConfiguredChangesToDatabaseAsync"); //Never executed in docker
}
catch (Exception ex)
{
    Console.WriteLine("Failed ApplyAllConfiguredChangesToDatabaseAsync");//Never executed in docker
    Console.WriteLine(ex);//Never executed in docker
    throw;
}
finally
{
    Console.WriteLine("Ended ApplyAllConfiguredChangesToDatabaseAsync");//Never executed in docker
}

[gfoidl]

Anywhere the exception get swallowed. So for the docker-scenario try to remove all exception-handling /-logic, so that the unhandled exception can escape, and then you should be able to see what's printed there.

Just from looking at the fragment of code you posted (BTW: in Github it's also possible to format the code so it's easier to read, thus for you easier to help) it's impossible to tell what is going on.

[gfoidl]

I meant in your code, like

I have a global UnhandledErrorHandler also

and also the cath (Exception)s, so that the unhandled exception won't be handled anywhere, so typically it gets printed to the console. Thus you hopefully get an exception (message, inner exception, and stacktraces) so that it gets actionable.

---

A different approach: can you share a minimal repro?

[ulf-melin-sveasolar]

Worth trying! So I did, and the result is the same, no output whatsoever. A minimal repro would of course be great if I could muster, but it would just take too much time (that I do not have at the moment). I am very greatful for the time and effort you put into this issue of mine so far.

Besides the connection string being wrong (which I am pretty certain it is not), can you see any other reason that can make ApplyAllConfiguredChangesToDatabaseAsync behave like that? Do you know if it works on another thread (or even appdomain) and if something goes wrong in the wrong way it cannot marshal the Exception back? Just brainstorming...

I can successfully execute the following, which I interpret as the connection string is working

var session = store.LightweightSession();
var searchResult = session.Search<WallboxChargerState>("ulfsw");

but then I found a nice looking method

await store.Storage.Database.AssertConnectivity();

and that one just exits the process. Does not tell me anything about what is wrong. Decompilation yields a description like:

        //
        // Summary:
        //     Attempts to connect to the actual database and throws an exception if that fails
        Task AssertConnectivity();

but I cannot catch that exception and it is not reported to console if I remove all error handling code.

I also tried setting the password to something wrong, and then I get a proper error to the console:

Npgsql.PostgresException (0x80004005): 28P01: password authentication failed for user "postgres"

So I am interpreting this as that the connection string is correct.

[gfoidl]

or even appdomain

Do you use multiple AppDomains?

Does it reproduce with the debugger attached too?
I.e. in Visual Studio with F5-debugging and all CLR-exception enabled

(note: the enabled checkbox in the marked section)
Then the debugger should halt on every exception throws and show the exception dialog -- there you can disable that type of exception (like OperationCanceledException so the irrelevant ones don't disturb that much).

The code you have shown so far looks fine, and I didn't run in similar problems when using ApplyAllConfiguredChangesToDatabaseAsync.

I am very greatful for the time and effort you put into this issue of mine so far.

Thanks 😃

[ulf-melin-sveasolar]

I do not use multiple appdomains (just curious if Marten or Weasel does it internally).

Well, if I run this from within Visual Studio debugger (just hit F5) then the progrm works fine, does the migration and everything. The problem is when I run it in a linux container, I attached the debugger to the console process in the container and enabled all exceptions:

But still, the process dies immediately when I step the debugger to
await store.Storage.Database.AssertConnectivity();

So I don't really think it is a bug or something in Marten, but something is wrong in this environment and I get no clue to what it is.

[gfoidl]

Marten or Weasel does it internally

No. And in .NET Core they aren't supported anyway.

something is wrong in this environment

Can you enable crash dumps?
Run the container with --cap-add=SYS_PTRACE and set -e DOTNET_EnableCrashReport=1 -e DOTNET_DbgEnableMiniDump=1.
Then when it crashes a dump should be created, which can be further investigated.

Sadly I don't have any other idea w/o repro.
Hopefully someone else has other ideas.