Errors when configuring DB replica access

[eloiferrer]

I have properly set up two mariadb pods in the k8s cluster and tested that replication works when changes are introduced in the main replica.
Starting docker compose also initializes all the tables in the my_wiki database. So the connection to the database is also working.
However, when accessing the local instance of the portal shows then these errors when navigating to any page:

[1608e11e86066f4c6fe54a52] /wiki/Special:RecentChanges Wikimedia\Rdbms\DBTransactionStateError: Cannot execute query from MediaWiki\Page\PageStore::getPageByNameViaLinkCache while transaction status is ERROR

Backtrace:

from /var/www/html/w/includes/libs/rdbms/database/TransactionManager.php(176)
#0 /var/www/html/w/includes/libs/rdbms/database/Database.php(980): Wikimedia\Rdbms\TransactionManager->assertTransactionStatus(Wikimedia\Rdbms\DatabaseMySQL, array, string)
#1 /var/www/html/w/includes/libs/rdbms/database/Database.php(633): Wikimedia\Rdbms\Database->assertQueryIsCurrentlyAllowed(string, string)
#2 /var/www/html/w/includes/libs/rdbms/database/Database.php(1340): Wikimedia\Rdbms\Database->query(Wikimedia\Rdbms\Query, string)
#3 /var/www/html/w/includes/libs/rdbms/database/Database.php(1349): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#4 /var/www/html/w/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\Database->selectRow(array, array, array, string, array, array)
#5 /var/www/html/w/includes/libs/rdbms/database/DBConnRef.php(369): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /var/www/html/w/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(793): Wikimedia\Rdbms\DBConnRef->selectRow(array, array, array, string, array, array)
#7 /var/www/html/w/includes/page/PageStore.php(201): Wikimedia\Rdbms\SelectQueryBuilder->fetchRow()
#8 /var/www/html/w/includes/cache/LinkCache.php(439): MediaWiki\Page\PageStore->MediaWiki\Page\{closure}(Wikimedia\Rdbms\DBConnRef, int, string, array)
#9 /var/www/html/w/includes/libs/objectcache/WANObjectCache.php(1805): MediaWiki\Cache\LinkCache->MediaWiki\Cache\{closure}(bool, int, array, null, array)
#10 /var/www/html/w/includes/libs/objectcache/WANObjectCache.php(1619): Wikimedia\ObjectCache\WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#11 /var/www/html/w/includes/cache/LinkCache.php(444): Wikimedia\ObjectCache\WANObjectCache->getWithSetCallback(string, int, Closure)
#12 /var/www/html/w/includes/cache/LinkCache.php(493): MediaWiki\Cache\LinkCache->getGoodLinkRowInternal(MediaWiki\Title\TitleValue, Closure, int)
#13 /var/www/html/w/includes/page/PageStore.php(190): MediaWiki\Cache\LinkCache->getGoodLinkRow(int, string, Closure, int)
#14 /var/www/html/w/includes/page/PageStore.php(156): MediaWiki\Page\PageStore->getPageByNameViaLinkCache(int, string, int)
#15 /var/www/html/w/includes/page/PageStore.php(328): MediaWiki\Page\PageStore->getPageByName(int, string, int)
#16 /var/www/html/w/includes/title/Title.php(3731): MediaWiki\Page\PageStore->getPageByReference(MediaWiki\Title\Title, int)
#17 /var/www/html/w/includes/title/Title.php(2593): MediaWiki\Title\Title->getFieldFromPageStore(string, int)
#18 /var/www/html/w/includes/page/LegacyArticleIdAccess.php(42): MediaWiki\Title\Title->getArticleID()
#19 /var/www/html/w/includes/Revision/RevisionStore.php(2962): MediaWiki\Revision\RevisionStore->getArticleId(MediaWiki\Title\Title)
#20 /var/www/html/w/includes/language/MessageCache.php(1427): MediaWiki\Revision\RevisionStore->getKnownCurrentRevision(MediaWiki\Title\Title)
#21 /var/www/html/w/includes/libs/objectcache/WANObjectCache.php(1805): MessageCache->{closure}(bool, int, array, null, array)
#22 /var/www/html/w/includes/libs/objectcache/WANObjectCache.php(1619): Wikimedia\ObjectCache\WANObjectCache->fetchOrRegenerate(string, int, Closure, array, array)
#23 /var/www/html/w/includes/language/MessageCache.php(1453): Wikimedia\ObjectCache\WANObjectCache->getWithSetCallback(string, int, Closure)
#24 /var/www/html/w/includes/libs/objectcache/BagOStuff.php(217): MessageCache->{closure}(int)
#25 /var/www/html/w/includes/language/MessageCache.php(1455): Wikimedia\ObjectCache\BagOStuff->getWithSetCallback(string, int, Closure)
#26 /var/www/html/w/includes/language/MessageCache.php(1365): MessageCache->loadCachedMessagePageEntry(string, string, string)
#27 /var/www/html/w/includes/language/MessageCache.php(1241): MessageCache->getMsgFromNamespace(string, string)
#28 /var/www/html/w/includes/language/MessageCache.php(1212): MessageCache->getMessageForLang(LanguageEn, string, bool, array)
#29 /var/www/html/w/includes/language/MessageCache.php(1109): MessageCache->getMessageFromFallbackChain(LanguageEn, string, bool)
#30 /var/www/html/w/includes/language/Message/Message.php(1498): MessageCache->get(string, bool, LanguageEn, string)
#31 /var/www/html/w/includes/language/Message/Message.php(1177): MediaWiki\Message\Message->fetchMessage()
#32 /var/www/html/w/includes/specialpage/SpecialPage.php(1046): MediaWiki\Message\Message->isDisabled()
#33 /var/www/html/w/includes/specials/SpecialRecentChanges.php(223): MediaWiki\SpecialPage\SpecialPage->addHelpLink(string, bool)
#34 /var/www/html/w/includes/specialpage/SpecialPage.php(729): MediaWiki\Specials\SpecialRecentChanges->execute(null)
#35 /var/www/html/w/includes/specialpage/SpecialPageFactory.php(1737): MediaWiki\SpecialPage\SpecialPage->run(null)
#36 /var/www/html/w/includes/actions/ActionEntryPoint.php(504): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#37 /var/www/html/w/includes/actions/ActionEntryPoint.php(146): MediaWiki\Actions\ActionEntryPoint->performRequest()
#38 /var/www/html/w/includes/MediaWikiEntryPoint.php(201): MediaWiki\Actions\ActionEntryPoint->execute()
#39 /var/www/html/w/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#40 {main}

Apart from updating the host and credentials in docker-compose.yml I've only added this to LocalSettings.php (with the right host, dbname, user and password):

$wgDBservers = [
    [
        'host' => "primary.serv.er",
        'dbname' => "wikidb",
        'user' => "wikiuser",
        'password' => "secret",
        'type' => "mysql",
        'flags' => DBO_DEFAULT,
        'load' => 0,
    ],
    [
        'host' => "replica1.serv.er",
        'dbname' => "wikidb",
        'user' => "wikiuser",
        'password' => "secret",
        'type' => "mysql",
        'flags' => DBO_DEFAULT,
        'load' => 1,
    ]
];

[eloiferrer]

I might be missing some other required configuration. Any tips would be appreciated @physikerwelt

[physikerwelt]

Does the wiki work if you remove the $wgDBservers variable (and it falls back to the $wgDBserver (without an s pointing to a single ip)

[eloiferrer]

Yes, everything works without errors if I remove the $wgDBservers configuration.

[physikerwelt]

The next step would be to verify that the mysql slave in in healthy state. There are several guides online how to verify this. It is possible that the master works but the replica (I think slave violates PC) has problems. Once that is figured out, I would switch to the servers (plural) variant with only a master node, and then add the slave in a next step. If all of the above does not help, I can have a look.

[eloiferrer]

SHOW SLAVE STATUS always shows the replica in a good state. When defining only one master with $wgDBservers, everything works, replication happens and SHOW SLAVE STATUS is also healthy. Things break when I add both nodes to $wgDBservers.

I am going to try next to just define replicas directly in the docker-compose.yml file. This does not seem to be a problem with Kubernetes or the db connection but rather a mediawiki configuration problem.

[physikerwelt]

Maybe you can share the details with me in PM and I can look around if I discover a typo, or something similar.

[physikerwelt]

This is the link to the configuration that worked before

https://github.com/MaRDI4NFDI/portal-compose/pull/428/files

[eloiferrer]

Can you take a look at this branch to see if you spot the problem? https://github.com/MaRDI4NFDI/portal-compose/tree/test_replica
It is a simple docker-compose project that shows the same error as above.
I have added a mysql-replica in docker-compose.yml, add the .cnf files and also the LocalSettings configuration.