forked from mittagessen/kraken
-
Notifications
You must be signed in to change notification settings - Fork 0
Training
Stefan Weil edited this page Oct 12, 2022
·
19 revisions
This sequence of examples was focused on getting a good kraken model for historic prints, especially for Fraktur script, but also with support for historic and modern Antiqua fonts.
- https://github.com/UB-Mannheim/AustrianNewspapers/wiki/Training-with-Kraken
- https://github.com/UB-Mannheim/digi-gt/wiki/Training#training-mit-kraken
- https://github.com/UB-Mannheim/reichsanzeiger-gt/wiki/Training-with-Kraken
- https://github.com/UB-Mannheim/digitue-gt/wiki/Training-with-Kraken
The training used the same data sets for training and evaluation as a previous training which was done with Tesseract.
It was done the latest kraken code plus additional fixes for a division by null and multithreading.
nohup time ketos train -t list.train -e list.eval -o frak2021_gpu -d cuda:0 | tee -a frak2021_gpu.log
nohup: Eingabe wird ignoriert und leite Standardfehlerausgabe auf Standardausgabe umgeleitet
Building training set
Building validation set
[52.2009] alphabet mismatch: chars in training set only: {'~', 'ὐ', 'ő', 'À', 'χ', '₆', 'Ò', 'ῳ', 'ï', 'ḡ', 'ώ', 'ἄ', 'μ', 'ϋ', '☛', 'Ἰ', 'φ', 'ō', 'ἰ', 'ξ', '͗', '№', '⅚', 'đ', 'ύ', 'ϖ', 'ὀ', 'Φ', 'Δ', '_', 'ϰ', 'ђ', 'ἀ', 'Τ', 'ᴐ', 'κ', 'Ó', 'Α', 'È', 'Ⅎ', 'ἐ', 'É', 'Ô', 'ὰ', '+', 'ϟ', '‣', 'Ž', 'η', '▲', 'Β', 'ί', '⅓', 'ž', 'Š', '⏓', 'ὁ', 'ό', '±', 'ά', 'ψ', '₈', 'î', '♂', '∙', 'θ', 'ⅎ', 'ℳ', 'Ç', 'є', 'ῆ', 'ϑ', '₄', 'ϱ', '✤', 'ὸ', '♀', '⅔', 'Ἀ', 'δ', 'ὴ', 'ἑ', 'ɔ', 'ζ', '᾽', 'ὶ', 'ì', 'ʞ', '⅙', '☚', '‛', '℔', 'ή'} (not included in accuracy test during training)
[52.2010] alphabet mismatch: chars in validation set only: {'ϊ'} (not trained)
Initializing model ✓
stage 1/∞
Accuracy report (1) 0.9612 1625450 63133
stage 2/∞
Accuracy report (2) 0.9604 1625450 64311
stage 3/∞
Accuracy report (3) 0.9578 1625450 68579
stage 4/∞
Accuracy report (4) 0.9519 1625450 78227
stage 5/∞
Accuracy report (5) 0.9428 1625450 92982
stage 6/∞
Accuracy report (6) 0.9421 1625450 94036
Moving best model frak2021_gpu_1.mlmodel (0.9611597061157227) to frak2021_gpu_best.mlmodel
154547.59user 1537.72system 29:15:33elapsed 148%CPU (0avgtext+0avgdata 5916304maxresident)k
385328inputs+453328outputs (68102177major+26511973minor)pagefaults 0swaps
The information here is no longer valid.
mkdir Juristische_Konsilien_Tuebingen
cd Juristische_Konsilien_Tuebingen
mkdir Training
cd Training
# Get image files.
wget --content-disposition https://files.transkribus.eu/Get?id=KWPGLJROIYFMCHXUKHROTBZI https://files.transkribus.eu/Get?id=FFZEEILUCMKWBMTOJNKXOQVO https://files.transkribus.eu/Get?id=DHTNZFGEQZVNFYSNNXSUGRZW https://files.transkribus.eu/Get?id=GLUHQPHRJZGIORAKNKSSHJHN https://files.transkribus.eu/Get?id=AXGGUKSGAASTMJLXHKLHTCMI https://files.transkribus.eu/Get?id=ZWNHITSTVYAYVRMXSACMYZSS https://files.transkribus.eu/Get?id=SJOIKRULJGZRUGOHANSBQCES https://files.transkribus.eu/Get?id=TXUOIQXJUFRBXPZXRIQKRGID https://files.transkribus.eu/Get?id=FBFDPQSANWKZHMEFPCAEBYGC https://files.transkribus.eu/Get?id=JJYAAMDIJMVXEMDQNQLFSNHP https://files.transkribus.eu/Get?id=CFHAJXQXKFTYWEMOXGMHIFTK https://files.transkribus.eu/Get?id=YJNIOPLPTVNXGLVKAYDWSZTJ https://files.transkribus.eu/Get?id=KSSNVVQMKCIPCOJNVRQIOIIH https://files.transkribus.eu/Get?id=OGZBGTHHLDZQRUFVQBDMDDPA https://files.transkribus.eu/Get?id=KYPDSMEVHCRVHUXKYNOXDTPT https://files.transkribus.eu/Get?id=UJUHLJMVEUIXOOVDPIQKPXFL https://files.transkribus.eu/Get?id=ZTTKWCFYHHJXPCJHVLGCFQCM https://files.transkribus.eu/Get?id=FOVUVMNPDJDZLKWIDYGJPLUX https://files.transkribus.eu/Get?id=DXRPIBLIMBBMIXVDXCSMOLDM https://files.transkribus.eu/Get?id=LZPWBKIXSJJIFAWPGUZSPVXT https://files.transkribus.eu/Get?id=HOHGKGGHOSQUUTAVACVQYWAL https://files.transkribus.eu/Get?id=GPQZHFWYWLZEXJCDTJKAALEE https://files.transkribus.eu/Get?id=TAAKRFOWGFBKCEFSHNQIVOIK https://files.transkribus.eu/Get?id=RSAIQAPCKZZTNAKLEHTWBPGM https://files.transkribus.eu/Get?id=NZGMAVAGNRHFMZNNUIJDTRLU https://files.transkribus.eu/Get?id=PCNIVHAJHFRBOYBPQERJTOED https://files.transkribus.eu/Get?id=OJKITNXIXTYUQPCZKYHQYFSD https://files.transkribus.eu/Get?id=SZJTDCPESLWRIHCZXMBEENTJ https://files.transkribus.eu/Get?id=LTREBTDWWJRERTPPUBONYEEH https://files.transkribus.eu/Get?id=KLINCPGDKEXBUQNBUDGJUOPY https://files.transkribus.eu/Get?id=ZGQKNMDFPAZNRBMMKSOZZHKA https://files.transkribus.eu/Get?id=HJFOROVZUNTZMHRZJJOOKJAM https://files.transkribus.eu/Get?id=JAOKFQRGZUHUSDRJUWEGEGUE https://files.transkribus.eu/Get?id=ULKQLRYWHGTQMGGDRWWVKNOG https://files.transkribus.eu/Get?id=FXHKSRQIETJMKXBQZHLEHHZZ https://files.transkribus.eu/Get?id=RQSXJYKCEUBGMPTEHSCBVAAT https://files.transkribus.eu/Get?id=LHMCRHNCINSMVCINCEKNACCG https://files.transkribus.eu/Get?id=LFGZRQTADQHPYNUJPNHFOLMR https://files.transkribus.eu/Get?id=XVTTWUSOAQFYRSZQVLQMMNYB https://files.transkribus.eu/Get?id=MQROMFPZURWSUWVWJAFNJWMS https://files.transkribus.eu/Get?id=XOIOSHOVCUSCCVHNGLOWBUHA https://files.transkribus.eu/Get?id=MQQGCETBKJUWDLIQYUKPRJRU https://files.transkribus.eu/Get?id=HOBEDMUTPAPQPGCVBFSSSUJS https://files.transkribus.eu/Get?id=RYRIDAQAQCWEUCFZSOJMIEVT https://files.transkribus.eu/Get?id=FKUXUMUMEVVMYEPKVXUTCKHY https://files.transkribus.eu/Get?id=ROOUZCYQSEZSDEBSHEMGPQCB https://files.transkribus.eu/Get?id=HYFOIAPWWUYVLBKKNCWWVQDR https://files.transkribus.eu/Get?id=MBLXSOIAFBZPRGCBEAWLSXIQ https://files.transkribus.eu/Get?id=JPTFEKQFLOIJABAHZPJGWTIX https://files.transkribus.eu/Get?id=GVOZSINXFOQINGIHOFLHZEXW https://files.transkribus.eu/Get?id=UIUAZQUCPLZMULJHNUGEYUKR https://files.transkribus.eu/Get?id=AJJVYMFQTWWIWJFCUKMJKJQP https://files.transkribus.eu/Get?id=GIJZOLKNHKZXBTBWPJJJLYYR https://files.transkribus.eu/Get?id=KRXUUEYRYVHLMMUFLKCGUQQH https://files.transkribus.eu/Get?id=RKUEWSQAKAWVGCJMBNOSIVNR https://files.transkribus.eu/Get?id=MNOPSPMISKMPTXUIJUSOEOCW https://files.transkribus.eu/Get?id=BFYUPHCCJDCUAZWITORDMLGX https://files.transkribus.eu/Get?id=APIACVAWACKQMPYJILFINMZT https://files.transkribus.eu/Get?id=TVBDRWEFCSENIIYFTLRYPEAA https://files.transkribus.eu/Get?id=GJYRHFRDWSHRLJVIGNKWWBZN https://files.transkribus.eu/Get?id=BQDKZIVQUJEXYYBCNVTBLHZM https://files.transkribus.eu/Get?id=DOIIRJFDXVHNAXFMUVFBPPYE https://files.transkribus.eu/Get?id=CSELZDAMABXKPWZNCSOHCBAF https://files.transkribus.eu/Get?id=SGFNBPQGXZMZAKGUSWXWOPVR https://files.transkribus.eu/Get?id=XQQIAQLQRMKMYCVHCLWHUYFQ https://files.transkribus.eu/Get?id=ROMIWFNFIBTGGCFLIZJJFWKU https://files.transkribus.eu/Get?id=KTUVRXMZBRHUYFVDAULYHKBC https://files.transkribus.eu/Get?id=AQHXCBAWQVKHUVUHZGDZPZUT https://files.transkribus.eu/Get?id=RXKIXYMKMLQBTJGXFYGCSGDY https://files.transkribus.eu/Get?id=UBEIKSGVTBMHIJPQOEGZSGWD https://files.transkribus.eu/Get?id=RWMVMWDCPNFBTQFGWQOAAVQQ https://files.transkribus.eu/Get?id=NYDHDAZGYHUDGXENOAASLFYF https://files.transkribus.eu/Get?id=SMVJNOMKZTUVAXKVDAJXSAZS https://files.transkribus.eu/Get?id=VIHRDXWGGOATSMCPZJJXAAXJ https://files.transkribus.eu/Get?id=YWFJNUVWHSZGMBDEZTJMCGFM https://files.transkribus.eu/Get?id=OXHBEFWYPNGMTTARWUPYSYUU https://files.transkribus.eu/Get?id=NDCENLOGOVSMHFPOTPOUCQKG https://files.transkribus.eu/Get?id=TIGUOIFRBCUMNOFZYARPGGRS https://files.transkribus.eu/Get?id=NKDMDKVMRTZWTKTRMCYDHFLW https://files.transkribus.eu/Get?id=IDIHIPZTDOXTOQCAWCBQMNOQ https://files.transkribus.eu/Get?id=IQOTZUSGWOKZEPBJBSYYJGTM https://files.transkribus.eu/Get?id=FRWNLSWXMYIPGQJGVPLWPIXT https://files.transkribus.eu/Get?id=QNXDOMQULVGUWUGHIKQDPCOB https://files.transkribus.eu/Get?id=EYISIDNNZSWKOQOYMQUTXVSK https://files.transkribus.eu/Get?id=RGFHPRMBPRYNEFLVEJTCOILG https://files.transkribus.eu/Get?id=FBZILEXIIRGVPPJLOYYSGHCD https://files.transkribus.eu/Get?id=KOIUIIXSXVJDFERSFIXMSZUL https://files.transkribus.eu/Get?id=OYLYXYLBLJYUJYOPLEFDHKLH https://files.transkribus.eu/Get?id=KJVSKBNNSJOQFGSRXVMRZYQR https://files.transkribus.eu/Get?id=UHAHGFFAMRNSTNQGMOBNQTUR https://files.transkribus.eu/Get?id=ADDQMOOXTWMOXGEEYMANWNJZ https://files.transkribus.eu/Get?id=HESNYZGFTYGUDFGZANLQLMOF https://files.transkribus.eu/Get?id=NQCXKFIDTGDACYHFUJZKJQDS https://files.transkribus.eu/Get?id=JZUMSCTEYEHMAOXDAXZNXGUX https://files.transkribus.eu/Get?id=IBERFVXKQCGEAHQNKPNCSNMA https://files.transkribus.eu/Get?id=FGLWDUKYAMWQIHEHKKQOOLMW https://files.transkribus.eu/Get?id=BCUVIEVSNFOUAYXSYCHNPXVP https://files.transkribus.eu/Get?id=XGPDURDECYWHYWZLJHDMIGLU https://files.transkribus.eu/Get?id=YSSYTELQQLMAHAVXQOGPUTUG https://files.transkribus.eu/Get?id=JOYKOTRPSTUFZXBORFKRCZJC https://files.transkribus.eu/Get?id=USOQSWUKNUEATWIZHMUYLDAF https://files.transkribus.eu/Get?id=VKIXEWUCBSZSKBBGIBOVUSUG https://files.transkribus.eu/Get?id=MEYLSJNTPSHUCXXUAJKCZPKA https://files.transkribus.eu/Get?id=XJDDZZAHHOSJGEPLXOBBRDOA https://files.transkribus.eu/Get?id=MYHOGTLJMXZUVRLJOSCUWWFC https://files.transkribus.eu/Get?id=BLASBUUDLXUNZMUIEVJUEJKJ https://files.transkribus.eu/Get?id=WGPNKBRBCKXVXKFONPETTISL https://files.transkribus.eu/Get?id=QMZIHCDKWOSFAPQVPENYRPHJ https://files.transkribus.eu/Get?id=KONWXWVKBGRGADHIGCFGFOXO https://files.transkribus.eu/Get?id=XXQAPZNSVNXIPGRGVHPTABFI https://files.transkribus.eu/Get?id=IPQPOXCWANMWHGBRMRKIYZLS https://files.transkribus.eu/Get?id=VIMEFFPNDFRWXEJQNXODRBEO https://files.transkribus.eu/Get?id=ZAABPSRTGFETETNGYMGLZCOL https://files.transkribus.eu/Get?id=VLSHOPUQJNHGWWSZKKXQAAHK https://files.transkribus.eu/Get?id=HZIGXWAUPNIABHHIBQHBTGEC https://files.transkribus.eu/Get?id=WDHNPKASTGLNHDIVJKSRVFMX https://files.transkribus.eu/Get?id=ZQWHSMWYZFDJUUDMGEYJYDDY https://files.transkribus.eu/Get?id=ZECCNUCTLIHMSLJPJVGAWUXP https://files.transkribus.eu/Get?id=XKKHGWQTAHWAULBAZBLBQCBQ https://files.transkribus.eu/Get?id=BDFTIFSQHNPGIQHLUSQBNZGL https://files.transkribus.eu/Get?id=NMFPJUPMQPKXBBLCBLNVDUQA https://files.transkribus.eu/Get?id=ZANEBGCYUOUQLYXHKOVSPMER https://files.transkribus.eu/Get?id=EGEUWBWWBYWLKJISSMKERPRZ https://files.transkribus.eu/Get?id=YFACZWEGHUMIJOSGFDIFIIXI https://files.transkribus.eu/Get?id=WXWUUGZKGYIKYFJVUPXLXNLN https://files.transkribus.eu/Get?id=BHERQQEMBDPKHNQCTTIABRAN https://files.transkribus.eu/Get?id=DKFQEVFTRHFUYTNWUWKGMDCC https://files.transkribus.eu/Get?id=MHUFOPWVDKPNENGOTPHMRBPG https://files.transkribus.eu/Get?id=PMKYLRJEIQBIIFQVGAAKBAYE https://files.transkribus.eu/Get?id=BTHWVSEFZFEQVWQXNLLVMQQX https://files.transkribus.eu/Get?id=YWYAOOYGHXSNXEMOWXFVFBBV https://files.transkribus.eu/Get?id=ZARBXUTLMISJJORKXYCCLKLJ https://files.transkribus.eu/Get?id=VIIJFLMYIRGLEPGPPAEUOHEG https://files.transkribus.eu/Get?id=BJAVSUFRATHIZMZJBFFVZLWY https://files.transkribus.eu/Get?id=XAFFBZVXTIOMURIZJANEDCRL https://files.transkribus.eu/Get?id=EADIASJDESBUIGBHBBRVBENC https://files.transkribus.eu/Get?id=UHXRNRWZURNNIITNZLBHIKAT https://files.transkribus.eu/Get?id=LYMSCWGIRTABXAXYSWMKLXFF https://files.transkribus.eu/Get?id=RVHBELCYHNQXRVOVSFHFPHUF https://files.transkribus.eu/Get?id=WYPAPFXUXDIYSHGCVUHPCDOE https://files.transkribus.eu/Get?id=RAAORHEZEHXFRPDCFVOUDSNE https://files.transkribus.eu/Get?id=AUMCSARBRKNJLWQGOEFAVEVN https://files.transkribus.eu/Get?id=LHPLORVBMFYIQXIFUOOQNWLF https://files.transkribus.eu/Get?id=YECVIISDTADFSAVYVVRULJED https://files.transkribus.eu/Get?id=BFQTOOGEOSYCFSTIJFYASZKN https://files.transkribus.eu/Get?id=PJXUZAFEEIUWRLDOENGGGYQM https://files.transkribus.eu/Get?id=YOHJAMQIAVSFHWRUESLIHEJH https://files.transkribus.eu/Get?id=EJXPQXSUQUIUSHKMECISPLRE https://files.transkribus.eu/Get?id=ETRCBMFJJFFSCBWGCWMKVYPU https://files.transkribus.eu/Get?id=RSYKJGUJTWKLWHUVWCZGVHBK https://files.transkribus.eu/Get?id=LHMYRDDJSVVHXCFZAPPIUYWS https://files.transkribus.eu/Get?id=JUQMUFIOQAYNHJSATWZVAVJK https://files.transkribus.eu/Get?id=WJLWJKXFVIOXLXEELWFAXEWS https://files.transkribus.eu/Get?id=XVFBFYOABAYUUBGEETZFNDUU https://files.transkribus.eu/Get?id=GSSITABWCVXAEYCPEGGAGWHO https://files.transkribus.eu/Get?id=HHQPEJLPUUSTOJVMGYHEMCWO https://files.transkribus.eu/Get?id=ETKNGULSXICMTXMSUNGUHAGR https://files.transkribus.eu/Get?id=EVVBDQATMBBOWZQVWWBGWFTE https://files.transkribus.eu/Get?id=JQNZVEEWHOTGPSGQWBJTYDXZ https://files.transkribus.eu/Get?id=HDJMNCKXBJTYRBANCDDNBOTV https://files.transkribus.eu/Get?id=FXWRRMVBGADVUBCLXQTUYPAT https://files.transkribus.eu/Get?id=YHIBTJOSJFQPMEFCMUASZYRR https://files.transkribus.eu/Get?id=WUALYWRDKBXDJKNGZGXEYHKQ https://files.transkribus.eu/Get?id=IENQXLHPZTOZJEENEIKRAKNU https://files.transkribus.eu/Get?id=ABWEDGWBHNRUHCDAZQBCHDHR https://files.transkribus.eu/Get?id=HOHZGRKUZEEHTQFOZIUVWRKG https://files.transkribus.eu/Get?id=TYEHTOEKTZMMCEFACTOKAUFI https://files.transkribus.eu/Get?id=DWSKPMUEJAIOJJYYGIRBIGCQ https://files.transkribus.eu/Get?id=JOUYTXSVTWDUMSRJHPBXFGKX https://files.transkribus.eu/Get?id=FGDLEJDZPQNYAMEAQXIQLDNK https://files.transkribus.eu/Get?id=JUXAJMEYQARQLPURIWTWGBBP https://files.transkribus.eu/Get?id=KRNJXKZJCTBMJJDLHHMMSINE https://files.transkribus.eu/Get?id=YULOHZIMFHEOJKWDHCFOLBUL https://files.transkribus.eu/Get?id=OFTLVWCMKVUPUZZJCRLXEIYS https://files.transkribus.eu/Get?id=YHRJAICVWUIQGVCSIPBMKEKH https://files.transkribus.eu/Get?id=JIYRQKQXDYYQJOFINPLTOQSZ https://files.transkribus.eu/Get?id=SSASFJCYPAANTCROPGETZUPP https://files.transkribus.eu/Get?id=QEXKWYLPSRKUWBCIGQNHHPTP https://files.transkribus.eu/Get?id=SVZAIDARQWLMQTKQVIOOSAKV https://files.transkribus.eu/Get?id=XADEUHKOGGIWCOCZMTGGFHDP https://files.transkribus.eu/Get?id=QRUEXOAFCJBJXSXVTYKDCVKF https://files.transkribus.eu/Get?id=TZWAQKCGYPBWVZNSHVAKWCLZ https://files.transkribus.eu/Get?id=HVTGPLBHUEUXGJMGBNHIJIVS https://files.transkribus.eu/Get?id=KGTWYGJEQVUTLYQJJHDYRSDR https://files.transkribus.eu/Get?id=UCCYYBWUNOZJTXPTLDSGMKED https://files.transkribus.eu/Get?id=FTRLAADUOFFQLRZKSCULMDKF https://files.transkribus.eu/Get?id=BYLJWEGQKLYXLDPIEERCHAHG https://files.transkribus.eu/Get?id=SYZATLLCHFOLZWDODQRXNKNF https://files.transkribus.eu/Get?id=WZJQYUAGMQINLJKWSLBKSGTG https://files.transkribus.eu/Get?id=WOTWYEYWCOLUIZDUZMKQOBPA https://files.transkribus.eu/Get?id=IZHOVYUENLZPLEPYGPDACJXR https://files.transkribus.eu/Get?id=ITMLHBAAKCXDNTLMKMXNWOVM https://files.transkribus.eu/Get?id=GTEWMTDGWLEQMNDTONNPWVAV https://files.transkribus.eu/Get?id=XHVQSIUMEGUOGCMLVIBNPDIZ https://files.transkribus.eu/Get?id=IYDRMOZDVMAEHZLCLYWJFKZU https://files.transkribus.eu/Get?id=OZGAGQKATLEVBZMKNPUSZJZA https://files.transkribus.eu/Get?id=BMYKTWTVVTYYYQQSTXWQHKLC https://files.transkribus.eu/Get?id=IYUGXNCSKYCGDZZATBMXIPRX https://files.transkribus.eu/Get?id=BBTDFLKQWLYLTYYEMEYWRCFX
# Get XML files.
wget --content-disposition https://files.transkribus.eu/Get?id=JHVUSEFVRTWKTDYBEFVBEPFB https://files.transkribus.eu/Get?id=BYFBODHLFRJDQBVOROTLEWET https://files.transkribus.eu/Get?id=RWSSDKJHNAPTDXVLREBGBCLW https://files.transkribus.eu/Get?id=YEPDJBAQSQUMXTBRNXGGFREU https://files.transkribus.eu/Get?id=FZQWWFZCNWCYAPQRVIZTAOSX https://files.transkribus.eu/Get?id=IMXDAERWVVGYBIOMIEFTXFMK https://files.transkribus.eu/Get?id=LVZAFONLFVUFHYVDCAHAERYP https://files.transkribus.eu/Get?id=CIMPTJQYTQESVVGCBQRALMEX https://files.transkribus.eu/Get?id=CVSIQYLKNTXSVXRHBRBHIKKM https://files.transkribus.eu/Get?id=UIOFSVNICSMOSBKPJBZOATHM https://files.transkribus.eu/Get?id=NIEOIIUKDXUNFYJHHXADOUZH https://files.transkribus.eu/Get?id=RECFLPURJRTWVCNLRPARZWCT https://files.transkribus.eu/Get?id=BIIMISGELCYEAVCSFBEOBEXE https://files.transkribus.eu/Get?id=YSCXXSWLKELAWMDBRLAXKZPM https://files.transkribus.eu/Get?id=PPKQXCNQTIPCWWKKYEYKKFWR https://files.transkribus.eu/Get?id=TMGWJSEMDVZZUHIMCJMBQSAS https://files.transkribus.eu/Get?id=RQQGZJRNLALMLBHXWZAQRJJB https://files.transkribus.eu/Get?id=NKRITEGYTBOJEQKHIYFVPIKG https://files.transkribus.eu/Get?id=OHMVVRMMPUDTYDRDFHJZPKHZ https://files.transkribus.eu/Get?id=IQWIGWEOZAOZGEBTROXULIFN https://files.transkribus.eu/Get?id=XKLRDXMZFCGBBDKXXWQUOGTP https://files.transkribus.eu/Get?id=KBBFFENIEDVCCEJKGINYYOIA https://files.transkribus.eu/Get?id=VHHUSIBLNZVDYJDEUGTFNFIB https://files.transkribus.eu/Get?id=NWOLIMGYYSFJGHASEFSAHPSU https://files.transkribus.eu/Get?id=CMXZGEPQCEQPOYXCVMUAAMXL https://files.transkribus.eu/Get?id=UJMUBEOLKJSRNPSVFBSPIIQE https://files.transkribus.eu/Get?id=QNKLLUDVYNWHQFYPDGDKOSWG https://files.transkribus.eu/Get?id=UBELSVTLYVTCJSWPBNAKXCWN https://files.transkribus.eu/Get?id=UAQBIBAIRXGBSPALZPERMJQN https://files.transkribus.eu/Get?id=ABGYGICZRGMJPDKBTQRFMZWY https://files.transkribus.eu/Get?id=IEOAQAMGVZGEDZPSJCWPXJOW https://files.transkribus.eu/Get?id=JLXONNHBYYYPWIACEFKYYIGW https://files.transkribus.eu/Get?id=VJUZWLEDNECLNPBDKLPQIXPP https://files.transkribus.eu/Get?id=IWOOMDSPRWIHAUSGCDHMDRDY https://files.transkribus.eu/Get?id=NNHSAVUBAHYCJNDDFERAQIPK https://files.transkribus.eu/Get?id=ORCJXYGXILWSHLOOHPSECMQL https://files.transkribus.eu/Get?id=DDJGAXYOUQPGAQUBYVKNPSZI https://files.transkribus.eu/Get?id=ONNXZWVCJEGUTVJFDUVLSPIG https://files.transkribus.eu/Get?id=IIAVPDPIIJCOHAZDKJDBAMHL https://files.transkribus.eu/Get?id=VNVJLQRTQLJKWIUJPFDMRNPT https://files.transkribus.eu/Get?id=UYHHWDUMYRMHUMQXPBAWJFFA https://files.transkribus.eu/Get?id=YSSLAKWGNUJKXPMFGMFCHPNW https://files.transkribus.eu/Get?id=AXPZFHVZZQROPNGGQEYZNZXO https://files.transkribus.eu/Get?id=HHQOOTMTFVPDCARHHHGBIAUX https://files.transkribus.eu/Get?id=POTHXBKQPCHCXROHCSKQIWBJ https://files.transkribus.eu/Get?id=DSCRKYNYRSZFFHCQSDFCYUCO https://files.transkribus.eu/Get?id=JZPDYBTUFJAFSUACHHIZBWXH https://files.transkribus.eu/Get?id=YOECZHMWXUKAVVZYRONOQLJX https://files.transkribus.eu/Get?id=URNYUAFDAUHQPAHFMUUBTKHT https://files.transkribus.eu/Get?id=REFTLJUUJNMDPSVNZTSBEJEB https://files.transkribus.eu/Get?id=XMUKGQJUGFOGGGPLQNECYVBK https://files.transkribus.eu/Get?id=TCPFSFMTPGDZEWSMJZBXQEGH https://files.transkribus.eu/Get?id=YHMLVPJHADPBXBWKVJFDNTEL https://files.transkribus.eu/Get?id=JYXIBXPCZESDNNSHCBETTLLB https://files.transkribus.eu/Get?id=RQVWZBXVHZHRBPDJLSYDINFB https://files.transkribus.eu/Get?id=KIUANWLYIDEJYTDSVUFAWUCV https://files.transkribus.eu/Get?id=POZKJCZOPOKGILWSBPFUYDNK https://files.transkribus.eu/Get?id=PLIDQKLLDDSTURVBSYUKEHOJ https://files.transkribus.eu/Get?id=IRPJIVREFPZBZYXFELHDZWTH https://files.transkribus.eu/Get?id=GCAWPLAWQVTKWUDKTXBCTRJY https://files.transkribus.eu/Get?id=LTGBEPJCSYDUNHIIFKLVUJQX https://files.transkribus.eu/Get?id=AYRNLZJFWCBVZOFUNVQBTDYE https://files.transkribus.eu/Get?id=FHLBDWEYSNDJJZOAOBJTSVBU https://files.transkribus.eu/Get?id=DLDNAFADKWZLSBRCBLVCXCSL https://files.transkribus.eu/Get?id=YQTEGXGRQLFNNOYEUWBFRYWL https://files.transkribus.eu/Get?id=XHVQSESDDDCAALHGRMUBTXHV https://files.transkribus.eu/Get?id=RQLANQXXACUDSKBSYKIQNODK https://files.transkribus.eu/Get?id=FYQCLMNREDKIKKOUOOLGWEAX https://files.transkribus.eu/Get?id=UNFTXZTWQDELXHLHXDZSDWYX https://files.transkribus.eu/Get?id=ORBMIMGYCXLYQMYIAEIOXPJV https://files.transkribus.eu/Get?id=VEITMPANUCBOUFALNYCRWHTI https://files.transkribus.eu/Get?id=ECNNPBOIQRTBDNLSPXDMPGLP https://files.transkribus.eu/Get?id=QKCSSFYBNNVCOSFIADEDCXDA https://files.transkribus.eu/Get?id=LBPAHTIOKUBENKAYUXZECLSN https://files.transkribus.eu/Get?id=UXPPULKUSALSNGREBVUEWAHK https://files.transkribus.eu/Get?id=EIPMIKZZMACPQAYNQHTJCNKX https://files.transkribus.eu/Get?id=NFVGPKGIVBNNHLUSALCJKVQD https://files.transkribus.eu/Get?id=DZHPULUWLVNRBEVGOOWJPHAY https://files.transkribus.eu/Get?id=DTPIETEDOPLJBMYMVJRASTVP https://files.transkribus.eu/Get?id=TLGKJAWVGBPWWSUPTXHZYTWA https://files.transkribus.eu/Get?id=VQLTSTPZFUURQZWASJKPPLRR https://files.transkribus.eu/Get?id=TVUSOUXQBJCRBWDGZWKBUJSL https://files.transkribus.eu/Get?id=PXWWTCWWTEWUONCKNUBJSJDQ https://files.transkribus.eu/Get?id=JJBWWXFHRTPDNVEJHVMEEPRO https://files.transkribus.eu/Get?id=BVNFMMNMGEOFCNOXKNKAODQH https://files.transkribus.eu/Get?id=VCZYYLYPGLGDUEEOLTSKGRCP https://files.transkribus.eu/Get?id=YRBFGZUNUWESRFIZAWFZDLXA https://files.transkribus.eu/Get?id=RTMXEFVCUSQYBUFGXJROOSQW https://files.transkribus.eu/Get?id=WHUQMBKMTMBXEAJTENUGAWTY https://files.transkribus.eu/Get?id=EOFPMFLTWMSWKVWNUOOBFPWG https://files.transkribus.eu/Get?id=XQFIWGCRJEASTRDJAEJUCKKD https://files.transkribus.eu/Get?id=KZRXAIFVAXZSUELIMALBPERE https://files.transkribus.eu/Get?id=AOZASWDDOAXNSZAZUUAKZUVW https://files.transkribus.eu/Get?id=NQHOQQTPPIGHMJOXLHZCMKIK https://files.transkribus.eu/Get?id=ZWGLSCBYSFNEBDLBAZUXLAEC https://files.transkribus.eu/Get?id=ZNKWVPEGXHBULCAOEKGGGJKR https://files.transkribus.eu/Get?id=TKANIGGKMDYJJBZJVOMVKBHL https://files.transkribus.eu/Get?id=WVQFNOXILOFEARUYWIHZDCVP https://files.transkribus.eu/Get?id=CGUNPASSZQOASOYNDPAPWUAI https://files.transkribus.eu/Get?id=WFYPPCWSRVAMVXIJROZTESKN https://files.transkribus.eu/Get?id=PLGVGOXDAMGBNCISZQHKBVVJ https://files.transkribus.eu/Get?id=BJXKMZOQAUZCFJGHDKIDIKWY https://files.transkribus.eu/Get?id=CMDZVVKNEJZKHZKZRTRQOLBC https://files.transkribus.eu/Get?id=IUCKTFOEMGCBWQMSNKBJBMLE https://files.transkribus.eu/Get?id=EIZMZATKMGACEAALZFSMTHQJ https://files.transkribus.eu/Get?id=YJLDTJWFIEZYUDLALRXVNIBK https://files.transkribus.eu/Get?id=QSJICLTFSXESUQEFXPMEVACO https://files.transkribus.eu/Get?id=JVEAELRWZRPXATUPJNMPSAIO https://files.transkribus.eu/Get?id=XLISFOZJUAXSOZXKWGTTHKDD https://files.transkribus.eu/Get?id=VVCOOOIHGMRDLSMDHMPGYLFE https://files.transkribus.eu/Get?id=GZEEXENYIDTXCTPBSAORHHSA https://files.transkribus.eu/Get?id=IDGJOMRHAFKFUEHXKQGCUFNG https://files.transkribus.eu/Get?id=UKXMGZJJCWNYHCVKXSOCWTBI https://files.transkribus.eu/Get?id=BARPNPDXBCVDSYLRXQITLSLU https://files.transkribus.eu/Get?id=HTWTCHRHVYIYHVMITIYNPGOZ https://files.transkribus.eu/Get?id=NZSTGDRQKMIECMYFKIXRGWQF https://files.transkribus.eu/Get?id=ICTGHWNQUZZZGZOMQBKWFDIP https://files.transkribus.eu/Get?id=QZYDJWWQDHCWRRWCIKBFHLVB https://files.transkribus.eu/Get?id=XDUTNFFNKXYTPZGFHFLEHTSY https://files.transkribus.eu/Get?id=ISXNKHWMHCVPRJLHBZPOWXNR https://files.transkribus.eu/Get?id=QXXAZLBIQPTNGSQBIAYVNGWB https://files.transkribus.eu/Get?id=NYSQTMYOGWTKDEPTPKNCJUWC https://files.transkribus.eu/Get?id=EONVKVKOIAHEROQVYEQZIXKR https://files.transkribus.eu/Get?id=XVNYPROHFPLWSUQKMCBYVOTI https://files.transkribus.eu/Get?id=HUDJOFZWWGELFRBJVVZNRJAS https://files.transkribus.eu/Get?id=FFDRJLMVQTQGUITZNOWSRFHT https://files.transkribus.eu/Get?id=OIAVJGZJSNRVPIQOQGKELTTS https://files.transkribus.eu/Get?id=NKYVSVOAKQUOCKKVRYHBGZYH https://files.transkribus.eu/Get?id=KJNSHSVFLRTXPBRLABNVJNEZ https://files.transkribus.eu/Get?id=MBSKPVBMWWDUAAXUEJDYNLPZ https://files.transkribus.eu/Get?id=MHLGDEADRYDIURDNKUFEGPOB https://files.transkribus.eu/Get?id=CTDTEHIJXZCTRQOMLHCQAJIE https://files.transkribus.eu/Get?id=SNZPGLFMDJIHJTIJWPUWMNLO https://files.transkribus.eu/Get?id=NICXUBXEUYOWGTCAOHUXLQYP https://files.transkribus.eu/Get?id=YKDBOQISRUEPRNOGCQRJSKZO https://files.transkribus.eu/Get?id=DCKSZDCZAELVYSCOGSYMZCRV https://files.transkribus.eu/Get?id=JAZYPAUWVLULYUMXQXSXPDUP https://files.transkribus.eu/Get?id=KJFYLNDRPHMNWWNBRASILZSO https://files.transkribus.eu/Get?id=GXZXDWEKOUMRRWVGCKBVOYDX https://files.transkribus.eu/Get?id=XXASBUPYNMROTAPFYCPEKTFO https://files.transkribus.eu/Get?id=UTWJIKOCOWILGTHAVQELNXZP https://files.transkribus.eu/Get?id=UQWTMHFCAGGNCRPDHMISHEZQ https://files.transkribus.eu/Get?id=QOLIGHKUCUPILRHJZTQBMAQW https://files.transkribus.eu/Get?id=QKGCVEHTCAHRULIYFKOKKCFY https://files.transkribus.eu/Get?id=KHQDPSYPSBRGAJUFRJMOYYHV https://files.transkribus.eu/Get?id=VKWKDCOHWOCNAROYRJKHLFYC https://files.transkribus.eu/Get?id=XNLDKCGFMCHVEZTAJQLOHVYS https://files.transkribus.eu/Get?id=LSUTSNAWNILOCLCJLJVHHPPP https://files.transkribus.eu/Get?id=VBICXRUQAIGVEXAVMWTWIUDK https://files.transkribus.eu/Get?id=KJFZNMXHNIVVXNRWRLNPFOOT https://files.transkribus.eu/Get?id=JBOQSXZLWOLKVMLXOVQDOKJU https://files.transkribus.eu/Get?id=IMDEJHWZKACFXIGKEFXPQLII https://files.transkribus.eu/Get?id=QZKWGDEXJPZKTDUCDFWDEOET https://files.transkribus.eu/Get?id=EQSKRRFMYFDNFAJKYSBEAGQZ https://files.transkribus.eu/Get?id=ECTQOJPXUKYFSXVWRRSGVQRO https://files.transkribus.eu/Get?id=XRDHUZNHGKFBUMJFWMZSNYGM https://files.transkribus.eu/Get?id=VXBCUSYDWISJNFPVLOPGMHQL https://files.transkribus.eu/Get?id=HASZIIPNMFNRBGRFNAHLLUBR https://files.transkribus.eu/Get?id=MUKOMKPEEGXLSMZSNNZVZQVF https://files.transkribus.eu/Get?id=TEYUPVVQHWBKCNMEBDOYBTNY https://files.transkribus.eu/Get?id=YYEOVUJKGVARJPNSIFGLTJDA https://files.transkribus.eu/Get?id=OZBOZQQGNFNCAIZRFBXBZTQU https://files.transkribus.eu/Get?id=SPDIYTHJUBLTXUFRDZHXVVNF https://files.transkribus.eu/Get?id=GBDRCZDQRFRWUHIBQMQQFWOB https://files.transkribus.eu/Get?id=PUXPSBYVMNNKUVFDVBXBUFGU https://files.transkribus.eu/Get?id=VWQGXYKCTNVODXWYFXCXIESE https://files.transkribus.eu/Get?id=QAGEQHUIBPCKCVZKCAYSKJAS https://files.transkribus.eu/Get?id=DXHLEYZKFOUPCGDJKILGLEEN https://files.transkribus.eu/Get?id=XJIGEZRWLZCHXNFMCJVGRJYO https://files.transkribus.eu/Get?id=CDFNKPNPFNLUDLMHNSVWIOJY https://files.transkribus.eu/Get?id=GMJIJQMKUDSKYOZBYNUSAXNB https://files.transkribus.eu/Get?id=ZIKEBIHOCZEPWRSXGLQIFANV https://files.transkribus.eu/Get?id=UMXWNECUPMBVUSYXQSQTLJXA https://files.transkribus.eu/Get?id=DOYOKEEZUPSEPOLQFFOSDXLI https://files.transkribus.eu/Get?id=BXVPCUAOJBDIXZTHJNVJQUVG https://files.transkribus.eu/Get?id=ZKGAJZFHGCNQPJACKBVUONZZ https://files.transkribus.eu/Get?id=KAAROZFEMHTPFBBZRROKOWBF https://files.transkribus.eu/Get?id=KGDGOZQBVHZIPFABKPUYMOPG https://files.transkribus.eu/Get?id=QEJFWXTMXGFJTFWCORHVCGCR https://files.transkribus.eu/Get?id=LPDOYQEQTNASZIWTBRVUZILS https://files.transkribus.eu/Get?id=YLQKKITJQKAOCOHWYOTYZUMC https://files.transkribus.eu/Get?id=LOTMIHEOLSGYTTJQPJCUDXIF https://files.transkribus.eu/Get?id=NLDJIZLJUKUYPWQZHQPRBACG https://files.transkribus.eu/Get?id=PNLSPNFUBYEFSRAMKEGHZRMX https://files.transkribus.eu/Get?id=TFLTRJJTYRGVLRUZKWXQJTSL https://files.transkribus.eu/Get?id=IGKKGDXZEUPHQEHRNVGGGSHF https://files.transkribus.eu/Get?id=KDNTAWHVDCRMWUJYHCXXNAEJ https://files.transkribus.eu/Get?id=CJNSNRLHKBUNFWVVDPFBOYGC https://files.transkribus.eu/Get?id=OZJOAVTDJANNGFWRCSPSSMNT https://files.transkribus.eu/Get?id=ZAKYZRJPMEUGAFMEETNWJHYD https://files.transkribus.eu/Get?id=AAKZVULGHVARRSMQCUZFRQOT https://files.transkribus.eu/Get?id=ANJKLTOEBRJAOFAZIQAOPDVL https://files.transkribus.eu/Get?id=QQENWYLYGEORJOOWXQGDDOYQ https://files.transkribus.eu/Get?id=RWKPNTKIKQTIMRNWWYGVBHXM https://files.transkribus.eu/Get?id=IAOLLMALNIHPDWGDSPCAKUFB https://files.transkribus.eu/Get?id=TOZWZVEROLTDQSSGQVVIDPFW https://files.transkribus.eu/Get?id=NZAQUQJMEOMMALKIAJTKYCEB https://files.transkribus.eu/Get?id=IYTYIEJULURSVZXZJOKNLGTP https://files.transkribus.eu/Get?id=PDDUAQELWDADQVLKYKTMUIEO
cd ..
mkdir Validation
cd Validation
# Get image files.
wget --content-disposition https://files.transkribus.eu/Get?id=VKHSBYKDRSPCXNYVVDUUJHIJ https://files.transkribus.eu/Get?id=ZFDKMFYHZSSWJRWJDBSKXLME https://files.transkribus.eu/Get?id=KUNKUUMWRVICWVMISJETKKSP https://files.transkribus.eu/Get?id=VCYDPLIKHZNSMBXZNEARDMQG https://files.transkribus.eu/Get?id=DSZCVDYESIQHIKUWNBMEGUMI https://files.transkribus.eu/Get?id=UMTPCFPNQPEMZGGSGOZWRHZZ https://files.transkribus.eu/Get?id=XGLLYKKRWGKDJTSTQZTQSBDB https://files.transkribus.eu/Get?id=GMDPHGGENKKZXGUNAYDUYWLO https://files.transkribus.eu/Get?id=YWGNYDFWVHJLFFSYFHFOPPWV https://files.transkribus.eu/Get?id=IVUIBDCTBUYBGAEWVULXXFST https://files.transkribus.eu/Get?id=SQCNQZEQTSUHTQWYPFYQALUH https://files.transkribus.eu/Get?id=SCOYGUKQFUEPENAKBDSOXYGF https://files.transkribus.eu/Get?id=UHGKWLEZJGEGEUDTJWHQHWGT https://files.transkribus.eu/Get?id=RLXCXHKMEZPDOPKXEQWALGPY https://files.transkribus.eu/Get?id=REUHQRVBWHKVAGYKNOBKCSNP https://files.transkribus.eu/Get?id=FARONPPATEJYGRXVRHXRBPPI https://files.transkribus.eu/Get?id=BYXLCAPXMRURUGUHGOCDUAWJ https://files.transkribus.eu/Get?id=RHZLBFIAGXPNVSEKDMXVNAZT https://files.transkribus.eu/Get?id=HMXEFFBXKKVTASJBJNLMYPNL https://files.transkribus.eu/Get?id=JRNXMKFSVZEOMVCMPGWFRPYJ https://files.transkribus.eu/Get?id=NTGOQDNHUKAKGEUWVHRHDUOR https://files.transkribus.eu/Get?id=JJLLOATKBIRFPVVUMQETOTTN https://files.transkribus.eu/Get?id=CAOYGGOXCZVJYMBKTZGNCFCT https://files.transkribus.eu/Get?id=WSAKKKUEQCSGPKCLCVHHCKVD
# Get XML files.
wget --content-disposition https://files.transkribus.eu/Get?id=CHDVOIIYLHWZEPXMAUJWWNKU https://files.transkribus.eu/Get?id=ZRLEVECHRBPRETNOKXOFWGPU https://files.transkribus.eu/Get?id=WCIRBUXVTCXPGZTDSGJLGVWR https://files.transkribus.eu/Get?id=LMXKCJUHLQTOTWZKQOYBXRRH https://files.transkribus.eu/Get?id=RVNPPKIXIUTUFYAYGDUMWPRB https://files.transkribus.eu/Get?id=BOCCILWYTULGOIMRGCUHLLBF https://files.transkribus.eu/Get?id=PHXCDTHIHWHPCPVZANZLMEZW https://files.transkribus.eu/Get?id=ERCWXYVDDXSQTNDNRADLLNSL https://files.transkribus.eu/Get?id=VWCEXDWQJWMZPPBWPGVUFLSU https://files.transkribus.eu/Get?id=REJFIMVSKOBVFYCGVPRYXSIL https://files.transkribus.eu/Get?id=QPDBOUSCBUNSAVIKCHKWKLOP https://files.transkribus.eu/Get?id=XVCPFSVVGHVLQYTRGUPECDBO https://files.transkribus.eu/Get?id=KAHBRRJHNECLTFIDYKEVHIKN https://files.transkribus.eu/Get?id=WPWCKGCRUCANVXXOAXMMMJHM https://files.transkribus.eu/Get?id=XQUUXHYOWQGAUAQIYITHIMBX https://files.transkribus.eu/Get?id=XJYJHKBQWTLIIBGYXZYMOMTE https://files.transkribus.eu/Get?id=RWJTCQCXVXQOTBIMHLYRZXYN https://files.transkribus.eu/Get?id=JBLGRLYTPUUOYLKCPNNSMGCY https://files.transkribus.eu/Get?id=AYSTOXAKCOUKGCLBAGEDTBQN https://files.transkribus.eu/Get?id=KRNQGZWNGROPPIIJXPATHOHH https://files.transkribus.eu/Get?id=QDARVKDOIPYISUFZBCOTHCXV https://files.transkribus.eu/Get?id=EMFJCMSQWEELQTLIQDVDHBHJ https://files.transkribus.eu/Get?id=FDIATJURTJQLLGMJTHTQIQTC https://files.transkribus.eu/Get?id=CUMMKSQMFFWKWEEBFRYKWNRL
cd ..
python3.9 -m venv venv3.9
source venv3.9/bin/activate
pip install -U pip setuptools wheel
pip install kraken
pip install -U torch torchvision --extra-index-url https://download.pytorch.org/whl/cu113
The training is running on a Debian GNU Linux server with Nvidia RTX A5000 GPU.
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen -d cuda:0 --lag 20 -r 0.0001 -B 1 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/24/22 17:38:52] WARNING alphabet mismatch: chars in training set only: {'‡', '[', '†', 'ꝟ', '=', 'û', 'ꝸ', 'º', 'X', '╒', 'Ü', '’', '♃', '½', ']', train.py:304
'ꝯ'} (not included in accuracy test during training)
WARNING alphabet mismatch: chars in validation set only: {'Ä', 'ù', 'ͦ'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/24/22 17:38:54] WARNING Non-encodable sequence ùltis... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence Är„... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence ͦ 166... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ net │ MultiParamSequential │ 4.0 M │
│ 1 │ net.C_0 │ ActConv2D │ 1.3 K │
│ 2 │ net.Do_1 │ Dropout │ 0 │
│ 3 │ net.Mp_2 │ MaxPool │ 0 │
│ 4 │ net.C_3 │ ActConv2D │ 40.0 K │
│ 5 │ net.Do_4 │ Dropout │ 0 │
│ 6 │ net.Mp_5 │ MaxPool │ 0 │
│ 7 │ net.C_6 │ ActConv2D │ 55.4 K │
│ 8 │ net.Do_7 │ Dropout │ 0 │
│ 9 │ net.Mp_8 │ MaxPool │ 0 │
│ 10 │ net.C_9 │ ActConv2D │ 110 K │
│ 11 │ net.Do_10 │ Dropout │ 0 │
│ 12 │ net.S_11 │ Reshape │ 0 │
│ 13 │ net.L_12 │ TransposedSummarizingRNN │ 1.9 M │
│ 14 │ net.Do_13 │ Dropout │ 0 │
│ 15 │ net.L_14 │ TransposedSummarizingRNN │ 963 K │
│ 16 │ net.Do_15 │ Dropout │ 0 │
│ 17 │ net.L_16 │ TransposedSummarizingRNN │ 963 K │
│ 18 │ net.Do_17 │ Dropout │ 0 │
│ 19 │ net.O_18 │ LinSoftmax │ 48.5 K │
└────┴───────────┴──────────────────────────┴────────┘
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:58 val_accuracy: 0.00000 early_stopping: 0/20 0.00000
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:57 val_accuracy: 0.27963 early_stopping: 0/20 0.27963
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:44 val_accuracy: 0.63872 early_stopping: 0/20 0.63872
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:02 val_accuracy: 0.77172 early_stopping: 0/20 0.77172
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:59 val_accuracy: 0.82858 early_stopping: 0/20 0.82858
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:05 val_accuracy: 0.86218 early_stopping: 0/20 0.86218
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:50 val_accuracy: 0.88192 early_stopping: 0/20 0.88192
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:01 val_accuracy: 0.89528 early_stopping: 0/20 0.89528
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:55 val_accuracy: 0.90674 early_stopping: 0/20 0.90674
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:02 val_accuracy: 0.90923 early_stopping: 0/20 0.90923
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:03 val_accuracy: 0.91599 early_stopping: 0/20 0.91599
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:48 val_accuracy: 0.91963 early_stopping: 0/20 0.91963
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:42 val_accuracy: 0.92259 early_stopping: 0/20 0.92259
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:35 val_accuracy: 0.92418 early_stopping: 0/20 0.92418
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:59 val_accuracy: 0.92860 early_stopping: 0/20 0.92860
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:46 val_accuracy: 0.92850 early_stopping: 1/20 0.92860
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:48 val_accuracy: 0.92978 early_stopping: 0/20 0.92978
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:49 val_accuracy: 0.93181 early_stopping: 0/20 0.93181
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:02 val_accuracy: 0.93016 early_stopping: 1/20 0.93181
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:55 val_accuracy: 0.93448 early_stopping: 0/20 0.93448
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:33 val_accuracy: 0.93439 early_stopping: 1/20 0.93448
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:27 val_accuracy: 0.93539 early_stopping: 0/20 0.93539
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:33 val_accuracy: 0.93567 early_stopping: 0/20 0.93567
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:32 val_accuracy: 0.93588 early_stopping: 0/20 0.93588
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:39 val_accuracy: 0.93722 early_stopping: 0/20 0.93722
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:41 val_accuracy: 0.93679 early_stopping: 1/20 0.93722
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:31 val_accuracy: 0.93707 early_stopping: 2/20 0.93722
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:34 val_accuracy: 0.93847 early_stopping: 0/20 0.93847
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:32 val_accuracy: 0.93747 early_stopping: 1/20 0.93847
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:42 val_accuracy: 0.93884 early_stopping: 0/20 0.93884
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:23 val_accuracy: 0.93884 early_stopping: 1/20 0.93884
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:21 val_accuracy: 0.93682 early_stopping: 2/20 0.93884
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:44 val_accuracy: 0.93869 early_stopping: 3/20 0.93884
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:49 val_accuracy: 0.93657 early_stopping: 4/20 0.93884
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:37 val_accuracy: 0.93919 early_stopping: 0/20 0.93919
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:39 val_accuracy: 0.93782 early_stopping: 1/20 0.93919
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:04:27 val_accuracy: 0.93875 early_stopping: 2/20 0.93919
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:49 val_accuracy: 0.94028 early_stopping: 0/20 0.94028
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:05:50 val_accuracy: 0.94112 early_stopping: 0/20 0.94112
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:06:05 val_accuracy: 0.93978 early_stopping: 1/20 0.94112
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:08:45 val_accuracy: 0.93922 early_stopping: 2/20 0.94112
stage 41/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:09:56 val_accuracy: 0.94046 early_stopping: 3/20 0.94112
stage 42/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:07:38 val_accuracy: 0.94186 early_stopping: 0/20 0.94186
stage 43/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:09:34 val_accuracy: 0.93940 early_stopping: 1/20 0.94186
stage 44/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:09:55 val_accuracy: 0.93909 early_stopping: 2/20 0.94186
stage 45/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 1:10:04 val_accuracy: 0.94040 early_stopping: 3/20 0.94186
stage 46/∞ ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1930/7366 0:51:25 0:18:01 val_accuracy: 0.94040 early_stopping: 3/20 0.94186
Using more than one worker process makes the training much faster, and augmentation should result in a better model.
pip install albumentations
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+ -d cuda:0 --augment --workers 4 -r 0.0001 -B 1 --min-epochs 20 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/26/22 13:43:36] WARNING alphabet mismatch: chars in training set only: {'’', 'ꝯ', 'û', ']', 'ꝟ', 'ꝸ', 'Ü', '[', '†', '=', '‡', 'º', '½', 'X', '♃', '╒'} (not included in accuracy test during train.py:304
training)
WARNING alphabet mismatch: chars in validation set only: {'ͦ', 'Ä', 'ù'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/26/22 13:43:38] WARNING Non-encodable sequence ùltis... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence Är„... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence ͦ 166... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ net │ MultiParamSequential │ 4.0 M │
│ 1 │ net.C_0 │ ActConv2D │ 1.3 K │
│ 2 │ net.Do_1 │ Dropout │ 0 │
│ 3 │ net.Mp_2 │ MaxPool │ 0 │
│ 4 │ net.C_3 │ ActConv2D │ 40.0 K │
│ 5 │ net.Do_4 │ Dropout │ 0 │
│ 6 │ net.Mp_5 │ MaxPool │ 0 │
│ 7 │ net.C_6 │ ActConv2D │ 55.4 K │
│ 8 │ net.Do_7 │ Dropout │ 0 │
│ 9 │ net.Mp_8 │ MaxPool │ 0 │
│ 10 │ net.C_9 │ ActConv2D │ 110 K │
│ 11 │ net.Do_10 │ Dropout │ 0 │
│ 12 │ net.S_11 │ Reshape │ 0 │
│ 13 │ net.L_12 │ TransposedSummarizingRNN │ 1.9 M │
│ 14 │ net.Do_13 │ Dropout │ 0 │
│ 15 │ net.L_14 │ TransposedSummarizingRNN │ 963 K │
│ 16 │ net.Do_15 │ Dropout │ 0 │
│ 17 │ net.L_16 │ TransposedSummarizingRNN │ 963 K │
│ 18 │ net.Do_17 │ Dropout │ 0 │
│ 19 │ net.O_18 │ LinSoftmax │ 48.5 K │
└────┴───────────┴──────────────────────────┴────────┘
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/∞ ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 673/7366 0:16:39 0:01:43 early_stopping: 0/5 -inf
Increasing the number of workers further reduces the time per epoch even more.
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+ -d cuda:0
--augment --workers 16 -r 0.0001 -B 1 --min-epochs 20 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/26/22 13:49:52] WARNING alphabet mismatch: chars in training set only: {'ꝯ', '½', 'ꝟ', '╒', '=', '[', 'û', '’', 'Ü', '†', '♃', 'ꝸ', '‡', 'X', 'º', ']'} (not included in accuracy test during train.py:304
training)
WARNING alphabet mismatch: chars in validation set only: {'Ä', 'ù', 'ͦ'} (not trained) train.py:308
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[05/26/22 13:49:55] WARNING Non-encodable sequence ùltis... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence Är„... encountered. Advancing one code point. codec.py:131
WARNING Non-encodable sequence ͦ 166... encountered. Advancing one code point. codec.py:131
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ net │ MultiParamSequential │ 4.0 M │
│ 1 │ net.C_0 │ ActConv2D │ 1.3 K │
│ 2 │ net.Do_1 │ Dropout │ 0 │
│ 3 │ net.Mp_2 │ MaxPool │ 0 │
│ 4 │ net.C_3 │ ActConv2D │ 40.0 K │
│ 5 │ net.Do_4 │ Dropout │ 0 │
│ 6 │ net.Mp_5 │ MaxPool │ 0 │
│ 7 │ net.C_6 │ ActConv2D │ 55.4 K │
│ 8 │ net.Do_7 │ Dropout │ 0 │
│ 9 │ net.Mp_8 │ MaxPool │ 0 │
│ 10 │ net.C_9 │ ActConv2D │ 110 K │
│ 11 │ net.Do_10 │ Dropout │ 0 │
│ 12 │ net.S_11 │ Reshape │ 0 │
│ 13 │ net.L_12 │ TransposedSummarizingRNN │ 1.9 M │
│ 14 │ net.Do_13 │ Dropout │ 0 │
│ 15 │ net.L_14 │ TransposedSummarizingRNN │ 963 K │
│ 16 │ net.Do_15 │ Dropout │ 0 │
│ 17 │ net.L_16 │ TransposedSummarizingRNN │ 963 K │
│ 18 │ net.Do_17 │ Dropout │ 0 │
│ 19 │ net.O_18 │ LinSoftmax │ 48.5 K │
└────┴───────────┴──────────────────────────┴────────┘
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.00000 early_stopping: 0/5 0.00000
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.20782 early_stopping: 0/5 0.20782
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.57978 early_stopping: 0/5 0.57978
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.71006 early_stopping: 0/5 0.71006
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:20 val_accuracy: 0.78446 early_stopping: 0/5 0.78446
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.82139 early_stopping: 0/5 0.82139
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:57 val_accuracy: 0.83555 early_stopping: 0/5 0.83555
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.86224 early_stopping: 0/5 0.86224
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.86741 early_stopping: 0/5 0.86741
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.88089 early_stopping: 0/5 0.88089
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.89192 early_stopping: 0/5 0.89192
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.87604 early_stopping: 1/5 0.89192
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.89733 early_stopping: 0/5 0.89733
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.90216 early_stopping: 0/5 0.90216
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.90222 early_stopping: 0/5 0.90222
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.90182 early_stopping: 1/5 0.90222
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:48 val_accuracy: 0.90608 early_stopping: 0/5 0.90608
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:44 val_accuracy: 0.91259 early_stopping: 0/5 0.91259
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.90496 early_stopping: 1/5 0.91259
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91141 early_stopping: 2/5 0.91259
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.90126 early_stopping: 3/5 0.91259
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91119 early_stopping: 4/5 0.91259
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91100 early_stopping: 5/5 0.91259
Moving best model Juristische_Konsilien_Tuebingen+_17.mlmodel (0.9125926494598389) to Juristische_Konsilien_Tuebingen+_best.mlmodel
real 151m8.293s
user 2131m47.894s
sys 1193m39.862s
Resume the training:
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -i Juristische_Konsilien_Tuebingen+_17.mlmodel -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+ -d cuda:0 --augment --workers 16 -r 0.0001 -B 1 --min-epochs 200 --lag 10 -w 0 -s '[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0
.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do.1,2 Lbx200 Do]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/26/22 16:56:31] WARNING alphabet mismatch: chars in training set only: {'Ü', '‡', 'û', '♃', '╒', 'ꝯ', 'º', 'ꝸ', '†', '=', 'X', '[', '½', ']', 'ꝟ', '’'} (not included in accuracy test during train.py:304
training)
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ net │ MultiParamSequential │ 4.0 M │
│ 1 │ net.C_0 │ ActConv2D │ 1.3 K │
│ 2 │ net.Do_1 │ Dropout │ 0 │
│ 3 │ net.Mp_2 │ MaxPool │ 0 │
│ 4 │ net.C_3 │ ActConv2D │ 40.0 K │
│ 5 │ net.Do_4 │ Dropout │ 0 │
│ 6 │ net.Mp_5 │ MaxPool │ 0 │
│ 7 │ net.C_6 │ ActConv2D │ 55.4 K │
│ 8 │ net.Do_7 │ Dropout │ 0 │
│ 9 │ net.Mp_8 │ MaxPool │ 0 │
│ 10 │ net.C_9 │ ActConv2D │ 110 K │
│ 11 │ net.Do_10 │ Dropout │ 0 │
│ 12 │ net.S_11 │ Reshape │ 0 │
│ 13 │ net.L_12 │ TransposedSummarizingRNN │ 1.9 M │
│ 14 │ net.Do_13 │ Dropout │ 0 │
│ 15 │ net.L_14 │ TransposedSummarizingRNN │ 963 K │
│ 16 │ net.Do_15 │ Dropout │ 0 │
│ 17 │ net.L_16 │ TransposedSummarizingRNN │ 963 K │
│ 18 │ net.Do_17 │ Dropout │ 0 │
│ 19 │ net.O_18 │ LinSoftmax │ 48.5 K │
└────┴───────────┴──────────────────────────┴────────┘
Trainable params: 4.0 M
Non-trainable params: 0
Total params: 4.0 M
Total estimated model params size (MB): 16
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:05 val_accuracy: 0.91661 early_stopping: 0/10 0.91661
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.89668 early_stopping: 1/10 0.91661
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.91540 early_stopping: 2/10 0.91661
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92044 early_stopping: 0/10 0.92044
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:19 val_accuracy: 0.90824 early_stopping: 1/10 0.92044
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91755 early_stopping: 2/10 0.92044
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.90014 early_stopping: 3/10 0.92044
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.91244 early_stopping: 4/10 0.92044
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.91060 early_stopping: 5/10 0.92044
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:02 val_accuracy: 0.91213 early_stopping: 6/10 0.92044
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.90397 early_stopping: 7/10 0.92044
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92128 early_stopping: 0/10 0.92128
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.90992 early_stopping: 1/10 0.92128
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92181 early_stopping: 0/10 0.92181
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.91278 early_stopping: 1/10 0.92181
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92343 early_stopping: 0/10 0.92343
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92577 early_stopping: 0/10 0.92577
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92166 early_stopping: 1/10 0.92577
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.92197 early_stopping: 2/10 0.92577
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92240 early_stopping: 3/10 0.92577
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:57 val_accuracy: 0.91297 early_stopping: 4/10 0.92577
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92078 early_stopping: 5/10 0.92577
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92418 early_stopping: 6/10 0.92577
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92611 early_stopping: 0/10 0.92611
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.91758 early_stopping: 1/10 0.92611
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92412 early_stopping: 2/10 0.92611
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91991 early_stopping: 3/10 0.92611
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92287 early_stopping: 4/10 0.92611
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92069 early_stopping: 5/10 0.92611
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92767 early_stopping: 0/10 0.92767
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:53 val_accuracy: 0.92234 early_stopping: 1/10 0.92767
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:41 val_accuracy: 0.92848 early_stopping: 0/10 0.92848
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91652 early_stopping: 1/10 0.92848
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.92704 early_stopping: 2/10 0.92848
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93280 early_stopping: 0/10 0.93280
stage 35/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91870 early_stopping: 1/10 0.93280
stage 36/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92726 early_stopping: 2/10 0.93280
stage 37/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92041 early_stopping: 3/10 0.93280
stage 38/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92673 early_stopping: 4/10 0.93280
stage 39/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92894 early_stopping: 5/10 0.93280
stage 40/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.92633 early_stopping: 6/10 0.93280
stage 41/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.92695 early_stopping: 7/10 0.93280
stage 42/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.91484 early_stopping: 8/10 0.93280
stage 43/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92997 early_stopping: 9/10 0.93280
stage 44/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93296 early_stopping: 0/10 0.93296
stage 45/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.91892 early_stopping: 1/10 0.93296
stage 46/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.91798 early_stopping: 2/10 0.93296
stage 47/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92888 early_stopping: 3/10 0.93296
stage 48/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93000 early_stopping: 4/10 0.93296
stage 49/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92135 early_stopping: 5/10 0.93296
stage 50/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92256 early_stopping: 6/10 0.93296
stage 51/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92975 early_stopping: 7/10 0.93296
stage 52/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.92468 early_stopping: 8/10 0.93296
stage 53/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.92589 early_stopping: 9/10 0.93296
stage 54/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92589 early_stopping: 9/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 54/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92265 early_stopping: 10/10 0.93296
stage 55/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92265 early_stopping: 10/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 55/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92901 early_stopping: 11/10 0.93296
stage 56/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92901 early_stopping: 11/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 56/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92493 early_stopping: 12/10 0.93296
stage 57/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92493 early_stopping: 12/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 57/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92159 early_stopping: 13/10 0.93296
stage 58/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92159 early_stopping: 13/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 58/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92097 early_stopping: 14/10 0.93296
stage 59/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92097 early_stopping: 14/10 0.93296Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 59/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92620 early_stopping: 15/10 0.93296
stage 60/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93495 early_stopping: 0/10 0.93495
stage 61/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92433 early_stopping: 1/10 0.93495
stage 62/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92957 early_stopping: 2/10 0.93495
stage 63/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:59 val_accuracy: 0.92882 early_stopping: 3/10 0.93495
stage 64/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92863 early_stopping: 4/10 0.93495
stage 65/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93377 early_stopping: 5/10 0.93495
stage 66/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93218 early_stopping: 6/10 0.93495
stage 67/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93744 early_stopping: 0/10 0.93744
stage 68/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92683 early_stopping: 1/10 0.93744
stage 69/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93293 early_stopping: 2/10 0.93744
stage 70/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93520 early_stopping: 3/10 0.93744
stage 71/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93246 early_stopping: 4/10 0.93744
stage 72/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92340 early_stopping: 5/10 0.93744
stage 73/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:10 val_accuracy: 0.93548 early_stopping: 6/10 0.93744
stage 74/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:16 val_accuracy: 0.92944 early_stopping: 7/10 0.93744
stage 75/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93078 early_stopping: 8/10 0.93744
stage 76/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93094 early_stopping: 9/10 0.93744
stage 77/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93094 early_stopping: 9/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 77/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92941 early_stopping: 10/10 0.93744
stage 78/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92941 early_stopping: 10/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 78/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92044 early_stopping: 11/10 0.93744
stage 79/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92044 early_stopping: 11/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 79/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93489 early_stopping: 12/10 0.93744
stage 80/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93489 early_stopping: 12/10 0.93744Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 80/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92359 early_stopping: 13/10 0.93744
stage 81/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93751 early_stopping: 0/10 0.93751
stage 82/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92848 early_stopping: 1/10 0.93751
stage 83/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93937 early_stopping: 0/10 0.93937
stage 84/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:00 val_accuracy: 0.92841 early_stopping: 1/10 0.93937
stage 85/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93716 early_stopping: 2/10 0.93937
stage 86/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92876 early_stopping: 3/10 0.93937
stage 87/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92412 early_stopping: 4/10 0.93937
stage 88/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93461 early_stopping: 5/10 0.93937
stage 89/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92627 early_stopping: 6/10 0.93937
stage 90/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93081 early_stopping: 7/10 0.93937
stage 91/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93542 early_stopping: 8/10 0.93937
stage 92/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93000 early_stopping: 9/10 0.93937
stage 93/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93000 early_stopping: 9/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 93/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93667 early_stopping: 10/10 0.93937
stage 94/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93667 early_stopping: 10/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 94/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93150 early_stopping: 11/10 0.93937
stage 95/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.93150 early_stopping: 11/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 95/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.93486 early_stopping: 12/10 0.93937
stage 96/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93486 early_stopping: 12/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 96/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93673 early_stopping: 13/10 0.93937
stage 97/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93673 early_stopping: 13/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 97/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93623 early_stopping: 14/10 0.93937
stage 98/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93623 early_stopping: 14/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 98/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.91889 early_stopping: 15/10 0.93937
stage 99/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.91889 early_stopping: 15/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 99/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93053 early_stopping: 16/10 0.93937
stage 100/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93053 early_stopping: 16/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 100/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93377 early_stopping: 17/10 0.93937
stage 101/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93377 early_stopping: 17/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 101/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93196 early_stopping: 18/10 0.93937
stage 102/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93196 early_stopping: 18/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 102/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93358 early_stopping: 19/10 0.93937
stage 103/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93358 early_stopping: 19/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 103/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93595 early_stopping: 20/10 0.93937
stage 104/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93595 early_stopping: 20/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 104/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92745 early_stopping: 21/10 0.93937
stage 105/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:45 val_accuracy: 0.92745 early_stopping: 21/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 105/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:45 val_accuracy: 0.93128 early_stopping: 22/10 0.93937
stage 106/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:50 val_accuracy: 0.93128 early_stopping: 22/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 106/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:50 val_accuracy: 0.92026 early_stopping: 23/10 0.93937
stage 107/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92026 early_stopping: 23/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 107/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93916 early_stopping: 24/10 0.93937
stage 108/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93916 early_stopping: 24/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 108/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93274 early_stopping: 25/10 0.93937
stage 109/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93274 early_stopping: 25/10 0.93937Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue...
stage 109/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92549 early_stopping: 26/10 0.93937
stage 110/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93635 early_stopping: 27/10 0.93937
stage 111/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93589 early_stopping: 28/10 0.93937
stage 112/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92695 early_stopping: 29/10 0.93937
stage 113/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93414 early_stopping: 30/10 0.93937
stage 114/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.92760 early_stopping: 31/10 0.93937
stage 115/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93772 early_stopping: 32/10 0.93937
stage 116/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:10 val_accuracy: 0.93368 early_stopping: 33/10 0.93937
stage 117/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93947 early_stopping: 0/10 0.93947
stage 118/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93816 early_stopping: 1/10 0.93947
stage 119/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93732 early_stopping: 2/10 0.93947
stage 120/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.92882 early_stopping: 3/10 0.93947
stage 121/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93084 early_stopping: 4/10 0.93947
stage 122/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93284 early_stopping: 5/10 0.93947
stage 123/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93271 early_stopping: 6/10 0.93947
stage 124/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93389 early_stopping: 7/10 0.93947
stage 125/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92788 early_stopping: 8/10 0.93947
stage 126/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.94028 early_stopping: 0/10 0.94028
stage 127/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:04 val_accuracy: 0.93582 early_stopping: 1/10 0.94028
stage 128/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92848 early_stopping: 2/10 0.94028
stage 129/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93598 early_stopping: 3/10 0.94028
stage 130/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92518 early_stopping: 4/10 0.94028
stage 131/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93461 early_stopping: 5/10 0.94028
stage 132/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92764 early_stopping: 6/10 0.94028
stage 133/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.94162 early_stopping: 0/10 0.94162
stage 134/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.94411 early_stopping: 0/10 0.94411
stage 135/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93835 early_stopping: 1/10 0.94411
stage 136/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93794 early_stopping: 2/10 0.94411
stage 137/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.94000 early_stopping: 3/10 0.94411
stage 138/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:57 val_accuracy: 0.93875 early_stopping: 4/10 0.94411
stage 139/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93735 early_stopping: 5/10 0.94411
stage 140/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93894 early_stopping: 6/10 0.94411
stage 141/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93327 early_stopping: 7/10 0.94411
stage 142/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93632 early_stopping: 8/10 0.94411
stage 143/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93729 early_stopping: 9/10 0.94411
stage 144/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.92988 early_stopping: 10/10 0.94411
stage 145/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93181 early_stopping: 11/10 0.94411
stage 146/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93701 early_stopping: 12/10 0.94411
stage 147/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.94056 early_stopping: 13/10 0.94411
stage 148/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:03 val_accuracy: 0.94401 early_stopping: 14/10 0.94411
stage 149/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:35 val_accuracy: 0.94548 early_stopping: 0/10 0.94548
stage 150/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.92340 early_stopping: 1/10 0.94548
stage 151/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93997 early_stopping: 2/10 0.94548
stage 152/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:29 val_accuracy: 0.93903 early_stopping: 3/10 0.94548
stage 153/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93421 early_stopping: 4/10 0.94548
stage 154/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:31 val_accuracy: 0.93856 early_stopping: 5/10 0.94548
stage 155/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93860 early_stopping: 6/10 0.94548
stage 156/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.92138 early_stopping: 7/10 0.94548
stage 157/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93825 early_stopping: 8/10 0.94548
stage 158/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93171 early_stopping: 9/10 0.94548
stage 159/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:59 val_accuracy: 0.93087 early_stopping: 10/10 0.94548
stage 160/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93695 early_stopping: 11/10 0.94548
stage 161/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93364 early_stopping: 12/10 0.94548
stage 162/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.94330 early_stopping: 13/10 0.94548
stage 163/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93249 early_stopping: 14/10 0.94548
stage 164/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.94252 early_stopping: 15/10 0.94548
stage 165/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93324 early_stopping: 16/10 0.94548
stage 166/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.94255 early_stopping: 17/10 0.94548
stage 167/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93212 early_stopping: 18/10 0.94548
stage 168/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93785 early_stopping: 19/10 0.94548
stage 169/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:32 val_accuracy: 0.93906 early_stopping: 20/10 0.94548
stage 170/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:08 val_accuracy: 0.93794 early_stopping: 21/10 0.94548
stage 171/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:33 val_accuracy: 0.93003 early_stopping: 22/10 0.94548
stage 172/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92888 early_stopping: 23/10 0.94548
stage 173/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93140 early_stopping: 24/10 0.94548
stage 174/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93941 early_stopping: 25/10 0.94548
stage 175/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93255 early_stopping: 26/10 0.94548
stage 176/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93190 early_stopping: 27/10 0.94548
stage 177/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93414 early_stopping: 28/10 0.94548
stage 178/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93682 early_stopping: 29/10 0.94548
stage 179/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93069 early_stopping: 30/10 0.94548
stage 180/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93327 early_stopping: 31/10 0.94548
stage 181/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:01 val_accuracy: 0.92605 early_stopping: 32/10 0.94548
stage 182/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93193 early_stopping: 33/10 0.94548
stage 183/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93760 early_stopping: 34/10 0.94548
stage 184/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93265 early_stopping: 35/10 0.94548
stage 185/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.92767 early_stopping: 36/10 0.94548
stage 186/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93747 early_stopping: 37/10 0.94548
stage 187/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:22 val_accuracy: 0.93909 early_stopping: 38/10 0.94548
stage 188/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93695 early_stopping: 39/10 0.94548
stage 189/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93844 early_stopping: 40/10 0.94548
stage 190/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93053 early_stopping: 41/10 0.94548
stage 191/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:16 val_accuracy: 0.94034 early_stopping: 42/10 0.94548
stage 192/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:58 val_accuracy: 0.93421 early_stopping: 43/10 0.94548
stage 193/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93455 early_stopping: 44/10 0.94548
stage 194/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93383 early_stopping: 45/10 0.94548
stage 195/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93090 early_stopping: 46/10 0.94548
stage 196/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.94563 early_stopping: 0/10 0.94563
stage 197/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.93657 early_stopping: 1/10 0.94563
stage 198/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.94068 early_stopping: 2/10 0.94563
stage 199/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93483 early_stopping: 3/10 0.94563
stage 200/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93050 early_stopping: 4/10 0.94563
stage 201/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93187 early_stopping: 5/10 0.94563
stage 202/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:07:09 val_accuracy: 0.94731 early_stopping: 0/10 0.94731
stage 203/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93396 early_stopping: 1/10 0.94731
stage 204/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:26 val_accuracy: 0.93642 early_stopping: 2/10 0.94731
stage 205/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:28 val_accuracy: 0.93501 early_stopping: 3/10 0.94731
stage 206/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:27 val_accuracy: 0.93224 early_stopping: 4/10 0.94731
stage 207/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:20 val_accuracy: 0.94221 early_stopping: 5/10 0.94731
stage 208/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93473 early_stopping: 6/10 0.94731
stage 209/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:21 val_accuracy: 0.93726 early_stopping: 7/10 0.94731
stage 210/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:25 val_accuracy: 0.94062 early_stopping: 8/10 0.94731
stage 211/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:23 val_accuracy: 0.93928 early_stopping: 9/10 0.94731
stage 212/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7366/7366 0:00:00 0:06:24 val_accuracy: 0.93931 early_stopping: 10/10 0.94731
Moving best model Juristische_Konsilien_Tuebingen+_202.mlmodel (0.9473143219947815) to Juristische_Konsilien_Tuebingen+_best.mlmodel
real 1387m15.438s
user 19775m16.197s
sys 11007m8.085s
(venv3.9) stweil@ocr-01:~/src/gitlab/scripta/escriptorium/Juristische_Konsilien_Tuebingen/Transkribus_Exporte$ time nice ketos train -f page -t list.train -e list.eval -o Juristische_Konsilien_Tuebingen+256 -d cuda:0 --augment --workers 24 -r 0.0001 -B 1 --min-epochs 200 --lag 20 -w 0 -s '[256,64,0,1 Cr4,2,8,4,2 Cr4,2,32,1,1 Mp4,2,4,2 Cr3,3,64,1,1 Mp1,2,1,2 S1(1x0)1,3 Lbx256 Do0.5 Lbx256 Do0.5 Lbx256 Do0.5 Cr255,1,85,1,1]'
WARNING:root:scikit-learn version 1.1.1 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:Torch version 1.11.0+cu113 has not been tested with coremltools. You may run into unexpected errors. Torch 1.10.2 is the most recent version that has been tested.
[05/27/22 17:05:44] WARNING alphabet mismatch: chars in training set only: {'½', 'ꝟ', '‡', 'Ü', '’', 'û', ']', '[', 'º', '♃', 'X', '╒', 'ꝸ', 'ꝯ', '=', '†'} (not included in accuracy test during train.py:304
training)
Trainer already configured with model summary callbacks: [<class 'pytorch_lightning.callbacks.rich_model_summary.RichModelSummary'>]. Skipping setting a default `ModelSummary` callback.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ Name ┃ Type ┃ Params ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 0 │ net │ MultiParamSequential │ 15.2 M │
│ 1 │ net.C_0 │ ActConv2D │ 72 │
│ 2 │ net.C_1 │ ActConv2D │ 2.1 K │
│ 3 │ net.Mp_2 │ MaxPool │ 0 │
│ 4 │ net.C_3 │ ActConv2D │ 18.5 K │
│ 5 │ net.Mp_4 │ MaxPool │ 0 │
│ 6 │ net.S_5 │ Reshape │ 0 │
│ 7 │ net.L_6 │ TransposedSummarizingRNN │ 921 K │
│ 8 │ net.Do_7 │ Dropout │ 0 │
│ 9 │ net.L_8 │ TransposedSummarizingRNN │ 1.6 M │
│ 10 │ net.Do_9 │ Dropout │ 0 │
│ 11 │ net.L_10 │ TransposedSummarizingRNN │ 1.6 M │
│ 12 │ net.Do_11 │ Dropout │ 0 │
│ 13 │ net.C_12 │ ActConv2D │ 11.1 M │
│ 14 │ net.O_13 │ LinSoftmax │ 10.4 K │
└────┴───────────┴──────────────────────────┴────────┘
Trainable params: 15.2 M
Non-trainable params: 0
Total params: 15.2 M
Total estimated model params size (MB): 60
stage 0/∞ ━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 956/7366 0:05:12 0:00:58 early_stopping: 0/20 -inf