Skip to content

Releases: megagonlabs/ginza

ginza-4.0.5

30 Sep 19:19
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v4.0.5
96efc4a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-4.0.5

ginza-4.0.5

  • 2020-10-01
  • Improvements
    • Add -d option, which disables spaCy's sentence separator, to ginza command line tool
Assets 3
Loading

ginza-4.0.4

11 Sep 10:58
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v4.0.4
85520ab
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-4.0.4

ginza-4.0.4

  • 2020-09-11
  • Improvements
    • ginza command line tool works correctly without BunsetuRecognizer in the pipeline
Assets 3
Loading

ginza-4.0.3

10 Sep 05:00
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v4.0.3
bf5faba
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-4.0.3

ginza-4.0.3

  • 2020-09-10
  • Improve bunsetu head identification accuracy over inconsistent deps in ent spans
Assets 3
Loading

ginza-4.0.2

03 Sep 16:11
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v4.0.2
4e731ac
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-4.0.2

ginza-4.0.2

  • 2020-09-04
  • Improvements
    • Serialization of CompoundSplitter for nlp.to_disk()
    • Bunsetu span detection accuracy
Assets 3
Loading

ginza-4.0.1

30 Aug 10:39
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v4.0.1
16f8452
Compare
Choose a tag to compare
ginza-4.0.1

ginza-4.0.1

  • 2020-08-30
  • Debug
    • Add type arguments for singledispatch register annotations (for Python 3.6)
Assets 3
Loading

LUW-4.0.0

18 Aug 12:29
@hiroshi-matsuda-rit hiroshi-matsuda-rit
ja_luw-4.0.0
3efaa69
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
LUW-4.0.0

The Ninjal's LUW (long-unit-word) NER model for GiNZA v4 and SudachiPy mode A.
The license of this model is the same as GiNZA and its models.

Usage: $ ginza -b ja_luw-4.0.0/

Accuracy:

        ent_f1:SPAN_LABEL=0.9551,SPAN_ONLY=0.9784
    ent_recall:SPAN_LABEL=0.9524,SPAN_ONLY=0.9757
 ent_precision:SPAN_LABEL=0.9578,SPAN_ONLY=0.9812

 ent_confusion
  URL(36): URL=30, _=3, 名詞-固有名詞-一般=2, 名詞-固有名詞-人名-一般=1
  web誤脱(31): _=14, 名詞-普通名詞-一般=7, 動詞-一般=4, 助動詞=2, 助詞-接続助詞=1, 助詞-格助詞=1, 助詞-終助詞=1, 感動詞-一般=1
  代名詞(1664): 代名詞=1606, _=43, 名詞-普通名詞-一般=11, 名詞-固有名詞-人名-名=1, 形状詞-一般=1, 副詞=1, 動詞-一般=1
  副詞(2841): 副詞=2604, _=114, 名詞-普通名詞-一般=53, 動詞-一般=17, 接続詞=14, 形容詞-一般=11, 名詞-固有名詞-人名-一般=10, 名詞-数詞=6, 形状詞-一般=4, 助詞-格助詞=2, 代名詞=2, 感動詞-一般=2, 連体詞=1, 名詞-固有名詞-一般=1
  助動詞(15394): 助動詞=15097, _=140, 助詞-格助詞=111, 助詞-副助詞=13, 動詞-一般=10, 助詞-終助詞=6, 形容詞-一般=6, 名詞-普通名詞-一般=5, 助詞-接続助詞=4, 助詞-準体助詞=1, 接続詞=1
  助詞-係助詞(4989): 助詞-係助詞=4906, _=80, 助詞-副助詞=1, 助動詞=1, 感動詞-一般=1
  助詞-副助詞(1841): 助詞-副助詞=1790, 助詞-終助詞=24, _=17, 助詞-接続助詞=4, 副詞=2, 動詞-一般=1, 形容詞-一般=1, 助動詞=1, 接続詞=1
  助詞-接続助詞(3354): 助詞-接続助詞=3201, _=105, 助詞-格助詞=41, 助動詞=5, 名詞-普通名詞-一般=1, 助詞-終助詞=1
  助詞-格助詞(21539): 助詞-格助詞=21268, _=159, 助動詞=72, 助詞-接続助詞=30, 助詞-準体助詞=4, 助詞-終助詞=3, 接続詞=2, 名詞-固有名詞-人名-名=1
  助詞-準体助詞(576): 助詞-準体助詞=565, _=5, 助詞-格助詞=5, 助詞-終助詞=1
  助詞-終助詞(1483): 助詞-終助詞=1443, _=14, 助詞-副助詞=9, 助動詞=5, 名詞-普通名詞-一般=5, 助詞-接続助詞=2, 形容詞-一般=2, 助詞-準体助詞=1, 副詞=1, 助詞-格助詞=1
  動詞-一般(12483): 動詞-一般=12005, _=364, 名詞-普通名詞-一般=72, 形容詞-一般=21, 副詞=10, 助動詞=2, 感動詞-一般=2, 形状詞-一般=2, 連体詞=1, 接続詞=1, 助詞-副助詞=1, 代名詞=1, 名詞-固有名詞-地名-一般=1
  名詞-助動詞語幹(29): 名詞-助動詞語幹=27, 形状詞-助動詞語幹=2
  名詞-固有名詞-一般(540): 名詞-固有名詞-一般=306, 名詞-普通名詞-一般=141, _=61, 名詞-固有名詞-地名-一般=13, 名詞-固有名詞-人名-一般=9, 副詞=4, 名詞-固有名詞-人名-姓=2, 名詞-固有名詞-地名-国=1, 名詞-固有名詞-人名-名=1, 動詞-一般=1,
形状詞-一般=1
  名詞-固有名詞-人名-一般(459): 名詞-普通名詞-一般=174, 名詞-固有名詞-人名-一般=141, _=55, 名詞-固有名詞-人名-姓=32, 名詞-固有名詞-一般=16, 代名詞=15, 名詞-固有名詞-人名-名=13, 名詞-固有名詞-地名-一般=5, 記号-文字=2, 補助記号-括弧開
=2, 接尾辞-名詞的-一般=1, 副詞=1, 形容詞-一般=1, 動詞-一般=1
  名詞-固有名詞-人名-名(497): 名詞-普通名詞-一般=345, 名詞-固有名詞-人名-名=73, 名詞-固有名詞-人名-姓=44, _=18, 名詞-固有名詞-人名-一般=7, 動詞-一般=3, 副詞=2, 名詞-固有名詞-地名-一般=1, 形容詞-一般=1, 名詞-数詞=1, 名詞-固有名詞-一
般=1, 感動詞-一般=1
  名詞-固有名詞-人名-姓(364): 名詞-固有名詞-人名-姓=194, 名詞-普通名詞-一般=108, 名詞-固有名詞-人名-名=26, 名詞-固有名詞-地名-一般=14, _=12, 名詞-固有名詞-人名-一般=2, 名詞-固有名詞-一般=2, 副詞=2, 形容詞-一般=2, 動詞-一般=1, 名詞-
数詞=1
  名詞-固有名詞-地名-一般(409): 名詞-固有名詞-地名-一般=257, 名詞-普通名詞-一般=89, _=24, 名詞-固有名詞-人名-一般=14, 名詞-固有名詞-一般=11, 副詞=4, 名詞-固有名詞-地名-国=2, 名詞-固有名詞-人名-姓=2, 形状詞-一般=1, 動詞-一般=1, 形容
詞-一般=1, 感動詞-一般=1, 接続詞=1, 名詞-数詞=1
  名詞-固有名詞-地名-国(230): 名詞-固有名詞-地名-国=215, _=7, 名詞-普通名詞-一般=3, 名詞-固有名詞-人名-一般=2, 名詞-固有名詞-地名-一般=2, 名詞-固有名詞-一般=1
  名詞-数詞(2308): 名詞-数詞=2096, _=165, 名詞-普通名詞-一般=34, 補助記号-ＡＡ-顔文字=4, 補助記号-一般=3, 名詞-固有名詞-地名-一般=2, 感動詞-一般=1, 接続詞=1, 補助記号-括弧閉=1, 副詞=1
  名詞-普通名詞-一般(24746): 名詞-普通名詞-一般=23234, _=1019, 名詞-固有名詞-一般=121, 形状詞-一般=83, 動詞-一般=64, 名詞-固有名詞-人名-一般=51, 副詞=42, 名詞-数詞=36, 名詞-固有名詞-地名-一般=27, 形容詞-一般=15, 名詞-固有名詞-人名-
姓=14, 名詞-固有名詞-人名-名=11, 感動詞-一般=5, 補助記号-一般=4, 名詞-固有名詞-地名-国=4, 代名詞=3, 形状詞-助動詞語幹=3, 助詞-終助詞=2, 助詞-格助詞=2, 助詞-副助詞=2, 形状詞-タリ=1, 補助記号-句点=1, 接続詞=1, 補助記号-括弧閉=1
  形容詞-一般(1646): 形容詞-一般=1515, _=56, 動詞-一般=33, 名詞-普通名詞-一般=20, 副詞=10, 助動詞=5, 形状詞-一般=2, 名詞-固有名詞-人名-一般=2, 感動詞-一般=1, 代名詞=1, 名詞-数詞=1
  形状詞-タリ(18): 形状詞-タリ=7, 名詞-普通名詞-一般=5, _=3, 代名詞=1, 動詞-一般=1, 感動詞-一般=1
  形状詞-一般(1582): 形状詞-一般=1454, 名詞-普通名詞-一般=84, _=27, 副詞=5, 形容詞-一般=5, 動詞-一般=3, 名詞-固有名詞-人名-姓=1, 連体詞=1, 感動詞-一般=1, 助動詞=1
  形状詞-助動詞語幹(465): 形状詞-助動詞語幹=450, _=10, 名詞-助動詞語幹=3, 副詞=2
  感動詞-フィラー(5): 感動詞-一般=3, 感動詞-フィラー=1, 接続詞=1
  感動詞-一般(161): 感動詞-一般=120, 名詞-普通名詞-一般=12, _=7, 形状詞-一般=4, 形容詞-一般=4, 動詞-一般=3, 副詞=3, 接続詞=2, 代名詞=2, 補助記号-一般=2, 助詞-終助詞=1, 名詞-固有名詞-人名-一般=1
  接尾辞-名詞的-一般(17): 接尾辞-名詞的-一般=7, 名詞-普通名詞-一般=6, _=4
  接尾辞-形容詞的(1): 名詞-普通名詞-一般=1
  接続詞(814): 接続詞=768, 副詞=31, _=11, 助詞-格助詞=2, 形容詞-一般=1, 名詞-普通名詞-一般=1
  接頭辞(2): _=1, 名詞-普通名詞-一般=1
  未知語(9): 名詞-固有名詞-一般=7, _=2
  漢文(1): 助詞-係助詞=1
  英単語(35): _=21, 名詞-固有名詞-一般=7, 名詞-普通名詞-一般=5, 補助記号-一般=1, 名詞-固有名詞-地名-一般=1
  補助記号-一般(1926): 補助記号-一般=1730, _=183, 補助記号-括弧閉=2, 助詞-終助詞=2, 助動詞=2, 動詞-一般=2, 感動詞-一般=2, 補助記号-ＡＡ-顔文字=1, 形容詞-一般=1, 名詞-固有名詞-人名-一般=1
  補助記号-句点(6322): 補助記号-句点=6215, _=107
  補助記号-括弧閉(2104): 補助記号-括弧閉=2100, _=4
  補助記号-括弧開(2067): 補助記号-括弧開=2064, _=3
  補助記号-読点(6992): 補助記号-読点=6991, _=1
  補助記号-ＡＡ-顔文字(103): 補助記号-ＡＡ-顔文字=73, _=8, 感動詞-一般=4, 名詞-普通名詞-一般=4, 副詞=4, 補助記号-一般=3, 名詞-数詞=3, 補助記号-括弧開=2, 補助記号-句点=1, 補助記号-ＡＡ-一般=1
  言いよどみ(2): 助詞-終助詞=1, 名詞-普通名詞-一般=1
  記号-一般(47): _=33, 名詞-普通名詞-一般=5, 名詞-固有名詞-一般=3, 記号-一般=3, 名詞-数詞=1, 英単語=1, 補助記号-一般=1
  記号-文字(103): _=58, 記号-文字=34, 名詞-普通名詞-一般=8, 補助記号-ＡＡ-一般=2, 名詞-数詞=1
  連体詞(1088): 連体詞=1069, _=13, 動詞-一般=4, 形容詞-一般=1, 副詞=1
Assets 5
Loading

ginza-4.0.0

16 Aug 13:15
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v4.0.0
a2cc7c4
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-4.0.0

ginza-4.0.0

  • 2020-08-16, Chrysoberyl
  • Important changes
    • Replace Japanese model with spacy.lang.ja of spaCy v2.3
      • Replace values of Token.lemma_ with the output of SudachiPy's Morpheme.dictionary_form()
    • Replace ja_ginza_dict with official SudachiDict-core package
      • You can deleteja_ginza_dict package safety
    • Change options and misc field contents of output of command line tool
      • Delete use_sentence_separator(-s) option
      • NE(OntoNotes) BI labels as B-GPE
      • Add subfields: Reading, Inf(inflection) and ENE(Extended NE)
    • Obsolete Token._.* and add some entries for Doc.user_data[] and accessors
      • inflections (ginza.inflection(Token))
      • reading_forms (ginza.reading_form(Token))
      • bunsetu_bi_labels (ginza.bunsetu_bi_label(Token))
      • bunsetu_position_types (ginza.bunsetu_position_type(Token))
      • bunsetu_heads (ginza.is_bunsetu_head(Token))
    • Change pipeline architecture
      • JapaneseCorrector was obsoleted
      • Add CompoundSplitter and BunsetuRecognizer
    • Upgrade UD_JAPANESE-BCCWJ to v2.6
    • Change word2vec to chiVe mc90
  • API Changes
    • Add bunsetu-unit APIs (from ginza import *)
      • bunsetu(Token)
      • phrase(Token)
      • sub_phrases(Token)
      • phrases(Span)
      • bunsetu_spans(Span)
      • bunsetu_phrase_spans(Span)
      • bunsetu_head_list(Span)
      • bunsetu_head_tokens(Span)
      • bunsetu_bi_labels(Span)
      • bunsetu_position_types(Span)
Assets 4
Loading

ginza-3.1.1

19 Jan 11:45
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v3.1.1
cec61bf
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-3.1.1

ginza-3.1.1

  • 2020-01-19
  • API Changes
    • Extension fields
      • The values of Token._.sudachi field would be set after calling SudachipyTokenizer.enable_ex_sudachi(True), to avoid serializtion errors
import spacy
import pickle
nlp = spacy.load('ja_ginza')
doc1 = nlp('This example will be serialized correctly.')
doc1.to_bytes()
with open('sample1.pickle', 'wb') as f:
    pickle.dump(doc1, f)

nlp.tokenizer.set_enable_ex_sudachi(True)
doc2 = nlp('This example will cause a serialization error.')
doc2.to_bytes()
with open('sample2.pickle', 'wb') as f:
    pickle.dump(doc2, f)
Assets 3
Loading

ginza-3.1.0

15 Jan 18:10
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v3.1.0
832b6e2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
ginza-3.1.0

ginza-3.1.0

  • 2020-01-16
  • Important changes
    • Distribute ja_ginza_dict from PyPI
  • API Changes
    • commands
      • ginza and ginzame
        • add -i option to initialize the files of ja_ginza_dict
Assets 5
Loading

ginza-3.0.0

14 Jan 20:54
@hiroshi-matsuda-rit hiroshi-matsuda-rit
v3.0.0
f620e5d
Compare
Choose a tag to compare
ginza-3.0.0

ginza-3.0.0

  • 2020-01-15
  • Important changes
    • Distribute ginza and ja_ginza from PyPI
      • Simple installation; pip install ginza, and run ginza
      • The model package, ja_ginza, is also available from PyPI.
    • Model improvements
      • Change NER training data-set to GSK2014-A (2019) BCCWJ edition
        • Improved accuracy of NER
        • token.ent_type_ value is changed to Sekine's Extended Named Entity Hierarchy
          • Add ENE7 attribute to the last field of the output of ginza
        • Move OntoNotes5 -based label to token._.ne
          • We extended the OntoNotes5 named entity labels with PHONE, EMAIL, URL, and PET_NAME
      • Overall accuracy is improved by executing spacy pretrain over 100 epochs
        • Multi-task learning of spacy train effectively working on UD Japanese BCCWJ
      • The newest SudachiDict_core-20191224
    • ginzame
      • Execute sudachipy by multiprocessing.Pool and output results with mecab like format
      • Now sudachipy command requires additional SudachiDict package installation
  • Breaking API Changes
    • commands
      • ginza (ginza.command_line.main_ginza)
        • change option mode to sudachipy_mode
        • drop options: disable_pipes and recreate_corrector
        • add options: hash_comment, parallel, files
        • add mecab to the choices for the argument of -f option
        • add parallel NUM_PROCESS option (EXPERIMENTAL)
        • add ENE7 attribute to conllu miscellaneous field
          • ginza.ent_type_mapping.ENE_NE_MAPPING is used to convert ENE7 label to NE
      • add ginzame (ginza.command_line.main_ginzame)
        • a multi-process tokenizer providing mecab like output format
    • spaCy field extensions
      • add token._.ne for ner label
    • ginza/sudachipy_tokenizer.py
      • change SudachiTokenizer to SudachipyTokenizer
      • use SUDACHI_DEFAULT_SPLIT_MODE instead of SUDACHI_DEFAULT_SPLITMODE or SUDACHI_DEFAULT_MODE
  • Dependencies
    • upgrade spacy to v2.2.3
    • upgrade sudachipy to v0.4.2
Assets 5
Loading