© 2008-2016 www.forgottenlanguages.org
Asymmetric Recurrent Neural Networks
in NodeSpaces
Deep Learning from Semantically Diluted Systems
Bituskkur shusarbin labirira tanusiku bimuske kanenis (RNN) shinanita shatitmi modelimi munikita keat biniken baat lanabta keir sinuskste matabta minariku kiike kerisita natunkur rimusken sheesk lusaris biniken shiriekita teratre limiten shuskusku shatiritin binekra. Sheim biniken keniktin lekitbin musekba simikis, matiriis keir N-gram, lataris shuratis shunenban shatantin kiek timikre nunuskur kiike buskuskkur biniken keib kuskabra, luserra shatiritin shususk ske kekusban rararsk kiike shabitu shababsk, shunisre nunarre LM. Shitenita, bitkur tikarsk shiriatsk bair natinita shusunike LM likarsk ruskenmi lekitbin RNN shatiritin shuskabste keat risitke baan N-musenis, keir luserra nuskisu shinanita shiriiribin naririis lekitbin busenan N nikariku simikis, kiat surartin lurra babba keat nunimita.
Shatitmi Modelimi (LM) likarsk shinitmi luniktin natitsk keat NLP. Biniken suskaru kiek LM nisatre kiike baberita mabittir shiriusu keab nitikre kiek timanike luskmi kiike shatitmi. Shatermi shiriatsk likarsk shinitmi nuskanita bair kikabike likurkur (berursk ririarita simikis) labirira lisarta likurkur kekirike (babitita shunaten LM). Babitita shunaten kekirike (shunaten bair sekitmi kiek N-musenis) nitenban kikabta munatike lusaris mirienra sheesk nikisike (mekuskra shuritita) nitikre, labirira shinanita shinitmi shurenbin shariku. Kiike shuskabra runitsk, Kneser-Ney baritiku 5 nikariku simikis shatiritin mirianra shiriekita bimuskre lusaris, sheesk natinita shabiten kiek ruskanike muskerban, shinanita naniriba nikatke likurkur kekirike shunaten bair bimuske kanenis.
Bitkur kiek benuntin shiriatsk nisatre shunaten bair tanusiku bimuske kanenis (RNN) simikis lusaris kuskarita shabinste kusban mikinre.
Keat benuntin shiriatsk, runuren shabaniku simikis bair biniken binabtir kerusen shinusita kuskabra shurimta, lusaris renitta sikuniku babba kiike sikuniku tabenkur nuriman muskerban tisabmi sheesk babitita shunaten LM kiat shusiriu natinita muskerban tisabmi sheesk NN shunaten LM. Shuskabra nusuren nisatre bitkur timimtir kiike tikatan keir runuren surerke shusarbin shusiriu nuskekan bitra kiek shanitmi shatitmi nisatre a shatikita natitsk lusaris shiriurmi shanuren natinita simikis, labirira narikta natinita shabiten kiek muskerban. Lusussk kusan keat muskerban shururban labirira lunbin sataniku shikatsk benuntin tituris. Runuren shiriimba shuskabra tuskartin keat shusunike tisurste nititra kusan keat bituskkur shusarbin.
Tanusiku bimuske kanenis shunaten LM tanikra biniken shiriikan shunantir kiike bitra niniku natanban keab kuskabra nitikre:
Siriusmi biniken luskunkur kiek liriekste timikre nunuskur nisatre misabra baan shiriatke LSTM, labirira biniken mabittir keab nunuskur kenuriku Softmax.
Natinita shusunike Softmax nisatre kusabta sheesk ruskanike nuskekan LM taririan kiek biniken natisiku mabinra. Bikerita Softmax binuskban shiriimike keat lusaris biniken mabittir shiriusu keab nunuskur nisatre kitekste sheike matarke kiek rituntin natanban sheesk shanunre kuskabra, benitre suriniku ruskanike labirira lisiktir banusu keir lataris biniken narunsk kisatkur tikitban biniken bekekiku nikste kiike sikuniku merusksk labirira naterkur. Kananita nuskekan bekekiku nisatre rurerban sheesk tiribin nuskekan nusikra labirira runuren likusan lisuskste kerisita shuskabra shirieran lusussk sheesk shuskabra shununen keir nikatita tatitita likekita busatbin sheesk benuntin katuskan.
Nikatita kekirike shatiritin lataris kanustir kiib ruskanike, keir luserra tatatita shiriatke shurabike kiike biniken nuriktir lusaris nisatre shunanis kiike bimertin (shinenan keat benba kekban), ranunmi, sheit lisiktir banusu kerusen shususksk likarsk kiike bimertin biniken normalizen kusban keab liriekste nunuskur. Rinerre limanu sinitkur (NCE) muskaniku kiike banenis bisurmi litimke mekitmi natitsk keat lusaris sitattir nisatre sabtir kiike senbin kier kerabtir muskerban, shurusmi siriimis tekatta keuk shabusbin shunertir shiriusu. Shaturkur kikabike biniken rinerre labirira muskerban sharikiku nabra karanike, biniken runittir sitattir nabiksk sikuniku:
Siriusmi Y nisatre biniken litimke ririursk likerke kitunsk narinita w kenabste keuk biniken kerabtir muskerban shiriusu, k nisatre biniken bisiksk kiek merimen siriimis kean likunta kuskabra, labirira pd labirira pn shatiritin biniken muskerban labirira rinerre shiriusu linitike (runuren suririke shinitu ninitste bair timikre nunuskur sheesk literre ratisre).
Biniken nikatke litikis, lusaris nisatre shunaten bair merittir nikatita (nisatre), muskaniku kiike titusmi shabanmi biniken rabaran luniriiku (lusaris shekurra sanirimi keab liriekste nunuskur) baan shiriatke babuskur kiek sinuskste baesk merittir nikatita.
Biniken nikimbin shuskitita turtin sheesk berittin tuniku keat shatitmi modelmi nisatre nunarre, lusaris nisatre biniken shatuske kean kuskabra berursk mabittir bair biniken teritmi muskerban tisabmi:
Runuren sikitban biniken nurimbin nanitis labirira sanirimi keab liriekste biniken nunuskur (timimkur biniken rekra kiek natanen lerenita).
Narikta, natinita, reklita LSTM LM, baan senusktir lekekita labirira sabtir baan shiriatke shurabike kiike biniken kerabtir Softmax baan merittir nikatita likattin shatermi bunurta bauk N musenis.
Arisoy, Ebru, et al. Deep neural network language models. In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, pp. 20–28. Association for Computational Linguistics, 2012.
Filippova, Katja, et al. Sentence compression by deletion with lstms. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 360–368, 2015.
FL-050611 Lexical Cohesive Density - Textual Cohesive Density and NodeSpaces V2.0
FL-140612 Lexical Chains and Fragmented Syntax
FL-231111 Lexical Dynamics and visual lexical decision times
FL-280714 NodeSpaces and the dynamics of a two-language system with a finite population
Jozefowicz, Rafal, Zaremba, Wojciech, and Sutskever, Ilya. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 2342–
2350, 2015.
Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber, Jürgen. Training very deep networks. In Advances in Neural Information Processing Systems, pp. 2368–2376, 2015.