![\"\"](\"https:\/\/colah.github.io\/posts\/2015-08-Understanding-LSTMs\/img\/LSTM3-chain.png\")
The top horizontal line is called the state of the cell, this is considered the long-term “memory” of the cell. Another special component of the architecture are the gates. Whenever you see a multiplication sign it means that it is a gate. <\/p>\n\n\n
![\"\"](\"https:\/\/thedataprocessblog.com\/wp-content\/uploads\/2023\/02\/image-1.png\")
Gates<\/h4>\n\n\n\n
In this case, they are preceded with a sigmoid activation function and the explanation for that is that the sigmoid function output goes from 0 to 1, therefore, the multiplication operation works as a gate for letting some information pass to the next stage or be forgotten. This way, the cell state can be changed and only the important things at the moment are remembered by the model.<\/p>\n\n\n\n
The output<\/h4>\n\n\n\n
The output of the model is the value h<\/em>, which can be viewed as a filtered version of the state of the cell. It is filtered at the last gate on the right, which has as input the cell state passed through a tanh<\/em> function.<\/p>\n\n\n\n Introduced by Gers & Schmidhuber (2000)<\/a>, the peephole approach provides the gates with the cell state itself as an input.<\/p>\n\n\n Introduced by Cho, et al. (2014)<\/a>, it merges the input and the forget gates into one gate, called the “update gate”. This achieves a simpler architecture, being easier to train and computationally cheaper. This variant has gained a lot of popularity over the years.<\/p>\n\n\n The introduction of the LSTM architecture brought far more possibilities to solve problems. The model proposed, and its variants, have been used successfully with various memory-related solutions.<\/p>\n\n\n\n The next big step that will be discussed in the next article is the Attention architecture introduced by Google. An approach that, again, changed everything in the machine learning world and created many more possibilities.<\/p>\n","protected":false},"excerpt":{"rendered":" After seeing the problems of the simple RNNs seen before researchers proposed a new approach to memory-related models. The new approach was named the LSTM model. Long Short Term Memory architecture (LSTM) Although the LSTM models are a variant of the Recurrent Neural Networks, they have very important changes. The whole idea is to separate … Continue reading Why transformers changed everything: LSTMs<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=\/wp\/v2\/posts\/71"}],"collection":[{"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=71"}],"version-history":[{"count":1,"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=\/wp\/v2\/posts\/71\/revisions"}],"predecessor-version":[{"id":75,"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=\/wp\/v2\/posts\/71\/revisions\/75"}],"wp:attachment":[{"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=71"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=71"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thedataprocessblog.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=71"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Other variants<\/h4>\n\n\n\n
Peephole connections<\/h5>\n\n\n\n
![\"\"](\"https:\/\/thedataprocessblog.com\/wp-content\/uploads\/2023\/02\/image-2-1024x409.png\")
Gated Recurrent unit (GRU)<\/h5>\n\n\n\n
![\"\"](\"https:\/\/thedataprocessblog.com\/wp-content\/uploads\/2023\/02\/image-3-1024x316.png\")
Conclusion<\/h3>\n\n\n\n