Curiosity Plug #01
Today was a rather mixed bag of topics, some of which were intentional while others were not. I've tried maintaining a daily journal and failed, repeatedly. Yet, the optimist in me never gives up. So, here we are, with another bag of stuff.
Complex Adaptive Systems
Simple systems are predictable, and complex systems are not. The reason is not mere semantics, but the underlying variables and mechanisms that power them. When you imagine a simple lever moving something from point A to B, you imagine a simple system. Add one more copy of the lever at another end, and you find yourself tracking chaotic movements through space-time. This, is a simplified model of a nonlinear system. For larger systems, there could be adaptive behavior as the system tries to minimize its energy, or Action, and reach a favorable dynamic. Nature needs to be interpreted in the same lens, if we are to make any headway towards preservation of endangered species and conservation of vital ecosystems. Ecology is not merely observation and research, but the end goal is to ensure development happens according to the limits naturally placed by ecosystems and adheres to the boundaries of the system. Tom Wessels is a favorite ecologist who has a wonderful series on Forest forensics in New England's Forests in and around Maine, USA. Here is a nice talk by him on this topic.
I shall be reading more into this in the coming days.
Transformers
LLMs are all the rage, but any cursory glance at the compute and its flow will tell you that these are definitely chalk dust. The real magic is hiding in the inference windows, and perhaps beyond. What LLMs excel at is contextualizing NLP using transformers, which are essentially layers of encoders and decoders which take input text, contextualize it and then run it through layers of deep neural nets all the while maintaining attention (or context), in order to ensure the response comes real close to the queries. Deepseek has upped the game by introducing Reinforcement Learning to improve the quality of responses, which is why I strongly believe RL is the OG. More on this later.
I'll be enjoying this wonderful deep dive by Karpathy Sensei meanwhile, and I'll be back with more in a day.
Lock in!