http://floreskul.name/posts/Recent content in Posts on Volodymyr FloreskulHugo -- gohugo.ioen-gbSun, 27 Apr 2025 00:00:00 +0000http://floreskul.name/posts/llm-memory-math/Sun, 27 Apr 2025 00:00:00 +0000http://floreskul.name/posts/llm-memory-math/When working with large language models, understanding memory requirements is crucial for choosing hardware, optimizing performance, and avoiding out-of-memory errors. Let’s break down the math behind LLM memory consumption during both training and inference. The Basics: Model Parameters Every LLM starts with its parameters. The memory needed for parameters alone is straightforward: Parameter Memory = num_parameters × bytes_per_parameter For a 7B-parameter model in float16 (2 bytes per parameter): Parameters = 7B × 2 bytes = 14 GBhttp://floreskul.name/posts/latent-space-strikes-again/Wed, 12 Feb 2025 00:00:00 +0000http://floreskul.name/posts/latent-space-strikes-again/DeepSeek’s recent releases sent ripples through the ML community—and the stock market, with NVIDIA dropping $600B in market cap in a single day. Under the hood of the demos and benchmarks sits a very practical idea: Multi-Head Latent Attention (MLA). It’s a way to run attention through a smaller latent space so you don’t hoard giant K/V caches. The Core Idea Standard multi-head attention does: $$q_t=W_Q x_t,\quad k_i=W_K x_i,\quad v_i=W_V x_i,\quad \text{Attn}(x_t)=\sum_i \operatorname{softmax}\left(\frac{q_t^\top k_i}{\sqrt{d_h}}\right) v_i.http://floreskul.name/posts/structure-from-noise/Thu, 05 Dec 2024 00:00:00 +0000http://floreskul.name/posts/structure-from-noise/Machine learning systems excel at finding patterns in chaos, extracting meaningful knowledge from what appears to be noise. Finding Businesses in a Haystack of Google Usernames During my time at Google, where I worked on spam and abuse prevention on Google Maps, one of the smaller projects was to identify Gmail accounts of businesses vs regular users. The challenge: usernames look like random character strings. But hidden patterns exist. Coffee shops create “hello.http://floreskul.name/posts/keys-queries-values/Fri, 21 Jun 2024 00:00:00 +0000http://floreskul.name/posts/keys-queries-values/“Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away.” - Antoine de Saint-Exupéry This principle underpins the split into Keys, Queries, and Values. It may look like extra complexity, but it’s a deliberate balance between expressiveness and simplicity. Over time we only added pieces when simpler setups stopped working. The Anatomy of Modern Attention At the heart of every Transformer lies the attention mechanism, which uses four key elements:http://floreskul.name/posts/ml-and-compression/Fri, 01 Mar 2024 00:00:00 +0000http://floreskul.name/posts/ml-and-compression/Machine Learning models are systems that learn patterns from data. This perspective has been prevalent in the era of traditional ML models, such as Logistic Regression and Random Forests. However, Deep Learning has significantly expanded the capabilities of ML models and enabled these models not only be optimised for the target metric but unintentionally serve as information compression systems for the training data. For example, the training process of a model like LLaMA 7B effectively leads to the compression of ~4.http://floreskul.name/posts/llms-in-web-data-extraction/Sat, 17 Feb 2024 00:00:00 +0000http://floreskul.name/posts/llms-in-web-data-extraction/The Web is estimated to contain 400B of documents, and this number continues to grow. With the Semantic Web largely failing to deliver on its promise to enable computers and people to work together on structured data, the need to automatically extract information from vast unstructured corpora of data has never been greater. LLMs offer new possibilities in this field. The tradeoff between their deep capacity but high costs underscores the need to identify system design patterns for efficient LLM-based information retrieval.http://floreskul.name/posts/embeddings-and-llms/Wed, 31 Jan 2024 00:00:00 +0000http://floreskul.name/posts/embeddings-and-llms/Large Language Models have not only revolutionized the field of Natural Language Processing but have also ignited a global wave of rapid breakthroughs in Generative AI. One intriguing question that arises in this landscape is: What is the current role of previous state-of-the-art NLP techniques like Text Embeddings? In simple terms, embeddings are numerical vector representations of information, whether it’s text, images, or even the output from a previous block of a Deep Neural Network.