Sivas Subramaniyan’s Post

View profile for Sivas Subramaniyan, graphic

Designing Applied AI solutions

The world of GenAI/ LLMs/ LMMs are governed by 𝐓𝐨𝐤𝐞𝐧𝐬. Tokens are the building blocks of Language as a LLM perceives it- they can be whole words, parts of words, or even punctuation. The LLM economy transacts and charges per Token-In/Out. An LLM can be imagined, (in a reductionist way) as a Lego set. Where each Lego set comes in multiple different sizes of lego blocks, which when put together in a meaningful way creates appreciable outputs. Only but, the number of unique Lego blocks (tokens) can run up to few 100,000s, which together make meaningful generations. It is to be noted that the final collection of tokens that characterizes an LLM are unique to it and are optimized. Check out - (https://lnkd.in/dqgn5HUN) to play around with different tokenizers for each different GenAI models. The token sets per GenAI model is determined using 1) the algorithm that merges characters such as BytePairEncoding (used by GPT) or Sentencpiece (used by Google & Llama) and 2) Most critically/ importantly the 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬. Most of the GenAIs models and their optimized tokens are derived from English Datasets (Mistral (french) and Jais (Arabic) are exceptions). Hence, when you try Tamil or Telugu on GPT-4 you will see that the GenAI consumes more tokens and making the usage extremely expensive in the token economy. GenAI models have set up themselves as indispensable tools towards societal growth. And it makes most sense for each cohesive societal unit that can adopt a Language Model to leverage it. It is in this is proactive thought and direction that the India AI mission is ushered in with a holistic view of establishing an Indian LLM stack - Compute Capacity, Indigenous Datasets & Models and Applications. Excited for the Brave New World Ahead ! |<EOS>| https://lnkd.in/ddYvUruf https://lnkd.in/dAwrhxvj

What is IndiaAI Mission, announced by govt to ‘bolster’ country's AI ecosystem?

What is IndiaAI Mission, announced by govt to ‘bolster’ country's AI ecosystem?

hindustantimes.com

Stay ahead of the curve with the evolving GenAI landscape! 🌐

Like
Reply

To view or add a comment, sign in

Explore topics