Prabeesh Keezhathra

Converting a list to string, with a delimiter. Implementation of inbuilt mkString() function in SCALA. #SCALA #SCALADEVELOPER #BIGDATADEVELOPER #BIGDATA Karthik K. def main(args: Array[String]): Unit ={ //var a = "Tanuj" //lowerCase(a) var s = "" var l = List(1,2,3,4,5,6,7,8) for( i <- 0 until l.length) { if( i == l.length -1) { s = s.concat(l(i).toString) } else { s = s.concat(l(i).toString) s = s.concat(':'.toString) } } print("The elements of list when converted to string are:-" + s) } } OUTPUT:

2

How Quora Scaled Writes with MySQL ? 1. Initial Setup: They began with a non-sharding setup, scaling reads using a leader-follower configuration + caching. 2. Increasing Write Throughput: They adopted two main approaches: Vertical Sharding: - They moved high-traffic tables to different partition groups, so multiple partitions ran different sets of tables using a master-follower setup. - They used Zookeeper to store information about partitions, their tables, and master and replica instances. - This method worked well for small tables but faced challenges with large tables, joins, and transactional operations. Horizontal Sharding: - This approach involves splitting a logical table into multiple physical tables, known as shards. - They used range-based sharding as they make frequent use of range queries. eg: If a shard of table T holds values >= x and < y, they named it T_shard_x_y. - They used the tool pt-online-schema-change to modify the schema, which requires 2x space as it creates a copy of the original table. Although it is an "online" schema change tool, it initially acquires a short-duration X lock on the entire table to create triggers. This X lock sometimes impacts production traffic. However, if the table has 10 shards, at any given time only about 10% of the table would be locked. This article provides an in-depth explanation and discusses the challenges involved. https://lnkd.in/gWcz26mY.

5

If you spend 5-10 mins on this post, then you can learn a really important topic in pyspark which a 1 hour video also cannot teach you this effectively. Lets say you have a 20 node spark cluster Each node is of size - 16 cpu cores / 64 gb RAM Let's say each node has 3 executors, with each executor of size - 5 cpu cores / 21 GB RAM => 1. What's the total capacity of cluster? We will have 20 * 3 = 60 executors Total CPU capacity: 60 * 5 = 300 cpu Cores Total Memory capacity: 60 * 21 = 1260 GB RAM => 2. How many parallel tasks can run on this cluster? We have 300 CPU cores, we can run 300 parallel tasks on this cluster. => 3. Let's say you requested for 4 executors then how many parallel tasks can run? so the capacity we got is 20 cpu cores 84 GB RAM so a total of 20 parallel tasks can run. => 4. Let's say we read a csv file of 10.1 GB stored in datalake and have to do some filtering of data, how many tasks will run? if we create a dataframe out of 10.1 GB file we will get 81 partitions in our dataframe. (will cover in my next post on how many partitions are created) so we have 81 partitions each of size 128 mb, the last partition will be a bit smaller. so our job will have 81 total tasks. but we have 20 cpu cores lets say each task takes around 10 second to process 128 mb data. so first 20 tasks run in parallel, once these 20 tasks are done the other 20 tasks are executed and so on... so totally 5 cycles, if we think the most ideal scenario. 10 sec + 10 sec + 10 sec + 10 sec + 8 sec first 4 cycles is to process 80 tasks all of 128 mb, last 8 sec is to process just one task of around 100 mb, so it takes little lesser but 19 cpu cores were free during this time. => 5. is there a possibility of, out of memory error in the above scenario? Each executor has 5 cpu cores and 21 gb ram. This 21 gb RAM is divided in various parts - 300 mb reserved memory, 40% user memory to store user defined variables/data. example hashmap 60% spark memory - this is divided 50:50 between storage memory and execution memory. so basically we are looking at execution memory and it will be around 28% roughly of the total memory allotted. so consider around 6 GB of 21 GB memory is meant for execution memory. per cpu core we have (6 GB / 5 cores) = 1.2 GB execution memory. That means our task can roughly handle around 1.2 GB of data. however, we are handling 128 mb so we are well under this range. I hope you liked the explanation :) My new big data batch is starting tomorrow. DM to know more. #bigdata #career #dataengineering #apachespark

732

26 Kommentare

𝐇𝐨𝐰 𝐭𝐨 𝐀𝐭𝐭𝐫𝐚𝐜𝐭 𝐓𝐨𝐩 𝐃𝐚𝐭𝐚 𝐓𝐚𝐥𝐞𝐧𝐭 𝐖𝐢𝐭𝐡𝐨𝐮𝐭 𝐁𝐫𝐞𝐚𝐤𝐢𝐧𝐠 𝐭𝐡𝐞 𝐁𝐚𝐧𝐤: So, you’re on a quest to hire the crème de la crème of data talent, but your budget is tighter than a data engineer's code on a Monday morning. Fear not! Here’s a guide to charm those data wizards without emptying your coffers: 1️⃣ 𝐒𝐭𝐞𝐩 𝟏: 𝐂𝐫𝐚𝐟𝐭 𝐭𝐡𝐞 𝐏𝐞𝐫𝐟𝐞𝐜𝐭 𝐉𝐨𝐛 𝐀𝐝 Start with a job ad that’s full of buzzwords and cutting-edge tools. It doesn’t matter if you’ve never heard of half of them or if your tech stack is older than a dial-up modem. This is a “visionary roadmap,” after all. Mention quantum computing, digital immune systems, and sprinkle in some “AI revolution” for good measure. It’s like a wishlist for Santa, but for tech. 🎁 2️⃣ 𝐒𝐭𝐞𝐩 𝟐: 𝐁𝐞 𝐕𝐚𝐠𝐮𝐞 𝐀𝐛𝐨𝐮𝐭 𝐒𝐚𝐥𝐚𝐫𝐲 You don’t want to scare off the top talent with something as tacky as a salary figure! Use the classic "DoE” to keep them guessing. If they push for details, just remind them you’re guarding against those “money motivated” types. It’s not about the paycheck, it’s about the passion for your dream, right? 💰 3️⃣ 𝐒𝐭𝐞𝐩 𝟑: 𝐒𝐞𝐥𝐥 𝐭𝐡𝐞 𝐃𝐫𝐞𝐚𝐦 Now that you’ve piqued their interest, it’s time to hype up the role. Make it sound like they’re the key to unlocking your company’s hidden potential. Emphasise your company’s grand ambitions for a data-driven utopia. You know, the kind where everyone rides unicorns and dashboards update themselves. 🦄 4️⃣ 𝐒𝐭𝐞𝐩 𝟒: 𝐓𝐨𝐮𝐭 𝐘𝐨𝐮𝐫 𝐂𝐮𝐥𝐭𝐮𝐫𝐞 𝐏𝐞𝐫𝐤𝐬 Here’s where you really hook them: your company culture. Highlight the perks that set you apart, like free office parking, Flexi-Work Fridays, and the infamous “optional” evening/weekend socials. And don’t forget the most irresistible perk of all – the chance to work with you! ✨ 5️⃣ 𝐒𝐭𝐞𝐩 𝟓: 𝐓𝐡𝐞 𝐎𝐟𝐟𝐞𝐫 𝐒𝐭𝐚𝐠𝐞 Congratulations! You’ve snagged their interest. Now, let them know they have incredible potential but aren’t quite there yet. Drop the bomb that the role will require a “slight” pay cut. But reassure them that it’s an opportunity of a lifetime. After all, who needs to pay rent when they’re working on something this groundbreaking? 🌱 6️⃣ 𝐒𝐭𝐞𝐩 𝟔: 𝐂𝐥𝐨𝐬𝐞 𝐭𝐡𝐞 𝐃𝐞𝐚𝐥 By now, your candidate is either fully invested in your vision or too exhausted to say no. This is your moment. They’ll say “YES” because, really, who could resist this dazzling opportunity? 🤩 --- 𝑇ℎ𝑒 𝑅𝑒𝑎𝑙 𝐷𝑒𝑎𝑙 Jokes aside, transparency is key. A mismatch between interview promises and reality on things like capabilities, salary, or culture can be a disaster. Be clear about what you offer and the challenges involved. After all, honesty not only saves everyone time but also builds trust – and that’s the real way to attract the best talent. #Recruitment #DataCulture --- Feel free to connect with me Sam Chaher; I’d love to hear your thoughts on these, and other recruiting strategies you’d rather were left behind!

11

1 Kommentar

Types of Indexing in Apache Hudi Index TypeDescriptionUse CasesBloom IndexUtilizes Bloom filters to quickly determine the potential presence of a record in a file group, reducing the search space during upserts and deletes.- High Throughput Upserts/Deletes: Ideal for scenarios with frequent data modifications. - Large Datasets: Efficiently handles big data volumes by minimizing file scans.Global Bloom IndexAn extension of the Bloom Index that maintains Bloom filters across all file groups globally, enhancing lookup performance for massive datasets.- Massive Scale Operations: Suitable for extremely large datasets where maintaining global state significantly improves performance. - Distributed Environments: Enhances consistency and speed across distributed file groups.HBase IndexLeverages HBase to store and manage index data externally, providing low-latency record lookups and high scalability.- Low-Latency Requirements: Perfect for applications needing rapid access to specific records. - External Index Management: When integrating with HBase ecosystems for robust indexing solutions.Simple IndexA non-index-based approach that scans all file groups to locate records. Acts as a fallback when other indexing mechanisms are not enabled or suitable.- Fallback Mechanism: Ensures data operations can proceed even without specialized indexing. - Smaller Datasets: Acceptable for datasets where full scans do not significantly impact performance.In-Memory Hash IndexMaintains an in-memory hash table mapping record keys to their respective file paths, enabling quick lookups without disk I/O.- Fast Lookups: Best for applications requiring rapid access to records with minimal latency. - Memory-Abundant Environments: Suitable when ample memory resources are available to store the hash index.SQL Query Examples Using Apache Hudi Indexes Apache Hudi integrates seamlessly with Spark SQL, allowing you to specify indexing types during table creation or data write operations. Below are examples demonstrating how to leverage different indexing types in your SQL workflows. Example - Creating a Hudi Table with Bloom Index sql Copy code CREATE TABLE hudi_bloom_index_table ( id STRING, name STRING, age INT, city STRING ) USING hudi OPTIONS ( primaryKey = 'id', preCombineField = 'timestamp', hoodie.index.type = 'BLOOM' ); Explanation: hoodie.index.type = 'BLOOM': Specifies the use of Bloom Index for efficient record lookups during upserts and deletes.

6

Insightful post by Heather Couture, PhD on dataset size and quality of the dataset. Thanks for sharing Heather 🙂 #RemoteSensing #EarthObservation #MachineLearning #DeepLearning #ComputerVision

2

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

22

Why Parquet Alone Is Not Enough: Iceberg & Hudi Explained with Example 🚀 🔍 Managing data lakes is hard. What happens when your datasets; - Evolve, - Need real-time updates, - or Require historical audits? Let’s explore this with a simple example and how Apache Iceberg and Apache Hudi enhance your data lake game. #Example: E-Commerce Sales Data Imagine you’re managing daily sales transactions for an e-commerce platform. #Goal’s: • Add a new column (campaign_id) to track promotions. • Debug a pricing error by querying historical data. • Update invalid records efficiently. #With_Parquet_Alone: 1. Adding campaign_id: You must rewrite all existing files, incurring time and cost. 2. Debugging historical data: You need manual backups or external versioning tools. 3. Updating invalid records: Entire files must be rewritten due to a lack of transactional support. #With_Iceberg_or_Hudi: 1. Adding campaign_id: Schema evolution allows you to add columns dynamically, without rewrites. 2. Debugging historical data: Use time travel to query the dataset before the error: SELECT * FROM sales_table FOR TIMESTAMP AS OF '2024-11-15'; 3. Updating invalid records: ACID transactions enable fast, reliable updates with simple SQL: MERGE INTO sales_table USING updates ON sales_table.order_id = updates.order_id WHEN MATCHED THEN UPDATE SET price = updates.price; #Result: Simplified operations, faster queries, and scalable data management. #Benefits of Iceberg & Hudi Over Parquet Both Iceberg and Hudi extend Parquet with: • Schema Evolution: Add, drop, or rename columns without rewriting files. • Time Travel: Query data as it existed at any point in time. • ACID Transactions: Perform reliable updates, deletes, and concurrent writes. • Partitioning Without Hassle: • Iceberg: Hidden partitions with automatic pruning. • Hudi: Optimized for upserts with built-in indexing. • Real-Time Support: • Hudi: Designed for streaming and CDC (Change Data Capture). • Iceberg: Best for batch analytics. • Metadata-Driven Optimization: Faster queries via centralized metadata. #Limitations of Parquet Alone: • No schema evolution. • No ACID transactions for updates or deletes. • Manual partitioning is error-prone. • No time travel for historical queries. • Limited real-time support for streaming or CDC. #Takeaway -If your data lake needs dynamic schemas, real-time updates, or historical insights, table formats like #Iceberg and #Hudi are game changers. They turn your static #Parquet files into a powerful data management system. Ready to make the switch? Let’s discuss how these formats can fit your data lake! 💬

55

10 Kommentare

CLASSES in SCALA are pretty much like CLASSES in C++ & Other OBJECT ORIENTED PROGRAMMING LANGUAGE. #SCALA #BIGDATA #CPP #C++ Karthik K. object CLASS { class Point{ private var _x = 0 private var _y = 0 private val bound = 100 def x: Int ={ return _x} 👈 //GETTER for private var _x def y: Int ={ return _y} 👈 //GETTER for private var _y def coord( x:Int, y:Int)={ 👈 //SETTER for private vars _x & _y this._x = x 👆👇 /"This" keyword store address of the object calling the function this._y = y println(this._x,this._y) } } def main(args: Array[String]):Unit = { val a = new Point 👈 //Object creation of Point in Heap Memory a.coord(20,30) print(a) } } OUTPUT:

4

3 Kommentare

TensorFlow vs. PyTorch: Finding the best Deep Learning framework for your AI journey 🚀 Which framework will boost your career goals? Whether you’re excited about diving into cutting-edge research with PyTorch or eager to create scalable AI models with TensorFlow, each option has its own perks! From how easy they are to use and the support you can get from the community to their flexibility and scalability, understanding these differences can really set you up for success in the fast-paced world of AI. I think Juliette Carreiro's article for @ironhack is a fantastic resource for anyone starting their journey in Data Science and Machine Learning! On another note, I’m still looking for a German-speaking Data Manager to join a leading company on an exciting automotive project in Frankfurt. If this sounds like a good fit for you or someone you know, please reach out here or at [email protected]. Happy to share more details! #DeepLearning #AI #DataScience #MachineLearning #TensorFlow #PyTorch #CareerInTech #Hiring #ironhack #redsofa #datamanager

12

1 Kommentar

Uber's Data Platform is an inspiration for many Data Engineers. It hosts around 👀 𝟭.𝟱 𝗲𝘅𝗮𝗯𝘆𝘁𝗲𝘀 of Apache Hadoop Distributed File System (HDFS). Uber has built 𝗗𝗮𝘁𝗮 𝗠𝗲𝘀𝗵 to handle such a huge amount of data, on which thousands of #Spark jobs are running and hundreds of concurrent users are running queries. ✨ 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗮 𝗗𝗮𝘁𝗮 𝗠𝗲𝘀𝗵? ☁ Data Mesh is a cloud resources management service that organizes data resources in an organization-centric hierarchical manner.  𝗖𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 𝗼𝗳 𝗗𝗮𝘁𝗮 𝗠𝗲𝘀𝗵: 1. 𝗗𝗼𝗺𝗮𝗶𝗻: A hierarchical unit of organizing all the resources in an organization or team. In data mesh terms, domains can be both at the organization level or at a sub-organization/team level. 2. 𝗗𝗼𝗺𝗮𝗶𝗻 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Resource containers, which are a unit of access control, resource grouping, and cost accounting. 3. 𝗗𝗮𝘁𝗮 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻: A logical container for data (DB/tables/files) belonging to the same category e.g., raw data, tables for internal consumption or modeled tables for external broad access. 4. 𝗗𝗼𝗺𝗮𝗶𝗻 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: The storage buckets which are a unit of access control, cost accounting, geographic location, storage class (hot, warm, cold), data lifecycle policies (TTL, legal holds, etc.), and data replication (to either standby store or archival store). 5. 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗱𝘂𝗰𝘁: A curated, discoverable, and usable representation of data, which includes the Hive tables, BI dashboards, workflow pipelines, etc. Read more about this here: https://lnkd.in/gkt7k5jJ #Uber #DataEngineering #DataMesh #architecture #cloud Arun M. Abhi Khune Sahana Bhat

22

2 Kommentare

Heterogeneous Graphs and Relational Graph Convolutional Networks (RGCNs): How can we prevent overfitting? In my post of last week [https://lnkd.in/d5cZn5ur], I spoke about Heterogeneous Graphs - examples of Heterogeneous Graphs being Knowledge Graphs which might have several node types or relationship types. I had mentioned that training such complex networks can become computationally intensive because each relationship type has to be associated with different weight/parameter thus increasing the number of  overall parameters to be learnt for a RGCN model and thus resulting in overfitting of the model. I had then highlighted about "Block Diagonal Matrices" And "Basis Learning" to reduce the number of parameters of a RGCN model. Let us try and put some more insight into "Block Diagonal and Basis Learning" and understand what these actually mean! Block Diagonal Matrices: The key idea with "Block Diagonal Matrices" is to make the Transformation matrices corresponding Wr as sparse as possible- that is: to enforce a block diagonal structure as shown in the Figure 1 below so that the non-zero elements lie only on the specific blocks of the transformation matrix "Wr". This reduces the number of non-zero elements which means that during computation you will have to only estimate the non-zero elements. This reduces the training time considerably What do you lose by using Block Diagonal Matrices? Well, we do lose some information by using Block Diagonal Matrices - the embedding dimension far from one another will not be able to interact with each other It’s only the embedding dimension that are along the non-zero entities that will be able to interact with each other that is: only the neurons along the non-zero entities will be able to interact with each other Basis Learning: The second idea to make the RCNs more scalable is to share the weights amongst the relationship types. That is: we do not want the relationships to be independent of each other, but we want the relations to share the information based on some magnitude of weighting. This can be achieved by elegantly representing the Transformation matrices Wr as a linear combination of "Basis" matrices. These "Basis" matrices are also termed "Dictionaries" In this approach, the weight matrix Wr will be represented as a linear combination i.e. a weighted sum of Dictionary matrices Vb as highlighted in the Equation below In the equation, arb is the "importance" weight corresponding to the relation for the matrix "b". Thus, every relation will have to learn its importance weight for each of the basis matrices. This significantly reduces the number of parameters - as the basis matrices may be 10 or a larger number whereas number of relation types may be thousands and arb are scalar numbers will have to be learnt #graphs #machinelearning #deeplearning My Technical Blogs Library 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on Engineering

6

Problem: There are thousands of language models (LMs) on 🤗 HuggingFace - but which one is the best for your NLP task? Solution: Use ⚖️ TransformerRanker ⚖️!

1

Knowledge Graphs for RAG (Retrieval- Augmented Generation): Some more insight | interpretation I have been talking of Knowledge Graphs for RAG (Retrieval-Augmented Generation) in my last couple of posts here: https://lnkd.in/dfxKezkw and https://lnkd.in/dx6jG__K which is also going to be my next blog in the series of Graphs. Another point to highlight here about why Knowledge Graphs with RAG scores over conventional RAGs is because in the Knowledge Graph we have relationships between chunks of documents defined (as below), then, whilst doing the similarity search we can pass a "window of chunks" through which the similarity search should be carried out (this is the Depth first search in a Graph) . We can thus ask to perform a similarity search with 2 (or more) chunks on either side of a node which will result in "more context" that can be passed to the LLM to retrieve a better response. That is, we are passing in a range of relationships - a "linked list" - to retrive the context . It is the not only the node with the highest score that we are passing but also extra information based on a path around the node of interest Context can further be expanded by passing the response obtained by the "windows of chunks" as context along with the original question to the LLM to obtain a better response. This is analogous to improving an LLMs response using its own response as context for a subsequent iteration. Incorporating the documents as chunks into the nodes of a KG and then building the realtioships, further makes, all Graph Traversal Algorithms like Shortest Path and Strongly connected , etc available to establish the right context #graphs #knowledgegraphs #llms #rag

8

**𝐋𝐞𝐯𝐞𝐥 𝐔𝐩 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐒𝐤𝐢𝐥𝐥𝐬 𝐰𝐢𝐭𝐡 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠!** A huge shoutout to my friends, Vishal Kaushal , a Snowflake Squad member, and Sachin Mittal, a 4x Snowflake Data Superhero, who are starting Snowflake Training: Zero to Hero course starting May 18th! . **Why Snowflake? ** Snowflake is more than just a data warehouse. It's a powerful, cloud-native data platform that empowers data engineers to build next-generation data solutions. Here's why learning Snowflake can be a game-changer for your career. By mastering Snowflake, you'll equip yourself with the skills to design and implement robust, scalable data solutions that are in high demand across industries. This positions you as a valuable asset in the ever-evolving world of data engineering. This comprehensive training, will equip you with the knowledge and skills to become a Snowflake master. Here's what you'll gain: Solid Foundation: Grasp the core concepts and functionalities of Snowflake. Hands-on Learning: Get practical experience through real-world exercises and projects. Unlimited doubt support Real time Projects : 2 End to End Real time projects , 1 using Matillion as well. Ready to take your data engineering skills to the next level? Don't miss out on this amazing opportunity! Sign up for the Snowflake Training: Zero to Hero course starting May 18th. 👉🏻 Enroll Here - https://lnkd.in/gcfxzNx4 📅 Live classes starts from 18th May 2024 Discount Coupon : LIVE20 PS : This is not a paid post. #Snowflake #DataEngineering #CloudDataWarehouse #Training #Upskilling #DataSuperhero

121

4 Kommentare

I came across an excellent GitHub repository packed with hands-on Jupyter notebooks for anyone eager to explore LLMs. It covers: 🔹 Introduction to LLMs – Understanding the basics. 🔹 Tokens & Embeddings – The foundation of language modeling. 🔹 Model Internals – Peeking under the hood of LLMs. 🔹 Text Applications – Classification, clustering, and topic modeling. 🔹 Prompt Engineering – Crafting prompts for better performance. 🔹 Advanced Text Generation – Tools and techniques. 🔹 Semantic Search – Intelligent information retrieval. 🔹 Multimodal Models – Combining text and image understanding. 🔹 Customization – Building embeddings, fine-tuning BERT, and generative models. Big thanks to the repo creator for sharing such a valuable resource. Here is the repo: https://lnkd.in/gUKg7Jp7

77

2 Kommentare

#PET #OWNERSHIP? Reminder of proxy variables Looking at map of pet dog ownership tells… … easily mistaken story. The clear geographical delineation in graph (matching the cold war Iron curtain line) can easily sway you into cultural root cause. 🤦🏻‍♂️ And that would be a great example of the correlation fellacy. Because 25 years after the Iron curtain is down is 2-3 generations of the dogs, long enough time for the political factors to wear off, if they ever had effect. 📖 The real learning of this graph is that there might be hidden factor that is not obvious and still it has profound effect on the actual “target variable”. 🐕 In our dog situation the underlying hidden root cause for the pet ownership is the share of agriculture on total labor market 1.5-2 generations back. Because if you had a farm, you were more likely to get used to living with animals or even needed some (shepherd dogs anybody?). The 2 generations lag is important, because you don’t make pet purchase decisions until you live in your own household and that is roughly, when you start your family (hence when you pass on the generation baton). 🔗 This phenomena is also known as #proxy #parameter and has significant impact also (in positive means) in #MachineLearning. Because proxy parameters are strong predictors, but are insights that markets tend to oversee. So if you know them, proxy parameters give you edge. 💡 Now you know the phenomenon I have a #puzzle for you: It is proven by data in many insurance companies (I worked on one of such studies) that #Zodiac sign is very strong predictor of the #car #accident rate. Yes, really true! Can you guess what it is proxy parameter of?

3