Tarun Khandagare

Speaking at #DES24, Pavan Nanjundaiah, Head – Tredence Studio, said today’s data engineer needs a beginner’s mindset to adapt to the growing change. “Today’s data engineers should expand left and right. Focus on your core foundational skills because they are not going to change. You don’t need to handcraft your code anymore.   Understand what is the business problem you’re trying to solve. My recommendation to data engineers is to shift left, which is the person who gave you the requirement and probably understands the business a little better. By doing this, you’re now zooming out of a core data engineer’s profile and understanding the business and the longer impact you can create.”   Read more here: https://lnkd.in/gKhtBwp6 #DataEngineers #Tredence #BeyondPossible

201

1 Comment

Read our new Blog on - "Apache Spark: RDD vs. DataFrame vs. Datasets" https://lnkd.in/gSXPxEhm A popular and effective distributed computing platform for handling massive data is called Apache Spark. Developers using Spark have three primary choices for processing data: DataFrame, Dataset, and RDD (Resilient Distributed Datasets). The variations between these choices, as well as their advantages and disadvantages, will be discussed in this article. For more detailed Blog, please refer to the our Website's Blog Section. https://lnkd.in/gSXPxEhm #TBC #ApacheSpark #BigData #DataScience #SparkRDD #SparkSQL #DataFrame #DataSet #DataEngineering #DataAnalysis #websitetraffic #onlinespeed #Windows11 #Windows10Pro #Angular #GooglePlay #agile #s #socialmedia #CyberThreats #digitalwellbeing #csharp #DotNet #softwaredevelopment #programming #vr #coding #webdevelopment

2

Databricks is a titan in data and AI. Through strategic acquisitions like Einblick, MosaicML, and Arcion, it is carving out a niche in generative AI. With a valuation of $43B, Databricks represents a visionary approach to technology: https://lnkd.in/er9DtRGe Generative AI is reshaping enterprise data applications. Databricks leads this charge, championing a future where data intelligence platforms redefine productivity. Databricks lives by the creed: “Cannibalize yourself before someone else does.” A strategy ensuring they stay ahead, innovating relentlessly. What fuels Databricks? A unique blend of mission-driven focus, strategic acquisitions, and a keen eye on the generative AI pulse. Their roadmap is not just about profit; it's about pioneering the future of AI. Looking ahead, Databricks envisions a dynamic AI industry landscape, where generative AI and data analytics converge to unlock new enterprise potentials. Behind Databricks are seven brilliant minds, each bringing a unique vision to the AI frontier. Join us as we dive deep into Databricks' story, exploring how it's shaping the future of Generative AI and data intelligence. 👇🏼 https://lnkd.in/er9DtRGe

1

NetApp Instaclustr's Anil Inamdar talks about how the developer experience of Apache Cassandra improves with the #opensource database's new 5.0 version: "It creates more efficient queries, and manages data with increased security and control." via HackerNoon https://lnkd.in/gckRy2QM

4

It has been a great year for Data Cloud, its adoption has been growing and new features have been added. Today, we are going to take a fresh look at Ingestion from Amazon S3. Since the beginning of the Data Cloud Diaries the product has been growing and some things have changed. Today, we are going to go step by step and see the current way that we bring data in through Amazon S3. This video is in the Data Cloud Diaries Series. Salesforce https://lnkd.in/gM4MXhfy

1

📣 At Databricks, we want to help customers build more #inference-friendly #llms. Our MixAttention architecture demonstrates that reductions in KV cache size make it possible to maintain model quality while improving inference speed and reducing memory footprint. Read our latest blog post to learn more: https://lnkd.in/gcHxM4DX

133

3 Comments

Nice Blog from Alex Merced at Dremio on how Apache XTable works in action and how to sync XTable-synced tables to Dremio's native catalog or open source Nessie 🔥 🔗 Blog Link: https://lnkd.in/deSd5qHG #dataengineering #softwareengineering #lakehouse

20

📢Excited to share our latest blog "PostgrSQL 17 is Here! A Major Release Packed with Performance Improvements" In this blog, Venkat Akhil Pavuluri explores about the Major Highlights of PostgreSQL 17 and why to upgrade it. Here is the blog : https://lnkd.in/g86q3_-8 🚀Aiming New Postgres Horizons🚀 Stay tuned!! For more info, reach out us at OpenSource DB or https://opensource-db.com/ #OpensourceDB #Database #PostgreSQL #PostgreSQL17 #Upgrade #NewRelease

21

Learn best practices for optimising Large Language Models (LLMs) and managing data pipelines with Sameer Rai of Tredence Inc. at the upcoming #TechgigWebinar. Discover tools like Kubernetes, TensorFlow, PyTorch, and Databricks to enhance scalability. Register Now:  https://bit.ly/45DUMR8 📅 Don’t miss out on this essential webinar! #DataScience #AI #MachineLearning #TechCommunity

2

Query optimization is critical in distributed data processing systems like #Presto. A framework used in #Presto is historical-based optimization (HBO) which leverages historical execution statistics to create a more efficient query strategy. Jay Narale of Uber Engineering shares more about optimizing queries in Presto with the HBO framework and some examples in his latest Presto blog 👏 https://lnkd.in/gUw_Bd2n

20

Building an AI application that deals with question answering task? Learn how RAG can help and how you can build one using AWS Bedrock and Langchain in this guide by Suyash Dubey. 🔗 Read here: https://hshno.de/RKXdtZG

1

Tomorrow at Confluent #Current24: Marc Farnsworth leads the Building Real-time Analytics Apps with Kafka and #Druid session. - Ingest streaming data natively at scale - Enhance and align data from multiple topics - Automatically detect data types and ingest into scalable tables - Execute SQL queries that combine real-time and historical data - Create interactive dashboards to visualize event data - Embed these dashboards into web applications https://bit.ly/4gvOkAH

5

🤔 Curious how RisingWave stacks up against Apache Flink? Join us on September 12 for the ultimate showdown by Yingjun Wu! 🎙 Yingjun will discuss: ▶ Stream processing: What it is and why it matters for real-time data; ▶ Design principles: How RisingWave’s #SQL-first design compares to Flink’s distributed state management; ▶ Cost comparison: Breakdown of infrastructure and operational costs; ▶ Use cases: Key industries and applications, including #IoT, finance, and #realtime analytics; 🎫 Save your spot here: https://lnkd.in/gb_EW8N8 #risingwave #apacheflink #flink #streamprocessing #dataprocessing

8

Spent some time today going through Meesho's blog on how they reduced 90% of their Redis infra cost and it is filled with interesting details ⚡ The blog covers their design, key decisions, and nuances around picking the right data structure and tuning Redis the right way. Give it a read, it is pretty informative. The high-level overview of it 1. key-value based initial design leading to bloated memory 2. moving to Redis Hash 3. it still bloats memory, but beyond a threshold 4. internals details on how Redis uses ziplist to implement hash 5. tuning `hash-max-ziplist-entries` configuration 6. minimizing calls to Redis using Lua script I have seen many people adopting KV based approach and then throwing money on infra to handle scale. But Meesho's blog highlighted how knowing database tunables can help you significantly reduce your infra costs. give it a read - https://lnkd.in/gp9N3JMh To be honest, this is a pretty interesting blog for anyone using Redis in production. I made a similar blunder back in 2020 and then spent 2 weeks optimizing it, so a hard relate :) #AsliEngineering #SystemDesign

663

16 Comments

👋 Hello Data Engineers! In our Spark Optimization Series, we’ve discussed Kafka, PostgreSQL, and MongoDB. Today, let’s explore Parquet optimizations: 1. Cost Savings: Compress data with formats like Snappy to reduce storage needs. 2. Faster Queries: Use features like predicate pushdown and column pruning to process only relevant data. 3. Efficient Organization: Partition data by key columns (e.g., year, month) for better performance. 4. Parquet’s columnar format is perfect for big data, offering improved storage and query efficiency. Missed the series? Check out our previous posts for more insights. Stay tuned for more Spark tips! 📊✨ #datatroops #spark #parquet #bigdata #optimization #datascience

13

🧰 Read more about how Amazon Managed Workflows for Apache Airflow (Amazon MWAA) uses Apache Airflow & Python to create workflows without having to manage the underlying infrastructure for: ⚖️ scalability ☁️ availability 🔒 security Read the blog by Radhika Jakkula & Sidhanth Muralidhar 👉 https://go.aws/4b8iHKh #AmazonRedshift #Apache #AmazonS3

170

2 Comments

Discover how to build scalable applications with Large Language Models (LLMs) in our upcoming webinar. From model optimization to efficient data pipeline management, Sameer Rai will cover it all.   Key Takeaways: 👉 Understanding LLMs 👉 Importance of scalability 👉 Best practices for optimization 👉 Tools like Kubernetes, TensorFlow, PyTorch, Databricks 👉 Case studies and future trends   When: 27 June, 2024  Time: 03:00 PM (GMT+5.5) Register Here: https://lnkd.in/e6Z4rrCz TechGig #LLM #Kubernetes #TensorFlow #Databricks #Tredence #BeyondPossible

151

5 Comments

Scaling Redis Clusters: Setup, Benefits, and Best Practices 🛠️ Redis is known for its high performance, but scaling is key to leveraging its full potential. In this case study, Senior Architect ArunSanthoshKumar Annamalai dives deep into Redis Clustering—how it works, how to set it up, and the key benefits it brings to modern infrastructure. From caching to horizontal scaling and automatic failover, Redis Clusters are transforming the way we handle real-time data processing. 👉 Read the full case study to explore best practices, real-world use cases, and step-by-step guidance on building resilient Redis Clusters. https://ow.ly/JGo950TMZ1R VenkataRamesh Nune Asis Patra Ranga Kanapathy Priyanka Baruah Farid Roshan Eric Pazdzierniak

15