Huxi LI

Small doubt, Why are Databricks not much interested in this area : Integrating Databricks Secrets with AWS secrets manager ? In the Azure cloud, Azure Key vault integration with Databricks Secrets made lives easier for secret management. Note : I know that we can use boto3 directly to retrieve the secrets from AWS secrets manager using IAM permissions. But it has its own limitations in case of shared access mode.

74

8 commentaires

Recently langchain4j-memory-store-redis was added to the langchain4j Quarkusio extension. Basically it uses Redis instead of the RAM to keep the chat context during a conversation. But you might think : What could be the use case ? Here is one : Imagine a serverless chatbot pod (so scaling back to 0) but that needs to keep the context ! Also in case it's needed it can scale up and keep track of multiple sessions/contexts You can see in this short screencast that even after that the pod scaled back to 0, it was able to remember my name at the second request. Some cool stuff about this demo (blog will follow) : - I'm using of course Quarkus/langchain4j compiled natively with GraalVM - I'm using the openshift sandbox because it's free and it has the operators I need - It's using my favourite project around the k8s ecosystem : Knative - For the Redis part, I'm using the blazing fast DragonflyDB running on Aiven

44

4 commentaires

Fantastic data points by Ajit Inamdar on Amazon ECS !! Using Amazon ECS from tiny workloads to large production grade deployments is a no-brainer, and you let AWS handle all the heavy lifting of container orchestration and yes, you can also leverage your existing AWS skills to get started. #aws #containers #amazonecs

17

2 commentaires

I'm very happy to share a technical article I co-created with Deeksha Gunde, that walks through enabling a federated Airflow environment using AWS SNS/SQS. https://lnkd.in/e4RwPirp #aws #dataengineering #analytics #bigdata

33

AutoMQ: Advancing Event Streaming Beyond Apache Kafka I've been exploring AutoMQ, a new event streaming platform that's fully compatible with Apache Kafka but addresses many of Kafka's limitations. Here's what you need to know: 𝗞𝗲𝘆 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀 𝗼𝗳 𝗔𝘂𝘁𝗼𝗠𝗤: ✅ 100% Apache Kafka compatible ✅  10x more cost-effective ✅  Instant auto-scaling capability ✅  Single-digit millisecond latency ✅  S3-based storage layer 𝗔𝘂𝘁𝗼𝗠𝗤 𝘀𝗼𝗹𝘃𝗲𝘀 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗞𝗮𝗳𝗸𝗮 𝗽𝗮𝗶𝗻 𝗽𝗼𝗶𝗻𝘁𝘀: 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀: Kafka's scaling involves time-consuming partition reassignment. AutoMQ uses stateless brokers for instant scaling. 𝗗𝗮𝘁𝗮 𝗦𝗸𝗲𝘄: Kafka's static partition distribution leads to skew. AutoMQ implements built-in self-balancing. 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗖𝗼𝘀𝘁𝘀: Kafka relies on expensive local SSDs. AutoMQ leverages cost-effective S3 storage. 𝗛𝗼𝘁/𝗖𝗼𝗹𝗱 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗲𝗻𝘁𝗶𝗼𝗻: AutoMQ separates hot and cold data access for optimized performance. 𝗟𝗶𝗺𝗶𝘁𝗲𝗱 𝗖𝗹𝗼𝘂𝗱-𝗡𝗮𝘁𝗶𝘃𝗲 𝗦𝘂𝗽𝗽𝗼𝗿𝘁: AutoMQ's architecture is designed for cloud environments and K8s. 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗜𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻𝘀: 𝗦𝘁𝗮𝘁𝗲𝗹𝗲𝘀𝘀 𝗕𝗿𝗼𝗸𝗲𝗿𝘀: Enable rapid scaling without data replication. 𝗦𝗲𝗹𝗳-𝗕𝗮𝗹𝗮𝗻𝗰𝗶𝗻𝗴: Automatically migrates partitions based on throughput. 𝗦𝗵𝗮𝗿𝗲𝗱 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: Utilizes cloud storage services for durability and performance. 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗔𝗰𝗰𝗲𝘀𝘀: Employs pre-reading, concurrency, and Direct I/O for superior performance. AutoMQ offers a compelling solution for teams looking to enhance their event streaming infrastructure without disrupting existing Kafka setups. Its architecture addresses scalability, cost-efficiency, and performance challenges inherent in traditional Kafka deployments. Interested in a deeper technical dive ? Here is the official website - https://www.automq.com/ Github Repository: https://lnkd.in/e5f2u7BF

326

19 commentaires

Observability with OpenTelemtry Serie: (Zero-Code instrumentation) with Jaeger in Java #Observability #OpenTelemtry #Java

11

On my way to apidays Paris. If you see me, please feel free to say hello 😃. I’ll be speaking tomorrow on the API Architecture track: “APIs are the backbone of modern software ecosystems, but the journey to mastering API architecture can be a maze of complexities. From API management and gateways to service mesh, zero-trust principles, and considerations for design quality, build, and deployment, this talk explores the journey of typical API architecture evolution. Discover how platform teams can transform these intricate layers into seamless experiences, enabling developers to focus on innovation. In this session, we’ll unveil how emerging Architecture-as-Code solutions from FINOS provide an alternative approach to defining API patterns. Explore how schema-driven tools and standardized patterns can simplify your API ecosystem while driving agility and scalability.” Also known as a lot for 30 minutes! Hoping to meet some of the OpenAPI Initiative committee to chat about a perceived open source tooling challenges we are seeing with new versions of the specification. Generation quickly supported, but tools for linting, validation, and diff are struggling in some ecosystems. We would like to help! If you want to talk about writing books, I’m also happy to discuss!

50

7 commentaires

Lesson learnt : If you need "reboot" in your #AWS #EC2 instance UserData script, it should be at the end or at least there should be a 5-10s sleep after it. Otherwise the next command in the script starts before the TERM signal reaches the cloud-init process and may stop mid-way potentially leaving the system in a broken state.

11

Will Bilal Aslam kill your job ? Of course not he is going to make you more productive with Databricks Lakeflow. Simplicity is the keyword here. Quentin Ambard Watch the full session here: https://lnkd.in/eMN7t6XT #Databricks

29

📢 New Video Alert: Keynote by Julien Le Dem on Apache Iceberg and the Deconstructed Database 🎥 Watch Julien's keynote from the #ChillDataSummit in #SanFrancisco. Julien Le Dem is one of the key innovators in the open-source community and a driving force behind projects like Apache Parquet. In this talk, Julien explores how Apache Iceberg is revolutionizing data lakes, breaking down the concept of the deconstructed database and its impact on large-scale data management. If you're passionate about data architecture or curious about the future of data processing, don’t miss Julien’s insights on how Iceberg is pushing the boundaries of what's possible in data lakes. 📽️ Watch the full video here: https://lnkd.in/d9eg2exq #ApacheIceberg #OpenSource #DataEngineering #DataLakes #BigData

26

1 commentaire

Dynamic Tables are generally available ! Dynamic tables allow you to create data pipelines that are easy to build, operate, and evolve. With general availability, the following new enhancements are added: Sharing and collaboration: Dynamic tables can now be shared across regions and clouds using Snowflake’s sharing and collaboration features. This makes it easy to share clean, enriched, and transformed data products with consumers in your organization, partner organizations, or the broader data cloud community, ensuring they stay updated according to your specified cadence. Disaster recovery and replication: Dynamic tables now support high availability through Snowflake’s replication infrastructure. You can build your production pipelines with peace of mind knowing that you’re supported with Snowflake’s disaster recovery solutions. Observability: New functionality added for better observability via Snowsight and programmatic interfaces. In Snowsight, there are new account-level views, visibility into warehouse consumption, improved graph and refresh history, and the ability to suspend and resume refreshes. Observability functions now include new account usage views, extended retention of information schema functions and added support for consistent metadata across Snowflake observability interfaces. Data Cloud integrations: Added support for clustering, transient dynamic tables, and governance policies (on sources of dynamic tables and dynamic tables themselves), allowing you to benefit from the best features of the Snowflake Data Cloud. Scalability: You can now create four times more dynamic tables in your account, and ten times more dynamic table sources feeding into another dynamic table. There are no longer any limits on the depth of a directed acyclic graph (DAG) that you can create. Query evolution and SELECT * support: Dynamic tables now automatically evolve to absorb new columns from base tables and incrementally repair without needing to rebuild the dynamic table when new columns are added. New documentation: We’ve added new articles to our documentation covering development best practices, performance optimization guides, troubleshooting pipeline issues, and other improvements. https://lnkd.in/epkbrVTJ Our partner Indexima offers automatic dynamic table creation&maintenance based on query patterns ! Learn more in the interview (in French) https://lnkd.in/eFKDSsxj

33

Here's a secret Snowflake tip 🤫: with the ability to host Notebooks on the new container runtime, we can now install arbitrary packages using !pip. 🔥 Yes, we can also install packages not available on the Anaconda Snowflake repo (with all the risks that come with it 😅). For instance, now you can install the Cinemagoer package and search the IMDb for all the movies containing the word "matrix" in the title 👇

93

3 commentaires

What's AIBI genie ? How does it work ? Who should use it ? Chao Cai Sr Director of Engineering at Databricks answers all this questions and share some best practices as well. If you have not tried it yet after this session you will definitely do it. If you're a Data Analyst, you shouldn't consider genie as threat it's more a companion in the data democratization journey. #Databricks

20

After three prep tests for AWS Certified Cloud Practitioner (CLF-C02) I can tell you concentrate very much on WAF (Well-Architected Framework) and CAF (Cloud Adoption Framework). I got so many questions about both that sometimes your brain starts spinning and you are not sure what is right or wrong. So, spend time really digesting that material in docs section because it is important. I handle other stuff somehow easily but whatever goes in this kind of pure theory it tends to get a bit of confusing at least for me. Also, way how questions are made you have to concentrate 100% otherwise you will miss small detail that makes your answer correct or not. I tried first prep test at night after really long and tiresome day, and it was really a challenge. One I did in the morning after full nights rest was much better. Another thing that I still haven't managed to do is actually going through the list of all AWS services and reading what they do. That is another important thing because sometimes some services that you never heard about pop up and you have no idea what they are for, so my advice is go again with docs and check all listed services in AWS Services Overview. Now I am still far away from finished with preparing for test, but I am really in that final finishing prep mode. This is it for today. Please follow, support, repost and do the same so we can reach more people and make this planet a happier place. See you tomorrow. #aws #skillbuilder #microsoftlearn #777dayschallenge #awstraining #TechHeroes #LearningIsFun #FutureTechLeaders #cloudcareerjourneys #awstronaut #CloudUltraPro #cloudjourney #cloudcertified #awscommunity #awscommunitybuilders #awscertified #digitaltransformation

28

2 commentaires

🚀 New Blog Post Alert! 🚀 I just published my first post on Medium: "PySpark Hacks 1: How Repartitioning with Persist Can Transform Performance of Big Data Joins". In this article, I share insights on how leveraging repartition with persist can optimize join performance in PySpark, reducing data shuffling and improving cluster efficiency. 📖 Read the full article here : https://lnkd.in/gQNr6nX2 I’d love to hear your thoughts and feedback! Feel free to leave your comments or connect with me to discuss best practices for optimizing large-scale data processing in Spark. Let’s keep the conversation going! #BigData #PySpark #DataEngineering #DataProcessing #SparkOptimizations

15

When I read this first, I was like, Tools updated for MySQL 10.1 ? That has not been released yet... https://lnkd.in/e9EDWHWq #mysql

1

Why Data Mesh Architecture is the Future of Data Management ? Imagine running a restaurant where each station has its own fresh ingredients tailored to their needs. That’s the essence of Data Mesh Architecture in data management. Instead of relying on a central team to handle all data, Data Mesh decentralizes it, giving each department—whether it’s marketing, finance, or HR—control over their own data. This approach addresses the bottlenecks of traditional, centralized data systems, where data processing delays can hinder timely decision-making. Problem with Traditional Data Management: In traditional setups, data flows through a central team, causing delays as departments wait for processed insights. This creates inefficiencies and slows down decision-making—a significant drawback in today’s fast-paced business environment. Data Mesh: A Game Changer Data Mesh shifts the paradigm by distributing data ownership across different domains within an organization. Each domain manages its data, allowing for real-time insights and faster decision-making. This decentralized approach is already being used by companies like Netflix and Spotify to stay agile and responsive. Key Benefits : - Speed and Scalability : Data Mesh eliminates bottlenecks, allowing companies to scale data operations smoothly. - Real-Time Insights : Departments access and analyze their data instantly, facilitating quicker, informed decisions. - Enhanced Governance: Each domain takes responsibility for its data, ensuring better governance and compliance. Challenges to Consider: Adopting Data Mesh requires a cultural shift. Teams must take ownership of their data, requiring new skills and a change in mindset. Clear governance policies are essential to maintain data integrity and security. Why Data Mesh is the Future : As data volumes grow, the inefficiencies of traditional architectures become more evident. Data Mesh offers a scalable, flexible solution, enabling organizations to react quickly and innovate continuously. It’s not just a new way of managing data; it’s a transformative approach that empowers businesses to thrive in a data-driven world. #DataMesh #Datastrategy #AI Sushant Rajput

32

Data Engineers, App Developers, Full Stack Developers... Tune in this Wednesday 30th October 3PM CET / 10 AM ET / 7 AM PDT on AWS Twitch channel Let's Talk About Data Show to learn how AWS in-memory databases like ElastiCache and MemoryDB can supercharge your applications! I will be joining by Shane Adams who will walk us through popular use cases, discuss the benefits of caching and real-time data access, and demonstrate how to set up and get the most out of these awesome systems. Tony Mullen Mai-Linh Le Prasad Matkar Ibrahim Emara #elasticache #cache #memorydb #aws #letstalkaboutdata #twitch #database #data

49