Vinay Rayini

Best Pre:invent announcements so far (Analytics/Database/Storage): Analytics: [1] Amazon MWAA microenvironments for Apache Airflow: AWS has launched mw1.micro, a cost-effective environment class for Amazon MWAA. It features 1 vCPU and 3GB RAM for both scheduler/worker and web server components. Running on AWS Fargate with Graviton2-powered Aurora PostgreSQL supports up to three concurrent tasks and 25 DAGs while maintaining essential MWAA features. https://lnkd.in/dkrzEmge [2] Replicate changes from databases to Apache Iceberg tables using Amazon Data Firehose (in preview) https://lnkd.in/d6T6rhdn Database: [1] Scaling to 0 capacity with Amazon Aurora Serverless v2 Amazon Aurora Serverless v2 now supports scaling database capacity down to 0 Aurora Capacity Units (ACUs) during periods of inactivity, with automatic pause and resume capabilities. Customers only pay for storage costs when paused, and the database automatically resumes within 15 seconds when connections are requested. https://lnkd.in/dFvRdivH [2] Amazon DynamoDB lowers pricing for on-demand throughput and global tables DynamoDB has reduced prices for on-demand throughput by 50% and global tables by up to 67%. https://lnkd.in/dHDvrvbq [3] Amazon Keyspaces (for Apache Cassandra) reduces prices by up to 75% Amazon Keyspaces has lowered prices for on-demand mode by up to 56% for single-Region and up to 65% for multi-Region usage, and for provisioned mode by up to 13% for single-Region and up to 20% for multi-Region usage. Additionally, to make data deletion more cost-effective, Keyspaces has lowered time-to-live (TTL) delete prices by 75%. https://lnkd.in/dq8G8Eg7 [4] AWS Compute Optimizer now supports rightsizing recommendations for Amazon Aurora AWS Compute Optimizer automatically analyzes Amazon CloudWatch metrics such as CPU utilization, network throughput, and database connections to generate recommendations for your DB instances running Amazon Aurora MySQL-Compatible Edition and Aurora PostgreSQL-Compatible Edition engines. https://lnkd.in/dGcAkWvA Storage: [1] Amazon S3 now supports up to 1 million buckets per AWS account Amazon S3 has increased the default bucket quota from 100 to 10,000 per AWS account. https://lnkd.in/dNQrKNYU

9

Attending #DataAISummit next week? Swing by the O'Reilly booth (#127) at the Data+AI Summit on June 12 from 3:15PM - 3:45PM PT for a book signing with me! We're giving away my earlier book "Modern Data Engineering with Apache Spark" but we can talk Delta, Spark, or anything really or all things The Delta Lake Definitive Guide. #opensource #oss #deltalake #freebooks

86

6 Comments

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

AlloyDB provides 2x better price-performance than self-managed PostgreSQL. See how in this article from Sridhar Ranganathan and Shesh Ranganath.

14

Completed next project in my AI/ML course with U Texas Austin. Analyze the data and come up with a predictive model to determine if a customer will leave the credit card services or not and the reason behind it #HandsOnProject #EDA #RANDOMFOREST #Bagging #Boosting #SMOTE #CrossValidation #DataPreprocessing #HyperparameterTuning #MyGreatLearning

13

3 Comments

TGIF, folks! Fridays are hands down my favorite day—a perfect mix of weekend anticipation and a well-deserved breather from the weekly hustle (which I also love, haha). So here's the latest, I've finally caved and set up a Topmate profile! 😅 Why? Because so many of you folks have been reaching out for 1-1 help with: 1. Navigating the wild world of data science and analytics careers 2. Prepping for those nerve-wracking interviews I’ve decided to dedicate few hours of my weekend to offering personalized support for these requests. I've opened up a few 1-on-1 slots on my Topmate profile for those interested. If you're looking for guidance or interview prep, feel free to check it out and book a session! Looking forward to helping you on your data science journey. 😊 https://lnkd.in/gy-fMC6Q #CareerGuidance #DataScience #Analytics #InterviewPrep #TopmateProfile

38

1 Comment

Learnt a good deal with this short workshop. Been working with Snowflake for 2+ years since joining System1. This workshop reinforced my opinion that Snowflake is a well-thought data-warehouse product.

21

1 Comment

https://lnkd.in/gUpFEbby One of the limitations of current generation #LLM is it's inability to consume non linear data at scale. #SpreadSheetLLM Attempts to get rid of this limitation by focusing on most commonly available non linear data, that is tabular data.

6

1 Comment

I took the AWS AI practitioner beta exam last Sunday and for me, it was challenging for a foundational cert. I had to answer 85 multiple choice questions but probably 35 questions were not graded. Besides, It required skills in Machine Learning, AWS Sagemaker, Responsible AI principles, and an overall understanding of AWS bedrock, including basic knowledge of how LLM's and foundational models work. A special thanks to Arturo Deza for the AI and ML concepts review at his classes, Harry Guillermo for motivating me to get into the AI field with AWS and Stéphane Maarek for his incredible course!!! #AWS #AI #MachineLearning #Sagemaker #ResponsibleAI #AWSBedrock #LLM https://lnkd.in/eQQtYJgz

44

1 Comment

Until a year ago I had a bias, 'Kubernetes is only relevant for late-stage companies'. During our market research, we were surprised to see startups as small as 20 employees using Kubernetes from day 1. Vedant Vyawahare & I have seen multiple reasons why companies moved to Kubernetes: 1. Serving unpredictable markets like travel and e-commerce. 2. Optimizing resources and avoiding provisioning. 3. Need to leverage Cloud credits across AWS, GCP & Azure Here are a few reasons why Kubernetes is becoming a household name. If you are planning to evaluate Kubernetes, feel free to ping me or sign up for our waitlist. (Link in comments) #Kubernetes #devops #devtools

30

3 Comments

Are you using tools like PyTorch, TensorFlow, Spark, Splunk, Snowflake, Kafka? ... the list goes on 🫶 They all have S3 connectors. Since Cloudian natively speaks S3, we easily integrate with your S3 AI/ML tools to simplify your workflows! Bring the storage team, analytics team, AI team 🤖, research team 🔬, and more together for a centralized data workflow your whole company can agree on! We offer a 🆓 trial cloudian.com/free-trial #ObjectStorage #S3API #S3Storage #S3Data #OnPremObjectStorage #S3ObjectStorage #Kafka #TensorFlow #PyTorch

1

Our trajectory with Google Cloud epitomizes a transformative strategic pivot. What began as a challenging market positioning has evolved into a compelling narrative of operational resilience and technological leadership. Over seven pivotal years, we've fundamentally reimagined our enterprise value proposition—transitioning from a peripheral cloud solution to a mission-critical AI innovation partner. The financial metrics tell a powerful story: we've systematically converted recurring quarterly deficits into a robust framework for sustainable profitability. Buckle up the next 7 years are going to be epic @ Google Cloud! 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 #google #googlecloud #GCP #gcpcloud

28

1 Comment

Big news for the AI community! Databricks has developed DBRX, a new open Large Language Model that surpasses models like GPT-3.5 and competes with Gemini 1.0 Pro. DBRX uses a sophisticated mixture-of-experts architecture that boosts both inference and training speeds while being more efficient than traditional dense models. This model is already integrated into Databricks’ GenAI-powered products and is available for access on Hugging Face. Enterprises can now develop their own MoE foundation models using Databricks tools and NVIDIA hardware, showcasing the powerful capabilities available for AI development. Read more about DBRX and its impressive capabilities on Databricks' blog: #ai #LLM #databricks #dbrx #genai

4

4 Comments

In an ideal world, we should not have the concept of warehouses in Snowflake. I don’t think I have ever met a data engineer who wants to delve deep into warehouse internals when they are already juggling tasks from different business stakeholders. Ideally, warehouses in Snowflake or DBUs in Databricks are just units of compute. One should be able to define what needs to be done and the underlying data warehouse should be capable enough to figure out the size of machine and the parallelism it needs to complete my task within a particular timeframe and minimum cost. No one cares about the configuration of the warehouse if the task succeeds within the business SLA and it’s priced fairly. Though to be honest, accurately predicting the memory and runtime requirements of an SQL before actually executing is a very tough technical problem to solve. Snowflake has always been famous for ‘it just works’ and my gut says that it’ll be the first to solve this problem (if it isn't solved by academia), but for now, it remains unsolved and the users need to take care of this! 

49

13 Comments