Nouran Younis

🚀 Automate Your CI/CD Pipeline with Snowflake and GitHub Actions 🚀 Our Senior Data Engineer, Jaime Salas, shares an insightful guide on integrating GitHub Actions with Snowflake to streamline your CI/CD pipeline. Learn how to: 🔹 Set up key pair authentication 🔹 Create and manage Snowflake users and schemas 🔹 Automate tests and merges using GitHub Actions Enhance your development workflow with these powerful automation techniques! https://hubs.ly/Q02BWXC10 👉 Read the full article #DataEngineering #Snowflake #GitHubActions #CICD #Automation

15

If you are ingesting data from Kafka/Kinesis into Snowflake you should watch this short video by Charles Tan and Shawn Gordon on how you can use DeltaStream to shift left your workloads to reduce cost and latency! #Flink #Kafka #Snowflake #ShiftLeft https://lnkd.in/gnZ45MTa

13

Handling missing data points is not as easy as it sounds for #streamprocessing "Fancy" mechanisms like "watermark" won't show you new result if the count is 0 for a tumble/hop window. Check the nice video from Confluent. Dan mentioned a solution to add special events in the data stream(say Kafka), for 0USD transactions. Well, it works but will impact your COUNT/SUM/AVG, unless you add many filtering. I would suggest use DataGen or RandomStream in Timeplus as a heartbeat stream, UNION 2 streams together, run TUMBLE window with GROUP BY window and type, then you will get count=0 and fire alerts. SQL like this: WITH cte AS (SELECT 'real' as type FROM stream1 UNION SELECT 'datagen' as type FROM datagen) SELECT window_start, type, count(1) as cnt FROM tumble(cte,1m) GROUP BY window_start, type HAVING cnt=0 Leave comments below if you have any thoughts, or join timeplus.com/slack for more discussions.

8

How did Dremio take on self-serve analytics with a "lakehouse" 🤔 Hear from Dremio Founder Tomer Shiran on how they made data more accessible through an open source data lakehouse approach using Apache Arrow and Apache Iceberg 💪 And built a $2B company while doing it 😲 Full episode 👉 https://lnkd.in/geQrmn2R

6

Dremio founder, Tomer Shiran, sat down with the Open Source Startup Podcast to talk about how we're making data more accessible through an open source data lakehouse approach using Apache Iceberg. Listen to it here!

7

How can you handle long pauses of your data in #ApacheFlink? Since your stream is stopped, setting an idle timeout won’t work. In the latest video in his Apache Flink in Action series, Dan Weston explores one possible solution. Give it a watch and then share your solution in the comments! Watch it here ➡️ https://lnkd.in/gjmAv46B

50

1 Comment

Come meet us at the Snowflake summit in SF next week and learn how we're applying specialized models (large, small, expert, etc.) to enterprise data giving your teams super data powers. #SnowflakeSummit #DataCloud #SanFrancisco #Networking #Data #BI #AI #GenAI #LLMs

28

2 Comments

We'd like to show you what Veza can do for critical enterprise systems, please reply in comments for systems you'd like to see and we'll start sharing some! Starting with ... "who can take what action on what data in Snowflake" Permissions unlocked on Snowflake! https://lnkd.in/gSxhH43f #cybersecurity #identitysecurity #IAM #IGA #PAM #VEZA

59

3 Comments

Developers can now store, index, and query vectors in PostgreSQL using pgvector, eliminating the need for a dedicated vector database. This blog from Yugabyte expert Brett Hoyer deep dives into pgvector, and explains how to store text embeddings as vectors, execute similarity search, choose between indexes, and scale with distributed SQL.💡 Find out more!⬇️ https://hubs.la/Q02JyYrM0 #YugabyteDB #pgvector #vector #AI #Postgresql #cloudnative #distributedsql

5

Big announcement today- we’re unlocking Snowflake data for everyone!  With Snowflake Native Amplitude, you get all the benefits of Amplitude without your data ever leaving Snowflake. That means one single source of truth for all your enterprise, financial, and behavioral data to inform better product decisions. Learn more:

357

35 Comments

I'm personally really excited about the newest Public Preview from Databricks and can't wait to test it out. With Databricks Apps, you can deploy custom Python applications directly within the Databricks environment on top of your Lakehouse data. No need for a separate web server to manage and host your application. In my previous consulting roles I spent a lot of time and effort making the results of ML models, custom analytics, and data engineering pipelines accessible to non-technical users and this feature would have been a true game-changer. Some sample applications that I'm excited to use Databricks Apps for: * Publish a custom, branded chatbot trained on your enterprise's data * Make results of ML models accessible to business users without the need to log into Databricks * Create interactive visualizations on top of your Lakehouse * Abstract complex query language with an easy to use interface

2

How does an organization benefit from Strategic Data Modeling? 💥 Consensus and visibility of the broader business model 💥More productive conversations with business teams 💥Better quality of requirements 💥Higher signal, lower noise in technical conversations 💥Cross-platform, cross-domain, and widely understood conventions 💥Big-picture visual overview of the business and its database footprint 💥Preliminary designs become implementation blueprints 💥Accelerating onboarding of new team members 💥Making data more accessible and unlocking self-service within organizations 💥Keeping the database landscape manageable at scale 💥Getting a handle on complex data pipelines Check out Serge Gershkovich's book, 𝐷𝑎𝑡𝑎 𝑀𝑜𝑑𝑒𝑙𝑖𝑛𝑔 𝑤𝑖𝑡ℎ 𝑆𝑛𝑜𝑤𝑓𝑙𝑎𝑘𝑒, to learn more! #DataModeling #SqlDBM #Cloud #Snowflake https://lnkd.in/gsQsJ_yb

14

1 Comment

🚀 Calling all data engineers! If you’re looking to get the most out of your Databricks Jobs clusters, check out these data-driven tips for optimizing compute clusters at scale. 💡Understand the tradeoffs of Photon and autoscaling 💸See how serverless compute compares to classic clusters 💪Learn how to save countless hours optimizing clusters More in this post: https://hubs.ly/Q02KQ1t50

8

Now on Databricks Runtime 14.3: the State Reader API, a new capability that allows users to access and analyze Structured Streaming's internal state data. Discover how this new capability is providing much-needed transparency, accessibility, and ease of use: https://dbricks.co/3UeoF6l

213

We showcased how easy it is to build and deploy GPU-powered ML workflows with any OSS package at #SnowflakeBUILD. Learn more from Dash DesAI as he demos how to speed up training for an XGBoost model with GPUs using Snowflake Notebooks on Container Runtime. Try it for yourself now: https://okt.to/sN4YAG

62

Real-Time #Analytics News from the Past Week: 1. Starburst announced a range of new capabilities for their Trino-based open hybrid lakehouse platform, Galaxy, including: -The general availability of fully managed streaming ingestion from #ApacheKafka to Apache Iceberg tables. -The public preview of fully managed ingestion from files landing in Amazon Web Services (AWS) S3 to Iceberg tables. -Multiple enhancements to performance and price-performance of their #lakehouse platform. 2. Cohesity introduced a patent-pending visual data exploration capability to Cohesity Gaia, its #AI-powered search assistant launched earlier this year. By providing customers with a visual categorization of the themes across documents and files within a #data set, the visual data explorer brings new context to the data and suggests queries that help users gain insights faster. 3. IBM announced the release of its most advanced family of AI models to date, Granite 3.0. IBM’s third-generation Granite flagship language models can outperform or match similarly sized models from leading model providers on many academic and industry benchmarks. 4. Confluent announced the launch of the Confluent for Startups AI Accelerator Program. The 10-week virtual program seeks to collaborate with 10 to 15 early-stage AI startups that are building real-time applications utilizing the power of #DataStreaming. 5. Semantic Web Company (SWC) and Ontotext announced that the two companies have merged to become the Graph AI provider, Graphwise. Read the rest on RTInsights ➡️ https://lnkd.in/eQKsrPCZ

12

Our last poll was about data and transparency. Today, Databricks has created The State Reader API , a new capability to help users have access and analyze internal state data for full transparency and ease of use __ Will this new capability make a real difference from what we are used to use so far? #Databricks #Newcapability #datatransparency #accessibility #easeofuse #nuboTed

2

2 Comments

#Airflow 2.10 is here! 🎉 Julian LaNeve, CTO at Astronomer, spoke with VentureBeat's Sean M. Kerner about what's new in #Airflow210, our latest Astro Platform release that adds #dbtsupport, and an early look into priorities for #Airflow3 in the age of #GenAI 💫 Check out the full article to learn about how #ApacheAirflow 2.10 will improve #dataorchestration with hybrid execution and why #datalineage in data orchestration matters for #AI 🤖

16

Starting our Savoir Labs series of Apache release updates, demos, and lab experiments. This episode we talk about the latest Apache CXF update releases. Savoir Labs: https://lnkd.in/e46amGvW #ApacheCXF #JAXWS #JAXRS #Savoirtech

4