Rajesh Kotian

Harmeet S. and Vishal Srivastava will be tackling the complexities of data catalogs at DataEngBytes Melbourne 2024. They’ll address key issues like the effectiveness of data catalogs, user adoption hurdles, and how to choose the right one for your needs. Expect a session filled with real-world examples, practical advice, and strategies to streamline your data catalog approach. If you’re navigating the data catalog landscape or looking for better solutions, this talk is for you. Join us to explore and discuss these critical topics! #deb2024

28

2 Comments

Traditional centralized data platform approaches are inadequate for handling the complexities of modern enterprises, leading to data quality problems and lack of clear data ownership. Read the blog by Chitra Subramanian which explores #DataMesh framework, #MicrosoftFabric and Infogain’s #iRAPID toolkit. https://lnkd.in/g_m3Wu4r #Infogain #EngineeringBusinessOutcomes #Data

79

1 Comment

#pandas on Snowflake lets you run your pandas code in a #distributed manner directly on your #data in Snowflake (Feature now GA in latest release) #analytics #datascience #textminning #snowflake ANZ Tech Talks Further details -

11

Unlock the power of data transformations in the modern data stack with Rishu Saxena and John (JC) Cosgrove. In this insightful session from Sydney DataEngBytes 2023, discover how Coalesce.io helps Snowflake customers implement data engineering and modelling best practices using a user-friendly GUI, flexible code, and efficient automation. Learn how to build robust data platforms for analytics, machine learning, and AI with ease. Watch the talk here: https://lnkd.in/gxxdKNnM #deb2023 #dataengineering #learning

29

 Harnessing the Power of Big Data with ABNTechCorp Big Data in 2024 will not be about volume—it’ll be about velocity and value. The ability to process massive amounts of data in real-time will become a critical competitive advantage. ABNTechCorp’s Big Data solutions help businesses make sense of the data deluge, turning raw information into actionable insights with predictive capabilities that keep you ahead of the market. Ready to make data your most powerful asset? At ABNTechCorp, we’re transforming Big Data from an overwhelming challenge into a strategic advantage that powers your business forward. #cybersecurity2024 #techstrategy #nextleveldefense #smartitsolutions #nextgeninfrastructure #futureproof #networksecurity #bigdatapower #smartautomation #advancedcybersecurity #5gfuture #predictiveanalytics #abntechcorp #digitaltransformation #itchallenges #zerotrust #cyberthreats #techrevamp #cyberdefense #digitalfuture #itinfrastructure #futurebusiness #trailblazingsecurity #bigdataanalytics #automationinnovation #cybersecuritysolutions #5gopportunities #empoweringdigital #itdominance #zerotrustsecurity

11

Recent events between UniSuper, a major pension fund, and Google Cloud highlight the importance of focusing on revenue-generating components of data strategy as well as revenue-preserving aspects like Disaster Recovery (DR). Read Clay Corcimiglia's article here for an in-depth exploration of how to embed resilience into your data strategy effectively: https://lnkd.in/e4HmuXBx #DataStrategy #UniSuper #GoogleCloud #DisasterRecovery

15

Mantel Group Achieves Elite Partnership Status with Databricks! We are thrilled to share our recently attained top level partnership status with Databricks, positioning us as one of only five Elite status consultancies in the Asia Pacific region. This Elite Consulting Partners Status is a well earned recognition of our deep expertise in the Databricks Data Intelligence Platform, along with wider data technologies to deliver comprehensive solutions and scalable outcomes. According to James Hayes, “Databricks' new program, designed to drive adoption of more features by clients within its data intelligence platform, has unlocked fresh incentives for organisations in ANZ, particularly those eyeing AI and ML use cases.” The track record of successful implementations, spanning data engineering to machine learning, demonstrates our commitment to helping clients succeed with Databricks. Together, we're shaping the future of data technology and driving meaningful change in the industry. Read about our Databricks Partnership, from Sasha Karen, at ARN - Channel News : https://lnkd.in/gE46Ne9q You can also read more about it from our website:  https://lnkd.in/g6zdt-Nv  #Databricks #ElitePartnership #DataInnovation #Lakehouse

128

1 Comment

Next Pathway successfully migrated a Fortune 500 financial services group from Netezza to Snowflake, showcasing our expertise in complex cloud transitions. This high-stakes project modernized the client's entire data ecosystem in just 6 months. Leveraging our AI-driven SHIFT Cloud, we successfully translated over 600,000 lines of Netezza SQL and Stored Procedure code to Snowflake. Additionally, we migrated over 5,800 DataStage and Informatica ETL pipelines to run efficiently in the Snowflake environment. Our comprehensive approach included: • 100% translation of Netezza SQL and Stored Procedure code • Complete migration of DataStage and Informatica ETL pipelines • Conversion of BI Report SQL Queries to work with Snowflake • Code optimization and QA support for a smooth parallel run and cut-over process This successful migration enabled the client to retire their legacy Netezza database, transition all ETL pipelines, and repoint existing BI Reports to Snowflake within the critical timeframe, avoiding extended licensing fees and ensuring business continuity. Explore the innovative solutions behind this large-scale transformation in our detailed case study: https://bit.ly/3IY52Hn #NextPathway #SHIFTCloud #CloudAutomation #CloudMigration #Snowflake #Onelake #DataFactory #AzureSynapes #Informatica #ETL #AI #Aiautomation #GenerativeAI

6

Aristotle Metadata is marking seven years of growth and innovation in 2024! Over the years, our team has worked tirelessly to simplify data governance and empower organisations worldwide to turn data into actionable insights. This milestone reflects a commitment to excellence, collaboration, and driving meaningful change in how we manage and use data. Read this IT Brief article on TechDay Australia to learn about our journey and vision for the future: https://lnkd.in/gVZDgkVs

16

Snowflake Kicks Off Data Cloud Summit 2024 with Major Moves in Open Source AI and Data Collaboration Snowflake made a series of bold moves toward open source, interoperability and customization in AI and data on Day 1 of its annual Summit. As CEO Sridhar Ramaswamy stated, Snowflake is positioning itself to make “enterprise AI easy, efficient and trusted” for customers across industries. The headliner was Snowflake’s expanded collaboration with NVIDIA to power customized, industry-specific AI applications. The companies are integrating NVIDIA’s AI software like NeMo Retriever into Snowflake’s Cortex AI platform. NVIDIA CEO Jensen Huang echoed the importance of this partnership for enterprise AI, stating “Data is the essential raw material of the AI industrial revolution. Together, NVIDIA and Snowflake will help enterprises refine their proprietary business data and transform it into valuable generative AI.” Another major reveal was Snowflake’s new Polaris Catalog, an open source catalog implementation for the Apache Iceberg data table format that is becoming the industry standard. “Organizations want open storage and interoperable query engines without lock-in,” said Christian Kleinerman, Snowflake EVP of Product. “With support from leaders like AWS, Microsoft, Google and Salesforce, we are simplifying how any organization can access data across systems with flexibility and control.” Read more - https://lnkd.in/gb4ZxBsj

5

Large Language Models (LLMs) on Structured Data Sets: A Synopsis of Current Challenges and Research Directions The integration of Large Language Models (LLMs) with structured data sets has become a focal point of discussion among senior technical leaders responsible for data architecture in various companies. This summary encapsulates the essence of recent conversations and research efforts aimed at understanding and optimizing the use of LLMs in the context of structured data. The primary application of LLMs in structured data environments is to interpret human queries and translate them into SQL or other query languages. This translation process is currently seen as inefficient, as it not only demands substantial computational resources but also does not fully leverage the capabilities of LLMs to address the underlying problem statements. The convolution arises from the mismatch between the unstructured nature of LLMs and the rigid, predefined schemas of structured data. Research in this domain is actively seeking to bridge the gap between the fluidity of LLMs and the rigidity of structured data. The goal is to enhance the user experience by creating more intuitive and efficient ways for LLMs to interact with and extract information from structured databases. This endeavor is not without its challenges, as the computational intensity of LLMs poses scalability issues. A striking comparison reveals that a typical LLM request consumes approximately 17 times more power than a standard Google search query. This disparity highlights the need for innovation in making LLMs more energy-efficient and cost-effective, especially when dealing with structured data. The sustainability and scalability of LLMs in data architecture are contingent upon advancements that can reconcile their high computational demands with the economic and environmental costs. One such research effort is the work by Wang et al. (2021), which proposes a novel framework for integrating LLMs with relational databases. Their approach involves a pre-processing step that transforms structured data into a format more suitable for LLMs, thereby reducing the computational overhead. Another significant contribution is the research by Zhang and Choi (2020), which focuses on optimizing the interaction between LLMs and structured data by introducing an intermediary layer that can effectively translate natural language queries into database operations. The ongoing research in this field is a testament to the potential of LLMs to revolutionize data architecture. The quest for a more harmonious integration of LLMs with structured data sets is not merely a convolution but a glimpse into the future of data management. The innovation that successfully mitigates the current limitations will undoubtedly pave the way for a new multi-billion dollar market, offering unprecedented opportunities for businesses and researchers alike. #LLMs #DataArchitecture #StructuredData #Innovation #Technology #Research

9

At the “Data Modernisation: Transforming into an AI-Powered Organisation” panel at Databricks' Data + AI World Tour Singapore event, experts discussed the challenges and strategies for AI-driven data modernisation, focusing on improving processes while maintaining data security. Andy Ta, Director of Data Analytics & AI and Chief Data Officer at Synapxe, shared, “For healthcare, it took us many years to secure the data, but we also need to tap into something that allows for experimentation. Additionally, we should be able to share the data with collaborators and partners so that it can be used at a larger scale to create more value.” Ritchie Ng, Research Scholar at NUS and Head of Data & AI at Eastspring Investments, explained, “First, we optimised all our processes, and then we standardised them. After those, we applied a unified data and AI platform. What happened was that we had a platform where you deployed any algorithm in a single use case, and you could expand it horizontally and vertically.” Vance Ng, Director of Data Science Accelerator at the CPF Board, said, “You may have the perfect system, but it tends to break the people. For us, what we look at is educating our end users in terms of upskilling them, and Databricks have done a great job in terms of lowering the boundaries and the barriers.” Bharathi V., Chief Digital & Information Officer at Suntory, said, “You have to look at your employee in the context of everything they do, and you have to really hone in on exactly what superpower are you uncovering for them?” #artificialintelligence #digitaltransformation

3

Techment’s Data Transformation in ADF Reduces Data Retrieval Times by 40%! . Techment’s Data Transformation in Azure Data Factory (ADF) has reduced data retrieval times by a remarkable 40%! By building a scalable, code-free ETL pipeline, we streamlined data integration from diverse sources, enhancing data accuracy and accessibility. Our client now enjoys comprehensive business insights with optimized Power BI dashboards. Curious to know how we did it? Dive into our latest blog and discover the magic of ADF transformation! https://lnkd.in/d-FZDkwm #DataTransformation #AzureDataFactory #ETL #PowerBI #DataAnalytics #Techment #BusinessInsights

3

At the AWS summit some of our team had the opportunity to meet and talk with the Snowflake team. We have looked forward to partnering with Snowflake to bring scalable, efficient, and secure data cloud services to the APAC region. Why Snowflake? It's simple. We wanted a cloud data solution that not only scales effortlessly with your growing data needs but also maintains peak performance without the operational overhead. Snowflake's architecture allows us to scale computing and storage independently, ensuring optimal performance and cost-efficiency. Their commitment to security and compliance aligns with our standards, providing peace of mind with every data interaction. Our team looks forward to bringing the best of Snowflake to drive innovation, improve data accessibility, and empower your team with real-time insights. Here's to making data-driven decisions better and faster! #DataCloud #Partnership #Innovation #Snowflake #DataDriven

8

What happens when different teams within an enterprise define and organize the same data differently? – Chaos sets in, data becomes messy, and data models become untenable! In his latest talk at IN/Clojure 2024, Anuj Kumar (FORMCEPT’s Co-founder and Chief Architect), revealed this to be the real reason why data teams keep reinventing the wheel again and again while building data pipelines. Take a sneak peek at what Anuj has to say about this in the short video below: https://lnkd.in/g33JmcKj Access the full video of Anuj’s talk on “Navigating #DataModels: A Journey into Unified, Scalable, Composable Data Architecture” here: https://lnkd.in/gyFm964F You can also send a request to download Anuj’s presentation here: https://lnkd.in/g4-upd_7

5

Oracle launches HeatWave #GenAI, featuring in-database #LLMs and a vector store. This tech lets developers build AI applications without data migration or extra costs. As per Oracle, it outperforms competitors like #Snowflake, Google #BigQuery, and #Databricks. Read details at https://lnkd.in/gqrcxGwH #Oracle #AI #GenAI #EnterpriseTech #HeatWaveGenAI

9

Ever feel like your data pipeline is more a traffic jam than a freeway? Say hello to Azure Data Factory—your ticket to smooth, scalable, and secure data integration. Whether you're juggling 90+ sources, automating complex workflows, or scaling for enterprise growth, ADF makes it seamless. From ETL to ELT, it’s not just about moving data; it’s about moving it with purpose. Pair it with Azure Synapse, and you’ve got a data dream team. #DataIntegration #AzureDataFactory #CloudInnovation

20