Santanu Bandyopadhyay

I had great time meeting with leaders and my team members at Experion offices in Kochi & Trivandrum last week. Exciting partner visits, customer discussions and townhall meetings were some of the highlights of the week and I looking forward to continuing the same positive momentum and vibes for the rest of FY25!! #WeAreExperion #experiontechnologies #DataAI #Data #AI #GenAI #machinelearning #DataEngineering #DataManagement

114

24 Comments

Here is a high level overview of snowflake's new features (DocAI, Universal Search & Copilot). #ATCI-DAITeam #ExpertsSpeak #accenturetechnology #Ruchi Soni

8

Looking forward to our next webinar with DataCamp next week! Tune in to listen to Piyush Mehta and I chat about how AI is effecting the world of data governance. Qualytics

21

Exploration of Databricks' Generative AI capabilities has been an insightful experience - summarized the key takeaways based on my learning/exploration of the Lakehouse AI platform along with Databricks IQ, MosaicAI, and more. https://lnkd.in/gGb3FAcC #databricks #genai #generativeai

17

1 Comment

I had the pleasure of interviewing Siddharth Rajagopal, Chief Architect at Informatica, on The Ravit Show during the Data Innovation Summit. Here's what we discussed -- -- Siddharth shared his excitement about participating in the summit, noting it as a hub for innovators looking to explore the frontiers of data and AI. We delved into several trending topics in the EMEA region, particularly the impact of the EU AI Act. -- A significant part of our conversation focused on the emergence of Data Products. Siddharth highlighted the key aspects that customers consider essential for scaling Data Products, such as data quality, governance, and integration capabilities. These elements are crucial for ensuring that Data Products deliver consistent value across various business functions. -- Siddharth also discussed how Informatica and its Intelligent Data Management Cloud (IDMC) platform are aligning with these trends. He shared how Informatica supports customers in adhering to the EU AI Act and in efficiently managing and scaling their Data Products. This support is critical for enterprises aiming to leverage data as a strategic asset while maintaining regulatory compliance. -- Our conversation wrapped up with Siddharth expressing optimism about the future of AI and data management. He emphasized the role of platforms like Informatica's IDMC in empowering organizations to harness the full potential of their data, paving the way for innovative and compliant data-driven solutions. Siddharth’s insights were not only informative but also indicative of the changes occurring within the data space, reflecting how businesses can adapt to and thrive in this evolving landscape. #data #ai #datainnovationsummit2024 #informatica #theravitshow

102

9 Comments

✨ Excited to share my latest achievement! ✨ I recently completed the Snowflake Education Services Hands-On Essentials: Data Engineering Workshop. This workshop provided in-depth insights into modern data engineering practices using Snowflake's powerful cloud data platform. With hands-on experience, I gained practical knowledge in areas like data ingestion, transformation, and optimization, which will greatly enhance my ability to work with data at scale. Looking forward to applying these skills to drive impactful data solutions! 🚀💡 #DataEngineering #Snowflake #CloudData #DataTransformation #ContinuousLearning #ProfessionalGrowth

28

7 Comments

SPARK code to find out the product, which generates the maximum revenue for a commercial store. #SCALA #SPARK #BIGDATA Karthik K. INPUT: PRODUCT_ID,QUANTITY_SOLD,PRICE 101,10,150 101,15,150 102,5,50 102,10,50 103,3,100 103,10,100 QUERY: def main(args: Array[String]):Unit = { val ss = SparkSession.builder.appName("two_dfS").master("local[*]").getOrCreate() val df_1 = ss.read.format("csv").option("header",true). option("path","C:/Users/hp/PRODUCT_ID.txt"). schema("PRODUCT_ID Int, QUANTITY_SOLD Int, PRICE Int").load() val df_2 = df_1.withColumn("REVENUE",col("QUANTITY_SOLD")*col("PRICE")) val df_3 =df_2.groupBy(col("PRODUCT_ID")). agg(sum(col("QUANTITY_SOLD")).as("TOTAL_QUANTITY_SOLD"), sum(col("REVENUE")).as("TOTAL_REVENUE")) df_3.show() val wind = Window.orderBy(col("TOTAL_REVENUE").desc) val df_4 = df_3.select(col("PRODUCT_ID"),col("TOTAL_REVENUE"),row_number().over(wind).as("ROW_NUMBER")) df_4.filter(col("ROW_NUMBER")<=1).show() OUTPUT:

2

Primary Key on Databricks. One of our customers insisted to add primary key in Databricks Table as part of migration process like traditional RDBMS System. Primary key added the primary key feature in DBR 11.3. The catch is that it is added as informational purpose only. Delta Lake itself will not enforce their uniqueness or non-nullability. You can add primary key, but it will work as expected.  Alternative approach implemented as workaround. Before Insertions/Updates: You can write custom logic to check for duplicate values in the primary key columns before inserting or updating records. This can be done using DataFrame operations in PySpark. Merge can be used in the update operation which can help manage data integrity by updating existing records or inserting new ones based on the primary key. Alternative Suggestions Data Validation and Cleaning-->Implement data validation and cleaning steps in your ETL (Extract, Transform, Load) Auditing and Monitoring -->Set up auditing and monitoring processes to regularly check for violations of primary key constraints. This can include periodic scans of your tables to identify and sends email notification of duplicate primary key values. As per Databricks. You can use primary key and foreign key relationships on fields in Unity Catalog tables. Primary and foreign keys are informational only and are not enforced. Foreign keys must reference a primary key in another table. You can declare primary keys and foreign keys as part of the table specification clause during table creation. This clause is not allowed during CTAS statements. You can also add constraints to existing tables.

11

In the rapidly evolving landscape of database management systems (DBMS), one constant has emerged: despite the introduction of numerous new technologies and data models, relational database management systems (RDBMS) have maintained their dominance. From the early hierarchical and network models to the rise of NoSQL and Columnar databases, each innovation has not replaced but rather contributed to the evolution of RDBMS. This article explores the progression of various database technologies and examines why RDBMS continues to be the backbone of data management. EQengineered Author: Ranjan Bhattacharya, Chief Data Officer. #databasemanagement #dataengineering #CIOs #CTOs #CDOs #CDAOs #CMOs #CFOs #CEOs #techinsights #EQengineered

16

3 Comments

The need for Application defines the standard of Learning!

21

I am thrilled to share our latest whitepaper on Master Data Management (MDM). This whitepaper explores strategies for streamlining data management, improving quality, and reducing IT overhead costs. Take a look to discover how MDM can revolutionize your data processes: #MasterYourData #MDM #DataManagement

70

2 Comments

A great summary of my talk at #DES2024

23

#GenAI models necessitates training on extensive datasets, which in turn requires robust #cloud storage solutions. As Gen AI continues to advance, demand for cloud services would surge correspondingly, creating a symbiotic growth relationship b/w the two. https://t.ly/9Rg-C

9

Good Overview!!! I highly recommend Vidhi Chugh's datacamp course—it's definitely worth checking out!

12

Learnt a good deal with this short workshop. Been working with Snowflake for 2+ years since joining System1. This workshop reinforced my opinion that Snowflake is a well-thought data-warehouse product.

22

1 Comment

Good takeaways from your post Shanthi Srinivasan! Thanks for sharing. AI Adoption Trends: AI adoption is rapidly increasing, with 34% of organizations already using AI and 22% planning to adopt it this year. Generative AI is also gaining traction. Industry-Specific Use Cases: AI is revolutionizing sectors like fintech, healthcare, and retail with applications such as interactive chatbots, drug discovery automation, and hyper-personalized product descriptions. Cross-Industry Applications: Conversational AI, company-specific large language models, and digital asset management are some of the cross-industry AI applications gaining momentum. Future Predictions: The future of AI includes agentic AI, affordable development of large language models, and a strong emphasis on ethical and responsible AI.

11

Bias in Machine Learning with EQengineered's Chief Data Officer, Ranjan Bhattacharya #datagovernance #ML #machinelearning #AI #artificialintelligence #ethicsinAI #dataengineering #digitaltransformation #EQengineered

22

7 Comments

⏰ Sharing this post with the interview by Logan Bartlett with Databricks CEO. It is very insightful. ⏩ Get your data right first. Don't skip this fundamental step to be successful in your AI journey. ⏩ Be extremely focused and critical of your AI use cases. #ai #genai #data #redpoint

2

1 Comment

🚀 Had an enriching experience at the Data Intelligence Day event by Databricks! Here's a deeper dive into what I learned: Insights from Databricks Leaders: Anil Bhasin, Ed Lenta, and Saroj V. provided invaluable insights into the Databricks platform. They highlighted its capabilities in data governance, warehousing, and GenAI. The platform's seamless integration across clouds and platforms stood out, offering organizations a unified solution for their data and AI needs. DBRX LLM: The discussion on DBRX, an open general-purpose LLM, was eye-opening. It was fascinating to learn about its potential to other established models like GPT, LLaMA, and Mistral. This signifies a significant leap forward in democratizing AI and making advanced AI capabilities more accessible to a wider audience. GenAI Impact in Banking and Multilingualism: - Rana Sinha Ray from HDFC Bank shared compelling use cases of GenAI in the banking sector. The potential for AI to revolutionize processes and customer experiences was evident. - Gautam Bhargava's presentation on Krutrim's multilingual GenAI model showcased the power of AI in addressing linguistic diversity. This innovation has immense implications, especially in a diverse country like India. Panel Discussion on Industry Trends: The panel discussion, moderated by Shweta Kamath, brought together industry leaders like Kamal K., Atul Sawant, Chintan Shah and Sudip Mazumder. Their insights on current trends and challenges in data intelligence and AI were enlightening. The discussion underscored the importance of continuous innovation and collaboration in driving industry growth. Key takeaways for me: - Krutrim's launch of a multi-lingual model is revolutionary and has the potential to make a significant impact in a diverse country like India. - Databricks's Unity Catalog emerged as a powerful tool for streamlining data governance and accelerating AI initiatives. - The increasing integration of GenAI into everyday life highlights the growing significance of data intelligence and AI technologies. Lastly a big thank you to Sathwik HV and Alankrit Pareek for inviting Fynd (Shopsense Retail Technologies Ltd.) team and managing registration process seamlessly. Your efforts ensured a smooth and enjoyable experience for all attendees! #DataIntelligenceDay #Databricks #AI #Innovation 📈✨

40