You're managing engineering data. How do you know which products are the best?

1. Always start with why: Why is this use case important? Is it tied to your top corporate priorities or part of your top organizational goals? Cost Benefits Analysis: Is this use case going to generate new revenue (directly or indirectly) or going to help with cost reduction/optimization and what is the cost of building this product? 2. Define and understand the audience/Users: Is this use case going to serve a specific user base or multiple teams/Line of businesses? The more you understand your audience and their needs, the better the product will be.

We need to define the KPI metrics, which can be categorised into: 1. Engineering related 2. Business metrics 3. Execution metrics For the above categories, identify top 3 critical stats which is makes sense for the product/business. Based on these metrics/KPIs you can do camparison between the products and draw actionable items from that.

Evaluating and determining the best products in the context of managing engineering data involves a multi-faceted approach. Here are key steps and considerations to help you identify the best products: Define Criteria for 'Best': Establish clear criteria for what makes a product the best for your needs. This could include factors like functionality, compatibility with existing systems, ease of use, scalability, security features, customer support, and cost-effectiveness.

There is an initial steps that is always a key to resolution is to understand the specific requirement and also the indicators. If you are able to crack than a solid and meaningful solution will be clearly evolve

Needs Assessment: Understand your specific requirements. Analyze the nature of the engineering data you are managing, the size of the datasets, the complexity of the processes, and the specific tasks the product needs to perform. Research and Market Analysis: Conduct thorough research to identify available products. Look into industry reports, product reviews, and case studies. Pay attention to products that are widely recognized and have a good reputation in your industry.

When you are able to understand the current requirement than another window open what to utilize to get optimum solution. At this point you have to choose wisely and also using your previous experience and also what are the requirements as an output.

Below are some of the best practices I have been following: 1. Develop repeatable ETL/ELT patterns for data acquisition. 2. Once you have procured data from different data sources and often in different formats, standardize data format to parquet/delta/iceberg (in near future) for big datasets or json for easy data consumption. 3.Apply data quality checks as a part of proactive data reliability/observability. 4. Data governance is an essential component for making data accessible, available and usable. 5. Make sure to apply right data security policies, follow enterprise privacy guidelines. 6. Collect user and platform telemetry to understand usage patterns. 7. Minimize waste by understanding the use case and collect regular user feedback.

User Feedback and Reviews: Gather feedback from users who have experience with the products. User reviews can provide insights into the real-world performance, reliability, and usability of the products. Trial and Testing: If possible, trial or conduct a pilot test with the products. Hands-on experience can be invaluable in assessing whether a product meets your needs and integrates well with your existing systems.

Scalability and Future-Proofing: Consider the scalability of the product and how well it can adapt to future changes in technology and your business needs. The best product should be able to grow and evolve with your requirements. Technical Support and Service: Evaluate the quality of customer support and technical service offered by the vendor. Good support can be crucial in resolving issues quickly and efficiently.

For maximizing value from your data: Make data easily accessible by providing right tools based on your user personas. For e.g. power users should have direct access to data, end users and executives should have data presented in visual reports/charts and data engineers should also have understanding of data. Data catalogs are key to provide context of data to user community. On going demos of new products and new features. Conduct regular feedback loops with user sessions and also by capturing telemetry.

Compliance and Security: Ensure that the products comply with relevant industry standards and regulations, especially regarding data security and privacy. Cost-Benefit Analysis: Conduct a cost-benefit analysis. The best product should offer a good return on investment, considering both direct costs and indirect benefits such as time savings, improved efficiency, or better data insights.

A big area that promotes quality issues is separation between manufacturing engineering and purchasing materials departments. Engineering needs to maintain responsibility for cost reduction efforts.

Integration with Existing Systems: Check how easily the product can be integrated with your existing systems and workflows. Seamless integration can significantly reduce implementation time and costs. Community and Ecosystem: Look into the community and ecosystem surrounding the product. A strong, active community can be a valuable resource for problem-solving and sharing best practices. By carefully considering these aspects, you can make an informed decision about which products are the best fit for managing your engineering data. The key is to align the product features and capabilities with your specific needs and long-term objectives.