Navigating Technical Due Diligence for AI Software Companies
At AKF Partners, when we perform technical due diligence on an AI software company, it's crucial for us to ask detailed questions about the target’s machine learning (ML) models, data practices, and deployment methodologies in addition to our typical diligence questions. This allows to gauge the novelty of the ML models as well as the engineering capabilities of the system. One of our past article’s discusses our approach to evaluating AI software product companies. Here’s a guide framed around a list of questions, inspired by the framework we've developed:
Machine Learning Models
1. What models do you have in your inventory, and what are their purposes?
2. How would you describe the structure of your models (Deep Neural Network, Convolution Neural Network, Recurrent Neural Network), including layers, nodes, and convolutions?
3. Which primary algorithms are employed in each of your models?
4. What are the precision and recall metrics for each model over the last 6-12 months?
5. How many users are receiving predictions from each model, and what percentage does this represent?
6. What tools are used for managing your AI models, and how were these tools selected?
7. What are the key features and predictions for each model?
Data Collection, Data Quality, and Preparation
1. Where does your data originate, and how is it labeled?
2. What is the accuracy of your data labeling process?
3. Can you describe your training dataset in terms of size, structure, and where it's stored?
4. What data storage technologies do you use (e.g., RDBMS, NoSQL, Cloud DBaaS), and why?
5. What data transport technologies are in use?
6. Which tools and cloud managed services are employed for data storage, management, and processing?
7. Do you have data governance documents? How do you classify data (e.g., PII, PCI)?
8. What are your data access policies?
AI/ML Processes
1. What is your process for training models, and how does it vary across different models?
2. Does the pipeline support near real time streaming? What are the use cases and what is the end-to-end cycle time from ingestion to presentation?
3. What performance metrics are you achieving with your models, and how do these compare to human benchmarks? (eg. AUC-ROC, Precision, Recall, F1)
4. How has model performance changed over time?
5. How often do you adjust, build, or test new models to adapt to changing data environments? How often are models retrained and what’s the process to do so?
6. How long does model retraining take?
7. What tools are used for AI model development and deployment, and how were these tools decided upon?
8. What tools are used to monitor models in production? What is monitored?
9. How is drift detected?
10. What is your AI and data science roadmap?
11. What guidelines do you follow for model development and evaluation?
12. Can you describe your deployment process for AI models?
13. What does your model release process look like, including validation, approval, and actual deployment?
14. How do you monitor model performance in production, and what's the frequency of degradation alerts?
15. What are your guidelines for AI production support?
Team Composition
1. What is the background of the Data Scientists and ML Engineers that are building software product?
2. How are they aligned and organized as a team?
Strategic Roadmap and Guidelines
1. What is your strategic roadmap for AI and data science?
2. How do you ensure your AI models align with future technological advancements?
3. What are your model development guidelines?
4. What, if anything, makes the models a long-term differentiator? How is the value measured?
At AKF Partners, conducting technical due diligence on an AI software company involves asking incisive questions about the target’s ML models, data strategies, and operational frameworks. This process helps us ensure that the technology not only functions well today but is also poised for future success. By exploring these questions, we gain insights into the novelty and depth of the AI software product. Drawing from our experience further enhances the due diligence process, ensuring it's both comprehensive and produces effective results for our clients. Contact Us for tech diligence help.
Consider attending our Free CTO Accelerator Webinars, designed to equip technology leaders with the tools they need to succeed.