Managing your AI-ML Projects the traditional way? You are doomed for failure
The usual pillars of Agile don’t align too well with the structure of AI/ML projects
Applying traditional Agile practices directly to AI/ML projects can result in significant challenges. This article examines why a one-size-fits-all approach is problematic. It identifies which aspects of AI/ML align with Agile principles and suggests alternative project management approaches better suited for AI/ML.
As a seasoned Agile practitioner with extensive experience in traditional software development, I’ve seen firsthand the transformative power of Agile. But having recently managed couple of AI/ML projects, including Generative AI initiatives, I couldn’t avoid noticing the challenges of imposing traditional Agile practices on these types of projects.
First, the planning process fell apart when certain tasks, such as Exploratory Data Analysis (EDA) and model tuning, took far more time than anticipated. Additionally, while some activities (such as a Chatbot GUI Development) could have benefited from a traditional Agile approach, others would have been better suited to a straight-forward watefall process. The usual pillars of Agile don’t align too well with the structure of AI/ML projects. Based on my own experiences and further readings, I would definitely approach these projects differently now.
Agile practitioners are already well aware of the various advantages that Agile methodologies offer, such as, increased flexibility for teams to adapt quickly & iteratively based on feedback, a faster time to market owing to short sprints of functional increments, providing competitive advantage, enhanced collaboration between cross-functional teams, ensuring alignment and swift problem-solving and improved quality owing to Continuous testing and integration.
However, AI/ML projects diverge a lot from traditional software development (Figure 1). The data-centric and experimental nature of AI/ML requires Agile adaptations to manage uncertainty effectively:
Data-Centric Development: AI/ML revolves around data rather than just code. Model development relies on data exploration, preprocessing, and experimentation, which can require flexible timelines.
Experimental Model Building: Building effective AI/ML models involves hyperparameter & prompt tuning, testing, and re-evaluating models. The iterative process doesn’t fit neatly into fixed-length sprints.
Continuous Model Improvement: AI models need ongoing refinement as more data becomes available. Traditional Agile’s focus on fixed-scope deliverables can hinder continuous improvements.
Pitfalls of Misapplying Agile in AI/ML Projects
Force-fitting traditional Agile practices into AI/ML projects often leads to the following problems:
Inaccurate Estimations: The unpredictable nature of AI/ML makes accurate estimation difficult, often resulting in rushed work and compromised quality.
Limited Experimentation: Strict Sprint boundaries can stifle necessary experimentation and data exploration, crucial for optimal model development.
Inadequate Focus on Data: Agile often prioritizes code delivery over data-related tasks, which may lead to insufficient attention on data quality and feature engineering.
Suitable and Unsuitable Aspects of AI/ML for Traditional Agile
Some aspects of AI/ML projects align well with Agile principles, while others require deviation from traditional approaches.
Suitable Aspects
Feature Engineering: Agile's incremental approach works well with the iterative process of creating and testing new features.
Model Architecture Development: Experimenting with various architectures can be managed effectively within Agile sprints, allowing rapid prototyping.
User Interface Design: Traditional Agile methodologies work well for developing user interfaces in AI/ML applications.
Unsuitable Aspects
Core Questions and Evaluation Metrics: Defining key questions and metrics should be done upfront, and frequent changes can result in significant rework.
Data Pipeline Development: The complexity of setting up data infrastructure often doesn’t fit within short sprint cycles.
Model Training and Evaluation: Training AI models can be time-consuming, making it difficult to predict completion within fixed sprints.
Model Deployment and Monitoring: These require ongoing maintenance, which exceeds typical Sprint boundaries.
Alternative Project Management Approaches for AI/ML
Recognizing the challenges of traditional Agile in AI/ML, several alternative methodologies have emerged, tailored to meet the unique needs of data science projects.
1. Data-Driven Scrum (DDS)
DDS is an Agile framework adapted for AI/ML projects. It uses iterations based on the completion of logical work chunks, allowing the flexibility necessary for data exploration and model experimentation. Results are collborayvely analyzed by the whole team, enabling data-driven decision-making. DDS accommodates overlapping iterations, allowing teams to start new work while other processes, like data collection, continue.
Managing DDS however requires a skilled Process Expert to guide the team effectively. Also, the flexibility can lead to scope creep if not carefully managed.
2. Team Data Science Process (TDSP)
TDSP is an agile, iterative framework from Microsoft for managing data science and AI/ML projects. It includes stages such as business understanding, data acquisition, modeling, deployment, and customer acceptance, integrating well with tools like Azure Machine Learning. TDSP standardizes workflows, supports collaboration, and enhances model management through MLflow.
While TDSP is a well-structured and scalable framwork promoting team collaboration and supporting MLOps. It requires deep integration with Azure tools and additional expertise in responsible AI practices.
3. Hypothesis-Driven Development (HDD)
HDD frames development as a series of experiments, treating assumptions as hypotheses to be validated. It focuses on validating assumptions through experimentation, aligning with AI/ML’s exploratory nature. HDD encourages continuous learning and improvement based on experimental outcomes.
However, HDD may struggle to manage larger, complex AI/ML systems as it demands careful experiment design to ensure valid conclusions.
4. Cognitive Project Management for AI (CPMAI)
CPMAI methodology is an iterative, data-centric framework for managing AI and machine learning projects. It addresses the unique challenges of AI, such as data preparation, model development, and operationalization. CPMAI focuses on delivering incremental model improvements, ensuring alignment with business goals while allowing for continuous iteration based on feedback.
The CPMAI methodology consists of 7 phases, namely business understanding, data understanding, data preparation, data modeling, model evaluation, and model operationalization. Its adoption may require extensive training and certification before the team is ready to apply.
Key factors for successly delivering AI/ML projects
Embrace Ambiguity: Accept that solutions emerge through data exploration, and allow for upfront data analysis, cleaning, and prototyping.
Iterative Model Versions: Plan for frequent model releases at milestones for stakeholder feedback, aligning with Agile's iterative focus.
Automate Workflows: Use MLOps (machine learning operations) to automate data pipelines, model training, and testing, increasing project velocity.
Monitor and Improve Continuously: AI/ML models require ongoing adjustments. Set up robust processes to monitor, retrain, and fine-tune models in production.
Foster Cross-Functional Collaboration: AI/ML projects need a diverse team with data engineers, scientists, developers, and domain experts collaborating closely.
Integrate AI Ethics and Security: Build AI governance, fairness, and privacy safeguards into the development process from the start.
Traditional Agile methodologies provide valuable principles for managing projects, but applying them rigidly to AI/ML projects can be counterproductive. AI/ML’s data-driven, experimental nature and the need for continuous model improvement require a more flexible, tailored approach.
By adopting methodologies such as DDS, HDD, TDSP, etc, and integrating Agile principles while accommodating AI/ML’s unique demands, teams can leverage the best of both worlds.