AICoP: Understanding Agentic AI

Columbia University’s AI Community of Practice Explores Agentic AI and Practical Applications

The AI Community of Practice at Columbia University, organized by the Emerging Technologies team, held its April 2025 session focused on agentic AI and how this evolving framework is shaping the next phase of artificial intelligence.

The featured presenter, Marc, a Machine Learning Engineer with the Emerging Technologies team, delivered a practical and technical look at how combining large language models (LLMs) with deterministic logic can create more controlled, action-oriented AI systems. Moving beyond simple chatbot interactions, the session explored how agentic AI introduces decision-making, tool use, and real-world task execution into AI workflows.

Demystifying Agentic AI

Marc opened the session by explaining the concept of agency in AI systems. Unlike traditional LLMs that simply predict the next token or word, agentic AI combines:

• Non-deterministic outputs (from LLMs)
• Deterministic rule-based logic (traditional code like if-then statements)

This hybrid approach allows AI systems to not only generate text but also decide what actions to take based on context, constraints, and goals.

Real-world examples, such as customer service chatbots, autonomous vehicles, and AI coding assistants like Cursor, illustrated how agentic systems are already at work today—balancing creativity with control to deliver practical outcomes.

Agentic Retrieval-Augmented Generation (RAG): A Live Demonstration

The core of the session featured a demonstration of an agentic RAG system built to classify users as either students or staff and then retrieve benefits information from the appropriate database.

Key features of the demo included:

• Router LLM: A specialized model tasked with deciding which database to query or when to reject a question entirely.
• Context Control: Only the correct benefits database was accessed based on user type, improving answer relevance and reducing hallucination risk.
• Transparent Decision-Making: The system logged every decision, offering clear insight into how each step was performed.

The demo showed how even simple agentic structures significantly enhance reliability, providing a foundation for more advanced agent-based systems.

Challenges and System Design Considerations

Marc highlighted several important challenges for building agentic AI systems:

• Similarity Scores: Semantic retrieval is imperfect, and tuning similarity thresholds is an ongoing area of experimentation.
• Evaluation Complexity: Each part of the system—routing, retrieval, answering—must be tested independently for robustness.
• Security Risks: Developers must consider prompt injection, excessive permissions, and API vulnerabilities when designing agentic systems.

The session emphasized that simplicity, transparency, and incremental complexity are crucial for creating useful and trustworthy agentic AI tools.

From Agentic Workflows to Full Autonomous Agents

The discussion shifted to autonomous AI agents, systems capable of creating their own action plans, adjusting based on feedback, and deciding when tasks are complete.

While full autonomous agents are still rare in production, early examples—like coding agents and experimental AI researchers—point to a future where AI systems are less about single prompts and more about ongoing collaboration between humans and machines.

Marc encouraged the group to start small, focusing on narrow, well-defined agentic systems before attempting larger-scale autonomy.

Practical Use Cases and Immediate Opportunities

The session concluded by suggesting low-risk agentic AI applications already feasible today, including:

• Research Assistance: Using agentic AI to triage large sets of documents for follow-ups.
• Administrative Tools: Moderating forums or verifying form submissions based on semantic understanding.
• Knowledge Management: Building personalized FAQ systems with controlled source material.

Faculty, staff, and students interested in experimenting with agentic AI systems or joining future sessions are encouraged to reach out to the Emerging Technologies team at [email protected].