AICoP - Local Compute, Real-World Impact for AI in Higher Education

The AI Community of Practice welcomed HP for a hands-on session exploring how universities can run powerful AI models entirely on local hardware—no internet connection required. Curtis Burkhalter, Technical Marketing Manager at HP, walked attendees through the ZGX Nano, a compact device capable of running cutting-edge open-source models, and demonstrated real-world applications spanning healthcare, education, and defense.

What's New

Desktop-Scale AI Hardware: HP's ZGX Nano packs NVIDIA's latest Blackwell GB10 GPU, an ARM-based processor, and a unified memory architecture into a device that fits in the palm of your hand—six inches by six inches by two inches. It runs on a Linux-based operating system (DGX OS) and is designed as a companion device, meaning researchers can work on their regular Windows or Mac machine and offload AI workloads to the Nano over a local network.

Unified Memory Changes the Game: Previous generations of hardware separated CPU and GPU memory, creating bottlenecks when transferring data between processors. The ZGX Nano's unified architecture shares a large memory pool across CPU and GPU with no bridge between them, enabling fine-tuning of models up to 70 billion parameters and inference on models up to 200 billion parameters. Two units can be linked to double that capacity.

One-Click Open-Source Stack: HP's free ZGX Toolkit, a VS Code extension, automatically discovers the Nano on your network, handles SSH key authentication, and provides one-click installation of popular open-source tools: Ollama for local inference, JupyterLab for notebooks, Open WebUI, and more. An engineer from Tesla reportedly went from unboxing to running models in minutes using the toolkit.

Complete Air-Gap Option: For the most sensitive environments, HP is the only OEM offering a unit with de-soldered Wi-Fi and Bluetooth antennas, ensuring the device is physically incapable of wireless communication. Ethernet remains available for controlled network access.

Why It Matters

Data Privacy Without Compromise: HIPAA-regulated patient data, FERPA-protected student records, sensitive donor information—all of these create compliance bottlenecks when cloud AI is involved. Local AI keeps everything within institutional control. As Burkhalter emphasized, the real cost isn't the compliance process itself—it's the research that doesn't happen while waiting for approvals.

Predictable Budgets for Grant-Funded Work: Cloud-based AI billing is per-token and notoriously unpredictable—Burkhalter shared a cautionary tale of racking up a $31,000 Databricks bill in two weeks at a previous role. Local hardware is a one-time purchase with zero marginal cost per query, making it far more compatible with the fixed budgets typical of grant-funded research.

Reproducibility as a Foundation: Open-source models from Hugging Face are version-locked by default. Unlike cloud providers that may update models or deprecate endpoints without warning—injecting unplanned variability into research—local deployments freeze the exact model weights, ensuring experiments can be faithfully reproduced. As Burkhalter put it, reproducibility "isn't an option in science. It's foundational."

Full Transparency and Control: When you send a prompt to a cloud-based model, you receive an answer but can't examine the system's architecture, fine-tune on your own data, or modify how the model operates. Local AI gives researchers access to every "nerd knob" under the hood—architecture selection, fine-tuning, evaluation metrics, and full auditability of outputs.

Real-World Applications in Higher Education

Texas A&M Foundation — Scholarship and Donor Operations: The foundation's donor relations team previously spent entire holiday breaks manually reviewing student transcripts to determine scholarship compliance. Using local AI, each transcript now processes in 17 seconds, reducing weeks of manual work to a few days. They've also automated the review of 2,500+ scholarship applications per year and built secure AI tools for processing sensitive donor information that cannot be shared with third-party cloud services.

Winona University — AI-Powered Teaching: Professor Patrick Paulson in Management Information Systems has built RAG-based chatbots for each of his courses, using open-source textbooks as the source of truth. Students query the chatbot instead of searching through textbook PDFs, with answers grounded in the actual course materials—significantly reducing hallucination risk. He's also planning an immersive tech lab where students share ZGX Nanos and Raspberry Pis as hands-on AI development resources. Perhaps most notably, he reduced his semester-long curriculum update process from six weeks to two days using AI tools.

APITA (France) — Medical Imaging Research: Researchers are using the ZGX Nano for 3D brain tumor segmentation from MRI scans—one of the most computationally demanding tasks in medical AI. Development iterations are 16x faster on the Nano's GB10 GPU compared to CPU-only processing and 5.5x faster than discrete GPU workloads, all while keeping patient data completely on-premises. The project's code and tools are freely available on GitHub.

Live Demos

Clinical Decision Support for Polypharmacy: Burkhalter demonstrated a tool he built by fine-tuning a 56 billion parameter Mistral model on FDA drug labels, the Stanford drug interaction database (48,000+ interactions), and PubMed clinical studies. The tool analyzes complex patient profiles—accounting for age, multiple medications, and pre-existing conditions—and generates a comprehensive clinical risk report in about 11 seconds. It provides drug-to-drug interaction analysis, risk scoring, monitoring recommendations, and a management plan. All processing runs locally on the ZGX Nano.

Maritime Surveillance Intelligence: A compound AI system that chains a vision language model with a large language model to analyze aerial drone imagery. The system identifies vessel types, assesses threat levels, and generates intelligence reports with contextual recommendations—all without fine-tuning, using only open-source models. The demo illustrated how multiple AI models can work together on a single local device and how similar architectures could be applied to radiology assistance or other image-analysis workflows.

Important Considerations Raised

Model Bias and Foreign Influence: A significant discussion emerged around the risks of using open-source models developed by foreign governments. Jim Cheng, Director of Columbia's Starr East Asian Library, shared that he tested DeepSeek's model in the city where it was developed and found it completely blocks responses about Tiananmen Square. Participants agreed that the community needs clear guidance on evaluating model bias, particularly for use in humanities and social science research where missing historical context could undermine scholarly work.

Security Best Practices: Participants emphasized that local hardware alone does not eliminate security concerns. Anyone handling PII or PHI should consult with CUIT for secure data management, regardless of whether they are using a purpose-built device like the ZGX Nano or a standard Mac or Windows machine. When downloading models from Hugging Face, stick to well-known, security-reviewed model families from reputable providers and check leaderboard rankings.

Practical Trade-offs: The ZGX Nano runs a customized Linux-based OS (DGX OS) that adds complexity to an institution's support ecosystem. Several attendees noted that for less technical users, a Mac Studio or Mac Mini with 16–64 GB of RAM could provide many of the same local AI benefits within a more familiar and supportable operating environment. The consensus was that the right tool depends on the use case and the user's technical comfort level.

What's Available at Columbia

HP has generously provided Columbia with a ZGX Nano unit that is available on-site for testing and experimentation. Researchers interested in utilizing the device for projects involving sensitive data or local model development can reach out to learn about availability. Curtis Burkhalter's demo code, fine-tuned models, and detailed setup documentation are all open-source and available on his GitHub. The HP team—including contacts Jerome and Shane who have been working with Columbia—will share presentation slides in a follow-up communication.

Takeaway

The hardware barrier to running serious AI locally has fallen dramatically. A device that fits in the palm of your hand can now fine-tune 70 billion parameter models and run inference on models up to 200 billion parameters—capabilities that required server rooms just months ago. For researchers and staff who work with regulated data, need reproducible results, or want full control over their AI workflows, local AI offers a compelling complement to cloud-based tools. As the AI landscape continues to evolve rapidly, Columbia's community benefits from having both options available.

Learn More

• Curtis Burkhalter's GitHub – Open-source demos, fine-tuned models, and setup guides
• APITA Brain Tumor Segmentation – Open-source 3D medical imaging project (GitHub)
• LLM Fit – Tool for checking if models will fit on your hardware
• Hugging Face – Open-source model repository with leaderboards
• HP ZGX Nano – Product information and specifications
   

For questions, contact: [email protected] For training and consult inquiries, contact: [email protected]