Artificial intelligence on the road, the future of autonomous vehicles

Urs Muller, Chief Software Architect at NVIDIA of autonomous vehicles visits Columbia where he outlined the current and future state of self-driving cars.

His work focuses on the development of end-to-end solutions for autonomous vehicles. He has 20+ years of experience in robotics, computer vision, machine learning, and high-performance computing. He received Ph. Ds in Electrical Engineering and Computer Science from the Swiss Federal Institute of Technology in 1993.

Autonomous vehicles will transform the way we live, work, and play, creating safer and more efficient roads. To realize these revolutionary benefits, the car of the future will require a massive amount of computational horsepower.

NVIDIA focuses on gaming, VR/AR, Data Center, and self-driving cars.

All of those initiatives are linked by the same underlying artificial intelligence and visual computing architecture.

HARDWARE: NVIDIA DRIVE AGX is a scalable, open autonomous vehicle computing platform that serves as the brain for autonomous vehicles.

Drive-AGX stack:

• Drive AV
• Driveworks SDK
• Drive OS
• Drive AGX Xavier and Pegasus Hardware
  • • Xavier - 1 processor, 30 TOPS for deep learning, 30 watts power
    • Pegasus - 2 Xavier processors

SOFTWARE: To augment and, eventually, replace the human driver, NVIDIA DRIVE software enables key self-driving functionalities such as sensor fusion and perception.

NVIDIA rents space on former Bell Labs campus in Holmdel, NJ. Bell Labs pioneered Deep Learning in 80's and 90's. Deep learning allows us to solve problems that we don't know how to program. 

Bell Works campus is ideal to test self-driving cars due to extensive private road network.

NVIDIA view of learning is as follows:

• choose the right structure
• allow machines to access prior knowledge

Lessons learned:

• look at the data during all processing steps
• solid debugging tools critical
• validate the training data
• the work is experimental
• work with real data

Milestones leading to today’s developments:

• 1995 - Holmdel deployed neural nets at Wachovia bank, and eventually processed 20% of checks in the US
• 1995 - AT&T Bell labs fractured
• 2002 - AT&T mass layoffs
• 2003 - Bell Labs work led to new programs at DARPA
• 2012 - Deep Learning becomes more popular and potentially viable in commercial applications of the Self-driving cars

Their goal is to solve the hard and unsolved autonomous vehicle problems using machine learning.

Simple driving rules fail due to numerous variable road conditions and unwritten behaviors.

It is hard to write down every single rule of safe driving, but relatively easy to collect training data through driving. Training data is gathered through pixel input (sensors) during actual human-driven path. Convolutional Neural Network prediction of path is compared to human path, and discrepancy is provided as feedback to the CNN for additional learning.

Open challenges:

• deal with ambiguous situations - there is often more than one correct answer (4-way stop signs)
• learn from imperfect behavior (several observations; most correct; some not correct)

The current cars being tested are hybrid, but the ultimate goal is to revolutionize the world by introducing self-driving electric cars. This is the journey to zero accidents. In the meantime, the current products are able to enhance driving safety and provide valuable lessons in construction of new vehicles.

Interview with Urs Muller, Chief Software Architect at NVIDIA

Parixit: Urs, thank you for coming to Columbia University. Can you give us a brief background about the work you do at NVIDIA?

Urs: I am leading the New Jersey self-driving team in Holmdel. What we do is focus on the unsolved problems in self driving and we have great hope we can solve those using machine learning.

Parixit: Can you talk to us about the not so distant future -- where we can see this technology taking us?

Urs: So, I think the first thing is enhancing safety in cars. The car drives you and if it drifts off the lane when you try to make a lane change or even break to avoid a collision or in later stage to drive around obstacles to avoid them.  Also possibly some driving assistance where the car steers, but we still need to pay attention.

Parixit: How many more years do you think we are at where it becomes completely autonomous?

Urs: So that you don't need a driver's license anymore?  I don’t know. This is open ended to a great degree and also depends if we see infrastructure changes to support self-driving cars.

Parixit: Can you talk about how the regulations differ from state to state, also how do they differ in other parts of the world?

Urs: Yes. They do differ, almost every state is different and other parts of the world differ also.

Parixit: Do you think people are looking towards self-driving cars or do you think there is still a push back, people wanting to drive?

Urs: Certainly, people are skeptical but the experience is actually what people want, they do want to do other things in the car, especially while commuting.

Parixit: What's the farthest the car has driven, has taken a road trip or do you do it in segments?

Urs: Yes, we do take road trips but it's not necessarily meaningful to quote those numbers because those are one offs, what really matters is that we consistently can do.

Parixit: Do you think it's possible to even go cross-country for self-driving car?

Urs: Yes, the highway part of it, if you picked the right highway is very close to doable. If you wanted to go from Los Angeles downtown or San Francisco downtown to downtown Manhattan, it’s a different story.

Parixit: So, highway driving is significantly easier?

Urs: I think there are 2 parts that are easier, nothing is easy. Highway driving with low traffic and clear lane markings is one of them, and the other one is low speed traffic in open areas. In between it’s really difficult.

Parixit: Can you talk about if you are a current computer science student, what are some of the skill sets and capabilities that you would look for to work on machine learning problems.

Urs: What we need in our team is mostly all rounded people who know not only machine learning, because majority of our work is not machine learning, it’s learning about controlling the car, car dynamics: if you set the steering wheel, the car does not always have the same curve, it depends on the tire pressure, the speed etc., those things matter, so you need to understand them. Then we have a great amount of diagnostic and visualization tools, which essentially is similar to exploring a new continent. Every day something new comes up and one day we need to be good at swimming thru rivers and next day we need to be good at climbing mountains. So, if somebody wants to do specifically deep learning, our team is less ideal for that, if somebody wants to really participate in the efforts to make it happen and enjoys doing different things, nothing will ever be a routine, it’s a great environment.

Parixit: So, it’s really fluid, a generalist would be a good match!

Urs: A generalist with the ability to go deep at a moment’s notice with any technology that comes up! We might not know yet what will it be.

Parixit: Thank you, and thank you for coming to Columbia Urs.

Urs: Thank you, my pleasure.