Emerging Technology Consortium

Apple@Columbia: CoreML and Machine Learning with Ashley Carroll and Kevin Ferrell

By
Nikki Pet
January 30, 2020
machine learning

For our last event of 2019, it was a pleasure to host Ashley Carroll and Kevin Ferrell from Apple’s App Dev Consulting team, where they introduced machine learning, in particular how to easily integrate machine learning into app development using Apple’s Core ML framework. To start the event, ETC co-founder Maneesha Aggarwal gave a recap of ETC’s 2019 activities, as well as previewing our events and goals for next year. Particularly exciting is a new grant from the Office of the Provost to fund projects involving emerging technologies in the classroom. Keep an eye out on our mailing list and Twitter to stay updated!

This event was particularly special for ETC, as it served as somewhat of a continuation from last February's event with Apple's Steve Hayman, where he introduced Apple’s ARKit framework and how to use it in app development. In this event, Ashley Carroll and Kevin Ferrell introduced us to the adjacent Core ML framework, used to integrate machine learning techniques into applications, especially useful for those with less expertise in machine learning. 

What is machine learning? Ms. Carroll explained that machine learning is a predictive technique that takes parameters from datasets in order to correctly identify unknown objects and classify them within given parameters. For example, if a machine learning program is designed to be able to classify types of fruits based on photos given, the user or programmer will feed large datasets of photos into the program with various parameters associated with the different types of fruits. For instance, strawberries will have parameters of being red, having a distinct shape, and a certain texture. The machine learning algorithm will then use those parameters to identify foreign images that fit those standards, learning how to filter edge cases, such as oddly shaped strawberries, off-color strawberries, and imposters, such as photos of pears or apples that have the same color and shape as strawberries. 

The model is trained to uniquely distinguish between varying input data, refining and adjusting its parameters as more and more data is fed into it. Models are also not meant to give absolutes, but will rather give percentages for how likely something belongs to a certain category. With the fruit identification example, an image of a strawberry could be processed as having a high percentage chance of being classified as a strawberry (e.g. 90%), a lower percentage chance of being an apple (e.g. 9.9%), and a negligible percentage chance of being a watermelon (e.g. 0.1%). 

Usage of machine learning models includes things like image recognition, as given in the above example, as well as encompassing numerous different types of classifications. These other types of models include:

  •  Object Recognition: where the model can recognize and classify multiple different types of objects in an image (as opposed to just one in the example above)

  • Text Classification: where models can read a block of text and detect the subject, tone, and inflection of the writing

  • Activity Classification: where models classify types of physical activity, such as running, walking, climbing, etc. using data collected by accelerometer and gyroscope from an Apple Watch or FitBit type product

  • Tabular Data Regression: this model is used by real estate websites like Zillow or Trulia to estimate rent and sale prices based on factors such as property tax, year of construction, etc. 

  • Content Recommendation: sites such as Netflix and Youtube use historical user preference to fine tune video recommendations, and apps designed for retail can also take in a desired product and give a list of similar items. Ms. Carroll gave the example of Boot Finder, an app that takes a photo of a pair of boots and returns similar products for purchase

 

Machine learning can also leverage individual user personalization. For example, Apple’s Core ML framework in particular has the ability to recognize user handwriting (as input through a stylus like the Apple Pen), and convert them to text or emojis. Many of these techniques can also be combined and connected within the same application.

Core ML is Apple’s machine learning framework that lets users unfamiliar with the specifics of machine learning utilize the technique as a tool in app development. Using Apple’s XCode IDE (integrated development environment), programmers can use Core ML to develop apps for iOS, MacOS, and other Apple operating systems, as well as being able to convert programmed apps for other operating systems (e.g. Android). Programmers can train existing Core ML models with their own data sets, with results being more refined and specific with larger data sets, and falling back on general pre-programming with smaller data sets. However, Carroll and Ferrell cautioned that users should be careful to regulate biases within the data set, so as not to create a biased machine learning model. 

To close, Kevin Ferrell demonstrated how to use Core ML to develop an app in real time, showing us an app he programmed that differentiates percussion noises in real time from a continuous audio input.