Quiet Capital’s Astasia Myers: ‘Phase 3 ML Infrastructure’ and the Expanding Machine Learning Market

ABSTRACT

KEY POINTS FROM ASTASIA MYER'S POV

Why is Phase 3 ML Infrastructure such an important category moving forward?

• ML infrastructure is moving “up the stack” and appealing to non-specialized software engineers. “Algorithms have been relatively commoditized and open-sourced, decreasing the need for sophisticated training infrastructure, as teams can focus on fine-tuning,” says Myers. “Model hubs, or ‘marts’ have arisen for discovery so teams can more quickly compare the algorithms they choose. Inference APIs are available so teams can easily integrate NLP, audio, and computer vision models into their products.”

• The adoption of ML by the software engineers at startups and large enterprises is expanding the addressable market. “Expanding ML’s accessibility beyond data scientists to software engineers significantly increases the total addressable market,” she says.
  • In ML’s first wave, Myers explains, adoption was limited to a few hyperscale players.
  • In the second wave, third party ML tools emerged and were implemented by specialized and motivated teams.
  • Now, in a third wave, as several trends drive further adoption, teams no longer need to be ML specialists to implement customized applications, opening the door for many new startup offerings as well as broader enterprise implementation. “It is easier to go from 0 to 1 with ML than the past waves and individuals do not need to be ML specialists to do so,” she says. “ML infrastructure is starting to be abstracted enough that software engineers can become involved in the process.”

What are the use cases and business models that might be attached to this category?

• Options such as API-as-a-service or tuning ML models based on nodes-per-hour will emerge, giving many organizations a less demanding option for adoption, according to Myers. Other pricing models will be based on CPU, memory, and GPU-per-second. Meanwhile, “teams will charge for specialized hardware and private self-hosted software for data security-sensitive business.”
• As a macro theme, players will leverage usage-based pricing to grow with the needs of teams, says Myers. With consumption-based pricing, providers can wedge into smaller-budget organizations and eliminate entry hurdles.
• UX-focused tooling based on developer experience will proliferate across the enterprise, according to Myers. Zeroing on use cases will be more about purpose-made and developer-friendly tooling than about underlying AI/ML capabilities, due to the broader availability of algorithms (see “In the Investor’s Own Words,” below).

What are some of the potential roadblocks?

• Organizational structures might inhibit software engineers from driving end-to-end ML processes, i.e. training, deploying, and monitoring models. “This would be particularly true for teams that are retrofitting ML onto their products, rather than building it in from day one,” she says. Additionally, software engineers still need education on many aspects of ML in order to effectively deploy applications.
• It will take time for established ML teams to migrate to new solutions. While companies committed to ML may want to adopt more user-friendly and accessible tools, they already have commitments to vendors and production pipelines. Replatforming will be a process according to Myers, so they will be using new offerings “for prototyping or exploration rather than in production in the near term.”

VISUAL: WE ARE ENTERING THE THIRD PHASE OF ML

IN THE INVESTOR’S OWN WORDS

AI has a long history. The Turing test was invented in 1950 and since then ML has experienced winters and summers. We are currently in a ML boom that began with the transition to Deep Learning starting in 2010. Since then we’ve experienced three ML infrastructure phases.

The first wave of ML infrastructure from 2010 to 2015 was reserved for a select few. In the beginning academics and researchers like Stanford’s Fei-Fei Li who created ImageNet and Ian Goodfellow who invented Generative Adversarial Networks (GANs) advanced training data sets and model’s capabilities. Businesses that had the finances and resources to take advantage of these advancements in Deep Learning were tech-forward, large companies like Google, Facebook, LinkedIn, and Netflix. They could hire PhDs and spend millions building internal ML infrastructure tooling. Leveraging ML to improve product experiences was a competitive advantage generating revenue lift so these corporations didn’t wait for third party vendors to emerge, but rather pushed the bounds of infrastructure themselves. These public companies became famous and well-regarded for their sophisticated ML teams and internal platforms.

The second wave of ML infrastructure, from roughly 2016-2020, led to the rise of ML infrastructure vendors that democratized access to tooling. During this phase product teams had real-world examples from the hyperscalers of how ML could improve user experiences. They started thinking creatively about how ML could be applied to their business. However, most product teams didn’t have the resources like time, talent, and expertise to build platforms themselves, which was often a multi-quarter, multi-million dollar effort, assuming they could even hire the team to make it happen. They needed third-party tooling.

Luckily, creators of some of the large scale ML systems were entrepreneurial and cloud service providers noticed the market opportunity. We saw a wave of vendors emerge to help fill the tooling gap. This included the rise of end-to-end ML platforms like AWS Sagemaker as well as specialized solutions like Tecton and Weights & Biases. Individuals implementing and leveraging these solutions sat in the data and data science domains. Professional specializations emerged, including ML engineer, MLOps, Data Scientist, and Data Engineer — each tasked with different responsibilities to help teams develop, test, and bring models into production. The development to production cycle was happening outside of the large technology companies, but teams were mostly training a model from scratch, a time-consuming and compute intensive process, which often took quarters.

The current, third wave of ML infrastructure further abstracts ML practitioners from core infrastructure. The emerging tooling is no longer simply focused on filling a void in the market but rather optimizing the experience. New solutions focus on ease of use, ergonomics, performance, and cost.

One reason new tooling focuses on user experience is because algorithms are more accessible and have advanced significantly. Model hubs like Hugging Face allow teams to more quickly compare and choose algorithms. Simultaneously, we’ve seen the emergence of foundational models that are available via open source and API. Fine-tuning these models with domain-specific data takes a fraction of the training time compared to training a model from scratch. Moreover, inference APIs make it possible for teams to easily integrate NLP, audio, and computer vision models into their products without training or fine-tuning. Examples include OpenAI’s GPT-3, Cohere.ai, AssemblyAI, Stability.ai, and Deepgram.

Today it is easier to go from 0 to 1 with ML than the past waves. Tools are simplifying and accelerating the process. Improved ergonomics means individuals do not need to be ML specialists to be ML practitioners. While cost remains a factor, ML infrastructure is starting to be abstracted enough that software engineers can become involved in the process. During our research, we often come across product engineers who are now fine-tuning models or leveraging inference APIs. This is a huge shift over the past two years. We have been excited about ML solutions since 2016 and like that the current wave of ML infrastructure further broadens ML’s accessibility. This is just the beginning.

WHAT ELSE TO WATCH FOR

Look for startups that help companies reduce costs across AI/ML implementations. “Performance and cost continue to be top of mind,” says Myers. “NVIDIA A100 GPUs are expensive so ML optimization startups have cropped up to eliminate inefficiencies at the algorithmic and systems levels across training and inference. Serverless ML runtimes like Modal allow users to train and deploy models for inference without having to configure or manage underlying infrastructure. These tactics help reduce infrastructure utilization and Total Cost of Ownership (TCO).”

COMPANIES MENTIONED IN THIS BRIEF

AssemblyAI, Co:here, Deepgram, Hugging Face, OpenAI, Stability ai, Weights & Biases, Tecton