How Hollow Thread Speeds Up AI | Microsoft Azure Blog

One of these technologies, which was presented at Microsoft Ignite this November, was Hollow Core Fiber, an innovative optical fiber designed to optimize Microsoft Azure’s global cloud infrastructure, offering superior network quality and secure data transmission.

This blog is part of the “Infrastructure for the AI Era” series, which focuses on emerging technologies and trends in large-scale computing. This piece takes a deeper dive into one of our newest technologies, Hollow Fiber (HCF).

Artificial intelligence is at the forefront of people’s minds and innovation is happening at lightning speed. But to keep pace with AI innovation, companies need the right infrastructure for the computationally intensive AI tasks they’re trying to run. This is what we call it “purpose infrastructure” for AI and it’s a commitment Microsoft has made to its customers. This commitment doesn’t just mean taking hardware developed by partners and placing it in your data centers; Microsoft is dedicated to working with partners, and occasionally with itself, to develop the latest and greatest technology that powers scientific discoveries and AI solutions.

One such technology that was highlighted at Microsoft Ignite in November was Hollow Core Fiber (HCF), an innovative optical fiber designed to optimize Microsoft Azure’s global cloud infrastructure, offering superior network quality, improved latency and secure data transmission.

Transmission by air

HCF technology was developed to meet the demanding requirements of workloads such as artificial intelligence and improve global latency and connectivity. It uses a proprietary design where light propagates through an air core, which has significant advantages over traditional fiber built with a solid glass core. An interesting bit is that the HCF structure has nested tubes to help reduce any unwanted light leakage and keep the light in a direct path through the core.

Because light travels faster through air than glass, HCF is 47% faster than standard silica glass, resulting in increased overall speed and lower latency. It also has higher bandwidth per thread, but what is the difference between speed, latency and bandwidth? While speed is the speed at which data is transferred over an optical medium, network latency is the time it takes to transfer data between two endpoints on a network. The lower the latency, the faster the response time. Additionally, bandwidth is the amount of data sent and received on the network. Imagine two vehicles leaving from point A to point B at the same time. The first vehicle is a car (representing single mode fiber (SMF)) and the second is a van (HCF). Both vehicles carry passengers (which is a figure); the car holds four passengers while the van holds 16. The vehicles can reach different speeds, with the van going faster than a car. This means that the van will take less time to travel to point B and therefore arrive at the destination first (providing lower latency).

For more than half a century, the industry has seen constant, yet small, advances in silicon fiber technology. Despite the progress, gains were modest due to the limitation of silica loss. A significant milestone with HCF technology was achieved in early 2024, when the lowest optical fiber loss (attenuation) recorded at 1550 nm was achieved, even lower than single-mode fiber with a pure silicon core (SMF). ¹ Along with low attenuation, HCF offers higher startup performance, wider spectral bandwidth, and improved signal integrity and data security compared to SMF.

The need for speed

Imagine you are playing an online video game. The game requires quick reactions and split-second decisions. If you have a high-speed, low-latency connection, your in-game actions will be quickly transmitted to the game server and to your friends, allowing you to react in real time and enjoy a smooth gaming experience. On the other hand, if you have a slow connection with high latency, there will be a delay between your actions and what is happening in the game, making it difficult to keep up with the fast pace of the game. Whether you’re missing key action times or falling behind others, lag is very annoying and can seriously disrupt gameplay. Similarly, in AI models, lower latency and high-speed connections can help models process data and make decisions faster, improving their performance.

Latency reduction for AI workloads

So how can HCF help AI infrastructure performance? AI workloads are tasks that involve processing large amounts of data using machine learning algorithms and neural networks. These tasks can range from image recognition, natural language processing, computer vision, speech synthesis and more. AI workloads require fast network connectivity and low latency because they often involve multiple data processing steps such as data ingestion, preprocessing, training, inference, and evaluation. Each step may involve sending and receiving data from various sources, such as cloud servers, edge devices, or other nodes in a distributed system. The speed and quality of your network connection affects how quickly and accurately data can be transferred and processed. If the network is slow or unreliable, it can cause delays, errors or failures in the AI workflow. This can result in poor performance, wasted resources, or inaccurate results. These models often need massive amounts of computing power and ultra-fast networking and storage to handle increasingly sophisticated workloads with trillions of parameters, so ultimately low latency and high-speed networking can help speed up model training and inference, improve performance and accuracy, and support AI innovation.

We help with AI workloads everywhere

Fast networking and low latency are especially important for AI workloads that require real-time or near-real-time responses, such as autonomous vehicles, video streaming, online gaming, or smart devices. These workloads need to process data and make decisions in milliseconds or seconds, which means they can’t afford any network delays or interruptions. Low latency and high-speed connections help ensure that data is delivered and processed in a timely manner, allowing AI models to deliver timely and accurate results. Autonomous vehicles are an example of a real-world application of artificial intelligence, relying on artificial intelligence models to quickly identify objects, predict movements, and plan routes in unpredictable environments. Fast data processing and transfer, facilitated by low latency and high-speed connectivity, enable near-real-time decision-making, improving security and performance. HCF technology can accelerate AI performance, providing faster, more reliable, and more secure networks for AI models and applications.

Regional implications

There are more implications beyond the direct hardware that runs your AI models. Data center regions are expensive, and both the distance between regions and between regions and the customer make a big difference for the customer and Azure as they decide where to build those data centers. When the region is located too far from the customer, it results in higher latency as the model waits for data to and from a more distant hub.

If we think about the example of a car versus a van and how it relates to a network, the combination of higher bandwidth and higher transfer speeds can transfer more data between two points on the network in two-thirds of the time. Alternatively, HCF offers longer reach by extending the transmission distance in an existing network up to 1.5x without impacting network performance. Ultimately, you can go the extra distance with the same latency envelope as traditional SMF and with more data. This has huge implications for Azure customers as it minimizes the need for data center proximity without increasing latency and reducing performance.

Infrastructure for the AI era

HCF technology was developed to improve global Azure connectivity to meet the demands of AI and future workloads. It offers several benefits to end users, including higher bandwidth, better signal integrity and increased security. In the context of AI infrastructure, HCF technology can enable fast, reliable and secure networking, helping to improve the performance of AI workloads.

As artificial intelligence continues to evolve, infrastructure technology remains a critical piece of the puzzle that ensures efficient and secure connectivity for the digital era. As advances in AI increasingly strain existing infrastructure, AI users are increasingly looking to leverage new technologies such as HCF, virtual machines such as the recently announced ND H100 v5, and silicon such as the first partner accelerator Azure AI, Azure Maia 100. Advances together enable more efficient processing, faster data transfer, and ultimately more powerful and responsive AI applications.

Follow our “Infrastructure for the Era of AI” series to better understand these new technologies, why we’re investing where we are, what these improvements mean for you, and how they’re enabling AI workloads.