Object storage for AI applications:
scalable storage for data-intensive workloads

The AI revolution demands new data strategie

AI models devour vast, often unstructured datasets – from images and videos to sensor data and texts. Today, unstructured data makes up around 80 to 90% of a company’s total data volume. The result: immense demands on data storage and constantly growing challenges for businesses.

The data flood of artificial intelligence

Imagine this: your company wants to deploy an intelligent chatbot that answers customer queries 24/7. To achieve that, it must learn from thousands of conversations, emails, and product data to deliver precise and helpful answers. Or perhaps you use AI to analyse your customers’ purchasing behaviour and deliver personalised recommendations in your online shop – which requires processing huge amounts of transactional data, clicks, and preferences.

The figures speak for themselves: A modern Large Language Model like GPT-o3 requires up to 45 terabytes of text data for training. Image recognition models like DALL-E process hundreds of millions of images – equivalent to several petabytes of raw data. Even “small” company AIs for chatbots or recommendation systems quickly consume hundreds of gigabytes of data per month. Traditional storage systems are quickly exhausted here.

The limits of traditional storage systems

Conventional storage solutions, as we know them from everyday office life, are optimised for structured data and manageable file volumes. These include:

• File systems (e.g. on NAS – Network Attached Storage): Data is stored in a tree structure, like folders on your computer. Intuitive for humans, but quickly becomes confusing for AI applications that must access millions of files. Each access requires traversing long directory paths – wasting valuable time.
• Block-based storage (e.g. SAN – Storage Area Network): Stores data in fixed blocks, like sectors on a hard drive. Very fast for single, defined data packets but less flexible when managing huge amounts of unstructured AI data.

Both approaches are usually “hierarchical” and scale “vertically”. That means: they become more powerful by upgrading individual components (e.g. bigger disks, faster processors). But once maximum capacity is reached, performance drops.

Bottlenecks in parallel access and metadata management

Reality is sobering: a consumer NAS reaches its limits with just 50 to 200 parallel accesses – but your AI clusters require thousands of simultaneous streams. Scale-out NAS or NVMe SAN are faster but far more expensive.

Another problem: file and block storage support only basic metadata – filename, creation date, access rights. AI applications need more context:

• What does an image show?
• Where was a sensor value recorded?
• Which patient belongs to a medical scan?

Rich, custom metadata is the key to making vast datasets usable for AI. New solutions are required.

Object storage: the storage architecture for the AI era

Object storage breaks with the old rules. Instead of cramming data into folders, it stores them as individual objects in a massive, flat address space called a “bucket”. Each object consists of:

• the data itself,
• a globally unique ID, and
• – the game-changer – extensible, rich metadata.

This is a blessing for AI applications. Detailed descriptions of the data simplify finding, categorising, and processing massive datasets. The flat architecture also enables virtually unlimited horizontal scalability. During training or inference, AI models often require simultaneous access to huge amounts of data. Because object storage is inherently distributed and massively parallelisable, many AI clients can access different objects simultaneously. Best of all: compared to traditional storage systems, object storage is more cost-efficient – especially for unstructured data. Many object storage solutions run on standard servers, lowering overall TCO even further.

Direct comparison: NAS, SAN, and object storage

Note: These values are general guidelines. Scalability, parallelism, and costs depend on specific implementation and vendor.