
\section{Emerging pre-trained and foundation models}
\label{sec:foundation}

As discussed in the previous sections, extracting geometric, structural, and semantic information from a single panorama (\autoref{sec:geostruct}) and augmenting these representations to support dynamic exploration and interaction (\autoref{sec:interactive}) involve solving highly ill-posed problems, which necessitate the use of strong priors. Learning such priors by leveraging massive, predominantly annotated datasets to extract latent information relevant to a given task has proven highly effective, but it also raises substantial practical challenges. Learning such priors from massive, task-specific datasets has proven highly effective, but it also raises substantial practical challenges. Suitable datasets are often limited to specific environments or unavailable for the targeted tasks, and creating them is costly, particularly when aiming to cover the wide variability of indoor environments. Moreover, even when appropriate data exists, training specialized models from scratch remains computationally expensive and time-consuming.

In many deep learning domains, recent years have seen significant progress in developing general-purpose pre-trained, or \emph{foundation}, models trained on large-scale, diverse data, often exploiting weakly or self-supervised approaches to extract general, reusable representations. Once trained, these models can be adapted or fine-tuned as reusable building blocks for a wide range of downstream tasks, exploiting the knowledge extracted from the original training data. Notably, the rapid advances in language-related tasks, particularly through large language models (LLMs), are largely attributable to the massive scaling of both training data and model capacity~\cite{Kaplan:2020:SLN,Brown:2020:LMF,Naveed:2025:COL}. Because images represent a higher-dimensional, noisier, and more redundant modality than text, the computer vision community has early on adopted large-scale unsupervised and self-supervised pre-training strategies~\cite{Radford:2021:LTV}, leading to widely used foundation models for many vision tasks, such as segmentation~\cite{Kirillov:2023:SA} and object detection~\cite{Xu:2021:ESO}. Building on these developments, pre-trained vision-language models such as CLIP~\cite{Chen:2020:GPP} combine both trends and have demonstrated strong zero-shot performance across a variety of downstream vision tasks, including image classification and object detection.

In the context of single-image $360^\circ$ inference, pre-trained or foundation models are leveraged in two main ways: by repurposing standard computer vision models trained on conventional imaging to support specific sub-tasks and combining them to address omnidirectional problems (\autoref{sec:foundation:standard}), and by developing dedicated $360^\circ$ models that explicitly exploit the holistic and intrinsic properties of omnidirectional images or videos (\autoref{sec:foundation:360}). 

%-----------------------------------------
\subsection{Transfer and composition of conventional vision foundation models for 360$^\circ$ tasks}
\label{sec:foundation:standard}

Leveraging standard vision foundation models for omnidirectional tasks offers a practical means to extend their applicability to more diverse environments and higher-resolution settings. Many existing models, whether designed for omnidirectional imagery or not, operate at relatively low spatial resolutions (e.g., $1024{\times}512$ pixels, approximately 0.5 MP), which are significantly below the requirements of modern virtual reality head-mounted displays (e.g., HTC Vive at $2448{\times}2448$ pixels per eye) and current omnidirectional imaging sensors, which can reach $8$K resolution ($8192{\times}4096$) or even 60 MP (e.g., Ricoh Theta X at $11008{\times}5504$). Simple post-hoc upsampling of holistic predictions only partially alleviates this limitation: fine geometric details, thin structures, small or distant objects, and high-frequency textures are often lost when depth or geometry is estimated at low resolution and upsampled afterward, thereby constraining downstream tasks such as depth estimation, SLAM, surface meshing, or neural rendering~\cite{Ai:2025:SRL}. To address these issues, composing partial inferences obtained from multiple limited–field-of-view perspective views or spherical tiles is emerging as an effective strategy for preserving spatial detail, while also addressing the limitations due to the scarcity of large-scale, high-resolution panoramic training datasets by transferring knowledge learned from much broader sets of examples.
 
For instance, pre-trained foundation depth models have recently led to substantial advances in zero-shot monocular depth estimation, notably improving cross-domain robustness and metric consistency (e.g., \emph{Depth Anything}~\cite{Yang:2024:DAU,Yang:2024:DAV}). In the context of panoramic imagery, several strategies have emerged to adapt such foundation models to omnidirectional depth estimation. 

A first line of work follows a \emph{distillation or transfer} paradigm, in which depth predictions from perspective models are used to supervise networks operating directly on equirectangular panoramas~\cite{Wang:2024:DAE,Jiang:2025:DA3}. This approach alleviates the scarcity of annotated panoramic data, but the resulting models often face scalability issues, as operating directly on high-resolution panoramas remains computationally demanding on standard hardware. 

A second family of methods adopts a \emph{projection-and-fuse} strategy, applying pre-trained depth models independently to overlapping perspective patches sampled from the panorama and subsequently reprojecting the predictions into the panoramic domain using seam-aware fusion~\cite{Li:2022:O3M,Rey:2022:3HR,Shen:2023:DOP,Ai:2023:HM3}. While this approach preserves high-frequency details, the independently generated depth maps are frequently inconsistent and difficult to reconcile, often requiring substantial post-processing to obtain a coherent global result~\cite{Rey:2022:3HR,Shen:2023:DOP}. To improve inter-patch consistency, ST$^2$360D~\cite{Cao:2025:SSC} reformulates the problem by converting panoramas into perspective video sequences and predicting depth using foundation video models~\cite{Luo:2020:CVD,Chen:2025:VDA}. Temporal continuity across frames helps enforce smoother transitions between adjacent views. However, the lack of a truly holistic representation can still hinder seamless global consistency. Peng and Zhang~\cite{Peng:2023:HDE} further address this issue by aligning perspective-based predictions to a low-resolution panoramic depth estimate, improving global registration while remaining dependent on the quality of the individual perspective inferences. Moreover, the subsequent composition and fusion stages may still introduce discontinuities and blurring near patch boundaries. 

Overall, this line of research reveals the emergence of a multi-resolution solving paradigm, in which a coarse, low-resolution full-panorama prediction may serve not only as global structural guidance for conditioning high-resolution local inferences from perspective foundation models, but also as an explicit scaffold to be exploited during the merging and fusion stages. By anchoring independent local predictions to a shared global representation, this approach would facilitate more consistent aggregation into a coherent solution and is likely to generalize to a wide range of omnidirectional inference problems. Conditioning local predictions from general limited-view foundation models, often also employing a globally constrained merging step, is, for instance, exploited by many of the view-synthesis approaches surveyed in~\autoref{sec:interactive:viewsynth}. In particular, conditioned open diffusion models~\cite{Rombach:2022:HIS,Gokaslan:2024:COD} are exploited for inpainting in novel view synthesis~\cite{Kulkarni:2023:3PN,Rey:2025:33P,Pu:2024:PNV} (\autoref{sec:interactive:viewsynth}).

%-----------------------------------------
\subsection{Native $360^\circ$ foundation models}
\label{sec:foundation:360}

Although the development of foundation models explicitly tailored to the intrinsic properties of $360^\circ$ imagery is a potentially promising research direction, it remains relatively underexplored compared to foundation models designed for conventional perspective images.

On the one hand, training foundation models is computationally and economically very expensive. To provide a reference, training a frontier foundation model can nowadays exceed the hundreds of millions of Euros~\cite{Maslej:2025:AII}, and costs are projected to increase~\cite{Cottier:2025:RCT}. Models trained on massive collections of web-scale perspective images or synthetic data have already demonstrated strong performance on a range of 3D tasks that can be transferred to omnidirectional image analysis and synthesis~\cite{Liu:2025:TUG}. In particular, recent work has shown that unified, generalist generative foundation models for videos can address a wide spectrum of analysis and synthesis problems, exhibiting emerging zero-shot perception and reasoning capabilities~\cite{Wiedemer:2025:VMZ}. As a result, both academic and industrial efforts have largely concentrated on leveraging and adapting such general-purpose models rather than developing omnidirectional-specific foundations from scratch.

On the other hand, despite the notable successes of models trained on synthetic imagery or conventional videos, the scarcity of large-scale, high-quality real-world 3D data remains a fundamental challenge for 360 and 3D tasks. Synthetic data can provide dense and accurate annotations for supervised learning, but its scale and diversity are inherently limited. Approaches such as PanDA~\cite{Cao:2025:PTP}, that aim to construct a foundation 360 variant of Depth Anything, tackle the challenge with a weakly supervised approach. First, a teacher model is obtained by fine-tuning Depth Anything~\cite{Yang:2024:DAU} on synthetic indoor and outdoor panoramic data. Then, a student model is trained on large-scale unlabeled data using teacher-generated pseudo-labels. DA360~\cite{Jiang:2025:DA3}, instead, fine-tunes the DAv2 zero-shot single pinhole depth estimation model~\cite{Yang:2024:DAV} with synthetic panorama depth datasets to produce scale-invariant globally consistent panoramic depth maps. The key innovation involves learning a shift parameter from the vision transform backbone, transforming the model’s scale- and shift-invariant output into a scale-invariant estimate that directly yields well-formed 3D point clouds.

Videos constitute a particularly promising alternative, as structural cues can be inferred from inter-frame correspondences, enabling self-supervised learning. However, acquiring diverse yet corresponding views of the same scene content at scale is difficult, and standard videos are captured from fixed viewpoints, restricting access to a broader range of informative perspectives. Large-scale $360^\circ$ videos naturally alleviate these limitations by enabling scalable extraction of corresponding views from multiple, diverse viewpoints. Initial efforts in this direction are emerging. For example, Wallingford et al.~\cite{Wallingford:2024:ISL} trained a diffusion-based model on one million $360^\circ$ YouTube videos and demonstrated applications to novel view synthesis as well as geometry and layout extraction. Further progress in this direction would be beneficial not only for video generation but for all tasks requiring holistic understanding. 

%-----------------------------------------
\subsection{Opportunities and limitations}
\label{sec:foundation:discussion}

Overall, foundation models are emerging as a powerful paradigm for single-view $360^\circ$ inference, as they enable the reuse of knowledge distilled from large and diverse datasets, improved cross-domain generalization, and, in some cases, zero-shot or few-shot capabilities in settings with limited panoramic supervision. This trend is currently more prominent in adapting general vision foundation models to omnidirectional tasks than in creating foundation models specifically designed for $360^\circ$ imagery. However, the development and deployment of foundation models incur substantial costs in terms of data, computation, and energy, placing them beyond the practical reach of many researchers and raising concerns about transparency, direction, and the concentration of power~\cite{Maslej:2025:AII}. Open models may offer only a partial solution~\cite{Bommasani:2024:CGO}. Furthermore, for many application scenarios, task-specific lightweight models remain significantly more efficient to train, execute, and deploy, particularly on constrained hardware. For these reasons, we anticipate continued research on integrating foundation models as reusable building blocks for omnidirectional problems and on devising lean, task-specific solutions that prioritize efficiency and accessibility.
