%---------------------------------------------------------------------------------
\section{Research trends and open challenges} 
\label{sec:challenges}
%-----------------------

As highlighted in the preceding sections, deep learning has driven major progress across both analysis and synthesis, and methods are reaching real-world applications. Nevertheless, many challenges remain open. Discussed throughout the survey and reflected in the cited literature, they outline the current frontier of the field. Below, we summarize several key trends and future research directions.

\paragraph*{Training data limitations} While enormous amounts of image data are available, available 360$^\circ$ imagery of interiors is much less abundant. Moreover, even though millions of indoor panoramas may exist in the wild and could, in theory, be harvested, curated research datasets remain small, less diverse, and costly to annotate. Synthetic datasets help expand scale and provide ground truth, but their diversity is still ultimately constrained by available 3D sources, which are currently limited and costly to generate. Key challenges include improving training data size and diversity through better annotation of real sources and faster generation of synthetic models, as well as, and most importantly, developing methods that work better in low-data regimes, exploiting labeled and unlabeled panoramas through semi- or self-supervised learning. An interesting related research direction is to use the more abundant regular perspective images for training 360$^\circ$ models through a combination of on-the-fly conversion to omnidirectional formats, augmentation, and masking of valid pixels. This approach has shown good potential for depth estimation~\cite{Guo:2025:DAC,Li:2026:DDA}, and may be extended to other tasks.

\paragraph*{Annotation accuracy on real-world data} In addition to its scarcity, real-world panoramic image annotation suffers from several limitations. While ground truth depth can be measured through instruments (i.e., depth sensors), room layouts and other structures generally come from human annotation and rely on per-dataset representation. It is not uncommon to have a mismatch between annotations and image features, and there is no universal way to use annotations from multiple datasets. Several efforts have strived to define and apply annotation workflows in different contexts (e.g., \cite{Cruz:2021:ZID,Deb:2024:ZTT}).

\paragraph*{Generalization challenges} Due to a variety of reasons, including the scarcity of datasets, many of the solutions surveyed in this report appear to be tested and trained using a single (or coherent) dataset, and leave out testing across domains, scene types, and capture conditions. In real cases, especially when scarce or synthetic data is used for training, data seen at inference time may often be very different from the one in the training set. Recent datasets have started to appear with training and testing sets coming from very different distributions (e.g.,  \emph{360MonoDepth}~\cite{Rey:2022:3HR} and Pano3D~\cite{Albanis:2021:PHB}, that mix synthetic and real data for cross-generalization purposes). While indoor pipelines do not require full open-domain generalization, enlarging the variety of supported interiors is an important goal for future research.

\paragraph*{Resolution} Most of the solutions presented for analysis (e.g., depth inference) and synthesis (e.g., renderable model generation) still work at relatively low resolutions (e.g., 512$\times$1024). This is due both to the lack of high-resolution training data available and to the complexity limitations imposed on the networks, especially when they need to work at high frequency on embedded devices. Current cameras and HMDs, however, have a significantly higher resolution (i.e., 4K), indicating a space for improvement of current methods.

\paragraph*{From single-task learning to holistic models} Very significant progress has been made on each of the individual low-level tasks, such as depth, layout, and semantics inference and object detection. At the same time, there is a clear trend to address them jointly, moving towards holistic scene understanding rather than combining individual task solutions. How to perform this combination is, however, still an open problem, both in terms of efficient network design (i.e., common or parallel branches, which architecture, ...) and loss combination (since losses may have different scales and conflicting gradients).  

\paragraph*{Cooperation between 360$^\circ$ and regular image processing} The wide context available in 360$^\circ$ images is essential for the understanding of indoor environments, with 360$^\circ$ solutions for depth and layout inference surpassing traditional constrained-FOV solutions. At the same time, many pre-trained solution exists to solve specific tasks using regular FOV images, from object detection to diffusion-based image generation. In the last few years, several combinations have been attempted, from diffusion-based floor plan generation to the conditional creation of pseudo-novel views for view synthesis. Given the increasing availability of such powerful models and the great cost for training them, we expect that the trend of combining panoramic and restricted-view processing pipelines will continue. 

\paragraph*{360$^\circ$ foundation models} foundation models are being created for regular images and videos. As for what happened with large language models (LLMs), research has shown that large, generative models trained on web-scale data have characteristics that allow them to be used not as task-specific models, but as unified, generalist foundation models that can solve a variety of analysis and synthesis tasks~\cite{Wiedemer:2025:VMZ}.  Some initial works are in progress for 360$^\circ$ imagery (\autoref{sec:foundation}), and further progress in this direction would be beneficial for all tasks requiring holistic views. Training and inference efficiency would be a concern, especially at high resolutions. Current works seem to be limited to relatively low resolution. The current prominence of very few industrial actors raises concerns about transparency, direction, and the concentration of power~\cite{Maslej:2025:AII}, and open models need coordination efforts~\cite{Bommasani:2024:CGO}.

\paragraph*{Balancing specialization and general-purpose solutions}
Large, pre-trained generic models are enabling a wide range of applications, but they are costly, energy-intensive, and demanding to both train and deploy. As their size and generality grow, such models are increasingly feasible only through substantial investments, typically accessible only to national organizations or corporations with massive computational resources. For tasks requiring broad reasoning or the generation of entirely novel data, these models may be indispensable. However, many indoor analysis and synthesis tasks operate within a narrower solution space, where leaner, task-specific models remain viable. Developing efficient, specialized approaches, ideally lightweight enough to run on embedded devices, and finding the right balance between pre-training and specialization are therefore important research goals.

\paragraph*{Uncertainty and explainability} All machine learning–based solutions discussed here produce plausible rather than strictly accurate models, inferring missing information from training data priors. As a result, assessing the uncertainty of predictions—whether depth, layout, or novel views—and explaining the rationale behind a given output, including possible alternatives, remains challenging. Recent work has begun to address these issues in the context of generative and neural scene representations (see, e.g.,~\cite{Klasson:2024:SU3,Aira:2025:MUG}). Extending these approaches to $360^\circ$ indoor imagery is promising but introduces new challenges in uncertainty visualization, such as identifying multiple plausible multi-room layouts, linking them to supporting evidence, and visualizing the alternative hypotheses.