\section{Automatic geometric and structural modeling}
\label{sec:geostruct}

The methods surveyed here all take as sole input a single $360^\circ$ image of an indoor scene or, for multi-room environments, one image per room. From this limited RGB data, the task is to infer geometric, structural, and visual properties that are not directly observable.
%
The first group of approaches focuses on pixel-level augmentation, enriching the panorama with co-registered maps such as depth, normals, or albedo. Depth estimation, in particular, is a cornerstone task, as it provides geometric cues that underpin most subsequent reconstruction and synthesis methods (\autoref{sec:geostruct:depth}).
%
A second group emphasizes structural abstraction, separating the architectural layout (walls, ceiling, floor) from contained objects. While some methods still annotate panoramas or floor-plan projections, many construct higher-level structured representations that also allow plausible reconstruction of occluded areas (\autoref{sec:geostruct:layout}).
%
Finally, we examine methods for multi-room understanding under very sparse sampling, where only one panorama per room is available and traditional multi-view pipelines fail. Here, the focus lies on recovering room layouts and inter-room connections, which are crucial for navigation and exploration (\autoref{sec:geostruct:connect}).

%===============================================================
\subsection{Depth and other pixel-wise information inference}
\label{sec:geostruct:depth}

%---------------------------------------------------------------------
\begin{table*}[t]
\centering
\small
\rowcolors{3}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|l|l|p{5.5cm}|}
\hline
\rowcolor{tblheadcolor}\textbf{Method} & \textbf{Representation} & \textbf{Backbone / Architecture} & \textbf{Key Contribution} \\
\hline
\rowcolor{tblheadcolor}\multicolumn{4}{|c|}{\textbf{Supervised distortion-aware methods}} \\
\hline
Su et al.~\cite{SU:2017:LSC} (2017) & Equirectangular & Learned spherical conv. & Distortion-aware convolution kernels \\
Cheng et al.~\cite{Cheng:2018:CPW} (2018) & Cube map & CNN encoder-decoder & Cube projection to reduce distortion \\
Tateno et al.~\cite{Tateno:2018:DAC} (2018) & Equirectangular & Multi-purpose CNN & Spherical-aware convolution kernels \\
Zioulis et al.~\cite{Zioulis:2018:ODD} (2018) & Equirectangular & Distortion-aware FCRN & Geometry-aware depth estimation \\
Su et al.~\cite{Su:2019:KTN} (2019) & Equirectangular & Kernel transformer net & Adaptive kernel learning for distortions \\
Wang et al.~\cite{Wang:2020:BFM} (2020) & Equirect.+Cubemap & Two-branch CNN (FCRN) & Dual-branch learning (BiFuse) \\
\hline
\rowcolor{tblheadcolor}\multicolumn{4}{|c|}{\textbf{Supervised structure-aware methods}} \\
\hline
Pintore et al.~\cite{Pintore:2021:SDD} (2021) & Equirectangular & Gravity-aware encoder-decoder & Indoor structural priors (SliceNet) \\
Shen et al.~\cite{Shen:2022:PPT} (2022) & Equirectangular & Panorama Transformer (PanoFormer) & ERP-transformer with spherical tangent patches \\
Li et al.~\cite{Li:2022:O3M} (2022) & Perspective patches & Transformer fusion & Sampled patches fused into panorama \\
Rey et al.~\cite{Rey:2022:3HR} (2022) & Perspective patches & CNN + fusion & Multi-resolution cubemap sampling \\
Shen et al.~\cite{Shen:2023:DOP} (2023) & Perspective patches & Patch-based ViT & Depth fusion through attention mechanisms \\
%Ai et al.~\cite{Ai:2023:HM3} (2023) & Perspective patches & Vision Transformer & Hierarchical attention over perspective patches \\
Ai et al.~\cite{Ai:2024:ETE} (2024) & Equirectangular & Transformer + bi-projection fusion & ERP-based with spherical point projection \\
Yan et al.~\cite{Yan:2025:SEP} (2025) & Equirect.+Spherical & ResNet + mesh encoder & SphereFusion with gated feature fusion \\
\hline
\rowcolor{tblheadcolor}\multicolumn{4}{|c|}{\textbf{Weakly-/Self-supervised methods}} \\
\hline
Wang et al.~\cite{Wang:2018:SSL} (2018) & Equirectangular & Distortion-aware CNN & First self-supervised framework (360-SelfNet) \\
Zioulis et al.~\cite{Zioulis:2019:SVS} (2019) & Equirectangular & CoordConv + view synthesis & Trinocular view synthesis \\
Wang et al.~\cite{Wang:2022:BSS} (2022) & Equirect.+Cubemap & BiFuse++ dual-branch CNN & Self-supervised with bi-projection fusion \\
Wang et al.~\cite{Wang:2024:DAE} (2024) & Equirectangular & Distillation framework & Knowledge transfer from perspective models \\
Wang et al.~\cite{Wang:2025:GSI} (2025) & Equirect.+Cubemap & Dual-domain collaborative CNN & Asymmetric dual-domain collaboration \\
Cao et al.~\cite{Cao:2025:PTP} (2025) & Equirectangular & Weakly-supervised CNN & Scene-structural knowledge transfer \\
\hline
\end{tabular}
\caption{\textbf{Representative methods for panoramic monocular depth estimation}. Approaches are grouped by input representation and architectural design.}
\label{tab:panoramic-depth}
\end{table*}
%---------------------------------------------------------------------
%---------------------------------------------------------------------
\begin{figure}[t]
    \centering
    {\includegraphics[width=0.9\linewidth]{img/scheme_depth.jpg}}
    \caption{\textbf{Schematic illustration of a typical encoder–transformer–decoder architecture for panoramic depth estimation.} The encoder extracts features directly from the equirectangular image or from patches of it. The final encoding layers shape the features as a feature sequence, ready to be processed by the transformer. The decoder recovers the predicted depth in equirectangular format.}
    \label{fig:ddd-arch}
\end{figure}
%---------------------------------------------------------------------
%
Pixel-wise reconstruction refers to the family of tasks aimed at recovering dense per-pixel information from a single panoramic image. These representations include depth, surface normals, semantic labels, and other descriptors that collectively provide the low-level building blocks for higher-level reconstructions such as room layouts or semantic parsing. Their relevance in the panoramic setting is amplified by the nature of $360^\circ$ imagery: while a single capture encodes the full visual field, perspective cues are heavily distorted or flattened, making the recovery of per-pixel geometric signals both indispensable and particularly challenging.  

Within this task family, monocular depth estimation has naturally emerged as the most central and widely studied problem. Depth is not only fundamental to recovering the underlying geometry of a scene, but also closely connected to the architectural structure of the scene (\autoref{sec:geostruct:layout}) and strictly required in downstream applications such as novel view synthesis (\autoref{sec:interactive:viewsynth}). Depth has thus become the pivot around which most panoramic reconstruction pipelines are built. Early approaches relied on handcrafted features, geometric constraints, and multi-scale reasoning to infer depth, but these were generally brittle in real-world conditions. With the advent of deep learning, large-scale training datasets have enabled models to extract non-trivial depth cues directly from image appearance, leading to a significant improvement in robustness and generalization~\cite{Pintore:2021:SDD}.  

It is worth noting, however, that the same architectural backbones used for depth prediction can be extended to other pixel-wise signals. For example, Eigen et al.~\cite{Eigen:2015:PDS} demonstrated that depth, surface normals, and semantic segmentation can all be jointly learned from a single image. These complementary outputs enrich both the geometric and semantic understanding of the environment: normals refine the perception of local structures, while semantics provide object-level meaning. Depending on the application, these signals often act as valuable complements to depth estimation~\cite{Lee:2025:HHP}. Their role becomes especially relevant in scenarios that extend beyond pure reconstruction, such as advanced scene editing or immersive manipulation. Concrete examples, including tasks such as emptying a complete scene, selective object removal, virtual staging, or style transfer, will be discussed in~\autoref{sec:interactive:editing}.

\paragraph*{Classic data-driven monocular depth estimation} Data-driven monocular depth estimation was introduced over a decade ago for limited field-of-view imagery, with pioneering works such as Make3D~\cite{Saxena:2009:M3D}. The emergence of deep learning and the availability of large-scale 3D datasets soon led to dramatic performance improvements. After the introduction of CNNs for regressing dense depth maps from a single image~\cite{Eigen:2014:ANI,Eigen:2015:PDS}, Laina et al.~\cite{Laina:2016:DPF} proposed the standard \emph{FCRN} encoder-decoder architecture, combining \emph{ResNet}~\cite{He:2016:DRL} for encoding, an up-projection module for decoding, and the reverse Huber loss~\cite{Lambert-Lacroix:2016:ABH} to improve depth prediction accuracy. Following these trends, many further solutions were introduced, including predicting depth from multiple perspective crops combined in the Fourier domain~\cite{Lee:2018:SID}, using ordinal regression losses to preserve spatial ordering among neighboring depth classes~\cite{Fu:2018:DOR}, or exploiting Conditional Random Fields (CRFs) for refined predictions~\cite{Liu:2015:DCN,Wang:2015:TUD,Cao:2018:EDF,Xu:2018:SAG}, among many others. While these methods were originally designed for perspective imagery, their success inspired early attempts to extend them to the panoramic domain. A common strategy was to project $360^\circ$ images into multiple perspective views, process them with pre-trained perspective networks, and then merge the results back into the equirectangular domain. Although effective as a first step, such direct adaptations did not fully exploit the unique properties of panoramic imagery, in particular, the global field of view and the associated geometric distortions, which often led to sub-optimal predictions~\cite{Zhang:PAN:2014,Zioulis:2018:ODD}. 
This limitation led to the emergence of $360^\circ$-specific approaches that explicitly model spherical geometry and global context, and now dominate research on indoor depth estimation from panoramic imagery. 
%

\paragraph*{Distortion-aware approaches}
Several solutions adapted perspective methods to 360$^\circ$ depth prediction by projecting panoramic images into cube maps~\cite{Cheng:2018:CPW} or by replacing standard convolutions with spherical/distortion-aware convolutions to cope with equirectangular distortions~\cite{SU:2017:LSC,Tateno:2018:DAC,Garanderie:2018:ETS,Zioulis:2018:ODD,Su:2019:KTN} (i.e., \emph{distortion-aware methods} -- \autoref{tab:panoramic-depth}). 
Hybrid approaches have also been proposed~\cite{Wang:2020:BFM,Jiang:2021:UUF,Huang:2025:GGP}: for instance, Wang et al.~\cite{Wang:2020:BFM} combined the two representations through a dual-branch network operating respectively on the equirectangular projection and on the cube map, coupling a distortion-aware encoder~\cite{Zioulis:2018:ODD} with the FCRN decoder~\cite{Laina:2016:DPF}. 

%%% indoor priors
\paragraph*{Structure-aware approaches}
An important line of research targets the direct processing of equirectangular panoramas with encoders that explicitly exploit the structural regularities of indoor environments (i.e., \emph{structured-aware methods} in \autoref{tab:panoramic-depth}). Representative methods include~\cite{Sun:2021:H3I,Pintore:2021:SDD,Pintore:2024:DDI,Pintore:2025:DED}, which leverage gravity-aligned features and geometry-aware constraints to reduce model complexity while still capturing both short- and long-range relations. An example is the \emph{SliceNet} architecture introduced by Pintore et al.~\cite{Pintore:2021:SDD}, which builds on the observation that man-made indoor structures are predominantly aligned with gravity, i.e., along the vertical direction. By applying a vertical compression of the equirectangular input and employing a recurrent neural network (RNN) to sequentially capture horizontal dependencies, SliceNet effectively incorporates gravity alignment into the network design. Furthermore, the feature sequence encoding and the recurrent component can be regarded as precursors to later transformer-based architectures that exploit long-range dependencies through attention mechanisms. 
Overall, such structure-aware methods demonstrate the benefits of embedding domain-specific priors inherent to indoor geometry.   

%%% transformers
\paragraph*{Transformer-based approaches}
The advent of transformer-based architectures has marked a turning point in panoramic depth estimation, introducing the ability to model long-range dependencies and global context more effectively than traditional convolutional networks. A first example is \emph{PanoFormer}~\cite{Shen:2022:PPT}, which adapts the transformer paradigm to panoramic imagery by introducing spherical tangent patches, learnable token flows, and panorama-specific metrics. 
More recent solutions increasingly leverage perspective sampling as a way to exploit the availability of powerful pretrained models on conventional imagery. Panoramic inputs are decomposed into a set of perspective views~\cite{Li:2022:O3M,Rey:2022:3HR}, which are processed independently and then fused through transformer-based architectures. 
In this direction, patch-wise vision transformers have been employed to aggregate local predictions into globally consistent depth maps~\cite{Shen:2023:DOP,Ai:2023:HM3,Ai:2024:ETE}, showing the potential of attention mechanisms to reason across spatially disjoint observations. 
%
In parallel, hybrid strategies have been explored to overcome the limitations of individual projections. SphereFusion~\cite{Yan:2025:SEP} is an example: by combining equirectangular and spherical representations through a gated fusion module, it can balance geometric faithfulness and texture detail, while also achieving state-of-the-art inference efficiency. This demonstrates the growing interest in projection-aware fusion schemes that can exploit the complementary strengths of different panoramic representations.

%%% self-supervised
\paragraph*{Self-supervised approaches}
The vast majority of existing methods for panoramic depth estimation are based on supervised learning. This creates a significant limitation, since collecting large-scale ground-truth depth data for $360^\circ$ imagery is particularly challenging, especially at high resolution. To address this, several attempts have been made to develop self-supervised or, at least, weakly supervised methods. While such approaches still lag behind fully supervised techniques in terms of accuracy, they represent a promising direction and are rapidly evolving (\autoref{tab:panoramic-depth}).  
%
Based on the cube padding strategy~\cite{Cheng:2018:CPW}, Wang et al.~\cite{Wang:2018:SSL} propose \emph{360-SelfNet}, the first framework for self-supervised $360^\circ$ depth estimation. Zioulis et al.~\cite{Zioulis:2019:SVS} adopt trinocular view synthesis to provide self-supervised depth inference. Building on this line, Wang et al.~\cite{Wang:2022:BSS} introduced \emph{BiFuse++}, extending the original BiFuse dual-branch design~\cite{Wang:2020:BFM} to self-supervised training with spherical padding and bi-projection fusion, showing that hybrid representations can also be effective even without ground-truth supervision.
%
Later, Wang et al.~\cite{Wang:2024:DAE} introduced \emph{Depth Anywhere}, a knowledge-distillation framework where a strong perspective depth estimator supervises the training of a panoramic model, enabling improved generalization across both domains. Complementary to this, Cao et al.~\cite{Cao:2025:PTP} tackled the challenge of high-resolution panoramic depth estimation in the absence of high-res annotations. Their weakly-supervised framework introduces a Scene Structural Knowledge Transfer (SSKT) module that injects structural cues at training time, thus allowing networks to recover fine-grained details without requiring dense ground-truth labels. 

\paragraph*{Overall architecture evolution}
The progression of panoramic depth estimation methods reflects a clear evolution in research focus (\autoref{tab:panoramic-depth}). Initial works mainly adapted solutions developed for conventional perspective imagery, concentrating on mitigating spherical distortions through cube map projections or distortion-aware convolutions, largely independent of scene contents. A subsequent trend has shifted towards explicitly incorporating the structural characteristics of indoor environments, with gravity alignment, layout consistency, and domain-specific priors emerging as central design choices. 
%
Overall, a typical architecture for panoramic monocular depth estimation follows an encoder–transformer–decoder scheme, as illustrated in \autoref{fig:ddd-arch}. First, an encoder extracts multi-scale features from the equirectangular input and reshapes them into a sequential representation suitable for transformer processing, mostly exploiting slicing~\cite{Sun:2021:H3I,Pintore:2021:SDD} or patching~\cite{Shen:2022:PPT,Li:2022:O3M,Ai:2023:HM3}. The transformer module then models long-range dependencies and global scene context across the entire panorama. Finally, a decoder maps the transformed features back to the equirectangular domain, producing dense per-pixel depth predictions. 
%
%------------------------------------------------------------------
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\rowcolors{2}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|}
\hline
\rowcolor{tblheadcolor}
\textbf{Method} & \textbf{Dataset} & \textbf{AbsRel} $\downarrow$ & \textbf{RMSE} $\downarrow$ & $\boldsymbol{\delta_1}$ $\uparrow$ & $\boldsymbol{\delta_2}$ $\uparrow$ & $\boldsymbol{\delta_3}$ $\uparrow$ & \textbf{GFLOPs} $\downarrow$ & \textbf{Params (M)} $\downarrow$ \\
\hline
OmniDepth~\cite{Zioulis:2018:ODD} & S2D3D & 0.200 & 0.615 & 0.688 & 0.889 & 0.958 & --- & --- \\
BiFuse~\cite{Wang:2020:BFM} & S2D3D & 0.121 & 0.414 & 0.866 & 0.958 & 0.986 & 682 & 253 \\
UniFuse~\cite{Jiang:2021:UUF} & S2D3D & 0.111 & 0.369 & 0.871 & 0.966 & 0.988 & 278 & 52 \\
SliceNet~\cite{Pintore:2021:SDD} & S2D3D & 0.100 & 0.373 & 0.904 & 0.962 & 0.984 & 101 & 79 \\
%HoHoNet~\cite{Sun:2021:HOH} & S2D3D & 0.101 & 0.383 & 0.905 & 0.969 & 0.977 & --- & --- \\
PanoFormer~\cite{Shen:2022:PPT} & S2D3D & ---   & 0.308 & 0.939 & 0.984 & 0.994 & 78 & 20 \\
OmniFusion~\cite{Li:2022:O3M} & S2D3D & 0.095 & 0.347 & 0.899 & 0.977 & 0.992 & --- & 42 \\
%SphereDepth~\cite{Yan:2022:SPD} & S2D3D & 0.116 & 0.451 & 0.867 & 0.964 & 0.986 & --- & --- \\
SphereFusion~\cite{Yan:2025:SEP} & S2D3D & \textbf{0.090} & \textbf{0.319} & \textbf{0.926} & \textbf{0.976} & \textbf{0.990} & 36 & 25 \\
\hline
OmniDepth~\cite{Zioulis:2018:ODD} & M3D & 0.290 & 0.764 & 0.683 & 0.879 & 0.943 & --- & --- \\
BiFuse~\cite{Wang:2020:BFM} & M3D & 0.205 & 0.626 & 0.845 & 0.932 & 0.963 & 682 & 253 \\
UniFuse~\cite{Jiang:2021:UUF} & M3D & 0.106 & 0.494 & 0.890 & 0.962 & 0.983 & 278 & 52 \\
SliceNet~\cite{Pintore:2021:SDD} & M3D & 0.176 & 0.613 & 0.872 & 0.948 & 0.972 & 101 & 79 \\
%HoHoNet~\cite{Sun:2021:HOH} & M3D & 0.149 & 0.514 & 0.879 & 0.952 & 0.977 & --- & --- \\
PanoFormer~\cite{Shen:2022:PPT} & M3D & ---   & 0.364 & 0.918 & 0.980 & 0.992 & 78 & 20 \\
OmniFusion~\cite{Li:2022:O3M} & M3D & 0.090 & 0.426 & 0.919 & 0.980 & 0.993 & --- & 42 \\
%SphereDepth~\cite{Yan:2022:SPD} & M3D & 0.121 & 0.592 & 0.862 & 0.952 & 0.977 & --- & --- \\
SphereFusion~\cite{Yan:2025:SEP} & M3D & \textbf{0.115} & \textbf{0.489} & \textbf{0.870} & \textbf{0.961} & \textbf{0.984} & 36 & 25 \\
\hline
\end{tabular}
\caption{\textbf{Accuracy and performance comparison for $360^\circ$ monocular depth estimation.} 
Results are reported on Stanford-2D-3D (S2D3D) and Matterport3D (M3D). 
values provided by SphereFusion~\cite{Yan:2025:SEP} official comparisons.
Metrics: lower is better for AbsRel, RMSE, GFLOPs; higher is better for $\delta$ thresholds. 
Computational costs (GFLOPs, Params) are taken from the original papers when available.}
\label{tab:panoramic-depth-benchmark}
\end{table*}
%
While supervised approaches still deliver the highest accuracy, there is a growing research effort in self- and weakly-supervised learning. These approaches, though less mature, are particularly promising in light of recent advances in foundation models, which may provide new pathways to reduce reliance on large-scale annotated data (\autoref{sec:foundation}). 

\paragraph*{Common metrics}
The evaluation of panoramic depth estimation still relies predominantly on standard metrics inherited from perspective-based depth regression since the introduction of FCRN, such as \textit{AbsRel}, \textit{RMSE}, and \textit{$\delta$ thresholds}~\cite{Eigen:2014:ANI,Laina:2016:DPF}. 
%
Specifically, the most common metrics include:  
\begin{itemize}
    \item \textbf{Absolute Relative Error (AbsRel)}: the average of the ratio between the absolute depth error and the ground-truth depth. It emphasizes proportional errors across depth ranges.  
    \item \textbf{Root Mean Squared Error (RMSE)}: the square root of the mean squared differences between predicted and ground-truth depth values, highlighting larger errors.  
    \item \textbf{Mean Relative Error (MRE)}: similar to AbsRel but normalized by predicted values, providing a complementary perspective on relative error magnitudes.  
    \item \textbf{$\delta$ thresholds}: the percentage of predicted depth values $d_p$ that satisfy 
    $
    \max\left(\frac{d_p}{d_{gt}}, \frac{d_{gt}}{d_p}\right) < \delta,
    $
    with typical thresholds $\delta = 1.25, 1.25^2, 1.25^3$. These metrics quantify prediction accuracy within multiplicative error bounds.  
\end{itemize}

It is important to note that while these metrics have become standard in monocular depth evaluation, their behavior may not be equally suitable across different domains. In indoor panoramic environments, where depth ranges are relatively short and errors at small distances can dominate the evaluation, \textit{AbsRel} tends to overweight inaccuracies in near-field geometry. Conversely, \textit{MRE} shifts the normalization to predicted values, which can mitigate this effect but may underestimate large relative errors at greater depths. This suggests that, despite their widespread adoption, a more tailored set of metrics could be beneficial for faithfully capturing reconstruction quality in panoramic indoor scenarios.

\paragraph*{Training objectives}  
The effectiveness of panoramic depth estimation methods strongly depends on the choice of loss functions employed during training. Early deep learning models for monocular depth estimation typically relied on the \emph{reverse Huber loss} (also known as the \emph{BerHu loss})~\cite{Laina:2016:DPF}, which balances sensitivity to small residuals with robustness to outliers. This formulation rapidly became a standard in indoor depth estimation, and remains at the core of several panoramic baselines such as \emph{OmniDepth}~\cite{Zioulis:2018:ODD} and \emph{360SD-Net}~\cite{Wang:2020:BFM}.  
Subsequent works have progressively enriched the objective functions with multi-scale supervision and structural regularizers. For instance, \emph{BiFuse}~\cite{Wang:2020:BFM} and \emph{UniFuse}~\cite{Jiang:2021:UUF} combine BerHu loss with smoothness penalties that encourage spatial consistency across neighboring pixels, while \emph{SliceNet}~\cite{Pintore:2021:SDD} further incorporates geometry-aware constraints aligned with indoor structural priors.  
With the advent of transformer-based architectures, methods such as \emph{PanoFormer}~\cite{Shen:2022:PPT} and \emph{Elite360D}~\cite{Ai:2024:ETE} have emphasized hybrid losses that combine point-wise regression with edge-aware or semantic-aware regularization, improving the fidelity of structural boundaries.  
More recently, \emph{SphereFusion}~\cite{Yan:2025:SEP} extends this trend by adopting distance-aware objectives that adapt the penalty according to depth ranges, leading to improved accuracy in large-scale indoor scenes.  
Overall, the evolution of training objectives mirrors the architectural trends: from simple pixel-wise regression to more sophisticated, context- and geometry-aware formulations, designed to reduce artifacts and enhance structural consistency in panoramic depth maps.
%
On the other hand, in the self- and weakly-supervised setting, the design of losses is even more critical, as ground-truth depth is not available.  
% Added EG
MIDAS~\cite{Ranftl:2022:TRM} propose to perform prediction in disparity space (inverse depth up to scale and shift) through a family of scale- and shift-invariant dense losses to handle ambiguities stemming from the merging of datasets acquired in the wild. Originally developed for perspective cameras, the method has also been employed for 360$^\circ$ depth estimation~\cite{Yang:2024:DAV,Jiang:2025:DA3}.
%
A common alternative strategy is the use of photometric reconstruction losses that compare synthesized views reconstructed from the predicted depth to the input panorama~\cite{Wang:2018:SSL}. Zioulis et al.~\cite{Zioulis:2019:SVS} extend this principle with a trinocular view synthesis loss, leveraging multiple synthesized perspectives to provide stronger self-supervision. Wang et al.~\cite{Wang:2022:BSS} introduce \emph{BiFuse++}, combining spherical padding and bi-projection fusion with reprojection-based losses, while more recent frameworks further refine this design. For example, Wang et al.~\cite{Wang:2025:GSI} propose an asymmetric dual-domain collaboration scheme, enforcing consistency across equirectangular and cubemap predictions, with geometry-aware self-supervised losses.  
Finally, weakly-supervised methods such as the one proposed by Cao et al.~\cite{Cao:2025:PTP} introduce structural knowledge-transfer modules, where geometric cues (e.g., room layout consistency) act as priors to constrain depth predictions without requiring dense annotations.  
%
Nonetheless, supervised methods remain dominant in terms of absolute accuracy, but the sophistication of loss functions in self-supervised learning is rapidly narrowing the gap.

\paragraph*{Representative methods performances}  
The comparative results in \autoref{tab:panoramic-depth-benchmark} highlight several consistent trends in $360^\circ$ monocular depth estimation. The reported performance values are taken from SphereFusion~\cite{Yan:2025:SEP} experiments, while computational stats were extracted from the original papers.
In terms of evaluation, nearly all methods are benchmarked with the same set of standard metrics inherited from perspective-based monocular depth estimation~\cite{Eigen:2014:ANI,Laina:2016:DPF}, namely \textit{AbsRel}, \textit{RMSE}, and the $\delta$ thresholds. These metrics allow for comparability across different approaches, but they remain agnostic to panoramic-specific distortions and structural regularities, which suggests that more tailored evaluation protocols may be beneficial.  
%
From a performance perspective, early \emph{distortion-aware} methods such as OmniDepth~\cite{Zioulis:2018:ODD} and BiFuse~\cite{Wang:2020:BFM} achieve reasonable accuracy but at the cost of higher residual errors, especially in challenging indoor scenarios. The introduction of \emph{structure-aware} methods, such as SliceNet~\cite{Pintore:2021:SDD}, brought noticeable improvements by embedding indoor geometry priors (e.g., gravity alignment), leading to better consistency and reduced computational overhead.  
%
The most recent generation of \emph{transformer-based} approaches, such as PanoFormer~\cite{Shen:2022:PPT} and SphereFusion~\cite{Yan:2025:SEP}, further advances the state of the art by modeling long-range dependencies and combining multiple projection domains. These architectures improve accuracy (with SphereFusion achieving the best balance across AbsRel, RMSE, and $\delta$ scores) while reducing computational cost, as reflected in their GFLOPs and parameter counts. Overall, this trend marks a shift from purely distortion-correcting networks to scalable structure- and context-aware architectures.
%===============================================================
\subsection{Single-image room layout estimation}
\label{sec:geostruct:layout}
%
\begin{figure*}[t]
    \centering
    {\includegraphics[width=0.9\linewidth]{img/redesigned-dong2024-panocontextformer.jpg}}
    \caption{\textbf{Illustration of a panoramic scene understanding pipeline}~\cite{Dong:2024:PPT}. The diagram highlights how depth estimation and layout parsing from a single panoramic image represent the two fundamental tasks for reconstructing a complete scene. The layout provides the structural backbone of the environment, enabling the placement of object instances and openings, while depth maps supply localized point clouds that allow for the modeling of individual objects. Once this pipeline is established, each object can be reconstructed using standard modeling techniques, largely independent of the panoramic indoor context.}
    \label{fig:scene_understanding}
\end{figure*}
%
Depth estimation is fundamental for panoramic scene modeling, providing the dense metric basis for reconstructing indoor environments. When the viewpoint is fixed,  a single panoramic depth map can support applications such as scene emptying, refurnishing, or style transfer (\autoref{sec:interactive}). More generally, however, it is necessary to separate permanent architectural structures (the \emph{layout}) from movable objects or clutter~\cite{Pintore:2020:SA3}. Layout reconstruction thus extracts a higher-level scene representation, typically consisting of floors, ceilings, walls, and connecting doors.

\paragraph*{Pixel-wise layout mapping}
In cases of small or null viewpoint motion, the separation between indoor layout and objects can be effectively performed at the pixel level, and several works in the literature adopt depth-guided segmentation masks that partition the equirectangular view into cluttered and uncluttered regions~\cite{Jin:2020:GSB,Xu:2021:LGN,Gkitsas:2021:PSP,Pintore:2022:IAE,Pintore:2025:PRI}, typically predicted using relatively simple encoder–decoder networks such as U-Net. A representative example is the work of Jin et al.~\cite{Jin:2020:GSB}, where layout masks are learned jointly with depth estimation. Their training strategy leverages specialized datasets that provide paired reconstructions of the same rooms in both furnished and empty configurations (\autoref{sec:background:data}), thus enabling the network to disentangle permanent structures from movable objects (\autoref{sec:background:data}).  

\paragraph*{Indoor-structured layout}
In more general contexts, pixel-wise masks alone are insufficient. 
Full layout estimation is significantly more complex than assigning one value per pixel, as for depth estimation, because it requires extrapolating occluded or invisible surfaces and reasoning about global structure beyond the directly observable data.  
%
The difficulty is accentuated in indoor panoramic capture, where occlusions caused by furniture and self-occluding concave geometries mean that large portions of the true structure are hidden from direct observation (\autoref{fig:background:indoor-occlusion-and-reconstriction}). Thus, layout reconstruction must plausibly hallucinate missing geometry, introducing a strong dependence on prior knowledge and structural constraints. 
%
To this end, a \emph{structured scene reconstruction} is required, where the separation between permanent architectural structures (layout) and movable objects is performed in a more robust and semantically informed manner~\cite{Ikehata:2015:SIM,Pintore:2020:SA3}. 
In the specific case of panoramic indoor scenes, this process leverages the global context provided by the single panoramic observation~\cite{Zhang:PAN:2014}, following pipelines such as the one illustrated in \autoref{fig:scene_understanding}. Within this framework, existing panoramic works~\cite{Zhang:PAN:2014,Zhang:2021:DPC,Dong:2024:PPT,Lee:2025:HHP} decompose the problem into two core tasks: estimating depth and predicting the layout from the input panorama. The layout serves as the structural scaffold of the environment, enabling the placement of object instances, including openings, while depth estimation provides point clouds for modeling individual objects. Once this pipeline is established, the objects themselves can be reconstructed using standard techniques largely independent of the panoramic setting. For this reason, most research in the panoramic domain has focused on depth estimation (as discussed in the previous section) and layout estimation.


%
%---------------------------------------------------------------------
\begin{table*}[!htbp]
\centering
\small
\rowcolors{2}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|c|c|c|c|c|}
\hline
\rowcolor{tblheadcolor}
\textbf{Method} & \textbf{Input} & \textbf{Pre-proc.} & \textbf{Backbone} & \textbf{Post-proc.} & \textbf{Output} \\
\hline
%Hedau09~\cite{Hedau:2009:RSL} & Persp. & MW align & Geom. reasoning, cuboid prior & Yes & Cuboid model \\
%Lee09~\cite{Lee:2009:GRF} & Persp. & MW align & IWM + corner reasoning & Yes & Struct. 3D layout \\
Zhang14~\cite{Zhang:PAN:2014} & ERP & OM + GC & Geom. reasoning & Yes & Cuboid layout \\
LayoutNet~\cite{Zou:2018:LNR} & ERP & MW+lines & CNN (corner/bound.) & Yes & 2D prob. maps \\
HorizonNet~\cite{Sun:2019:HNL} & ERP & MW align & CNN+RNN (1D scanline) & Yes & 1D vectors \\
DuLa-Net~\cite{Yang:2019:DNE} & ERP+ceil & MW align & Dual CNN (ERP+top) & Yes & Floor-plan map \\
AtlantaNet~\cite{Pintore:2020:AI3} & Ceil+floor & None & CNN proj. planes & Light & 3D layout (AWM) \\
Deep3DLayout~\cite{Pintore:2021:D3R} & ERP & None & Graph NN & None & Watertight mesh \\
SSLayout360~\cite{Tran:2021:SSI} & ERP & MW align & Semi-sup. CNN & Yes & Corners/bound. \\
LGT-Net~\cite{Jiang:2022:LGT} & ERP & MW align & SWG-Transformer & Light & 1D seq. \\
DMH-Net~\cite{Zhao:2022:3RL} & Cubemap & MW align & CNN+Hough space & Yes & Line-based 3D \\
DOP-Net~\cite{Shen:2023:DOP} & ERP & MW align & Plane-disent. CNN & Yes & Plane-wise feat. \\
PanelNet~\cite{Yu:2023:PU3} & Panels & None & Panel-wise CNN & Yes & Cont. panels \\
Bi-Layout~\cite{Tsai:2024:NMA} & ERP & MW align & Dual embed.+guidance & Yes & 2 layouts \\
Seg2Reg~\cite{Sun:2024:SD2} & ERP & None & CNN+occl.-aware render & Min. & 1D depth \\
HUSH~\cite{Lee:2025:HHP} & ERP & None & SH-Transformer & None & Multi-task (D/N/L) \\
\hline
\end{tabular}
\caption{\textbf{Comparison of representative panoramic layout estimation methods.} 
Acronyms: ERP = Equirectangular Projection, Persp. = Perspective, Ceil = Ceiling plane projection, Floor = Floor plane projection, MW = Manhattan World, IWM = Indoor World Model, AWM = Atlanta World Model}
\label{tab:layout_methods}
\end{table*}
%---------------------------------------------------------------------

%%%layout related
\paragraph*{Geometric reasoning approaches}
Early approaches to indoor layout estimation exploited the strong regularities of man-made environments, relying on geometric reasoning to align image features with simplified constrained 3D models. Hedau et al.~\shortcite{Hedau:2009:RSL} pioneered the use of a \emph{cuboid prior} for pixel labeling, while Lee et al.~\shortcite{Lee:2009:GRF} exploited the Manhattan-World Model (MWM) to infer 3D structures from detected corners. 
%
Moving to panoramic imagery, Zhang et al.~\shortcite{Zhang:PAN:2014} were among the first to leverage $360^{\circ}$ captures to overcome the contextual limitations of narrow field-of-view images. Their method mapped an entire panorama to a cuboid room model by combining \emph{Orientation Maps} (OM)~\cite{Lee:2009:GRF} for the upper hemisphere and a \emph{geometric context} (GC) analysis~\cite{Hoiem:2007:RSL} for the lower part. Xu et al.~\shortcite{Xu:2017:P2C} later extended this reasoning to the IWM, while Yang et al.~\shortcite{Yang:2016:E3R} generalized to MWM layouts using partially oriented super-pixel facets and line segments. Several follow-ups~\cite{Pintore:2020:SA3} adopted similar strategies, though the effectiveness of these purely geometric methods strongly depends on the quantity and quality of extracted features (e.g., corners, edges, planar patches). This limitation has motivated a progressive shift towards data-driven approaches~\cite{Zou:2019:3DM}.

\paragraph*{Hybrid data-driven approaches}
As highlighted by Zou et al.~\shortcite{Zou:2021:MRL}, most data-driven layout pipelines still follow a similar structure: (i) a pre-processing stage, often involving MWM alignment (e.g., based on~\cite{Zhang:PAN:2014}); (ii) prediction of layout elements in image space; and (iii) post-processing to regularize a 3D model from the predicted 2D layout elements.
%
Representative examples include \emph{LayoutNet}~\cite{Zou:2018:LNR}, which directly predicts corner and boundary probability maps from panoramas, and \emph{HorizonNet}~\cite{Sun:2019:HNL}, which encodes the layout as three 1D vectors representing, column-wise, the floor-wall, ceiling-wall, and wall-wall boundaries. The final layout is then recovered by fitting MWM segments to the predicted corners. \emph{DuLaNet}~\cite{Yang:2019:DNE} combines features from the equirectangular panorama and a ceiling-plane projection to produce a floor-plan probability map, which is regularized under MWM assumptions. Several recent works have extended HorizonNet, including \emph{Led$^{2}$Net}~\cite{Wang:2021:LM3}, which leverages rendered horizon depth maps to improve predictions, achieving state-of-the-art performance under IWM constraints. Other methods jointly exploit depth, layout, and semantics to refine their predictions. For instance, Zeng et al.~\shortcite{Zeng:2020:J3L} estimated a layout depth map by integrating semantic segmentation and full-scene depth cues to regularize the IWM layout.

\paragraph*{Geometry-aware data-driven approaches}
Despite their effectiveness, many of these methods rely on heavy pre-processing, such as vanishing line detection for Manhattan-world alignment~\cite{Zou:2019:3DM,Zhang:PAN:2014,Lee:2009:GRF}, or on complex post-processing steps to enforce structural regularity~\cite{Zou:2018:LNR,Sun:2019:HNL,Yang:2019:DNE}. \emph{AtlantaNet}~\cite{Pintore:2020:AI3} addressed these limitations by relaxing the strict MWM/IWM assumptions: given panoramas roughly aligned with gravity, it estimates layouts under the more flexible Atlanta World Model (AWM), by projecting the spherical image onto both horizontal planes (floor and ceiling). Gravity-aligned panoramas are common across all major datasets (\autoref{sec:background:data}), and alignment can be efficiently recovered in preprocessing when missing (\autoref{sec:background:gravity-alignment}).  

Overall, while restrictive priors such as MWM, IWM, or AWM enable simpler formulations and robust constraints, they inherently limit the expressivity of inferred layouts.
Several efforts have attempted to expand the solution space. Graph-based neural networks have been proposed to directly infer watertight 3D meshes of room layouts (\emph{Deep3Dlayout}~\cite{Pintore:2021:D3R}), moving beyond predefined priors toward more generalizable representations. This line of work reflects a broader shift: from rigid constraint-based formulations to flexible data-driven architectures capable of handling clutter, irregular geometries, and diverse architectural patterns. However, recovering back a structured layout (e.g., ceiling, walls, floor) from the predicted 3D mesh is not immediate, so research still follows direct regression of the layout. 
%
In parallel, \emph{SSLayout360}~\cite{Tran:2021:SSI} showed that semi-supervised signals (corners/boundaries learned from mixtures of labeled and unlabeled panoramas) can approach fully supervised performance with far fewer labels—useful for domains where dense GT is costly.
%

\paragraph*{Transformers with geometry awareness}
\emph{LGT-Net}~\cite{Jiang:2022:LGT} proposed a geometry-aware transformer (SWG-Transformer) tailored to 360 layouts, capturing long-range dependencies across the horizontal scanline while respecting Manhattan constraints. LGT-Net demonstrated that self-attention over the compressed 1D sequence is effective for both simple and complex topologies; it remains a common, high-performing reference architecture and a building block for later methods.
%Cubemap and learnable Hough reasoning 
\emph{DMH-Net}~\cite{Zhao:2022:3RL} converts panoramas to cubemaps and maps features into a \emph{learnable Manhattan Hough space}, detecting semantic lines and assembling them into 3D layouts. Working in line/Hough space improves robustness to occlusions and long straight structures, and an optimization-based post-process enforces global consistency. The trade-off is added complexity (projection, multi-face fusion) and sensitivity to non-Manhattan cases.
%
%Plane disentangling and alternative 360 representations
\emph{DOP-Net}~\cite{Shen:2023:DOP} explicitly disentangles features into orthogonal plane streams (floor/ceiling/walls), with cross-scale distortion awareness and triple attention to improve 3D reasoning. By structuring features around planes, it yields strong 3D IoU and clearer interpretability than purely 1D heads. 
Beyond classic horizon-depth, \emph{PanelNet}~\cite{Yu:2023:PU3} models the 360 image as continuous horizontal panels, a representation designed to leverage the panorama’s seamless azimuth continuity; while broader than layout alone, it indicates a trend toward representations that better respect spherical-image structure.
%Disambiguating annotations with bi-layout prediction
To directly address ambiguous supervision/evaluation, \emph{Bi-Layout}~\cite{Tsai:2024:NMA} predicts \emph{two} complementary layouts per image—an \emph{enclosed} one that stops at ambiguous openings and an \emph{extended} one that spans visible adjacent areas—via two global context embeddings and a shared guidance module that conditions the image feature on the target layout type. A ``disambiguate'' metric then selects the best-matching hypothesis against GT, improving both overall accuracy and particularly the hardest, ambiguity-dominated subsets. This line suggests a pragmatic path forward: rather than collapsing ambiguity into a single target, learn multiple valid hypotheses with lightweight specialization and evaluate accordingly.
%
\emph{Seg2Reg}~\cite{Sun:2024:SD2} reconciles 2D segmentation and 1D regression: while the former is easier to learn and captures occlusions but needs heavy post-processing, the latter yields better geometric consistency. Seg2Reg predicts a 2D \emph{floor-plan density} on the ERP panorama and converts it into 1D layout-depth signals through an \emph{occlusion-aware flattened volume rendering} module, combining dense supervision with stable training and accurate geometry. It further introduces a \emph{3D warping augmentation} and a unified re-implementation of strong baselines, consistently outperforming LED2-Net and horizon-depth pipelines by reducing the mismatch between training (2D) and evaluation (1D/3D).  
%%%example of multi-signal
More recently, \emph{HUSH}~\cite{Lee:2025:HHP} instead aligns representation and geometry using \emph{spherical harmonics} (SH) as basis functions on the unit sphere. Scene-adaptive SH coefficients serve as structured queries in a hierarchical attention module fused with image features, and an SH indexing module selects task-relevant bases for multi-head outputs (depth, normals, layout). This SH-driven inductive bias improves geometric consistency, achieves state-of-the-art depth, and remains competitive on normals and layout, offering a distortion-robust multi-task alternative to horizon-depth and transformer-only approaches. Multi-task loss combination is a simple static weighted sum of the individual task losses.

\paragraph*{Overall architecture evolution}
As summarized in \autoref{tab:layout_methods}, existing approaches differ significantly in terms of input representation (e.g., ERP, cubemap, or hybrid views), reliance on geometric priors (MW/IWM/AWM), and network design (from CNNs to graph-based and transformer architectures). 
Classical methods depend heavily on pre- and post-processing to enforce structural constraints, while recent data-driven solutions such as Deep3DLayout, Seg2Reg and HUSH reduce this gap by integrating geometry-awareness directly into the learning process. 
Overall, the trend shows a gradual shift from rigid cuboid/MW assumptions toward more flexible, geometry-aligned representations capable of handling complex room shapes and supporting multi-task predictions.

\paragraph*{Layout completion}
%
Once a panoramic depth map or point cloud and a consistent structural layout are available, the reconstruction of scene contents (i.e., object instances, their placement, and shapes) can be obtained by adapting existing \emph{scene understanding} techniques originally developed for perspective
images. Such pipelines typically rely on object detection, 3D box fitting, and CAD-model retrieval or implicit shape representations, and can be straightforwardly applied to the panoramic setting
after suitable geometric adaptation.  
%
It is important to stress that the recovery and detailed reconstruction of individual objects constitutes a major research topic on its own, involving challenges such as fine-grained geometry
and semantics. While panoramic context provides valuable cues for these tasks, they fall beyond the scope of the present survey, which concentrates on the layout reconstruction problem as the structural backbone for subsequent scene modeling.

% --------------------------------------------------------
In this context, 3D object detection approaches ~\cite{Song:2016:SSC, Qi:2018:FRN} aimed to infer 3D bounding boxes and object poses from 2D image representations, typically through a 2D detection stage~\cite{Ren:2015:FRT}.  
For object reconstruction, early pipelines matched CAD models from large repositories (e.g., ShapeNet) to detected 2D proposals~\cite{Chang:2015:SNI, Tulsiani:2018:MP3D}, while more recent methods demonstrated that \emph{implicit neural representations} outperform grid-, point-, or mesh-based encodings in capturing geometry and in learning shape priors~\cite{Mescheder:2019:ONL, Sitzmann:2020:IFR}.
%
A second line of research has explored \emph{joint learning} of multiple tasks to exploit contextual dependencies between layout and objects. For instance, CooP~\cite{Huang:2018:CHS} introduced a
cooperative training scheme for object poses and indoor layouts, while Total3D~\cite{Nie:2020:T3U} was the first to address layout estimation, object detection/pose, and object reconstruction jointly from a single
view. Follow-up works further improved reconstruction fidelity by coupling implicit functions with graph-based reasoning~\cite{Zhang:2021:DPC, Liu:2022:THF}.  
Despite their progress, these methods operated exclusively on perspective images, which inherently limit contextual coverage.

The introduction of panoramic imagery enabled holistic parsing of indoor scenes. PanoContext~\cite{Zhang:PAN:2014} pioneered the use of $360^{\circ}$ panoramas for layout estimation, while subsequent works~\cite{Zhang:2021:DPC} leveraged hybrid frameworks combining image features with scene graphs to jointly estimate layouts, object shapes, and poses from a single panorama. More recently,
PanoContext-Former~\cite{Dong:2024:PPT} proposed to directly lift a panoramic image into a volumetric 3D representation and to employ a transformer-based context module, which models global interactions
between layout and object tokens for holistic panoramic scene understanding.

%{Transformers in scene understanding.}
Transformers have become the dominant backbone across many domains, initially in NLP and later in vision
tasks with the Vision Transformer (ViT)~\cite{Dosovitskiy:2020:AII}.
Numerous hybrid CNN–Transformer architectures~\cite{Liu:2021:STH, Touvron:2021:TDI} have since been proposed to capture both local and global context efficiently.  
In multimodal learning, CLIP~\cite{Radford:2021:LTV} jointly trained vision and text encoders, framing image classification as text retrieval. CLIP uses a contrastive learning approach that, in a pretraining base, learns a multi-modal embedding space by jointly training an image
encoder and text encoder to maximize the cosine similarity of the image and text embeddings of the $N$ correct pairs in the batch while minimizing the cosine similarity of the embeddings of the $N^2-N$ incorrect pairings. The encoder is then specialized per task in the downstream phase. By contrast, Hu and Singh~\cite{Hu:2021:MMT} exploited pre-trained language models~\cite{Raffel:2020:ELT,Lewis:2019:BDS} within a transformer encoder-decoder architecture~\cite{Vaswani:2017:AIA} to encode visual and text inputs and generate label text. For both pretraining and downstream tasks, the model parameters are optimized by minimizing the negative log-likelihood of label text tokens given an input text and an image. In 3D scene understanding, transformers have been successfully applied to fuse object- and point-level features, as in Group-Free~\cite{Liu:2021:G3O} and Mask3D~\cite{Schult:2023:MT3} for 3D detection and semantic segmentation. PQ-Transformer~\cite{Chen:2022:PJP} and AnchorRec~\cite{Dong:2023:SAG} further addressed joint 3D object and layout prediction from point clouds, though with limitations in handling complex non-Manhattan structures. Conditional object queries~\cite{Wang:2022:BTF} have also been used to fuse point cloud and image features for robust 3D detection.  
%
%===================================
\begin{table}[t]
\centering
\small
\rowcolors{2}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|c|p{3.3cm}|}
\hline
\rowcolor{tblheadcolor}
\textbf{Method} & \textbf{Input} & \textbf{Key Idea / Contribution} \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{3}{|c|}{\textbf{3D Detection / Reconstruction}} \\
\hline
Song16~\cite{Song:2016:SSC} & Persp. & Sliding-shape 3D boxes \\
Qi18~\cite{Qi:2018:FRN} & Persp. & Frustum PointNets \\
Chang15~\cite{Chang:2015:SNI} & Persp. & ShapeNet CAD retrieval \\
Tulsiani18~\cite{Tulsiani:2018:MP3D} & Persp. & Multi-view supervision \\
Mescheder19~\cite{Mescheder:2019:ONL} & Persp. & Implicit occupancy networks \\
Sitzmann20~\cite{Sitzmann:2020:IFR} & Persp. & SIREN implicit functions \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{3}{|c|}{\textbf{Joint Layout + Objects}} \\
\hline
CooP~\cite{Huang:2018:CHS} & Persp. & Coop. training layout+poses \\
Total3D~\cite{Nie:2020:T3U} & Persp. & Joint layout, boxes, reconstr. \\
Zhang21~\cite{Zhang:2021:DPC} & Persp. & Implicit + graph reasoning \\
Liu22~\cite{Liu:2022:THF} & Persp. & Hi-fi holistic implicit recon. \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{3}{|c|}{\textbf{Panoramic Holistic}} \\
\hline
PanoContext~\cite{Zhang:PAN:2014} & ERP & First $360^\circ$ layout parsing \\
DeepPanoContext~\cite{Zhang:2021:DPC} & ERP & Image+scene graph joint est. \\
PC-Former~\cite{Dong:2024:PPT} & ERP & Volumetric lift + transformer \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{3}{|c|}{\textbf{Transformer-based}} \\
\hline
ViT~\cite{Dosovitskiy:2020:AII} & RGB & Patch transformer backbone \\
Swin~\cite{Liu:2021:STH} & RGB & Shifted-window transformer \\
CLIP~\cite{Radford:2021:LTV} & RGB+Txt & Vision--language pretraining \\
Group-Free~\cite{Liu:2021:G3O} & Pcloud & Obj/point fusion for det. \\
Mask3D~\cite{Schult:2023:MT3} & Pcloud & Transformer 3D segmentation \\
PQ-Transf.~\cite{Chen:2022:PJP} & Pcloud & Joint objects + layouts \\ AnchorRec~\cite{Dong:2023:SAG} & Pcloud & Anchor-based reconstr. \\
Cond.Queries~\cite{Wang:2022:BTF} & RGB+Pcl & Multimodal det. w/ queries \\
\hline
\end{tabular}
\caption{\textbf{Summary of layout completion and holistic scene understanding methods for panoramic imaging.} 
Grouped into detection/reconstruction, joint layout+objects, panoramic approaches, and transformer-based frameworks.}
\label{tab:layout_completion_compact}
\end{table}
%===============================
%
Overall, transformers offer clear advantages in modeling contextual relationships across modalities and tasks, and represent the current state-of-the-art backbone for 360$^\circ$ holistic scene understanding.
%

Unlike depth estimation or layout recovery, where benchmarks and metrics such as 2D/3D IoU or RMSE are relatively standardized, the evaluation of holistic scene understanding methods remains heterogeneous. Different works often adopt distinct datasets or emphasize different components of the task—whether layout fidelity, object detection accuracy, or reconstruction quality—making direct comparisons less straightforward. Importantly, these methods are not an end in themselves but serve as functional complements to layout estimation, enabling the reconstruction of scene contents when such detail is required. 
%
\paragraph*{Evaluation metrics}
Most panoramic layout estimation works report performance using a combination of 2D- and 3D-based measures, including:
%
\begin{itemize}
    \item \textbf{2D IoU}: overlap between the predicted and ground-truth floor-plan, usually in bird’s-eye view.
    \item \textbf{3D IoU}: volumetric consistency between the predicted and ground-truth 3D room geometry.
    \item \textbf{Corner error}: Euclidean or angular distance between predicted and ground-truth layout corners.
    \item \textbf{Boundary accuracy}: pixel-level error rate of predicted structural boundaries in ERP space.
\end{itemize}
%
These metrics jointly capture both the global structural fidelity and the local geometric accuracy of layout predictions.
%
In contrast to layout estimation, where evaluation is usually based on standardized geometric metrics, object-level recovery and holistic scene understanding require additional objectives tailored to detection and reconstruction. 
Common objective functions employed across these approaches include:
\begin{itemize}
    \item \textbf{Cross-entropy / focal loss} – for semantic segmentation and classification tasks.  
    \item \textbf{L1/L2 regression losses} – for depth prediction, layout parameters, and bounding box regression.  
    \item \textbf{Chamfer Distance (CD)} – for aligning reconstructed and ground-truth 3D shapes.  
    \item \textbf{Earth Mover’s Distance (EMD)} – for fine-grained shape alignment in point-based reconstructions.  
    \item \textbf{Adversarial losses} – to encourage realism in generative or implicit reconstructions.  
    \item \textbf{Perceptual losses} – leveraging pretrained networks to preserve high-level structural features.  
\end{itemize}
%
\paragraph*{Training objectives}
In terms of loss design, most methods optimize combinations of pixel- or feature-level supervision signals. 
\emph{Cross-entropy loss} is typically adopted for corner or boundary probability maps, while \emph{L1/L2 regression losses} are used for continuous predictions such as scanline heights or depth values. 
Several methods incorporate \emph{geometric consistency losses}, enforcing alignment between predicted corners, boundaries, or floor-plan projections and their ground-truth counterparts. 
Recent works also employ \emph{auxiliary depth- or semantic-guided losses} to improve structural reasoning, or introduce \emph{regularization terms} that enforce Manhattan/Atlanta alignment constraints. 
When semi-supervised or weak supervision is considered, \emph{consistency losses} (e.g., between ERP and cubemap predictions, or across multiple layout hypotheses) are additionally used to stabilize training.
%
%layout completion training
On the other hand, training objectives for object-level recovery in panoramic scenes largely follow standard practices from point-cloud-based reconstruction and detection. Typical formulations combine cross-entropy or focal losses for semantic components with L1/L2 regression for depth and bounding boxes, while Chamfer Distance and Earth Mover’s Distance are used to align predicted and ground-truth shapes. Adversarial or perceptual losses may be added to enhance realism. Overall, these objectives mirror established pipelines in generic 3D scene understanding and thus fall outside the specific panoramic focus of this survey, serving primarily as complementary tools for extending layout estimation into full scene modeling. 

\paragraph*{Multiple objectives and multi-task combination}
Most of the multi-task pipelines surveyed, as well as single-task pipelines that integrate multiple losses or regularization terms, aggregate objectives using a statically weighted sum. While fixed weights are conceptually simple, selecting effective values often relies on heuristics or exhaustive grid search~\cite{Sener:2018:MLM}. Challenges arise not only from differences in loss magnitudes due to units or scales~\cite{Kendall:2018:MLU}, but also from heterogeneity across task types (e.g., classification vs. regression)~\cite{Dai:2023:IGB} and from conflicting gradient directions between objectives~\cite{Liu:2021:CDG}. When such issues are ignored, some task-specific losses may increase, harming individual task performance even as the overall loss decreases. To address these limitations, researchers have proposed dynamic reweighting, multi-objective optimization, and gradient combination strategies~\cite{Chen:2018:GGN,Yu:2020:GSM,Lee:2021:ICCV}, which offer promising directions for application to 360$^\circ$ pipelines.

%----------------------------------------------------------------
\begin{table}[t]
\centering
\small
\rowcolors{2}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|c|c|c|}
\hline
\rowcolor{tblheadcolor}
\textbf{Method} & \textbf{Backbone} & \textbf{2D IoU(\%)} & \textbf{3D IoU(\%)}  \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{4}{|c|}{\textbf{(a) MatterportLayout test set results}} \\
\hline
LayoutNet v2~\cite{Zou:2021:MRL} & ResNet-34 & 78.73 & 75.82 \\
DuLaNet v2~\cite{Zou:2021:MRL} & ResNet-50 & 78.82 & 75.05 \\
HorizonNet~\cite{Sun:2019:HNL} & ResNet-50 & 81.71 & 79.11 \\
%HoHoNet~\cite{Sun:2021:HNL} & ResNet-34 & 82.32 & 79.88 \\
AtlantaNet~\cite{Pintore:2020:AI3} & ResNet-50 & 82.09 & 80.02 \\
LED2-Net~\cite{Wang:2021:LM3} & ResNet-50 & 82.61 & 80.14 \\
LGT-Net~\cite{Jiang:2022:LGT} & ResNet-50 & 83.52 & 81.11 \\
Seg2Reg~\cite{Sun:2024:SD2} & ResNet-34 & 83.39 & 81.08 \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{4}{|c|}{\textbf{(b) ZInD test set results}} \\
\hline
HorizonNet~\cite{Sun:2019:HNL} & ResNet-50 & 90.44 & 88.59 \\
LED2-Net~\cite{Wang:2021:LM3} & ResNet-50 & 90.36 & 88.49 \\
LGT-Net~\cite{Jiang:2022:LGT} & ResNet-50 & 91.77 & 89.95 \\
Seg2Reg~\cite{Sun:2024:SD2} & HRNet-18 & 92.50 & 90.73 \\
\hline
\rowcolor{tblheadcolor}
\multicolumn{4}{|c|}{\textbf{(c) PanoContext and Stanford2D-3D-S  test set results (3D IoU)}} \\
\hline
\rowcolor{tblheadcolor}
 & & \textbf{PanoContext} & \textbf{Stanford} \\
\hline
LayoutNet v2~\cite{Zou:2021:MRL} & ResNet-34 & 85.02 & 82.66 \\
DuLaNet v2~\cite{Zou:2021:MRL} & ResNet-50 & 83.77 & 86.60 \\
HorizonNet~\cite{Sun:2019:HNL} & ResNet-50 & 82.63 & 82.72 \\
AtlantaNet~\cite{Pintore:2020:AI3} & ResNet-50 & -- & 83.94 \\
LGT-Net~\cite{Jiang:2022:LGT} & ResNet-50 & 85.16 & 86.03 \\
Seg2Reg~\cite{Sun:2024:SD2} & HRNet-18 & 87.23 & 87.24 \\
\hline
\end{tabular}
\caption{\textbf{Benchmark results of representative panoramic layout
estimation methods on three widely used datasets.} (a) MatterportLayout,
(b) ZInD, and (c) PanoContext and Stanford2D3D. The table reports
2D and 3D IoU scores whenever available, including only methods
providing directly comparable results in the original
papers. Numerical comparison from Seg2Reg~\cite{Sun:2024:SD2} experiments.}
\label{tab:layout_results}
\end{table}
%-------------------------------------------
\paragraph*{Performance of representative methods}
\autoref{tab:layout_results} summarizes the performance of several
representative methods for panoramic room layout estimation across
multiple benchmark datasets. We report both 2D and 3D IoU scores
whenever available, focusing on methods for which comparable
results were published in the original papers. The results highlight
the steady progression from early CNN- and RNN-based baselines
(LayoutNet v2, DuLaNet v2, HorizonNet, AtlantaNet) toward more geometry-aware
and transformer-based approaches (LED2-Net, LGT-Net).
Seg2Reg further improves consistency by bridging segmentation and
regression, achieving the best overall results across the ZInD,
PanoContext, and Stanford2D3D benchmarks.
%360MonoDepth/Replica~\cite{Data:360MonoDepthReplica:2022} point clouds.
While these results provide useful insights into how layout-based methods can be extended to full-scene parsing, their focus remains on generic 3D object detection protocols rather than on metrics tailored to panoramic layout estimation. For this reason, we include them here primarily as an illustrative complement, highlighting representative outcomes in a panoramic setting that, although informative, remain tangential to the central scope of this survey.
%---------------------------------------------
\begin{table*}[t]
\centering
\small
\rowcolors{2}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|c|}
\hline
\rowcolor{tblheadcolor}
\textbf{Method} & \textbf{cab.} & \textbf{door} & \textbf{chair} & \textbf{curtain} & \textbf{lamp} & \textbf{rug} & \textbf{sofa} & \textbf{table} & \textbf{trash} & \textbf{TV} & \textbf{mAP}$\uparrow$ \\
\hline
DeepPanoContext~\cite{Zhang:2021:DPC} & 35.33 & 6.78 & 47.04 & 13.60 & 12.15 & 4.49 & 26.87 & 73.34 & 39.59 & 4.86 & 26.41 \\
DeepPanoContext-3D~\cite{Zhang:2021:DPC} & 52.49 & 11.42 & 70.39 & 32.38 & 20.02 & 9.10 & 30.13 & 82.24 & 63.22 & 12.19 & 38.36 \\
Group-Free~\cite{Liu:2021:G3O} & 59.56 & 42.21 & 52.83 & 34.07 & 19.65 & 32.90 & 80.59 & 51.47 & 44.64 & 52.76 & 47.07 \\
PC-Former~\cite{Dong:2024:PPT} & 63.69 & 46.74 & 54.02 & 30.41 & 20.04 & 48.53 & 80.96 & 46.42 & 51.53 & 47.82 & 49.02 \\
\hline
\end{tabular}
\caption{\textbf{Object detection results} on the 360MonoDepth/Replica~\cite{Data:360MonoDepthReplica:2022} dataset, where mAP is a 3D IoU threshold of 0.15 over 10 representative categories, reported by PC-Former~\cite{Dong:2024:PPT}. 
DeepPanoContext~\cite{Zhang:2021:DPC} predicts 2D object layouts from panoramic images, while DeepPanoContext-3D extends the pipeline by lifting predictions into a volumetric 3D space for direct object reconstruction. 
Unlike layout estimation, these benchmarks follow point-cloud-based detection protocols; we include them here as illustrative examples of object evaluation in the panoramic context, which remains ancillary to the main focus of this survey.}
\label{tab:replicapano_detection}
\end{table*}
%--------------------------------------------
For reference, \autoref{tab:replicapano_detection} shows the evaluation of object detection in panoramic scenes. It typically follows the conventions established in point-cloud benchmarks, relying on 3D IoU thresholds to compute mean Average Precision (mAP) across object categories. 
In this case, we summarize the evaluation presented in PC-Former~\cite{Dong:2024:PPT} on 

%-------------------------------------------
\begin{table*}[htbp]
\centering
\small
\label{tab:multiroom-comparison}
\rowcolors{2}{tbloddrowcolor}{tblevnrowcolor}
\begin{tabular}{|l|p{4.5cm}|p{3.5cm}|p{5.5cm}|}
\hline
\rowcolor{tblheadcolor}
\textbf{Method} & \textbf{Input} & \textbf{Architecture} & \textbf{Output} \\
\hline
Extreme SfM~\cite{Shabani:2021:ESF} & Sparse panoramas without visual overlap & Single-image layout + Geometric optimization& Displacements of rooms; Coarse global floorplan\\
SALVe~\cite{Lambert:2022:SSA} & Sparse panoramas with minimal overlap & Learned structural priors + optimization & Panorama positioning and layout alignment \\
BADGR~\cite{Li:2025:BBA} & Wide-baseline panoramas & Diffusion-based bundle adjustment & Accurate relative localization of rooms \\
{NadirFloorNet}~\cite{Pintore:2025:NRM} & Single panorama per room + positions & Single-image layout + deformable attention & Metrically coherent 3D floorplan \\
{PanoFloor}~\cite{Pintore:2025:PRI} & Sparse panoramas with relative positions & Diffused density map + deformable attention & Floor plan layout, doors, windows, clutter; connectivity graph for VR-ready model \\
\hline
\end{tabular}
\caption{\textbf{Comparison of representative methods for sparse multi-room panoramic reconstruction.} Each method is characterized by its inputs, core architecture, and produced outputs.}
\end{table*}
%--------------------------------------------

%===============================================================
\subsection{Positioning and connecting single rooms}
\label{sec:geostruct:connect}
While single-room depth and layout estimation provide essential building blocks, many practical applications require structured models that span entire apartments, offices, or floors.

Classical multi-room reconstruction pipelines rely on dense image collections with significant visual overlap to establish reliable correspondences and recover camera poses using methods such as Structure-from-Motion (SfM)~\cite{Pintore:2019:AMC}. These approaches are fundamentally different from the single-view setting, where only one panorama per room is typically available and overlap across rooms is minimal or even absent. In such conditions, conventional SfM completely fails~\cite{Shabani:2021:ESF}, as feature-based matching cannot be applied. Our focus is therefore on single-view-centric approaches, where each panorama is independently reconstructed into a partial 3D model, and global consistency is enforced at a higher level. This is usually achieved by estimating the room layout, detecting openings such as doors and windows, and aligning them into a topological graph that connects adjacent panoramas. The resulting graph-based representation enables the assembly of a coherent floorplan and supports interactive navigation across multiple panoramic viewpoints, even without dense multi-view coverage.

The main challenge stems from the fact that a single panorama is inherently confined by the bounds of its capture location. Inferring the existence of multiple connected rooms, therefore, requires going beyond local observations and combining geometric cues (e.g., openings, vanishing lines, occlusion boundaries) with strong structural priors and learned patterns of indoor organization. While elements such as doors and corridors can act as weak indicators of adjacency, they are frequently occluded, cluttered, or only partially visible. Robust solutions must therefore integrate local geometric evidence with global reasoning strategies to hypothesize plausible multi-room connectivity from incomplete or noisy inputs.

Early efforts in this direction mainly relied on explicit geometric reasoning and energy-minimization frameworks to extrapolate single-room layouts into larger structures~\cite{Furukawa:2009:RBI,Silberman:2012:ISS,Cabral:2014:PPC,Monszpart:2015:RRM,Ikehata:2015:SIM}. However, these pipelines tended to be brittle, requiring clean inputs and often failing under the clutter and noise typical of real-world indoor panoramas. 
%
Recent research has therefore reformulated the problem into two complementary tasks. The first, \emph{positioning single rooms}, focuses on estimating the relative placement of panoramas by leveraging coarse geometric priors and global alignment strategies. The second, \emph{connecting single rooms}, goes one step further by requiring the accurate reconstruction of structural elements such as room layouts and door locations, which act as critical anchors for establishing adjacency and continuity between spaces. While positioning provides a global scaffold for arranging panoramas, connecting requires a higher level of structural reasoning to ensure that individual reconstructions can be seamlessly assembled into a coherent floor plan.

\paragraph*{Positioning single rooms}
%
%
\begin{figure*}[t]
    \centering
    {\includegraphics[width=0.9\linewidth]{img/redesigned-shabani2021-extreme.jpg}}
    \caption{\textbf{Positioning single room layouts example}~\cite{Shabani:2021:ESF}. The pipeline leverages geometric cues, such as estimated single-room layouts (\autoref{sec:geostruct:layout}) and detected openings, and combines them with semantic priors and architectural regularities to hypothesize pairwise room connections. These hypotheses are then integrated into a global optimization framework that jointly refines both room positions and adjacencies, ultimately assembling a coherent and metrically consistent floorplan.}
    \label{fig:positioning_rooms}
\end{figure*}
%
%
To achieve this goal, an important task when working with sparse views is spatial registration across multiple panoramas: given sparse single-view captures, methods such as Shabani et al.~\cite{Shabani:2021:ESF} attempt to estimate plausible relative room displacements to reconstruct a global layout of a building. Their pipeline departs from traditional SfM by explicitly addressing the lack of visual overlap between panoramas. Instead, it leverages geometric cues such as estimated single room layouts (\autoref{sec:geostruct:layout}) and detected openings, combined with semantic priors and architectural regularities, to hypothesize pairwise connections between rooms. These hypotheses are then embedded into a global optimization framework, which jointly refines room positions and adjacency to assemble a coherent floorplan (\autoref{fig:positioning_rooms}). 
%
More recent approaches have pushed this direction by introducing learned verification and generative refinement, in particular exploiting diffusion models~\cite{Pu:2024:PNV}. For instance, SALVe~\cite{Lambert:2022:SSA} introduces a semantic alignment verifier that exploits windows, doors, and openings (W/D/O) as robust cues for hypothesizing pairwise panorama alignment under extreme wide baselines. Candidate alignments are validated through a deep CNN operating on bird’s-eye views of aligned floor and ceiling renderings, before constructing a global pose graph optimized with GTSAM. In parallel, BADGR~\cite{Li:2025:BBA} proposes a diffusion-based bundle adjustment framework that refines both camera poses and floor plan layouts from sparse panoramas. Unlike prior guided diffusion strategies, BADGR integrates a planar BA layer into the denoising process, enforcing view-consistency while injecting learned structural priors (e.g., collinearity, wall adjacency) to plausibly complete occluded parts of the layout. Together, these methods exemplify a shift from purely geometric heuristics towards hybrid pipelines where semantic cues and generative models enable both positioning and floorplan reconstruction in the challenging regime of sparse, wide-baseline panoramic captures.

\paragraph*{Connecting single rooms}
%
\begin{figure*}[t]
    \centering
    {\includegraphics[width=0.9\linewidth]{img/redesigned-pintore2025-panofloor.jpg}}
    \caption{\textbf{Floorplan layout and connectivity graph recovery example}~\cite{Pintore:2025:PRI}. The pipeline takes as input the relative positions and individually predicted depths of multiple panoramas to reconstruct a global floor plan enriched with semantic elements (layouts, doors, windows, and clutter), also building a connectivity graph that describes the navigable structure of the environment.}
    \label{fig:connecting_rooms}
\end{figure*}
%
To reliably connect multiple panoramas, it is not sufficient to estimate their relative placement: some form of floorplan reconstruction, even in a simplified representation, is required to provide the structural scaffold that links individual rooms together.
Floor plan reconstruction is a more complex problem than just relative localization.
%Recovering such models from a single panoramic input per room is extremely challenging, as it requires not only analyzing the geometry of the visible space, but also inferring the connectivity and arrangement of adjacent, non-visible rooms. This makes \emph{single-view multi-room reconstruction} a fundamental research direction for extending the scope of structured indoor modeling beyond isolated spaces.
%
State-of-the-art multi-room reconstructors usually require the availability of reliable density maps for their analyses~\cite{Chen:2022:HHE,Yue:2023:CTD}. These maps, accumulating the occurrence of 3D points projected onto the floorplan, are built from dense point clouds that are hard to generate with sufficient precision using single-view inference from purely visual data~\cite{Pintore:2025:NRM}. 
%
In the context of dense coverage (i.e., density maps from point clouds),  recent hybrid approaches leverage deep networks to detect low-level primitives (e.g., corners, edges, or room regions) and assemble them into structured floor plans through optimization. Floor-SP~\cite{Chen:2019:FIC} and Nauata et al.~\cite{Nauata:2020:VWB} employ Mask R-CNN~\cite{He:2017:MRC} to segment room regions and reconstruct polygons via shortest-path optimization, while MonteFloor~\cite{Stekovic:2021:MEM} explores multiple room arrangements with Monte-Carlo Tree Search. Similarly, FloorNet~\cite{Liu:2018:FAU} detects corners and generates wall segments using integer programming to yield coherent layouts. End-to-end solutions further extend this trend: HEAT~\cite{Chen:2022:HHE} predicts corners and edges with a transformer-based backbone, RoomFormer~\cite{Yue:2023:CTD} generates polygonal sequences from sparse 3D cues, SLIBO-Net~\cite{Su:2023:SFR} integrates Manhattan World priors to refine geometry, and PolyDiffuse~\cite{Chen:2024:PPS} applies conditional diffusion to infer plausible multi-room structures from density maps.
%In particular, RoomFormer~\cite{Yue:2023:CTD} is able to simultaneous recover the floor plan layout, the position of doors and windows and the room type.
%
%
In the context of extreme multi-room reconstruction from sparse panoramas, \emph{NadirFloorNet}~\cite{Pintore:2025:NRM} extends the transformer-based \emph{RoomFormer}~\cite{Yue:2023:CTD} to single-image-per-room settings. 
The method first predicts a clutter-free nadir depth projection to recover room geometry, and then fuses these representations into a consistent global floorplan using deformable attention, given the image positions~\cite{Shabani:2021:ESF,Lambert:2022:SSA,Li:2025:BBA}. By jointly estimating ceiling heights and 2D polygons, it achieves metrically coherent 3D layouts from extremely sparse inputs.  
\emph{PanoFloor}~\cite{Pintore:2025:PRI}, instead, takes as input the relative positions and individually predicted depths of multiple panoramas (\autoref{sec:geostruct:depth}) and reconstructs a global floor plan enriched with semantic elements. Specifically, the method estimates room layouts, places doors and windows, and accounts for clutter, while also building a connectivity graph that describes the navigable structure of the environment. This representation enables the creation of a complete, VR-explorable model that integrates geometry, semantics, and accessibility (\autoref{sec:interactive:viewsynth:multiroom}).
%
%
In summary, as shown in~\autoref{tab:multiroom-comparison}, these approaches span from geometry-based positioning methods to transformer- and diffusion-based pipelines that jointly recover localization and structured floorplans, progressively moving towards metrically consistent, semantically enriched, and VR-ready reconstructions. These pipelines further highlight how depth and layout estimation remain fundamental tasks, as they provide the geometric and structural backbone required for reasoning about room connectivity and assembling coherent multi-room models under such sparse conditions.  





